Compare commits

...
278 Commits
Author SHA1 Message Date
beatzaplenty b9d3b51028 Merge pull request 'fix(ha): resolve data disk device via by-id symlink even in dry-run' (#105) from worktree-partitioned-swimming-pizza into main
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Reviewed-on: #105
2026-07-29 04:01:49 +00:00
beatzaplentyandClaude Sonnet 4.6 539bdf9833 fix(ha): resolve data disk device via by-id symlink even in dry-run
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m21s
The by-id lookup is read-only so it's safe to run in dry-run mode.
Previously it was gated behind `if ! $DRY_RUN`, which always triggered
the sdb fallback warning in dry-run — making it look like the device
path wasn't reliable when the symlink actually exists on both servers.

Now the lookup always runs and the script errors out with a clear message
if the by-id symlink is genuinely missing, instead of silently falling
back to a guessed /dev/sd* name.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:58:53 +10:00
beatzaplenty dc6c37ed2e Merge pull request 'fix(ha): prefix qm commands with sudo for non-root Proxmox SSH user' (#104) from worktree-partitioned-swimming-pizza into main
Check NixOS configurations / eval-hosts (push) Failing after 4m22s
Reviewed-on: #104
2026-07-29 03:55:28 +00:00
beatzaplenty b5e61d62bd Merge pull request 'fix(gc-hosts): fix pve1 PATH and discovery sudo prompts' (#103) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Reviewed-on: #103
2026-07-29 03:53:50 +00:00
beatzaplentyandClaude Sonnet 4.6 ced1657407 fix(ha): prefix qm commands with sudo for non-root Proxmox SSH user
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m25s
qm lives at /usr/sbin/qm, which is not in the default PATH for
non-interactive SSH sessions as a non-root user.  Add PVE_SUDO (set to
"sudo" when PVE_SSH_USER != root, matching create-proxmox-resource.sh's
own sudo_prefix pattern) and prepend it to all three qm invocations in
the script.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:53:41 +10:00
beatzaplentyandClaude Sonnet 4.6 7b4794211d fix(gc-hosts): fix pve1 PATH and discovery sudo prompts
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m45s
Two bugs:

1. Dynamic discovery used BatchMode=yes for the pct/qm list SSH call, which
   silently suppressed any sudo password prompt and returned empty output.
   Fix: split SSH_OPTS into SSH_OPTS (BatchMode, for gc) and SSH_QUERY_OPTS
   (no BatchMode, for discovery) so sudo can prompt when needed.

2. pve1 gc used a non-login SSH session which doesn't source /etc/profile,
   so nix-collect-garbage was never on PATH. The Nix installer registers
   itself via /etc/profile.d/nix-daemon.sh which only runs in login shells.
   Fix: use "bash -l -c 'nix-collect-garbage -d'" instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:53:02 +10:00
beatzaplenty fdf41c659c Merge pull request 'Worktree crispy churning kernighan' (#102) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m50s
Reviewed-on: #102
2026-07-29 03:46:05 +00:00
beatzaplentyandClaude Sonnet 4.6 0ef8259225 fix(gc-hosts): source nix-daemon profile on pve1 before running gc
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
BatchMode SSH sessions don't source /etc/profile on non-NixOS hosts, so
nix-collect-garbage isn't on PATH for the wayne user. Source the nix-daemon
profile script explicitly, matching the pattern in scripts/lib/nix-bootstrap.sh.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:45:48 +10:00
beatzaplentyandClaude Sonnet 4.6 629c1457a9 refactor(gc-hosts): discover running pve1 guests dynamically each run
Replace the static host list with dynamic discovery: workstation (nixos)
and pve1 are hard-wired first and second; remaining hosts are discovered
on every run by SSHing to pve1, listing running VMs/containers via
pct/qm list, and resolving their NixOS hostnames from a single flake eval.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:44:26 +10:00
beatzaplenty 7ba603c005 Merge pull request 'feat(scripts): add gc-hosts.sh for parallel nix gc across all live hosts' (#101) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #101
2026-07-29 03:36:09 +00:00
beatzaplentyandClaude Sonnet 4.6 decf3ddff9 feat(scripts): add gc-hosts.sh for parallel nix gc across all live hosts
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m29s
Runs nix-collect-garbage -d on all deployed NixOS hosts and pve1 in
parallel, skipping nix-cache to avoid evicting shared cache paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:35:24 +10:00
beatzaplenty 4490dfab7d Merge pull request 'feat(nix-cache): authorize pve1's wayne key as remote builder client' (#100) from worktree-pve1-nix-cache-builder into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #100
2026-07-29 03:16:53 +00:00
beatzaplentyandClaude Sonnet 4.6 fa163b613d feat(nix-cache): authorize pve1's wayne key as remote builder client
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
Generated /home/wayne/.ssh/id_ed25519 on pve1 (single-user Nix install
owned by wayne, no daemon — builds run as wayne not root). Adds the
public key to remoteBuilderAuthorizedKeys so nix-cache accepts it.

After deploying nix-cache, re-run configure-nix-cache-client.sh as
wayne on pve1 to wire in the builder line and known_hosts entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:16:30 +10:00
beatzaplenty 15bc5bd369 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
2026-07-29 13:02:03 +10:00
beatzaplenty 5a4fbbf7ad enable beszel agent 2026-07-29 13:01:53 +10:00
beatzaplenty e720bbb018 Merge pull request 'fix(nfs): mask rpc-svcgssd on server and ha-server build types' (#99) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Failing after 9m46s
Reviewed-on: #99
2026-07-29 02:54:12 +00:00
beatzaplentyandClaude Sonnet 4.6 40cdf724b4 fix(nfs): mask rpc-svcgssd on server and ha-server build types
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m43s
nfs-utils changed ConditionPathExists from !/etc/krb5.keytab to
/etc/krb5.keytab, so on IPA-joined hosts the service now starts instead
of skipping. Exports use standard auth (no sec=krb5) so the nfs/ Kerberos
principal is never provisioned and the service fails.

enable = false masks the unit so nfs-server's Wants= can't pull it in.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 12:51:50 +10:00
beatzaplenty aaa193645b Merge pull request 'refactor(beszel): move KEY and TOKEN fully into enable-agent.nix' (#98) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Successful in 10m45s
Reviewed-on: #98
2026-07-29 02:38:19 +00:00
beatzaplentyandClaude Sonnet 4.6 f22ff7db79 refactor(beszel): move KEY and TOKEN fully into enable-agent.nix
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m59s
KEY (hub's SSH public key) is now stored in variables.nix as beszelHubKey
and set by enable-agent.nix — no host file ever needs to touch it.
TOKEN was already universal; this removes the last per-host beszel config.

Importing enable-agent.nix in a build type is now the only step needed to
add a new host to beszel monitoring.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 12:37:54 +10:00
beatzaplenty 0da3c3070f updated flake.lock and added beszel agent environment
Check NixOS configurations / eval-hosts (push) Failing after 16m56s
2026-07-29 12:18:38 +10:00
beatzaplenty e1551feefd enabled beszel-agent
Check NixOS configurations / eval-hosts (push) Successful in 10m34s
2026-07-29 12:14:49 +10:00
beatzaplentyandClaude Sonnet 4.6 b4bc30cb2c chore(secrets): remove per-host beszel-token secrets
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
Five files contained only beszel-token and are deleted entirely.
nix-cache.yaml retains cache-priv-key; beszel-token removed from it.
The universal token in secrets/common.yaml replaces all of these.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 12:00:38 +10:00
beatzaplenty 5c8d55bd78 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Failing after 4m33s
2026-07-29 11:50:44 +10:00
beatzaplenty 999c9a3151 add beszel universal token 2026-07-29 11:50:01 +10:00
beatzaplenty 1e0fff1b26 Merge pull request 'refactor(beszel): switch to universal token via secrets/common.yaml' (#97) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Failing after 13m26s
Reviewed-on: #97
2026-07-29 01:47:15 +00:00
beatzaplentyandClaude Sonnet 4.6 24c6469f10 refactor(beszel): switch to universal token via secrets/common.yaml
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m40s
Replace per-host host-token.nix imports with a single beszel-token secret
in secrets/common.yaml, wired once in enable-agent.nix. Host files now only
need services.beszel.agent.environment.KEY — no imports block required.

Delete modules/beszel/host-token.nix (no longer referenced anywhere).

Action needed: run `sops secrets/common.yaml` and add `beszel-token: <value>`
from the beszel hub UI before deploying.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 10:58:15 +10:00
beatzaplenty c9458ac8a6 Merge pull request 'docs: add beszel agent setup guide' (#96) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
Reviewed-on: #96
2026-07-29 00:41:22 +00:00
beatzaplentyandClaude Sonnet 4.6 99ba0ed52b docs: add beszel agent setup guide
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s
Documents how to wire beszel-agent into any flake build: the two-module
split (enable-agent.nix in build types, host-token.nix in host files),
sops secret creation, hub pairing flow, and optional EXTRA_FILESYSTEMS /
Docker monitoring knobs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 10:41:03 +10:00
beatzaplenty 1e4072029e update rmote-builder keys
Check NixOS configurations / eval-hosts (push) Successful in 10m38s
2026-07-29 10:39:53 +10:00
beatzaplenty 46649fc7e0 update remote-builder client keys
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
2026-07-29 10:29:40 +10:00
beatzaplenty 4cbcc3beb9 updated .gitignore and change default editor in home.nix
Check NixOS configurations / eval-hosts (push) Failing after 13m5s
2026-07-29 10:17:37 +10:00
beatzaplenty 35f696ccd5 Merge pull request 'feat(ha): add resize-data-disk.sh for online data disk expansion' (#95) from worktree-ha-resize-disk into main
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
Reviewed-on: #95
2026-07-29 00:01:49 +00:00
beatzaplentyandClaude Sonnet 4.6 e18b605706 feat(ha): add resize-data-disk.sh for online data disk expansion
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m27s
Three-phase online resize: Proxmox qm resize → guest block device
rescan → drbdadm resize + xfs_growfs. No downtime required.
Detects active node via crm_mon, pre-checks DRBD UpToDate/UpToDate,
auto-discovers VMIDs from qm list, and resolves the block device name
from the stable scsi-0QEMU... by-id symlink with a slot-index fallback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 09:59:39 +10:00
beatzaplenty ee2451d87c remove stray pihole backup
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
2026-07-29 09:50:50 +10:00
beatzaplentyandClaude Sonnet 4.6 71a6c4738c fix(ha): fix crm_mon active-node detection in all three scripts
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
crm_mon 2.x formats the Promoted line as "    * Promoted: [ node ]" — the
asterisk bullet means the previous grep -E '^\s*(Promoted|Masters):' never
matched, so active-node detection silently returned empty in health.sh,
failover.sh, and acceptance-tests.sh.

Fix: pipe through grep -v Unpromoted first, then grep -E '(Promoted|Masters):'
without anchoring to start-of-line.

Also: remove the SSH_OPTS=-i ~/.ssh/... variable pattern in health.sh and
failover.sh; tilde is not expanded inside double-quoted strings, so $SSH_OPTS
was passing a literal ~ to SSH.  Inline the key path in each function
definition instead (same as acceptance-tests.sh already did).

Also: drop the 2>/dev/null embedded in the crm_mon argument string — the
outer 2>/dev/null on the n1/n2 call already suppresses SSH stderr; the
embedded one was harmless but noisy to reason about.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 02:36:22 +10:00
beatzaplentyandClaude Sonnet 4.6 793a2b5924 feat(ha): add failover.sh and health.sh operator scripts
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
failover.sh: graceful active→standby resource migration with pre-checks
(DRBD UpToDate/UpToDate, quorum), spinner wait for XFS mount on target,
rollback on timeout, --to/--force/--dry-run/--timeout flags.

health.sh: read-only status panel — node reachability, quorum, DRBD
role/dstate/cs per node, Pacemaker resources, failure history, XFS mount
usage, and service port checks (NFS 2049, iSCSI 3260) via VIP.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 02:23:47 +10:00
beatzaplenty 98b5429fb9 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 02:04:51 +10:00
beatzaplentyandClaude Sonnet 4.6 f658bdbabc fix(ha/cluster-init): enable Pacemaker maintenance-mode during DRBD sync
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Even with both nodes in standby, Pacemaker's monitor operations keep running.
When the monitor sees DRBD is Primary on a standby node (that it didn't start),
it triggers a stop action — killing the initial sync after ~10 s.

Enable maintenance-mode after standby stops the existing resources but before
DRBD is manually brought up for the sync.  Maintenance-mode suspends all
start/stop/monitor actions so Pacemaker is completely hands-off during the
sync.  Disable it alongside crm_standby -v off once UpToDate/UpToDate is
confirmed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:49:15 +10:00
beatzaplentyandClaude Sonnet 4.6 d7e63cd80e fix(ha/cluster-init): fix DRBD metadata skip condition and StandAlone grace period
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
The metadata creation check used grep -q "UpToDate" which matches when the
resource is DOWN — drbdadm dstate reads metadata directly and returns just
"UpToDate" (no slash) for a down-but-previously-synced resource.  This caused
metadata creation to be silently skipped, leaving mismatched UUIDs from an
interrupted sync which cause DRBD to go WFConnection→StandAlone immediately.

Fix: require exact "UpToDate/UpToDate" (with slash, only possible when the
resource is UP and both disks are confirmed synced) before skipping.

Also give the StandAlone detection a 5-iteration (15 s) grace period before
dying, to avoid false-positive on the first few ticks while the peer is still
coming up.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:45:03 +10:00
beatzaplentyandClaude Sonnet 4.6 b88ea49880 fix(ha/cluster-init): keep Pacemaker in standby until DRBD sync completes
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Clearing crm_standby before the initial sync finished caused Pacemaker's OCF
DRBD agent to race with the manual drbdadm up/primary calls.  The agent saw
DRBD in WFConnection or SyncSource and tore it down, driving the resource back
to StandAlone and killing the sync in seconds.

Move the crm_standby -v off calls to immediately after the sync-complete
break, so Pacemaker only resumes once DRBD is UpToDate/UpToDate and safe
to hand back.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:41:15 +10:00
beatzaplentyandClaude Sonnet 4.6 c7268ade8e fix(ha/cluster-init): detect StandAlone DRBD state in sync wait loop
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
If the DRBD peer connection drops during the initial sync wait (cs:StandAlone),
the loop would spin forever printing "waiting for sync progress" with no
indication of what's wrong. Now parses cs: from /proc/drbd each tick and
dies with a clear error if StandAlone is detected. Also shows the cs: field
in the non-syncing message so the actual connection state is always visible.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:29:04 +10:00
beatzaplentyandClaude Sonnet 4.6 367158548e fix(ha): wrap crm-fence-peer.sh to set PATH for kernel UMH callout
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
When the DRBD kernel module invokes fence-peer via the User Mode Helper
mechanism it provides a minimal PATH that omits /run/current-system/sw/bin.
crm-fence-peer.sh calls cibadmin, crm_mon etc.; without those in PATH a
pipeline breaks with SIGPIPE.  A signal-killed process has WEXITSTATUS()==0,
so the kernel sees exit code 0, logs "fence-peer helper broken, returned 0",
and retries forever — blocking Pacemaker failover completely.

Fix: use pkgs.writeShellScript to create thin PATH-fixing wrappers in the
Nix store.  The wrappers prepend /run/current-system/sw/bin before exec-ing
the real crm-fence-peer.sh / crm-unfence-peer.sh, giving them a working
Pacemaker toolchain regardless of what PATH the kernel provides.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:22:46 +10:00
beatzaplenty 585126beee secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 01:11:23 +10:00
beatzaplentyandClaude Sonnet 4.6 ceb491b18d fix(ha/cluster-init): drop TTY detection, always use \r for sync progress
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
The TTY check ([[ -t 1 ]]) evaluated false under sudo, causing each update
to print on a new line via log(). Since \r worked correctly in the original
code (user confirmed), drop the branching and always use printf \r.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:55:31 +10:00
beatzaplentyandClaude Sonnet 4.6 6d16eaff89 fix(ha/cluster-init): fix /proc/drbd sync progress parsing and TTY display
Check NixOS configurations / eval-hosts (push) Successful in 10m29s
The sync progress grep was silently failing because /proc/drbd formats the
percentage line as "sync'ed:  5.2%" (two spaces after colon), but the
pattern matched only one.  Use [[:space:]]+ throughout to tolerate any
whitespace.

Also make the display TTY-aware: interactive sessions get the spinning
\r in-place update; piped/logged sessions get a plain log line printed
once per unique percentage point (avoiding scroll spam).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:51:35 +10:00
beatzaplentyandClaude Sonnet 4.6 e085d4707c fix(ha): add DRBD sync progress indicator and pre-flight sync gate
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
cluster-init.sh: replace the fixed 300-iteration sync wait with an
indefinite loop that prints an in-place progress line (%done, ETA, speed)
from /proc/drbd every 3 s. Clears the line with printf \r before logging
completion, so the output stays clean alongside the [cluster-init] log
lines.

acceptance-tests.sh: add a pre-flight check that hard-exits if
drbdadm dstate is not UpToDate/UpToDate, with a hint to the watch
command for monitoring progress. Tests cannot give accurate results
while the initial full-sync is in progress.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:46:57 +10:00
beatzaplenty c8d4440787 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:40:11 +10:00
beatzaplentyandClaude Sonnet 4.6 e7531b276e fix(ha): use crm_mon for Active node detection and add cleanup before VIP wait
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
acceptance-tests.sh: replace drbdadm-role-based Active node detection with
crm_mon-based detection. DRBD role briefly reads Secondary during Pacemaker
transitions, causing T2/T3/T4 to target the wrong node. crm_mon Promoted:/
Masters: lines are authoritative; wait up to 90 s for Pacemaker to settle
before falling back to NODE1.

cluster-init.sh: add crm_resource --cleanup before the VIP wait loop so
stale migration-threshold failure counters from previous cluster-init runs
are cleared before Pacemaker decides resource placement.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:24:47 +10:00
beatzaplenty 49a10d7cc5 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:16:51 +10:00
beatzaplentyandClaude Sonnet 4.6 ead4f55805 fix(ha): use stable by-id disk path for DRBD instead of /dev/sd*
Check NixOS configurations / eval-hosts (push) Failing after 9m44s
/dev/sda and /dev/sdb are assigned by the OS based on Proxmox disk-add
order, which is not consistent across VMs. Use the SCSI controller path
instead — drive-scsi1 is always the dedicated data disk on all HA nodes
regardless of which sda/sdb it gets assigned to.

variables.nix: replace per-node haServer{1,2}DrbdDisk with a single
haServerDrbdDisk using /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1.

cluster-config.nix: revert to shared volume block (per-node block was
needed for /dev/sd* but the by-id path is identical on both nodes).

cluster-init.sh:
- single DRBD_DISK variable (matching haServerDrbdDisk)
- robust /etc/drbd.conf patch: NixOS manages this file as a symlink to
  a read-only Nix store path; cp --remove-destination breaks the symlink
  before sed -i so the edit actually takes effect
- scp helper script to NODE2 rather than bash -c over SSH to avoid
  quoting complexity

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:11:13 +10:00
beatzaplenty cac5ec45cc secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:02:48 +10:00
beatzaplentyandClaude Sonnet 4.6 ec90753a09 fix(ha): per-node DRBD disk assignment (NODE2 data disk is /dev/sda not /dev/sdb)
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Proxmox VM disk ordering differs between the two HA nodes:
  ha-server-1: sda=OS (50G), sdb=DRBD data (32G)
  ha-server-2: sda=DRBD data (32G), sdb=OS (50G)

The DRBD resource config was using a shared disk=/dev/sdb which targeted
the OS disk on ha-server-2, causing drbdmeta and drbdadm up to operate
on the mounted root filesystem (hence "Device or resource busy").

Changes:
- variables.nix: add haServer1DrbdDisk/haServer2DrbdDisk
- cluster-config.nix: move volume block inside per-host on{} sections
  so each node uses the correct backing disk
- cluster-init.sh: use NODE1_DRBD_DISK/NODE2_DRBD_DISK variables;
  add runtime check that patches /etc/drbd.d/*.res on the running nodes
  if the deployed config points to the wrong disk (workaround for VMs
  built before this fix; redeploy with --force-rebuild to make permanent)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:57:07 +10:00
beatzaplenty df1ddee735 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:48:23 +10:00
beatzaplentyandClaude Sonnet 4.6 8e8261be31 fix(ha/cluster-init): drop bash -c wrapper for NODE2 drbdmeta calls
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
drbdmeta is in /run/current-system/sw/bin which is already on sudo's
PATH on the HA nodes; the bash -c wrapper with PATH manipulation was
adding complexity without benefit and introduced quoting hazards that
likely caused write-dev-uuid to fail or run in the wrong context.

Use direct n2_ssh calls instead — the same pattern that works when
tested manually on NODE2.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:42:36 +10:00
beatzaplenty 5e9541741f secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:30:12 +10:00
beatzaplentyandClaude Sonnet 4.6 07543d7d56 fix(ha/cluster-init): add drbdmeta to PATH and remove openssl dependency
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
drbdmeta lives in the same Nix store dir as drbdadm but sudo doesn't
inherit the full PATH, so drbdmeta was not found (exit 127) even though
drbdadm was.  Resolve drbdmeta's directory from drbdadm's location and
prepend it to PATH.

Replace openssl rand for UUID generation with /proc/sys/kernel/random/uuid
— openssl is not guaranteed to be on PATH in a minimal NixOS root
environment, but /proc/sys/kernel/random/uuid is always present.

Apply the same PATH fix on NODE2 inline in the bash -c invocations that
call drbdmeta over SSH.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:24:01 +10:00
beatzaplentyandClaude Sonnet 4.6 a7c4a24fc3 fix(ha): use drbdmeta --force directly; fix T4 grep-c arithmetic bug
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
cluster-init.sh: replace 'drbdadm create-md --force' with direct drbdmeta
calls using --force on both steps.  drbdadm create-md --force passes --force
to the create-md sub-call but NOT to write-dev-uuid, which then fails when
/dev/sdb is busy and stdin is not a TTY ("stdin not a TTY, not waiting for
confirmation" → exit 20).  write-dev-uuid failing means DRBD has no UUID,
so the peer can't identify the device → connection fails → no sync → wrong
Active node detected by acceptance tests.

acceptance-tests.sh T4: grep -c returns exit code 1 when the count is 0
(no matches), triggering '|| echo "0"' and producing "0\n0" which breaks
[[ "$IQN_COUNT" -ge 1 ]] with "arithmetic syntax error".  Fixed by running
the pipe inside bash -c with '|| true' on the grep to suppress the
non-zero exit code.  Same fix applied to T5's IQN_ON_STANDBY check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:09:06 +10:00
beatzaplenty a98955a8da secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:08:32 +10:00
beatzaplenty 7d60741e73 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:00:18 +10:00
beatzaplenty 427be2b287 secrets(ha): refresh sops host-key registrations for new VM instances 2026-07-28 22:55:02 +10:00
beatzaplentyandClaude Sonnet 4.6 684351b89b fix(ha): pre-seed SSH host key in disko image; fix DRBD init race
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
Root cause of recurring sops failures on new VM boots: disko builds raw
disk images, and create-proxmox-resource.sh only syncs the clan-var SSH
host key to the Proxmox node (for proxmox.nix to bake into the image) when
it actually builds — reusing a cached image skips sync_remote_host_keys, so
destroy+recreate reuses a stale image with the wrong or randomly-generated
key baked in.  On first boot the VM gets a different key than what .sops.yaml
was encrypted for, and sops fails permanently.

Fix 1 — deploy.sh Phase 3: always pass --force-rebuild so every VM creation
rebuilds the disko image fresh with the current clan-var key baked in via
NIXOS_HOST_KEYS_DIR (proxmox.nix already reads this under --impure).

Fix 2 — deploy.sh Phase 5.5: after VMs boot, scan their actual ed25519 host
keys and, if they drift from clan vars, update the clan var pub-key files,
rewrite the .sops.yaml age anchors, and re-encrypt all affected sops files.
Defence-in-depth: normally a no-op after Fix 1, but catches any residual
mismatch (e.g. --skip-create-vms reuse of an older image).

Fix 3 — cluster-init.sh: add crm_standby -v on for both nodes before DRBD
metadata init.  Without this, Pacemaker's OCF DRBD agent races: it sees
drbdadm down as a failure and immediately calls drbdadm up again, leaving
/dev/sdb busy when create-md / write-dev-uuid runs (drbdmeta exits 20 with
"stdin not a TTY, not waiting for confirmation").  Standby suppresses
resource scheduling during init; crm_standby -v off restores it after DRBD
is up on both nodes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 22:51:29 +10:00
beatzaplenty 327ff0b44d secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 22:27:29 +10:00
beatzaplenty 43b4cc6aa6 secrets(ha): regenerate IPA keytabs for ha-server-1 and ha-server-2
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
2026-07-28 21:54:00 +10:00
beatzaplentyandClaude Sonnet 4.6 dd984019a1 fix(ha): prevent drbd.service from starting alongside Pacemaker
Check NixOS configurations / eval-hosts (push) Failing after 9m47s
drbd.service runs drbdadm up all at activation time, but Pacemaker's
OCF drbd agent manages the resource lifecycle (up/down/promote/demote).
When both run simultaneously the device is busy, causing drbdmeta
apply-al to abort with exit 20.

Set wantedBy = [] so the kernel module and config are still present
(via services.drbd.enable = true) but systemd doesn't auto-start the
service. Pacemaker's OCF agent calls drbdadm directly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 21:48:43 +10:00
beatzaplenty cb8a51b2fd updated sops keys
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
2026-07-28 21:41:31 +10:00
beatzaplenty 18ab0ff254 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 21:19:22 +10:00
beatzaplenty 5c5f22f84a +x on scripts and remove beszel from ha for now
Check NixOS configurations / eval-hosts (push) Failing after 5m35s
2026-07-28 20:48:51 +10:00
beatzaplenty 094eaa752b secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 20:35:12 +10:00
beatzaplentyandClaude Sonnet 4.6 fe9fc7364b fix(ha/cluster-init): explicitly start pacemaker after corosync restart
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
At first boot, pacemaker fails with 'Dependency failed' because ipa-activation
fails (IPA not enrolled yet) before corosync/authkey are ready, causing a
systemd boot-ordering race.  The service recovers fine when started manually.
Add an explicit 'systemctl start pacemaker' on both nodes immediately after
restarting corosync so cluster-init doesn't time out waiting for it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:47:33 +10:00
beatzaplentyandClaude Sonnet 4.6 1ce4589830 fix(ha/deploy): remove _patch_targetctl — iscsi-target.nix now correct
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
The _patch_targetctl runtime dropin was a workaround for the wrong targetctl
binary path in the original iscsi-target.nix (used pkgs.targetcli-fb, but
targetctl is in rtslib-fb's python3 env).  The module is now fixed to use
\${python3}/bin/targetctl, so new VM images have the correct ExecStart/ExecStop
baked in and the patch finds nothing — failing with exit 1 and killing the deploy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:41:20 +10:00
beatzaplenty 7417cc1b0a Merge pull request 'feat(gui): add wayne's GUI Home Manager config matching the nixos user' (#94) from worktree-functional-mapping-russell into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #94
2026-07-28 09:29:28 +00:00
beatzaplentyandClaude Sonnet 4.6 ec44b7955b feat(gui): add wayne's GUI Home Manager config matching the nixos user
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
Adds nextcloud-client, chromium, git, vim, claude-code, fish, sops,
the Nextcloud tray service, SOPS_AGE_KEY_FILE, and Proxmox/PBS desktop
shortcuts to wayne's HM config on GUI hosts. Uses lib.mkDefault on the
EDITOR setting in ipa/client.nix so the GUI layer can override to vim.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 19:28:22 +10:00
beatzaplentyandClaude Sonnet 4.6 cf3d8ea9a5 fix(sops): resolve merge conflicts in ha-server keytab files
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
Both secrets/ha-server-{1,2}.keytab had unresolved conflict markers from
the stash/merge interaction.  Keep the upstream version (2 age recipients:
admin + the host's own key) so each host can decrypt its own keytab on boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:26:39 +10:00
beatzaplentyandClaude Sonnet 4.6 756e45743c fix(sops): resolve merge conflict in .sops.yaml keytab recipients
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
The HA branch merge left conflict markers in the ha-server-{1,2}.keytab
path_regex entries.  Keep the upstream side (adds *proxmox-ha-server-{1,2}
as recipients) so each host can decrypt its own IPA keytab on boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:23:07 +10:00
beatzaplenty 7e8755d141 added ipa keytabs for ha servers
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
2026-07-28 19:12:23 +10:00
beatzaplenty 8194707478 Merge pull request 'Worktree ha file server test' (#93) from worktree-ha-file-server-test into main
Check NixOS configurations / eval-hosts (push) Successful in 10m37s
Reviewed-on: #93
2026-07-28 09:09:04 +00:00
beatzaplentyandClaude Sonnet 4.6 d740064a35 fix(ha): all 7 acceptance tests pass — targetctl, fencing, failover, data integrity
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s
Deploy/init fixes:
- iscsi-target.nix: targetctl binary is in rtslib-fb (python3 env), not
  targetcli-fb — fixes ExecStart and ExecStop for the targetctl.service
- deploy.sh: _patch_targetctl() applies runtime dropin to both nodes before
  cluster-init so Pacemaker can manage the iSCSI target from first start
- cluster-init.sh: replace crm configure heredoc with cibadmin --replace XML
  (pacemaker-4.0 schema: globally-unique in meta_attributes, promoted-max/
  promoted-node-max, Promoted role in constraints); force_unmount=true on
  xfs-data; DRBD promote timeout 240s
- cluster-config.nix: add crm-fence-peer.sh/crm-unfence-peer.sh handlers;
  update fencing comment to reflect resource-only + Pacemaker-aware handler
  replacing STONITH during testing phase
- ha-server.nix: add openiscsi to systemPackages for T4 iscsiadm availability

Acceptance test fixes:
- acceptance-tests.sh: fix ((PASS++)) set -e bug → PASS=$((PASS+1));
  detect Active/Standby dynamically via drbdadm role (Pacemaker can promote
  either node); T4 bash TCP probe instead of iscsiadm; T5 timeout 120s;
  T6 echo|sudo tee for root-owned XFS write (bash -c redirect runs as nixos
  not sudo — permission denied); use ns cat / ns rm for root-owned reads

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:03:39 +10:00
beatzaplenty 164d14eb87 Merge pull request 'fix(docker): declaratively apply groupmod --non-unique for IPA GID conflict' (#92) from worktree-parsed-mapping-raven into main
Check NixOS configurations / eval-hosts (push) Successful in 10m36s
Reviewed-on: #92
2026-07-28 08:19:53 +00:00
beatzaplentyandClaude Sonnet 4.6 720399b00d fix(docker): declaratively apply groupmod --non-unique for IPA GID conflict
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
NixOS's group activation uses plain groupmod, which silently skips the
GID change when SSSD exposes the same GID (50010) via the IPA
docker-access group through NSS.  Add an activation script that runs
after the normal 'groups' step and applies groupmod --non-unique so the
local docker group can share GID 50010 with the SSSD-provided IPA group.
If the GID actually changes on a live system the script also restarts
docker.socket + docker.service so the socket is recreated with the new
GID.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 18:17:43 +10:00
beatzaplentyandClaude Sonnet 4.6 da4d808c6a fix(ha/cluster-init): use crm configure instead of cibadmin XML for resources
cibadmin raw XML fails schema validation under pacemaker-4.0: globally-unique
is not a valid direct <clone> attribute, and master-max/master-node-max are
renamed.  Switch to crm configure commands which are schema-version-aware:
- promotable clone with promoted-max/promoted-node-max
- order/colocation constraints using Promoted role (Pacemaker 4.0 naming)
- crm configure handles schema differences automatically

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:57:49 +10:00
beatzaplentyandClaude Sonnet 4.6 79cde50e27 fix(ha/cluster-init): use targetcli delete commands for LIO teardown
clearconfig does not reliably clear kernel LIO configfs state — the kernel
still holds backing-file references, blocking umount.  Replace clearconfig
with explicit targetcli delete commands (/iscsi delete, /backstores/fileio
delete) which do release kernel state.  Also make DRBD secondary demote
idempotent (skip if already Secondary).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:55:26 +10:00
beatzaplentyandClaude Sonnet 4.6 d31d9fa584 fix(ha/cluster-init): fix LIO teardown, mount idempotency, VIP portal
- Remove VIP-specific portal binding — the VIP doesn't exist until Pacemaker
  assigns it; the default all-IPs portal (::0:3260) is correct for Pacemaker
  to manage
- Clear existing LIO targets before re-running targetcli (idempotent on
  partial failures)
- Tear down LIO kernel objects after saveconfig so umount succeeds (LIO holds
  the backing file open otherwise)
- Guard mount with mountpoint check so re-runs don't fail when already mounted
- Use --replace for cibadmin constraints (idempotent vs --create)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:50:24 +10:00
beatzaplentyandClaude Sonnet 4.6 c76698efb7 fix(ha): add xfsprogs to system packages; fix cluster-init PATH and DRBD check
- ha-server.nix: add xfsprogs to systemPackages so mkfs.xfs is on PATH for
  root (needed by cluster-init.sh during initial setup)
- cluster-config.nix: create /var/lib/drbd via tmpfiles to silence
  lk_bdev_save warnings from drbd-utils
- cluster-init.sh: dynamically find xfsprogs in /nix/store if not on PATH
  (fallback for running VMs before xfsprogs is in the system profile)
- cluster-init.sh: fix DRBD metadata check on node2 — broken regex now uses
  grep -E for ERE alternation to correctly skip create-md when DRBD is already
  set up (previous regex would have triggered create-md on a live secondary)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:43:27 +10:00
beatzaplenty 601db79689 Merge pull request 'fix(ipa): restore IPA password login via LightDM and su' (#91) from worktree-parsed-mapping-raven into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #91
2026-07-28 07:31:46 +00:00
beatzaplentyandClaude Sonnet 4.6 acebbdbe26 fix(ha/deploy): use nixos+sudo instead of root SSH; temp key for inter-node comms
Root SSH was failing because only the RSA admin key was authorized but the
local dev box only has an ed25519 key. Fix:

- cluster-config.nix: add ed25519 keys to root (same set as nixos user) so
  future deployments work without the temp-key workaround
- deploy.sh/acceptance-tests.sh: SSH as nixos user with sudo instead of root@
- cluster-init.sh: HA_USER/HA_KEY env vars + n2_ssh()/n2_scp() helpers so
  inter-node SSH works regardless of whether root-to-root is available
- deploy.sh Phase 6: generate temp keypair, authorize on node2, place on node1
  for root to use during cluster-init, clean up afterward

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:30:53 +10:00
beatzaplentyandClaude Sonnet 4.6 fc8f7baf3e fix(ipa): restore IPA password login via LightDM and su
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
Two independent fixes:

1. hashedPassword = "!" on the IPA user stub
   Without any shadow entry, pam_unix returns PAM_AUTHINFO_UNAVAIL
   before prompting, so PAM_AUTHTOK is never set. The "!" marker
   (account locked, not a real hash) ensures a shadow entry exists so
   pam_unix prompts and sets PAM_AUTHTOK — which the subsequent
   pam_sss module can then use.

2. pam_sss try_first_pass instead of use_first_pass (login + su)
   use_first_pass silently fails when PAM_AUTHTOK is unset.
   try_first_pass prompts independently in that case, making IPA
   password auth work even if pam_unix returns early for any reason.

Root cause found during incident: wayne (uid 50002) was outside IPA's
auto-SID range (163800000+), so sidgen never assigned him an
ipaNTSecurityIdentifier. Without it, ipadb's handle_authdata step fails
with "Generic error" after SPAKE pre-auth succeeds. Fixed by manually
adding objectClass: ipaNTUserAttrs + ipaNTSecurityIdentifier RID 550002
to wayne's LDAP entry on domain-controller. Any future IPA user with a
manually-assigned uid outside the auto-range needs the same treatment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 17:25:37 +10:00
beatzaplentyandClaude Sonnet 4.6 3f9b968a41 fix(ha/deploy): use non-sudo SSH for /nix writability check
pve_check() always uses sudo, so "sudo test -w /nix" passes as root
regardless of whether the SSH user can actually write there.  Use a
direct non-sudo SSH command for the writability probe so the check
reflects wayne's own access, not root's.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:15:20 +10:00
beatzaplentyandClaude Sonnet 4.6 1263c7540c fix(ha/deploy): ensure remote clone is on correct branch before building
When create-proxmox-resource.sh clones the repo to pve1, it stays on
whatever branch was checked out. Add a pre-build phase that detects
branch mismatch and switches the remote clone to the current local
branch before building, so the Proxmox node always builds from the
same commits we're deploying.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:12:39 +10:00
beatzaplentyandClaude Sonnet 4.6 a8d8b1465c fix(ha/deploy): fix /nix ownership on pve1 before codex-setup
pve1 has a pre-existing /nix store owned by a different UID; wayne's IPA
UID (50002) can't write to it.  Add a pre-phase-3 check that uses wayne's
passwordless sudo to chown -R the store before create-proxmox-resource.sh
runs codex-setup.sh on the node.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:11:05 +10:00
beatzaplentyandClaude Sonnet 4.6 b76d54e702 fix(ha): fix deploy script bugs and correct cluster configuration
- Add root SSH key + passwordless sudo to ha-server nodes (needed for
  deploy script to run cluster-init.sh via SSH as root on node1)
- Fix cluster-init.sh: correct default IPs (228/227/229 per variables.nix),
  use \${VAR:-default} for all config so deploy.sh can override via env
- Fix acceptance-tests.sh: same IP corrections, add -i flag to SSH calls,
  use \${VAR:-default} pattern
- Fix deploy.sh dry-run bugs: pve_check() always runs SSH for read-only
  probes so bridge existence check is accurate; wait_for_ssh skips in
  dry-run instead of timing out
- Fix cluster-init invocation: upload script via scp and run via SSH as
  root on node1 (was incorrectly trying to run the script locally)
- Fix acceptance-tests invocation: pass IP env vars from deploy.sh

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:04:13 +10:00
beatzaplentyandClaude Sonnet 4.6 59854a0229 feat(ha): implement clan vars and fully encrypt all HA secrets
- Run sync-host-keys.sh for proxmox-ha-server-{1,2}: generates SSH host
  key pairs in vars/per-machine/, registers age anchors in .sops.yaml,
  adds both hosts as recipients for common.yaml, ha-corosync-authkey,
  and per-host secrets/keytab files
- Re-encrypt secrets/common.yaml with both new host keys
- Convert all stub secrets to real sops-encrypted files:
    secrets/ha-server-{1,2}.yaml    (YAML, beszel-token = PLACEHOLDER)
    secrets/ha-server-{1,2}.keytab  (binary, stub text encrypted)
    secrets/ha-corosync-authkey      (binary, stub text encrypted)
- Add scripts/ha/deploy.sh: full lifecycle script (bridge setup, VM
  creation, DRBD disk + storage NIC attachment, boot wait, cluster-init,
  acceptance tests, --destroy)

Bootstrap order (operator runs these before first deploy):
  1. bash scripts/ha/deploy.sh            # deploys, tests
  # Post-deploy secret replacement:
  2. sops secrets/ha-server-{1,2}.yaml   (set real beszel-token)
  3. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
  4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
  5. Set services.beszel.agent.environment.KEY in host.nix after hub pairing
  6. nixos-rebuild switch on both nodes to pick up real secrets

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 16:48:04 +10:00
beatzaplentyandClaude Sonnet 4.6 dc83333526 feat(ha): pre-stage IPA enrollment for ha-server-1 and ha-server-2
- Add stub keytab files (secrets/ha-server-{1,2}.keytab) so modules/ipa/client.nix
  activates for both nodes — IPA security.ipa.enable = true confirmed by eval
- Add .sops.yaml creation rules for both keytabs (admin-only until
  sync-host-keys.sh registers the host age keys)

Eval: both proxmox-ha-server-1 and proxmox-ha-server-2 build cleanly
with IPA enabled, beszel agent enabled, corosync authkey + beszel-token
+ ipa-host-keytab all wired via sops-nix.

Bootstrap order before first deploy:
  1. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
  2. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
  3. sops updatekeys secrets/common.yaml
  4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
  5. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
  6. sops secrets/ha-server-1.yaml  (set beszel-token)
  7. sops secrets/ha-server-2.yaml  (set beszel-token)
  8. On node1 after first boot: corosync-keygen, then
     sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey
  9. Set services.beszel.agent.environment.KEY in each host.nix after hub pairing
 10. git add/commit the real secrets, then nixos-rebuild switch on both nodes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 16:30:29 +10:00
beatzaplenty a960c662f0 Merge pull request 'feat(ipa/docker): grant docker access via IPA group membership' (#90) from worktree-docker-ipa-group into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #90
2026-07-28 05:48:45 +00:00
beatzaplenty 6f602b2245 Merge pull request 'chore: full sweep — docs sync, SSH key module extraction, NFS dedup, dead code removal' (#89) from worktree-full-sweep into main
Check NixOS configurations / eval-hosts (push) Successful in 10m39s
Reviewed-on: #89
2026-07-28 05:26:57 +00:00
beatzaplentyandClaude Sonnet 4.6 e62e9c9a6a chore: full sweep — docs sync, SSH key module extraction, NFS dedup, dead code removal
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m40s
Documentation fixes:
- README/AGENTS: rename tailscale-exit-node → tailscale-router, add ha-server
  build type and proxmox-ha-server-{1,2} host table rows, add baremetal to
  platform list, remove references to non-existent flake-target-refactor-spec.md
  and remove-sensetive-info-refactor.md
- docs/auto-installer.md: fix lxc-tailscale-exit-node → lxc-tailscale-router,
  add pxe-minimal to the flake outputs list
- variables.nix: fix domainControllerIp comment — IPA is the authoritative DNS
  at .253 (Pi-hole is gone), not a forwarding intermediary

Code deduplication:
- Extract duplicate SSH host-key preservation activation scripts from
  modules/platforms/lxc.nix and modules/platforms/proxmox.nix into a shared
  modules/common/preserve-ssh-host-key.nix; both platforms now import it
- Replace 8-line hand-enumerated NFS export lists in server.nix and ha-server.nix
  with a mkNfsExports helper that generates exports from vars.nfsShares — adding
  a share to variables.nix now propagates to both exporters automatically

Dead code removal:
- modules/common/configuration.nix: remove leftover NixOS skeleton comments
  (hardware-configuration import, grub lines) that were never used
- modules/docker/enable-service.nix: remove commented-out listenOptions and
  daemon.settings blocks
- hosts/server/host.nix, hosts/nix-cache/host.nix: remove #DOCKER_HOST comments

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 15:18:06 +10:00
beatzaplentyandClaude Sonnet 4.6 5beed2d75c feat(ipa/docker): grant docker access via IPA group membership
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m39s
Create an IPA group "docker-access" (GID 50010) and pin the local
"docker" group to that GID on all Docker hosts.  Any IPA user in the
docker-access group automatically gains docker socket access through
SSSD supplementary-group resolution — no per-host docker.members
entry needed.

Specific changes:
- variables.nix: add dockerAccessGid = 50010
- modules/docker/enable-service.nix: lib.mkForce docker GID to
  dockerAccessGid, removing the need to name individual IPA users
- modules/build-types/docker.nix: remove direct wayne docker.members
  entry (access now comes from IPA group)
- modules/ipa/client.nix: refactor repeated security.* / systemd.*
  top-level keys into merged attribute sets (fixes statix W20); add
  security.pam.services.lightdm.makeHomeDir so the GUI login path
  also creates the home dir on first login

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 15:16:05 +10:00
beatzaplentyandClaude Sonnet 4.6 fe7fc55c04 feat(docker): add IPA wayne user to docker group
Check NixOS configurations / eval-hosts (push) Failing after 12m13s
Grants the domain wayne account docker socket access on docker.sweet.home
without needing sudo, alongside the existing nixos local user membership.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 14:07:34 +10:00
beatzaplentyandClaude Sonnet 4.6 6cdae391f4 fix(docker): pin IPA dyndns to eth0 to prevent Docker bridge IP registration
SSSD's dyndns_iface defaults to "*" which registers every interface's IP in
IPA DNS, including all Docker bridge networks (172.x.x.x). This caused
docker.sweet.home to resolve to Docker bridge IPs instead of 192.168.2.225.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 14:07:34 +10:00
beatzaplenty 95eb494370 Merge pull request 'Worktree ipa lxc kerberos fix' (#88) from worktree-ipa-lxc-kerberos-fix into main
Check NixOS configurations / eval-hosts (push) Failing after 9m47s
Reviewed-on: #88
2026-07-28 03:39:23 +00:00
beatzaplentyandClaude Sonnet 4.6 9294d2fd25 fix(ipa): skip home-manager-wayne service when home dir absent
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m42s
On first enrollment /home/wayne doesn't exist until the IPA user's first
login (pam_mkhomedir creates it). home-manager-<user>.service cd's into
the home dir immediately and fails with ENOENT, causing the whole rebuild
activation to return exit code 4.

Add ConditionPathExists so systemd skips the service (condition not met,
no failure) instead. After first login the dir exists and subsequent
rebuilds activate Home Manager normally.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 13:38:49 +10:00
beatzaplentyandClaude Sonnet 4.6 0fe7ddf6e8 fix(ipa): reject FQDN input in create-nixos-ipa-host-account.sh
Passing a FQDN like "nixos.sweet.home" instead of the short hostname
"nixos" caused the script to create a double-FQDN IPA host account
(nixos.sweet.home.sweet.home). Add an early check that rejects any
TARGET containing a dot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 13:23:48 +10:00
beatzaplenty 5c2d61d35a Merge pull request 'fix(ipa): use NOPASSWD sudo for IPA user to bypass broken PAM path' (#87) from worktree-ipa-lxc-kerberos-fix into main
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Reviewed-on: #87
2026-07-28 03:16:03 +00:00
beatzaplentyandClaude Sonnet 4.6 186e9187ce fix(ipa): use NOPASSWD sudo for IPA user to bypass broken PAM path
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m44s
HM's useUserPackages creates a users.users stub for every configured HM
user, which lands wayne in /etc/passwd. NixOS adds pam_sss.so with the
"localusers" flag to the sudo PAM stack when SSSD is enabled; that flag
causes pam_sss to skip SSSD for any user found in local /etc/passwd,
falling through to pam_unix which has no shadow password for the stub.
Result: sudo auth always fails for the IPA user despite being in wheel.

Use NOPASSWD for the IPA user in sudoers instead. The IPA user already
authenticated to reach a shell (SSH key from IPA or Kerberos), so
re-prompting via a broken PAM path is security theater on a homelab.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 13:12:42 +10:00
beatzaplenty e6f15404f5 Merge pull request 'fix(ipa): add wheel group to IPA user stub for sudo access' (#86) from worktree-ipa-lxc-kerberos-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 10m54s
Reviewed-on: #86
2026-07-28 03:00:33 +00:00
beatzaplentyandClaude Sonnet 4.6 44a0acc18f fix(ipa): add wheel group to IPA user stub for sudo access
Check NixOS configurations / eval-hosts (pull_request) Failing after 28m20s
initgroups() uses NSS (groups: files sss) to build the supplemental
group list at login. Adding wheel to the local users.users stub means
the IPA user gets wheel membership from /etc/group, satisfying sudo's
group check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 12:59:46 +10:00
beatzaplenty 1fc6e63178 Merge pull request 'Worktree ipa lxc kerberos fix' (#85) from worktree-ipa-lxc-kerberos-fix into main
Check NixOS configurations / eval-hosts (push) Failing after 33m33s
Reviewed-on: #85
2026-07-28 02:55:20 +00:00
beatzaplentyandClaude Sonnet 4.6 3589fc31d7 feat(ipa): add Home Manager config for IPA primary user
Check NixOS configurations / eval-hosts (pull_request) Failing after 40m50s
Any enrolled host now automatically gets a Home Manager profile for the
IPA primary user (vars.ipaUser = "wayne"), covering what IPA doesn't:
dotfiles, user-scoped packages (tmux, sshfs), and EDITOR variable.

The home directory is pre-created by systemd-tmpfiles so HM activation
succeeds on steady-state systems before first login; pam_mkhomedir remains
as a fallback for fresh deploys where SSSD hasn't cached the user yet.

A minimal users.users stub satisfies NixOS's assertion requirements
(isNormalUser + group) that arise because home-manager.useUserPackages
creates a users.users entry to install packages to /etc/profiles/per-user/.
The stub is shadowed by SSSD at runtime (security.ipa sets passwd: sss files).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 12:53:10 +10:00
beatzaplentyandClaude Sonnet 4.6 89746718a9 feat(shell): make Switch-nix/Test-nix/buildImage system-wide
IPA users (e.g. wayne@) don't get Home Manager so the aliases defined in
aliases.nix were invisible to them. Move Switch-nix, Test-nix, and
buildImage into programs.bash in configuration.nix so every user on every
host gets them via /etc/bashrc. Stub out aliases.nix for future per-user
HM-only additions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 12:32:33 +10:00
beatzaplenty 232d0e7c40 Merge pull request 'fix(ipa): suppress Kerberos NFS services in LXC containers' (#84) from worktree-ipa-lxc-kerberos-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #84
2026-07-28 02:13:47 +00:00
beatzaplentyandClaude Sonnet 4.6 0b60d3ff2d fix(ipa): suppress Kerberos NFS services in LXC containers
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m30s
The NixOS IPA module adds ConditionPathExists=/etc/krb5.keytab drop-ins
for auth-rpcgss-module.service and rpc-gssd.service via systemd.units.
In LXC containers with keytabs those conditions pass, the services start,
and then fail because auth_rpcgss can't be loaded and rpc_pipefs doesn't
exist in the container namespace.

Use lib.mkForce on our systemd.units text to win the conflict with NixOS's
existing definitions, and include ConditionVirtualization=!container
alongside the ConditionPathExists conditions so the services are skipped
(inactive, not failed) in containers that have a keytab.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 12:07:58 +10:00
beatzaplenty ff82e8885d added remaining ipa keytabs
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
2026-07-28 11:45:06 +10:00
beatzaplenty 9a7411d1dd Merge pull request 'Worktree ipa common config' (#83) from worktree-ipa-common-config into main
Check NixOS configurations / eval-hosts (push) Failing after 9m46s
Reviewed-on: #83
2026-07-28 01:43:53 +00:00
beatzaplentyandClaude Sonnet 4.6 2f829ba3e7 refactor(ipa): move FreeIPA client to common config with auto-enrollment
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m48s
modules/ipa/client.nix is now a self-contained NixOS module (no longer a
parameterized function): it checks builtins.pathExists for
secrets/<hostname>.keytab at eval time and enables itself automatically if
found, making it a no-op for hosts without a keytab.

modules/common/configuration.nix imports it so every host in the flake is
a candidate for IPA enrollment — no per-host wiring needed. Adding a
keytab (via scripts/ipa/create-nixos-ipa-host-account.sh) is now the
only step required to enroll a host.

The module also sets networking.domain and networking.nameservers via
mkDefault when active, so new hosts don't need those set explicitly.

Also:
- Remove explicit IPA imports from hosts/nix-cache and hosts/tailscale-router
- Add secrets/pxe-boot.keytab + creation rule; remove incorrect
  secrets/nixos.sweet.home.keytab and its creation rule
- Add .sops.yaml creation rules for all remaining host keytabs
  (server, docker, tor-relay, nix-minimal, nixos) so the creation script
  can target them without manual .sops.yaml edits
- Fix duplicate tailscale-router.keytab rule and corrupted gui.yaml comment
  block in .sops.yaml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 11:15:59 +10:00
beatzaplenty dedd69dc42 secrets: add IPA keytab for nixos.sweet.home 2026-07-28 10:43:43 +10:00
beatzaplenty f7f670ca4c Merge remote-tracking branch 'origin/worktree-docker-gui-fix'
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
2026-07-28 10:33:15 +10:00
beatzaplenty 55ba283c82 Merge branch 'main' into worktree-docker-gui-fix
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m24s
2026-07-28 00:32:06 +00:00
beatzaplenty 2d46d25a67 Merge pull request 'fix(ipa): harden script and update module docs' (#82) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Reviewed-on: #82
2026-07-28 00:25:13 +00:00
beatzaplentyandClaude Sonnet 4.6 f5d29be041 fix(ipa): harden script and update module docs
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m53s
Script fixes:
- Rename HOSTNAME variable to TARGET (shadowed the bash builtin)
- Fix ipa-getkeytab -s to always use IPA_SERVER, not DC_HOST (diverge if
  --dc is overridden to a jump host)
- Remove dead REALM variable
- Add EXIT trap to delete the plaintext keytab if the script aborts before
  sops encryption completes; cleared after successful encrypt
- Distinguish real ipa host-add failures from "already exists" instead of
  swallowing all errors with || true
- Warn explicitly when no platform age keys exist for the target (keytab
  would be admin-only and the host couldn't decrypt it at boot)
- Fix sops fallback from pinned nixos-25.11 channel to nixpkgs (uses the
  repo's own flake.lock)
- Expand "next steps" output to include networking.domain and nameservers
  lines that host.nix requires for IPA membership

Module docs:
- Point to the script as the primary setup path; move manual steps to a
  fallback section
- Note that certs/ipa-ca.crt is already committed (no need to re-fetch)
- Document the networking.domain and nameservers requirements in the header
- Add sync-host-keys.sh as explicit step 0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 10:24:23 +10:00
beatzaplenty e10a1572c3 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
2026-07-28 10:23:34 +10:00
beatzaplenty 578ef70aa9 updated flake.lock 2026-07-28 10:23:23 +10:00
beatzaplenty e9fcbbbbcb Merge pull request 'Worktree ipa client module' (#80) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Failing after 9m47s
Reviewed-on: #80
2026-07-28 00:13:08 +00:00
beatzaplentyandClaude Sonnet 4.6 f46ae18672 fix(ipa): work around OpenSSH 10 AuthorizedKeysCommand path check
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m43s
OpenSSH 10.0 tightened AuthorizedKeysCommand security by checking every
path component of the command binary for group/world-write permission.
/nix/store is 1775 (group-writable by nixbld), so sshd silently skips
the command for any binary in the Nix store — causing IPA pubkey auth to
silently fail with no diagnostic.

Fix: copy sss_ssh_authorizedkeys to /usr/local/bin via systemd tmpfiles
(C+ copies the file rather than symlinking, so the path at runtime is
root-owned/755 throughout), and point AuthorizedKeysCommand at the copy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 10:08:57 +10:00
beatzaplenty fc277294f3 add vars to module
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
2026-07-28 10:02:58 +10:00
beatzaplenty f3e5ea67a0 add docker group to user in enable docker service module
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
2026-07-28 09:59:48 +10:00
beatzaplenty 7a8aebf679 Merge branch 'worktree-docker-gui-fix' 2026-07-28 09:59:40 +10:00
beatzaplenty a8d95aad02 enable docker
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
2026-07-28 09:49:14 +10:00
beatzaplentyandClaude Sonnet 4.6 dac5fbd574 feat(gui): enable docker service on workstation
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m45s
Imports docker/enable-service.nix and pins virtualisation.docker to
docker_29 via a nixpkgs overlay. Fixes indentation on `imports` block
and `programs.direnv.enable` line so nixpkgs-fmt passes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:48:59 +10:00
beatzaplenty da0c651c60 Merge branch 'worktree-ipa-client-module' 2026-07-28 09:48:55 +10:00
beatzaplentyandClaude Sonnet 4.6 97019205da fix(ipa): create home dir on first login + AuthorizedKeysCommand
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m45s
- security.pam.services.sshd.makeHomeDir: IPA users have no pre-created
  home directory on the host; without this, sshd opens a session to a
  missing directory and resets the connection immediately after auth
- AuthorizedKeysCommand was already added in previous commit

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:47:11 +10:00
beatzaplentyandClaude Sonnet 4.6 5487490b8e feat(ipa): add AuthorizedKeysCommand + enroll tailscale-router
- modules/ipa/client.nix: add AuthorizedKeysCommand so sshd fetches SSH
  public keys from IPA via sss_ssh_authorizedkeys, enabling pubkey login
  without per-host authorized_keys files

- hosts/tailscale-router/host.nix: add IPA client module + networking.domain
  so SSSD runs and wayne can authenticate on this host

- secrets/tailscale-router.keytab: sops-encrypted keytab for
  tailscale-router.sweet.home (generated by create-nixos-ipa-host-account.sh)

- .sops.yaml: creation rule for secrets/tailscale-router.keytab

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:42:38 +10:00
beatzaplenty 543ea432f0 Merge pull request 'fix(tailscale-router): stop dnsmasq from intercepting host DNS queries' (#79) from worktree-peaceful-stirring-noodle into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #79
2026-07-27 23:27:34 +00:00
beatzaplentyandClaude Sonnet 4.6 c7bbf88dce fix(tailscale-router): stop dnsmasq from intercepting host DNS queries
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m30s
NixOS's dnsmasq module defaults resolveLocalQueries to true, which adds
127.0.0.1 to networking.nameservers and binds dnsmasq to listen-address=127.0.0.1.
This made the host route all its own DNS through dnsmasq, which had
no-resolv=true and no upstream for anything outside the tailnet domain —
so every non-tailscale DNS query from the host itself (including SSSD
resolving the IPA server FQDN after the IPA client module was added) failed.

Setting resolveLocalQueries=false limits dnsmasq to its intended role: a
forwarding proxy reachable on the LAN interface for IPA's conditional
forwarder. The host uses domainControllerIp directly (already set in
networking.nameservers in host.nix).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:26:50 +10:00
beatzaplenty d57145b31e add tailscale-router to domain
Check NixOS configurations / eval-hosts (push) Failing after 9m44s
2026-07-28 09:17:29 +10:00
beatzaplenty 1cbe80c0ca secrets: add IPA keytab for tailscale-router 2026-07-28 09:15:21 +10:00
beatzaplenty 01ee261cf8 Merge pull request 'fix(ipa): stream keytab via sudo cat instead of scp' (#78) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Reviewed-on: #78
2026-07-27 23:14:41 +00:00
beatzaplentyandClaude Sonnet 4.6 f6f30c675f fix(ipa): stream keytab via sudo cat instead of scp
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m25s
ipa-getkeytab runs as root via sudo so the temp file is root-owned;
scp as wayne gets Permission denied. Pipe through `sudo cat` over SSH
instead, which reads as root but writes locally as the invoking user.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:08:19 +10:00
beatzaplenty 60b80cbd96 Merge pull request 'fix(ipa): SSH as wayne with sudo instead of root on domain controller' (#77) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
Reviewed-on: #77
2026-07-27 23:06:49 +00:00
beatzaplentyandClaude Sonnet 4.6 27a8c7fad9 fix(ipa): SSH as wayne with sudo instead of root on domain controller
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m23s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:02:32 +10:00
beatzaplenty d9cee0a674 Merge pull request 'feat(ipa): add create-nixos-ipa-host-account script' (#76) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Reviewed-on: #76
2026-07-27 22:58:46 +00:00
beatzaplentyandClaude Sonnet 4.6 6c1891812e feat(ipa): add create-nixos-ipa-host-account script
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m24s
Single command to enroll a NixOS host in FreeIPA and produce a
sops-encrypted keytab at secrets/<hostname>.keytab:
  - Adds the .sops.yaml creation rule automatically (with all registered
    platform-variant age keys as recipients)
  - SSHes to the domain controller to run ipa host-add + ipa-getkeytab
  - Refreshes the admin Kerberos ticket via `ssh -t ... kinit admin` if
    missing or expired, so no manual kinit step is needed
  - SCPs the keytab and encrypts it in-place with sops (file must be at
    secrets/<hostname>.keytab before encryption so the path-based creation
    rule matches — the common failure point when doing this manually)

Also adds HOME_DOMAIN and IPA_SERVER to scripts/env.sh, matching
variables.nix's homeDomain/ipaServer (same manual-sync pattern as
NIX_CACHE_HOST/LAN_DOMAIN).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 08:57:55 +10:00
beatzaplenty 59316c982e Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
2026-07-28 07:26:40 +10:00
beatzaplenty 9eca3bd719 Merge pull request 'Worktree ipa client module' (#75) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Reviewed-on: #75
2026-07-27 21:25:57 +00:00
beatzaplenty f2f0fcf756 certs: add FreeIPA CA certificate
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m45s
2026-07-28 07:25:28 +10:00
beatzaplenty b4fb9c25f2 secrets(nix-cache): add sops-encrypted IPA host keytab 2026-07-28 07:25:25 +10:00
beatzaplentyandClaude Sonnet 4.6 8955840f0a fix(ipa): use in-place sops encryption in module docs
sops matches creation rules against the input file path, so encrypting
/tmp/<host>.keytab directly with stdout redirect fails to find the rule.
Copy to secrets/ first, then use -i to encrypt in-place.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 07:19:53 +10:00
beatzaplentyandClaude Sonnet 4.6 a4b49c9909 fix(ipa): correct ipaServer to domain-controller.sweet.home
ipa-getkeytab confirmed ipa.sweet.home doesn't respond to LDAP;
domain-controller.sweet.home is the actual IPA server FQDN.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 07:16:53 +10:00
beatzaplenty c0936a10e7 Merge pull request 'feat(ipa): add reusable declarative FreeIPA client module' (#74) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Reviewed-on: #74
2026-07-27 21:06:28 +00:00
beatzaplentyandClaude Sonnet 4.6 f4bbd6331d feat(ipa): add reusable declarative FreeIPA client module
Check NixOS configurations / eval-hosts (pull_request) Failing after 10m0s
Adds modules/ipa/client.nix — a parameterized module that joins a NixOS host
to the sweet.home FreeIPA domain without ipa-client-install. It configures
security.ipa (SSSD, Kerberos, PAM, NSSwitch) and places a pre-provisioned host
keytab via sops-nix binary secret so enrollment is fully reproducible from the
flake.

- variables.nix: adds ipaServer (FQDN of the FreeIPA KDC; security.ipa.server
  requires a hostname, not an IP, for Kerberos/TLS)
- certs/ipa-ca.crt: placeholder for the IPA CA public certificate (operator
  replaces with: curl http://<ipa-server>/ipa/config/ca.crt)
- secrets/nix-cache.keytab: placeholder binary sops file (operator replaces
  with the encrypted keytab after ipa host-add + ipa-getkeytab)
- .sops.yaml: adds creation rule for secrets/nix-cache.keytab (same recipients
  as secrets/nix-cache.yaml)
- hosts/nix-cache/host.nix: imports the IPA client module; adds
  networking.domain so the host's FQDN resolves correctly

Module header documents the three operator steps needed per host before deploy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 23:31:21 +10:00
beatzaplenty b99ba87cf6 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos 2026-07-27 23:05:35 +10:00
beatzaplenty 2128353f9f Merge pull request 'Fix/static ips' (#73) from fix/static-ips into main
Check NixOS configurations / eval-hosts (push) Failing after 42m27s
Reviewed-on: #73
2026-07-27 12:06:33 +00:00
beatzaplentyandClaude Sonnet 4.6 7b4ce0ab3d fix(server): prevent zfs-init-tank from wiping pool on udev race
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m46s
zpool create -f was called if `zpool import -d /dev/disk/by-id` failed,
which could happen due to a race with systemd-udev-settle. The disk
would then be visible by the time zpool create ran, silently destroying
all data on an otherwise-intact pool.

Fix: locate the data disk first, retry the import directly against it
as a fallback, then check zdb -l for existing ZFS label metadata before
concluding the disk is blank. Remove -f so zpool create refuses rather
than overwrites if a pool is present.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 21:52:39 +10:00
beatzaplentyandClaude Sonnet 4.6 3123565011 fix(tailscale-router): scope ts.net forwarder to tailnet subdomain
IPA refuses to create a forward zone for ts.net because it's a real
public TLD with DNSimple nameservers. The forward zone must use the
tailnet-specific subdomain (vars.tailnetDomain, e.g. tail13f623.ts.net)
instead. Update dnsmasq server selector and comments to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 19:23:46 +10:00
beatzaplentyandClaude Sonnet 4.6 36f5ebdf86 feat(tailscale-router): serve ts.net DNS forward zone for LAN hosts
FreeIPA (the new authoritative DNS) cannot reach 100.100.100.100
(Tailscale's internal MagicDNS resolver) directly because the DC is not
a Tailscale node. The tailscale-router IS a Tailscale node and can
reach 100.100.100.100 via tailscale0, so it now runs a dnsmasq
instance on its LAN interface that forwards all ts.net queries to
Tailscale's resolver.

After deploying this host, configure FreeIPA with:
  kinit admin
  ipa dnsforwardzone-add ts.net \
    --forwarder=192.168.2.222 \
    --forward-policy=only

This replaces Pi-hole's conditional forwarder for ts.net and restores
resolution of Tailscale MagicDNS names (e.g. raspberrypi.tail13f623.ts.net)
for all LAN hosts using FreeIPA as their DNS server.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 19:20:12 +10:00
beatzaplenty bba054db85 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
# Conflicts:
#	variables.nix
2026-07-27 18:29:07 +10:00
beatzaplenty b63a529a1d Merge branch 'fix/static-ips' 2026-07-27 18:25:37 +10:00
beatzaplentyandClaude Sonnet 4.6 ee96713a50 feat(networking): declare static IPs for all NixOS-managed hosts in flake
Adds static IP configuration to every NixOS host in the flake that
has a fixed LAN address, and centralises all network primitives
(IPs, gateway, prefix length, interface names) in variables.nix so
there is one place to update if any of them change.

variables.nix additions:
- lanGateway / lanPrefixLength — LAN gateway and /24 prefix, replacing
  every hardcoded 192.168.2.254 / 24 across host files
- lxcLanInterface / vmLanInterface / vmStorageInterface — NIC names for
  LXC containers (eth0), Proxmox VMs (ens18), and the HA storage NIC
  (ens19), used as attribute keys so changing the name is a one-line edit
- haStoragePrefixLength — /29 for the storage subnet, mirrors haStorageCidr
- Per-host IP variables: nixCacheIp (.224), tailscaleRouterIp (.222),
  torRelayIp (.221), serverIp (.226), dockerIp (.225)

host.nix changes:
- tailscale-router, tor-relay, nix-cache, pxe-boot: useDHCP = false,
  static address on eth0 (lxcLanInterface), struct-form defaultGateway
  (required when using systemd-networkd which LXC containers use)
- server, docker: useDHCP = false, static address on ens18 (vmLanInterface),
  struct-form defaultGateway (works for both scripted networking and networkd)
- ha-server-1, ha-server-2: replace hardcoded 192.168.2.254 / 24 / 29
  with the new variables; no functional change for these hosts

modules/build-types/pxe-boot.nix:
- Domain-controller kickstart template: replace hardcoded 192.168.2.138
  and 192.168.2.254 with vars.domainControllerIp / vars.lanGateway /
  vars.lanPrefixLength / vars.homeDomain so the template stays correct
  if the DC IP or domain is ever changed again

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 18:14:31 +10:00
beatzaplenty 1ece0c75d9 Merge pull request 'chore(pxe-boot): remove log-dhcp debug flag now that PXE boot is confirmed working' (#86) from fix/pxe-dnsmasq-port-conflict into main
Reviewed-on: #86
2026-07-27 05:52:18 +00:00
beatzaplentyandClaude Sonnet 4.6 548c6f5041 chore(pxe-boot): remove log-dhcp debug flag now that PXE boot is confirmed working
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 15:50:19 +10:00
beatzaplenty bae4c8171f Merge pull request 'fix(pxe-boot): use current pxe-boot IP (.247) in dnsmasq until Stage 5 renumber' (#85) from fix/pxe-dnsmasq-port-conflict into main
Reviewed-on: #85
2026-07-27 05:41:38 +00:00
beatzaplentyandClaude Sonnet 4.6 3748c86049 fix(pxe-boot): use current pxe-boot IP (.247) in dnsmasq until Stage 5 renumber
vars.pxeServerIp was already set to the post-renumber target (.223) but the
pxe-boot CT is still at .247, so dnsmasq was advertising .223 as the TFTP server
and the VM couldn't reach it.  Update to .247 so PXE boot works now; the comment
reminds us to flip it back to .223 when Stage 5 step 6 renumbers the CT.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 15:40:54 +10:00
beatzaplenty 9dd969cf4c Merge pull request 'fix(pxe-boot): open UDP 4011 for PXE boot service discovery' (#84) from fix/pxe-dnsmasq-port-conflict into main
Reviewed-on: #84
2026-07-27 05:22:46 +00:00
beatzaplentyandClaude Sonnet 4.6 7e4b2d33fb fix(pxe-boot): open UDP 4011 for PXE boot service discovery
After the proxy DHCP handshake, the UEFI client queries the PXE boot
server on UDP port 4011 to get the actual boot filename. This port was
not open in the NixOS firewall, so all four PXE discovery attempts from
the test VM silently timed out. dnsmasq was bound to 0.0.0.0:4011 but
never received the packets.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 15:12:12 +10:00
beatzaplenty 288d50fd33 Merge pull request 'Fix/pxe dnsmasq port conflict' (#83) from fix/pxe-dnsmasq-port-conflict into main
Reviewed-on: #83
2026-07-27 05:05:42 +00:00
beatzaplentyandClaude Sonnet 4.6 f0e76f8aff fix(pxe-boot): use pxe-service for proxy DHCP initial boot offers
dnsmasq 2.93 does not send proxy DHCP offers from dhcp-boot alone in proxy
mode — pxe-service is required to trigger proxy offers for the initial boot.

Replace the dhcp-boot-based arch detection with pxe-service entries for
arch 0 (BIOS), 7 (EFI BC / OVMF), and 9 (EFI x86-64), then use dhcp-boot
only for the iPXE second-boot HTTP chain (tag:ipxe clients). Add log-dhcp
for post-deploy verification.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 15:03:01 +10:00
beatzaplenty e80d195284 Merge branch 'fix/pxe-dnsmasq-port-conflict' 2026-07-27 14:32:00 +10:00
beatzaplentyandClaude Sonnet 4.6 c770feebc9 fix(pxe-boot): disable dnsmasq DNS listener to avoid port 53 conflict
dnsmasq binds port 53 by default even when only proxy DHCP is needed.
systemd-resolved already owns port 53 on the pxe-boot container, so
dnsmasq crashes at startup. Setting port=0 disables the DNS listener
entirely — pxe-boot only needs the proxy DHCP path for iPXE chainloading.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 14:29:30 +10:00
beatzaplenty 7fd6d558d5 updated dnsmasq service 2026-07-27 14:07:28 +10:00
beatzaplentyandClaude Sonnet 4.6 58c40292e2 fix(pxe-boot): move dnsmasq out of systemd.services into services.dnsmasq
dnsmasq config was placed inside systemd.services{} block, making Nix
interpret it as systemd.services.dnsmasq.settings (which doesn't exist).
Move to top-level services.dnsmasq and add UDP 67 to firewall for DHCP
proxy mode.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 14:02:41 +10:00
beatzaplenty 94842875d0 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos into worktree-network-cutover-plan 2026-07-27 14:01:24 +10:00
beatzaplenty cfa36b97fc added dnsmasq to pxe-boot 2026-07-27 13:44:29 +10:00
beatzaplentyandClaude Sonnet 4.6 4444398cac fix(network): wire correct IPs throughout and add cutover plan
variables.nix:
- HA server LAN IPs: .200/.201/.202 → .228/.227/.229 (from ip-addressing.md)
- Add haServer1StorageIp (.228), haServer2StorageIp (.227) for 192.168.4.0/29
- Add haStorageCidr for firewall rules

ha-server host.nix (both nodes):
- Add ens19 interface on storage subnet (/29)
- Fix defaultGateway: 192.168.2.1 → 192.168.2.254
- Fix nameservers: 192.168.2.1/8.8.8.8 → domainControllerIp (.253)

cluster-config.nix:
- DRBD replication addresses: LAN IPs → storage IPs (keep replication off LAN)
- Corosync ring_addrs: LAN IPs → storage IPs
- Firewall: add haStorageCidr to allowed sources

docs/network-cutover.md: step-by-step cutover plan with rollback at every stage

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 12:32:54 +10:00
beatzaplenty f80378f92f Merge pull request 'docs(network): establish IP addressing scheme and update infra IPs' (#82) from worktree-network-ip-scheme into main
Reviewed-on: #82
2026-07-27 02:23:01 +00:00
beatzaplentyandClaude Sonnet 4.6 cd4997f429 docs(network): establish IP addressing scheme and update infra IPs
Defines the new structured 192.168.2.0/24 layout:
- .10–.59   client DHCP (router-assigned, DNS → .253)
- .220–.229 virtual nodes (VMs / LXC containers)
- .230–.239 expansion buffer
- .240–.249 physical nodes (pve1 at .245, PBS at .244)
- .250–.253 network services (router .254, FreeIPA/DC .253)

Storage network 192.168.4.0/29 defined for HA DRBD replication
(internal vmbr1 bridge, no uplink). Host octet matches LAN throughout.

Updates variables.nix: pxeServerIp .247→.223, pbsIp .108→.244,
adds domainControllerIp .253. Updates pxe-boot.md IP references.
Full migration before/after table in docs/ip-addressing.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 12:19:45 +10:00
beatzaplenty 6ce4784376 Merge pull request 'Worktree ha file server test' (#81) from worktree-ha-file-server-test into main
Reviewed-on: #81
2026-07-27 02:18:06 +00:00
beatzaplentyandClaude Sonnet 4.6 5856d45575 feat(ha): wire sops secrets and disable NetworkManager for HA servers
- cluster-config.nix: add corosync_authkey sops binary secret
  (/etc/corosync/authkey, mode 0400) and force-disable NetworkManager
  (common config enables it; HA nodes need stable static IP networking)
- hosts/ha-server-{1,2}/host.nix: add host-token.nix import for
  sops-managed beszel-token; add KEY placeholder for beszel hub pairing
- .sops.yaml: add creation rules for secrets/ha-server-{1,2}.yaml and
  secrets/ha-corosync-authkey (admin-only until sync-host-keys.sh runs)
- secrets/ha-server-{1,2}.yaml, secrets/ha-corosync-authkey: stub files
  so eval passes before real secrets are provisioned

Bootstrap order (post-merge):
  1. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
  2. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
  3. sops updatekeys secrets/common.yaml  (grants HA nodes common secrets)
  4. sops secrets/ha-server-{1,2}.yaml   (set beszel-token values)
  5. On node1: corosync-keygen; sops -e --input-type binary
     /etc/corosync/authkey > secrets/ha-corosync-authkey; git add/commit

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 11:45:36 +10:00
beatzaplentyandClaude Sonnet 4.6 e3498b1087 feat(ha): promote HA file server to production flake targets
Adds proxmox-ha-server-1 and proxmox-ha-server-2 as real mkTarget entries
alongside the existing proxmox-server, backed by a new ha-server build type.

New modules
  modules/ha/cluster-config.nix — DRBD resource + corosync nodelist sourced
    from vars (haServer1Host/Ip, haServer2Host/Ip); resource-only fencing for
    production STONITH; HA port firewall rules for DRBD, iSCSI, Corosync, pcsd
  modules/build-types/ha-server.nix — imports pacemaker-stack + iscsi-target
    + cluster-config + beszel; NFS exports from vars.haStorageRoot (XFS-over-DRBD
    mount); nfs-server.service.wantedBy force-cleared so Pacemaker controls
    start/stop on the Active node only

New hosts
  hosts/ha-server-{1,2}/host.nix — static IP from vars, unique hostId; sops
    secrets (beszel, corosync authkey) are TODOs pending sync-host-keys.sh

variables.nix
  haServer1/2Host, haServer1/2Ip, haServerVip, haStorageRoot, haIscsiIqn
  ports.haServerDrbd/Iscsi/Corosync{1,2,Crypto}/PacemakerRemoted/Pcsd

scripts/ha/ (migrated + updated from test-lab/ha/)
  cluster-init.sh — generates corosync authkey, initialises DRBD/XFS/iSCSI,
    creates NFS dataset dirs, configures Pacemaker with DRBD + XFS + iSCSI
    + nfs-server + VIP; STONITH disabled initially (enable separately)
  cluster-enable-stonith.sh — enables fence_pve_ssh STONITH after key deploy
  fence-pve-ssh.py — Proxmox SSH fence agent (node names updated to ha-server-1/2)
  acceptance-tests.sh — T1–T7 production acceptance tests

test-lab/ha/ removed — all Nix config moved to modules/ha/ and
  modules/build-types/; scripts moved to scripts/ha/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 11:26:37 +10:00
beatzaplentyandClaude Sonnet 4.6 724d9a45af feat(ha): add NixOS modules for DRBD+XFS+LIO+Corosync+Pacemaker HA stack
All 7 acceptance tests pass on live NixOS 25.11 VMs (VMIDs 200/201 on
pve1).  Failover completes in ~5 s with data integrity verified.

modules/ha/pacemaker-stack.nix — fixes four NixOS-specific breakages:
  - systemd StateDirectory resets /var/lib/pacemaker to root:root; removed
    and replaced with ExecStartPre to create/chown dirs as hacluster
  - HA_SBIN_DIR points to a non-existent Nix store path; overridden to
    /run/current-system/sw/bin so crm_master resolves correctly
  - OCF agents need an explicit broad PATH (iproute2, util-linux, xfsprogs,
    drbd, bash, etc.) — NixOS services have no implicit PATH
  - FUSER=true bypasses the psmisc fuser check_binary call in the
    Filesystem OCF agent (psmisc not installed on minimal hosts)

modules/ha/iscsi-target.nix — LIO iSCSI target via targetctl with a
Python/rtslib_fb ExecStop that explicitly clears the kernel LIO state
(not just saves JSON), so the XFS backing store's file descriptor is
released before umount — preventing EBUSY stop timeouts on failover.
Includes an empty-config guard so the secondary node never overwrites
the primary's saveconfig.json with an empty one.

test-lab/ha/common.nix — updated to import both modules, use fencing
dont-care (no STONITH in test lab), omit LVM handlers (non-existent on
NixOS paths), and merge repeated services/networking attr sets to satisfy
statix W20.  test-lab/ha/acceptance-tests.sh — final v4 with crm_standby
fix (pacemaker 3.x API).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 09:13:55 +10:00
beatzaplenty 109429c7da Merge pull request 'feat(pxe-boot): add FreeIPA Server (Rocky Linux 9) iPXE menu entry' (#80) from worktree-freeipa-pxe into main
Reviewed-on: #80
2026-07-26 21:36:58 +00:00
beatzaplentyandClaude Sonnet 4.6 40856b2e5e feat(pxe-boot): add FreeIPA Server (Rocky Linux 9) iPXE menu entry
Adds an unattended install option to the PXE boot menu that installs
Rocky Linux 9 and configures FreeIPA on the domain-controller.sweet.home
host without any operator interaction after selecting the menu entry.

How it works:
- fetch-rocky-pxeboot.service downloads the Rocky 9 Anaconda pxeboot
  kernel and initrd from the Rocky mirror on first pxe-boot deploy
  (idempotent, same pattern as fetch-debian-netboot)
- rocky-freeipa.ipxe boots Anaconda with inst.ks pointing at the
  hosted Kickstart and net.ifnames=0 biosdevname=0 for stable eth0
- rocky-freeipa.ks (generated, includes vars.adminSshKey) performs:
  - Minimal Rocky 9 install with ipa-server + ipa-server-dns
  - Static IP 192.168.2.138 via NM connection file written in %post
  - /etc/hosts fixed for FreeIPA FQDN requirement
  - Random DM + admin passwords generated and saved to
    /root/ipa-credentials.txt (chmod 600, never hardcoded)
  - freeipa-first-boot.service oneshot enabled to run
    ipa-server-install on the first real boot (~20 min)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Jbvxx4xbHVcx1NkK3vtmK
2026-07-27 07:35:13 +10:00
beatzaplentyandClaude Sonnet 4.6 23634134f0 fix(ha-test): remove Debian LVM handlers and fix corosync authkey length
- Remove before/after-resync-target LVM handlers (snapshot-resync-target-lvm.sh
  doesn't exist in NixOS; exit code 127 caused DRBD to drop connections on sync)
- Remove split-brain handler pointing to /usr/lib/drbd/ (Debian path)
- Fix testAuthKey from 126 to 128 bytes (corosync minimum is 1024 bits)
- Fix cluster-init.sh quorum check: corosync-quorumtool has no -q flag;
  use -s | grep 'Quorate: Yes' instead

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 06:40:57 +10:00
beatzaplentyandClaude Sonnet 4.6 34c55f27ca test-lab/ha: nixpkgs-fmt formatting
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 06:18:42 +10:00
beatzaplentyandClaude Sonnet 4.6 dc36a47ac9 test-lab/ha: add session key + enable qemu-guest-agent
Add nixos@nixos session key so the Claude Code session can SSH into
test VMs directly.  Also enable services.qemuGuest.enable so
qm guest exec works as a fallback for key injection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 06:06:53 +10:00
beatzaplentyandClaude Sonnet 4.6 750121e9dd test-lab: add two-node HA file-server test cluster config
Disposable test VMs (ha-test-node1 / ha-test-node2, VMIDs 200/201 on pve1)
to evaluate whether the DRBD + XFS + LIO + Corosync + Pacemaker stack
runs correctly on NixOS before deciding NixOS vs Debian for production.

Includes:
- test-lab/ha/disko.nix: 20G boot disk layout (smaller than production)
- test-lab/ha/common.nix: shared HA stack (drbd, corosync, pacemaker,
  targetcli-fb, xfsprogs), OCF PATH workaround for nixpkgs#207891
- test-lab/ha/node1.nix / node2.nix: per-node hostname + static IP
- test-lab/ha/fence-pve-ssh.py: Proxmox SSH fence agent for STONITH
- test-lab/ha/cluster-init.sh: one-shot cluster bootstrap script
- test-lab/ha/cluster-enable-stonith.sh: enables STONITH post-key-deploy
- flake.nix: adds ha-test-node1 / ha-test-node2 nixosConfigurations
  (bypasses mkTarget / clan-core / sops-nix — test-only)

These VMs must be destroyed once acceptance testing is complete.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:46:04 +10:00
beatzaplenty 479444d26a Merge pull request 'fix(sops): hard-fail on missing admin key and expand literal ~ in key path' (#79) from worktree-structured-nibbling-nova into main
Reviewed-on: #79
2026-07-26 19:43:57 +00:00
beatzaplentyandClaude Sonnet 4.6 8c19ee9d72 fix(sops): hard-fail on missing admin key and expand literal ~ in key path
SOPS_AGE_KEY_FILE was set in hosts/nixos/home.nix sessionVariables with a
literal ~ that Home Manager injects as-is into the environment.  In bash,
tilde expansion does not happen inside double-quoted variable references, so
DEFAULT_SOPS_AGE_KEY_FILE resolved to ~/... literally and the -s file-existence
check in ensure_admin_decrypt_key silently failed.  The script then generated
a brand-new age key (to ~/... relative to the repo root) while the real admin
key at ~/.config/sops/age/keys.txt went untouched -- making it appear the key
was lost when it was actually still intact.

Fix the home.nix root cause by using config.home.homeDirectory so the path
is fully resolved.  Add tilde expansion in ensure_admin_decrypt_key as a
belt-and-suspenders guard for any caller whose environment has the same issue.

Also replace the auto-generate-a-new-key fallback with a hard failure: auto-
generating a new admin key is never useful (it cannot decrypt existing secrets)
and created serious confusion about whether the original key was lost.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:42:32 +10:00
beatzaplenty 2514d3bc89 Merge pull request 'fix(lxc-pxe-boot): make privileged so NFS mounts work' (#78) from worktree-debian-pxe into main
Reviewed-on: #78
2026-07-26 19:29:25 +00:00
beatzaplentyandClaude Sonnet 4.6 48d6d6f7a2 fix(lxc): auto-derive privileged from NFS fileSystems, not hostname list
Replace the hardcoded hostname check (docker, pxe-boot) with a check
on config.fileSystems: any lxc-* host whose NixOS config declares an
NFS fileSystem entry is automatically made privileged. The script
already reads proxmoxLXC.privileged dynamically via
flake_target_lxc_privileged, so no logic change is needed there —
only the comment is updated to describe the new derivation.

Result: lxc-docker and lxc-pxe-boot (the two with NFS mounts) evaluate
as privileged=true; lxc-nix-cache, lxc-minimal, lxc-server,
lxc-tailscale-router, lxc-tor-relay evaluate as privileged=false.
Any future lxc-* host that declares an NFS mount gets the correct
privilege level for free without a separate manual edit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:27:56 +10:00
beatzaplentyandClaude Sonnet 4.6 cf0d62696f fix(lxc-pxe-boot): make privileged so NFS mounts work
The kernel's NFS client (FS_USERNS_MOUNT not set) rejects NFS mounts
from inside any unprivileged container's user namespace with EPERM —
AppArmor's mount=nfs feature only whitelists the AppArmor layer; the
VFS-level rejection happens before AppArmor is consulted.

lxc-pxe-boot mounts server.sweet.home:/tank/pxe-boot/images at
/mnt/pxe-images so nginx can serve large ISOs without filling the
container's root disk. Same pattern as lxc-docker (already privileged
for the same reason since it also mounts several NFS shares).

Operator action required: VMID 103 must be recreated as a privileged
container (the UID mapping on disk differs between privileged and
unprivileged; changing it in-place with pct set is unsafe). Rebuild
the tarball and use create-proxmox-resource.sh to replace it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:23:22 +10:00
beatzaplenty b2a3d5fbdd Merge pull request 'docs(pxe-boot): fix NFS path, note nesting=1 requirement for LXC' (#77) from worktree-debian-pxe into main
Reviewed-on: #77
2026-07-26 19:18:48 +00:00
beatzaplentyandClaude Sonnet 4.6 86e55ff954 docs(pxe-boot): fix NFS path, clarify NFSv3/NFSv4 split, document nesting=1 requirement
- Correct the NFS path from /tank/proxmox/pxe-images to /tank/pxe-boot/images
  (matches variables.nix's proxmoxPxeImages.subpath)
- Clarify that LXC uses NFSv3+nolock while VM uses NFSv4.2+automount
- Add explicit note that lxc-pxe-boot needs features: nesting=1,mount=nfs and
  why: nesting=1 is required by systemd 260+ for userns/credential isolation
  (AppArmor denies userns_create without it), mount=nfs for NFSv3 access.
  pct set replaces the whole features string — include both or the container
  will fail to boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:13:21 +10:00
beatzaplenty d3d6382360 Merge pull request 'fix: wire vars.nfsShares.options and correct stale share key names' (#76) from worktree-tidy-giggling-pizza into main
Reviewed-on: #76
2026-07-26 18:45:53 +00:00
beatzaplentyandClaude Sonnet 4.6 b8d21d78d9 fix: wire vars.nfsShares.options and correct stale share key names
- Filter non-attrset values from nfsShares in server.nix so the
  poolDatasets loop skips the new `options` string entry
- Fix typo proxomoxLxcImages → proxmoxLxcImages in server.nix exports
- Rename proxmoxPxeImages → pxebootImages in mount-pxe-images.nix and
  pxe-boot.nix to match the actual key in variables.nix

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 04:45:16 +10:00
beatzaplenty dcce023b14 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos 2026-07-27 04:34:53 +10:00
beatzaplenty 4c5ade5605 updated NFS shares and added vars.nfsShares.options 2026-07-27 04:34:32 +10:00
beatzaplenty b232daf5e1 Merge pull request 'fix(pxe-boot): fix NFS mount in LXC — NFSv3+nolock and skip rpc_pipefs' (#75) from worktree-debian-pxe into main
Reviewed-on: #75
2026-07-26 18:15:58 +00:00
beatzaplentyandClaude Sonnet 4.6 b65736c0dc fix(pxe-boot): fix NFS mount in LXC — NFSv3+nolock and skip rpc_pipefs
Proxmox LXC containers block the sunrpc filesystem (rpc_pipefs) via AppArmor
unless the container has `features: mount=nfs` set. NFSv4 requires rpc_pipefs
for client state management, so the mount fails outright in a default LXC.

Two fixes for the LXC case (config.boot.isContainer):
- Switch from nfsvers=4.2 to nfsvers=3,proto=tcp,nolock,nofail: NFSv3 doesn't
  need rpc_pipefs at the protocol level, and nofail keeps boot clean if the
  NFS server is unreachable.
- Add ConditionVirtualization=!container to var-lib-nfs-rpc_pipefs.mount via
  systemd drop-in: NixOS pulls this unit into nfs-client.target for any NFS
  fileSystems entry. With the condition, systemd skips (not fails) the unit in
  containers, keeping nfs-client.target green and activation reporting clean.

Proxmox VM hosts (not isContainer) continue to use nfsvers=4.2 with
x-systemd.automount unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 04:14:18 +10:00
beatzaplenty 4f54a1f0cd Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos 2026-07-27 04:05:15 +10:00
beatzaplenty 2d85ecec8f update pxe-boot images share path 2026-07-27 04:04:59 +10:00
beatzaplenty 34bb14d9f6 Merge pull request 'fix(server): open mountd port 20048 so NFS clients can scan and mount' (#74) from worktree-fix-nfs-mountd-firewall into main
Reviewed-on: #74
2026-07-26 17:57:40 +00:00
beatzaplenty 515da66db9 Merge pull request 'Worktree debian pxe' (#73) from worktree-debian-pxe into main
Reviewed-on: #73
2026-07-26 17:57:31 +00:00
beatzaplentyandClaude Sonnet 4.6 2e9d3da301 fix(server): open mountd port 20048 so NFS clients can scan and mount
showmount and the NFSv3 mount protocol need mountd reachable after querying
portmapper on 111; the server was only opening TCP 111 and 2049, causing
clients (e.g. Proxmox GUI NFS storage scan) to time out connecting to
mountd on 20048. Also adds UDP for all three ports — portmapper, nfsd, and
mountd all use both protocols.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 03:56:39 +10:00
beatzaplentyandClaude Sonnet 4.6 321048626e feat(pxe-boot): mount pxe-images NFS share and symlink to /srv/pxe/http/images
Adds modules/pxe-boot/mount-pxe-images.nix, which mounts
server.sweet.home:/tank/proxmox/pxe-images at /mnt/pxe-images via NFSv4.2
(x-systemd.automount on Proxmox VMs, nofail on LXC containers — same pattern
as docker/mount-data.nix). The pxe-boot build-type now imports this module and
replaces the previous local /srv/pxe/http/images directory rule with an L+
symlink pointing to /mnt/pxe-images, so large images (ISOs, disk images) live
on the NFS share rather than the host's own root disk.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 03:54:37 +10:00
beatzaplenty 16d6baea5f Merge remote-tracking branch 'origin/main' into worktree-debian-pxe 2026-07-27 03:50:56 +10:00
beatzaplenty d082c6a084 updated nfs shares on server build type 2026-07-27 03:40:19 +10:00
beatzaplenty ee93322ae2 add proxmox nfs shares 2026-07-27 03:28:46 +10:00
beatzaplenty d1ce8d3e71 add proxmox iso share 2026-07-27 03:26:31 +10:00
beatzaplentyandClaude Sonnet 4.6 f7c32aff12 feat(pxe-boot): add Debian bookworm minimal netboot menu entry
Adds a `fetch-debian-netboot.service` oneshot that downloads the Debian
bookworm netboot kernel and initrd from deb.debian.org on first boot,
stages them under /srv/pxe/http/debian/, and serves them via a generated
debian.ipxe chain script. The service is idempotent — it skips the
download if both files are already present.

Also merges the previously split systemd.tmpfiles.rules and
systemd.services blocks into a single systemd = { ... } attrset to
satisfy statix's repeated-keys lint.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 03:23:14 +10:00
beatzaplenty bf88a6ebb0 added tailscale-router beszel secret
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
2026-07-26 11:14:05 +10:00
beatzaplenty de508141a8 removed tailscale from docker updated secrets
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
2026-07-26 11:12:46 +10:00
beatzaplenty 18cd9c2342 Merge pull request 'feat(beszel): add beszel agent to tailscale-router' (#72) from worktree-beszel-tailscale-router into main
Check NixOS configurations / eval-hosts (push) Failing after 11m27s
Reviewed-on: #72
2026-07-26 01:08:29 +00:00
beatzaplentyandClaude Sonnet 4.6 997918e2f7 feat(beszel): add beszel agent to tailscale-router
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m33s
Wires beszel-agent into all tailscale-router variants (lxc/linode/proxmox)
by importing enable-agent.nix in the build type and host-token.nix in the
host file. Adds the sops creation rule for secrets/tailscale-router.yaml
(all three platform variants as recipients). The secrets file must be
created manually before deploying — see instructions in PR.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 11:07:51 +10:00
beatzaplenty 9872b8ff1d Merge pull request 'fix(tailscale-router): add MASQUERADE to POSTROUTING, not nixos-nat-post' (#71) from worktree-lovely-spinning-bubble into main
Check NixOS configurations / eval-hosts (push) Successful in 10m43s
Reviewed-on: #71
2026-07-26 00:01:48 +00:00
beatzaplentyandClaude Sonnet 4.6 e9b225d2d6 fix(tailscale-router): add MASQUERADE to POSTROUTING, not nixos-nat-post
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
extraCommands runs after nixos-nat-post is deleted but before it is
re-created, so -A nixos-nat-post silently fails every time.  POSTROUTING
is a built-in chain that always exists; target it directly instead.

The -C idempotency check prevents duplicate rules on firewall reloads.
Drop networking.nat.enable -- it was only needed for the sub-chain that
turned out to be the wrong target.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 10:00:47 +10:00
beatzaplenty 2860f750b4 Merge pull request 'fix(tailscale-router): actually insert MASQUERADE rule via extraCommands' (#70) from worktree-lovely-spinning-bubble into main
Check NixOS configurations / eval-hosts (push) Successful in 10m34s
Reviewed-on: #70
2026-07-25 23:56:11 +00:00
beatzaplentyandClaude Sonnet 4.6 781b1d324e fix(tailscale-router): actually insert MASQUERADE rule via extraCommands
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
networking.nat.externalInterface without internalInterfaces creates the
nixos-nat-post chain but inserts no MASQUERADE rule into it — confirmed
by inspecting the live firewall-start script on the deployed host.

Add the rule explicitly via firewall.extraCommands targeting nixos-nat-post,
scoped to LAN source traffic (vars.lanCidr) going out tailscale0.
extraStopCommands removes it on firewall stop.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 09:54:33 +10:00
beatzaplenty 3014a45936 Merge pull request 'fix(tailscale-router): masquerade LAN traffic into Tailscale' (#69) from worktree-lovely-spinning-bubble into main
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
Reviewed-on: #69
2026-07-25 23:48:13 +00:00
beatzaplentyandClaude Sonnet 4.6 cda2132d6a fix(tailscale-router): masquerade LAN traffic into Tailscale
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
Without SNAT on tailscale0, Tailscale drops forwarded packets from LAN
source IPs (192.168.2.x) because they are not recognised Tailscale
addresses.  With networking.nat.externalInterface = "tailscale0", all
traffic leaving through the Tailscale tunnel is masqueraded to the
router's own Tailscale IP (100.x.x.x), making it indistinguishable from
locally-originated traffic.  Conntrack handles the return path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 09:45:07 +10:00
beatzaplenty 6c1cc821a0 Merge pull request 'feat(tailscale): enable UDP GRO forwarding on subnet router uplink' (#68) from worktree-polished-wandering-toast into main
Check NixOS configurations / eval-hosts (push) Successful in 10m34s
Reviewed-on: #68
2026-07-25 23:14:33 +00:00
beatzaplentyandClaude Sonnet 4.6 4be064572d feat(tailscale): enable UDP GRO forwarding on subnet router uplink
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m42s
Adds a oneshot systemd service that sets ethtool rx-udp-gro-forwarding on
and rx-gro-list off on the default-route interface at boot, silencing
Tailscale's warning about suboptimal UDP GRO forwarding on subnet routers.
Interface is discovered dynamically via `ip route get` so it works on all
platforms regardless of NIC naming.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 09:13:28 +10:00
beatzaplenty 89d506180d Merge pull request 'fix(create-proxmox-resource): fix VM disk never attaching after import' (#67) from worktree-warm-discovering-moon into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #67
2026-07-25 22:57:21 +00:00
beatzaplentyandClaude Sonnet 4.6 6e1e992652 fix(proxmox): embed SSH host key via NIXOS_HOST_KEYS_DIR so sops can decrypt on first boot
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m30s
--pre-format-files placed the key on the QEMU builder VM's rootfs, not the
target disk. nixos-install chroots into the target and runs sshd-keygen, which
found no key in the chroot and generated a fresh (unregistered) one. sops then
could not decrypt on first boot because the key didn't match .sops.yaml, leaving
both root and nixos with '!' in /etc/shadow even after mutableUsers = false was
set (hashedPasswordFile pointed to paths sops never wrote).

Fix modules/platforms/proxmox.nix to embed the clan SSH host key in
environment.etc via NIXOS_HOST_KEYS_DIR at eval time -- the same pattern
lxc.nix uses. nixos-install's own activation places the key on the target disk,
sshd-keygen finds it already present and skips generation, and sops decrypts
correctly on first boot. Includes the same preserveSshHostKey/restoreSshHostKey
activation scripts as lxc.nix so subsequent nixos-rebuild switch calls (without
NIXOS_HOST_KEYS_DIR) don't remove the key as "obsolete" from environment.etc.

Update create-proxmox-resource.sh: switch VM builds from
  ./result-<target> --pre-format-files ... --build-memory 2048
to
  NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build --impure ... diskoImagesScript
  ./result-<target> --build-memory 2048
matching the LXC build path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
2026-07-26 08:54:39 +10:00
beatzaplentyandClaude Sonnet 4.6 5467c2e140 fix(create-proxmox-resource): fix VM disk never attaching after import
Three bugs combined to leave every VM build with a shell but no boot disk:

1. The remote build script moved the raw image to /var/lib/vz/import/ before
   qm importdisk could use it. If the mv failed (cross-filesystem copy, sudo
   path, or any other reason) the remote script exited non-zero -- but the
   local script's set -e handling of the SSH heredoc was inconsistent, so
   qm create sometimes ran anyway, leaving a diskless VM shell.

   Fix: skip the mv entirely. The diskoImagesScript writes <hostname>.raw into
   its CWD (the remote repo dir, $out = $PWD at invocation). Import directly
   from that path; clean it up after a successful import.

2. The qm importdisk output regex expected "Successfully imported disk as '...'"
   but current Proxmox emits "unusedN: successfully imported disk '...'"
   (lowercase, no "as"). The grep returned no match and exited 1.

3. The disk_id assignment used $(... | grep ...) without || true inside the
   substitution. With set -euo pipefail, a non-zero grep exit aborts the
   script before the fallback could run -- so the VM was always left with an
   unattached unused0 disk.

   Fix: update the primary regex to match the actual PVE format; add || true
   inside the substitution so set -e never fires on a grep miss; add a qm
   config fallback (scan for unusedN: lines) that works regardless of PVE
   output format changes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
2026-07-26 08:54:39 +10:00
beatzaplenty 123cd2b3d7 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Successful in 10m28s
2026-07-26 08:23:44 +10:00
beatzaplenty 006dd8097a updated vm HDD size 2026-07-26 08:23:30 +10:00
beatzaplenty 18cd6e884e Merge pull request 'fix(common): set mutableUsers = false to fix password setup on disk images' (#65) from worktree-warm-discovering-moon into main
Check NixOS configurations / eval-hosts (push) Successful in 10m36s
Reviewed-on: #65
2026-07-25 21:37:34 +00:00
beatzaplenty 3102d66337 Merge pull request 'fix(create-proxmox-resource): case-insensitive importdisk parse + warn on --disk-size for VMs' (#64) from worktree-gentle-cuddling-hippo into main
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Reviewed-on: #64
2026-07-25 21:37:14 +00:00
beatzaplentyandClaude Sonnet 4.6 dfa5452af5 fix(common): set mutableUsers = false to fix password setup on disk images
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
When a proxmox-* disk image is built, activation runs during the image
build without a valid sops age key (the SSH host key doesn't exist yet),
so root and nixos land in /etc/shadow with locked '!' entries. With the
default mutableUsers = true, update-users-groups.pl preserves existing
shadow entries for accounts that already exist, so hashedPasswordFile is
silently ignored on every subsequent boot — passwords are never fixed.

Setting mutableUsers = false forces update-users-groups.pl to apply
hashedPasswordFile unconditionally on every activation. On first real
boot the sops-decrypted hash is now written regardless of whether the
account already existed in shadow from the image build.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
2026-07-26 07:36:06 +10:00
beatzaplentyandClaude Sonnet 4.6 852ba2240f fix(create-proxmox-resource): case-insensitive importdisk parse + warn on --disk-size for VMs
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m22s
qm importdisk in QEMU 11.x outputs lowercase "successfully imported disk
as '...'" rather than the capitalised form the original grep expected.
The case mismatch made disk_id always empty, which caused the script to
exit 1 after qm create had already run -- leaving the VM with only an
EFI disk, no scsi0, and boot order still set to net0.

Fix by adding -i (case-insensitive) to the grep. Both the old capitalised
format (where the disk id had an "unused0:" prefix inside the quotes) and
the new lowercase format are handled correctly: the sed strip of unused0:
is preserved for backward compatibility, and the regex result is identical
either way.

Also add an early warning when --disk-size is passed for --type vm: the
flag is LXC-only for create mode and was silently ignored, leaving users
expecting a different size than the proxmoxImageSize in variables.nix.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 06:40:54 +10:00
beatzaplenty 096dff4fa0 Merge pull request 'docs(sync-host-keys): fix stale host-keys/ references in comments and usage' (#63) from fix-stale-wording into main
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Reviewed-on: #63
2026-07-25 14:36:31 +00:00
beatzaplentyandClaude Sonnet 4.6 b5f749daa9 docs(sync-host-keys): fix stale host-keys/ references in comments and usage
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m24s
After the clan vars migration all keys are in vars/per-machine/, not
host-keys/. Update:
- File header: "existing clan var is never overwritten" (not host-keys/ file)
- Header --remove/--regenerate description: mention clan vars as primary
- usage() --remove, --regenerate-all-keys, --dry-run text
- cmd_remove/cmd_regenerate_all empty-guard messages
- README.md vars/per-machine/ row: "all deployed hosts" (not "LXC hosts")

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-26 00:11:03 +10:00
beatzaplenty adaf53d647 Merge pull request 'fix(sync-host-keys): extend --remove/--regenerate to cover clan vars' (#62) from fix-sync-host-keys-clan-vars into main
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Reviewed-on: #62
2026-07-25 14:02:11 +00:00
beatzaplentyandClaude Sonnet 4.6 01679f1639 fix(sync-host-keys): extend --remove/--regenerate to cover clan vars
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m26s
locally_managed_hosts() only scanned host-keys/ (now empty for all
current targets), so --remove and --regenerate-all-keys silently did
nothing. Fix:

- locally_managed_hosts(): also yields targets from
  vars/per-machine/*/openssh/ssh_host_ed25519_key/secret, deduped
- cmd_remove: shows [clan-vars] or [host-keys/] label per entry;
  deletes vars/per-machine/<target>/openssh/ in addition to host-keys/
- cmd_regenerate_all: same -- removes clan vars dirs before regenerating

Also update CLAUDE.md and README.md to reflect that all flake targets
now use clan vars (not just lxc-*); host-keys/ is only for the
auto-installer's own pre-seeding path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 23:50:38 +10:00
beatzaplenty 8f4c88347d Merge pull request 'Worktree phase0 provision ordering fix' (#61) from worktree-phase0-provision-ordering-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 10m19s
Reviewed-on: #61
2026-07-25 12:20:21 +00:00
beatzaplentyandClaude Sonnet 4.6 e9832d87c4 chore(vars): bulk clan vars SSH host keys for all remaining 20 flake targets
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m24s
Adds vars/per-machine/<target>/openssh/ for every flake target except
lxc-tor-relay and lxc-nix-cache (already committed). 18 targets recovered
from pve1 host-keys/ backup; lxc-gui and proxmox-minimal have no prior
live deployment and no backup key, so fresh ed25519 keys were generated —
their .sops.yaml anchors were updated to match.

All secrets are admin-only encrypted (matching clan_generate_ssh_key
convention). Age fingerprints verified against .sops.yaml anchors.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 21:54:37 +10:00
beatzaplentyandClaude Sonnet 4.6 08aac4261f fix(secrets): correct corrupted sops fingerprints for proxmox-minimal and lxc-gui
The previous Phase 4 commit had sed-mangled fingerprints for these two
targets (old and new fingerprints concatenated into one line). The correct
new fingerprints are:
  - proxmox-minimal: age19m0m7vdfg... (freshly generated, no prior key on pve1)
  - lxc-gui:        age1rrxqea6q6... (freshly generated, no prior key on pve1)

Re-run sops updatekeys on common.yaml and gui.yaml to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 21:54:25 +10:00
beatzaplenty 2526b2dca7 Merge pull request 'chore(secrets): Phase 4 — remove stale sops.yaml anchors and re-encrypt' (#60) from worktree-phase0-provision-ordering-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 10m19s
Merge PR #60: Phase 4 — remove stale sops.yaml anchors
2026-07-25 11:32:25 +00:00
beatzaplentyandClaude Sonnet 4.6 2df53fd5d7 chore(secrets): Phase 4 — remove stale sops.yaml anchors and re-encrypt
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m21s
Remove four stale age key anchors from .sops.yaml that correspond to
non-lxc build-type variants that were never deployed (or are now
superseded by their lxc-* counterparts):

  &docker   → superseded by &lxc-docker (active running host)
  &server   → superseded by &lxc-server (active running host)
  &nix-cache → superseded by &lxc-nix-cache (active running host)
  &nix-minimal → superseded by &lxc-minimal (active running host)

Also remove the secrets/docker.yaml creation_rules block entirely since
that file does not exist.

Re-encrypt secrets/common.yaml, secrets/nix-cache.yaml, and
secrets/server.yaml with sops updatekeys to drop the stale recipients.
The four removed keys can no longer decrypt these files.

Update README.md and CLAUDE.md to clarify that deployed lxc-* hosts
now use clan vars (vars/per-machine/<target>/openssh/) rather than the
gitignored host-keys/ directory.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 21:31:56 +10:00
beatzaplenty 5e2ff76cf7 Merge pull request 'refactor(provision): Phase 3 — remove legacy host-keys/ fallback' (#59) from worktree-phase0-provision-ordering-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Merge PR #59: Phase 3 — remove legacy host-keys/ fallback
2026-07-25 11:24:55 +00:00
beatzaplentyandClaude Sonnet 4.6 e8c4122460 refactor(provision): Phase 3 — remove legacy host-keys/ fallback
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m22s
All actively deployed lxc-* hosts now have clan vars. Remove the legacy
scp -pr host-keys/ fallback in sync_remote_host_keys(): instead of
silently copying the gitignored directory, error clearly if no clan var
exists for the target and tell the operator how to generate one.

Also extend the uncommitted-changes check to cover vars/per-machine/ in
addition to .sops.yaml and secrets/, since clan vars must be committed
before the remote build git-pulls them.

Update the script header and sync_remote_host_keys comment to reflect
the new clan-only key flow.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 21:24:32 +10:00
beatzaplenty 5db41b1166 Merge pull request 'chore(vars): clan vars SSH host key for lxc-nix-cache' (#58) from worktree-phase0-provision-ordering-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Merge PR #58: clan vars SSH host key for lxc-nix-cache
2026-07-25 11:18:19 +00:00
beatzaplentyandClaude Sonnet 4.6 ea7794dc05 chore(vars): commit clan vars SSH host key for lxc-nix-cache
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m27s
Encrypted private key matches the running container's key and the
vars.nixCacheHostKey in variables.nix (no rotation). Age fingerprint
age1ufg390... matches the &lxc-nix-cache anchor in .sops.yaml.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 21:10:46 +10:00
beatzaplenty 67752fb1e8 Merge pull request 'fix(lxc): clan vars for lxc-tor-relay + sops-reinstall service fix (network.target)' (#57) from worktree-phase0-provision-ordering-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
Merge PR #57: fix(lxc): sops-reinstall to network.target + clan vars for lxc-tor-relay
2026-07-25 09:55:25 +00:00
beatzaplentyandClaude Sonnet 4.6 1a14b1d4d3 fix(lxc): move sops-reinstall service from sysinit to network.target
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
nixos-lxc-sops-reinstall.service called switch-to-configuration test at
sysinit.target time (DefaultDependencies=false), before D-Bus was up.
D-Bus is required to restart systemd targets after activation scripts
run. The service reported failure on every boot (exit 1: "Failed to open
dbus connection") even though secrets were correctly installed, because
the D-Bus call happens after activation scripts complete.

Move the service to network.target so basic.target (which includes
dbus-broker.service) runs first. Also drop DefaultDependencies=false so
systemd auto-adds After=basic.target. Add SuccessExitStatus=11 to handle
the edge case where switch-to-configuration holds the lock during a
concurrent rebuild (exit 11 = "Could not acquire lock" -- the rebuild's
own activation already installed the secrets, so treating it as success
is correct).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 19:50:35 +10:00
beatzaplentyandClaude Sonnet 4.6 68aea4cdcc chore(vars): commit clan vars SSH host key for lxc-tor-relay
The key was generated in a prior session but not committed — the clan vars
files existed only in that session's working tree. Recovered the original
private key from pve1's host-keys/ backup (fingerprint age16kqf... matches
the &lxc-tor-relay anchor already in .sops.yaml), re-encrypted for admin
age key only, and stored in the canonical clan vars layout.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 19:29:21 +10:00
beatzaplenty 0853952269 Merge pull request 'feat(provision): Phase 0-2 + fix — clan-core SSH host keys, activation ordering, boot-time sops' (#56) from worktree-phase0-provision-ordering-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Merge feat(provision): Phase 0-2 + fix — clan-core SSH host keys, activation ordering, boot-time sops
2026-07-25 09:16:43 +00:00
beatzaplentyandClaude Sonnet 4.6 055577ee91 fix(lxc): fix activation ordering and add boot-time sops reinstall
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
Two bugs prevented nixos-rebuild switch from working on lxc-* hosts after
first boot, both confirmed live on a deployed lxc-tor-relay container:

1. Ordering bug: preserveSshHostKey had no explicit deps, so the topological
   sort placed it at position 7 — after etc at position 5. By the time it
   tried to save the SSH key, etc had already removed it as "obsolete"
   (absent from the current generation's environment.etc when built without
   NIXOS_HOST_KEYS_DIR). Consolidate all four system.activationScripts entries
   into one block and add etc = { deps = ["preserveSshHostKey"]; } and
   setupSecrets = { deps = ["restoreSshHostKey"]; } to enforce the correct
   save→etc→restore→sops chain.

2. No boot-time secrets: /run/secrets is a tmpfs cleared on every reboot, and
   sops-nix does NOT generate a boot-time service in this configuration
   (confirmed live: no sops-nix.service in systemctl list-unit-files).
   Add nixos-lxc-sops-reinstall.service, modelled after sops-nix's own service
   placement (wantedBy/before sysinit.target, DefaultDependencies=false), so
   secrets are reinstalled before basic.target on every non-first boot.
   ConditionPathExists skips it on first boot; nixos-lxc-first-boot-activate
   handles that case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 19:07:30 +10:00
beatzaplentyandClaude Sonnet 4.6 a63e1c70c3 feat(provision): Phase 2 — migrate SSH host keys to clan vars
Replaces the gitignored host-keys/ directory with clan vars as the
authoritative storage for SSH host keys. Keys are now generated as
sops-binary-encrypted clan var files (admin-key only) and checked into
vars/per-machine/<target>/openssh/, eliminating the plaintext private
key that previously had to live outside the repo.

Changes:
- modules/clan/ssh-host-key.nix: clan vars generator for the ed25519
  SSH host key pair (neededFor="activation" — not mapped to sops.secrets,
  delivered via tarball baking for LXC or --pre-format-files for VMs)
- flake.nix: add clanCore module + required settings to every mkTarget;
  deduplicate bundled disko/sops-nix via follows; all 27 hosts eval clean
- flake.lock: updated to reflect the new follows constraints
- scripts/lib/clan-vars.sh: new helper library with
  clan_ssh_key_exists / clan_ssh_pubkey_path / clan_decrypt_ssh_key /
  clan_generate_ssh_key for use by the provisioning and sync scripts
- scripts/secrets/sync-host-keys.sh: queue_host_sync() now checks clan
  vars first; generates via clan_generate_ssh_key if no key exists;
  derives age fingerprint from clan pub key for .sops.yaml registration
- scripts/proxmox/create-proxmox-resource.sh: key management simplified
  (sync-host-keys.sh now generates the key if missing, so the inline
  prepare-host-key.sh call is gone); sync_remote_host_keys() decrypts
  the clan key into a temp dir and scps just the two files to the node
  when a clan key exists, falling back to the old host-keys/ scp for
  any remaining legacy entries

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 18:22:33 +10:00
beatzaplentyandClaude Sonnet 4.6 a9745b594b feat(flake): add clan-core 26.05 as a flake input (Phase 1, no behavior change)
Introduces clan-core pinned to its 26.05 release alongside the existing nixpkgs
26.05 input. No host configuration is changed — this is a pure dependency
addition so Phase 2 (per-host vars/secret management migration) has the input
available without a separate flake.lock bump.

clan-core.inputs.nixpkgs.follows = "nixpkgs" keeps a single nixpkgs closure.
sops-nix remains as a flake input; vars layers on top of it rather than
replacing it (clan's sops storage backend still needs sops-nix).

All hosts evaluate cleanly (codex-maintenance.sh --full-check equivalent
triggered by the flake.nix change).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 17:29:29 +10:00
beatzaplentyandClaude Sonnet 4.6 78bb784265 fix(provision): block build until sops changes are committed, guard missing host keys
Three ordering-related fixes to the Proxmox provisioning flow:

1. prepare-host-key.sh: make idempotent -- if the key already exists, print
   a note and exit 0 instead of erroring. The caller (create-proxmox-resource.sh)
   already guards standalone calls, but the script itself should be safe to
   run directly on a host that was already keyed.

2. create-proxmox-resource.sh: after sync-host-keys.sh updates .sops.yaml /
   secrets/, detect uncommitted changes and block with a prompt until the
   operator confirms they've committed and pushed. The PVE node's git pull
   only picks up committed+pushed state; without this gate, a new host's sops
   recipient is missing from the secrets files the image build uses, so the
   host can't decrypt secrets on first boot.

3. create-proxmox-resource.sh: add an explicit existence check for the host
   key in both the LXC and VM remote build heredocs, before it's passed as
   --pre-format-files / NIXOS_HOST_KEYS_DIR input. Gives a clear error
   pointing at sync-host-keys.sh instead of a raw `cp: cannot stat` from
   disko deep in the build.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 17:14:25 +10:00
beatzaplentyandClaude Sonnet 4.6 784e064fa2 fix(create-proxmox-resource): use absolute path for --pre-format-files
Check NixOS configurations / eval-hosts (push) Successful in 10m20s
The disko images script does `cd "$TMPDIR"` before parsing its arguments,
so relative paths passed to --pre-format-files resolve against the temp
dir instead of the repo root. Use $(pwd) to capture the absolute repo
path before the disko script changes directory.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 16:45:04 +10:00
beatzaplenty 4a5afa6c1c Merge pull request 'fix(tailscale): rename tailscale-subnet-router to tailscale-router everywhere' (#55) from worktree-fix-tailscale-host-dir into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
fix(tailscale): rename tailscale-subnet-router to tailscale-router everywhere
2026-07-25 06:37:18 +00:00
beatzaplentyandClaude Sonnet 4.6 c844ccc4e3 fix(tailscale): rename tailscale-subnet-router → tailscale-router everywhere
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m28s
Complete the rename so all identifiers match networking.hostName:
- flake.nix: attribute names and buildType strings
  (linode/proxmox/lxc-tailscale-subnet-router → *-tailscale-router)
- modules/build-types/tailscale-subnet-router.nix → tailscale-router.nix
- .sops.yaml: anchor and alias names (age keys unchanged, no re-encrypt needed)
- host-keys/: local gitignored key files renamed (not committed)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 16:33:03 +10:00
beatzaplentyandClaude Sonnet 4.6 9a0aea8f89 fix(create-proxmox-resource): auto-generate missing host key before sync
Check NixOS configurations / eval-hosts (push) Successful in 10m19s
If host-keys/<target>_ssh_host_ed25519_key doesn't exist, run
prepare-host-key.sh to generate it before sync-host-keys.sh runs.
Prevents sync-host-keys.sh from hitting its SKIP/exit-1 path (anchor
in .sops.yaml but no local key) and the downstream disko build failure
when --pre-format-files can't find the key file.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 16:26:49 +10:00
beatzaplenty b013e28dcd Merge pull request 'fix(tailscale): rename hosts/tailscale-subnet-router → hosts/tailscale-router' (#54) from worktree-fix-tailscale-host-dir into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #54
2026-07-25 06:24:59 +00:00
beatzaplentyandClaude Sonnet 4.6 5a56030f6e fix(tailscale): rename hosts/tailscale-subnet-router → hosts/tailscale-router
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m30s
The directory name was tailscale-subnet-router but networking.hostName
was already tailscale-router, causing a mismatch that confused scripts
comparing directory paths against actual hostnames. Flake attribute
names (linode/proxmox/lxc-tailscale-subnet-router) and .sops.yaml
anchors are unchanged — they describe the build type, not the hostname.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 16:24:13 +10:00
beatzaplentyandClaude Sonnet 4.6 f8719437ba fix(nix-cache): use FQDN to fix hostname resolution on clients
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
systemd-resolved only uses LLMNR for single-label hostnames, never DNS —
same issue mount-data.nix already documented and fixed for NFS by switching
to server.sweet.home. Change the substituter URL, SSH knownHosts, and
remote-builder hostName from bare "nix-cache" to "nix-cache.sweet.home",
and update nginx's virtualHost to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 16:01:56 +10:00
beatzaplentyandClaude Sonnet 4.6 9e34b9cbb9 fix(push-host-keys): detect non-interactive stdin, direct to SUDO_PASS
Check NixOS configurations / eval-hosts (push) Successful in 10m19s
read exits non-zero when stdin is not a terminal (set -e killed the
script silently). Catch that and emit a clear error pointing to the
SUDO_PASS environment variable rather than crashing with no output.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 15:55:50 +10:00
beatzaplenty 4dabd725f0 sync nix-cache ssh key
Check NixOS configurations / eval-hosts (push) Successful in 10m34s
2026-07-25 15:55:04 +10:00
beatzaplentyandClaude Sonnet 4.6 2743d664a5 fix(push-host-keys): remove /dev/tty probe, plain read is sufficient
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
/dev/tty exists as a device node even without a controlling terminal,
so -r/-w tests pass but opening it fails. Plain 'read -r -s' from stdin
is enough: works interactively from a real terminal, and from a non-tty
context the caller should set SUDO_PASS in the environment instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 15:53:00 +10:00
beatzaplentyandClaude Sonnet 4.6 cfa34f3565 fix(push-host-keys): fall back to stdin when /dev/tty unavailable
Check NixOS configurations / eval-hosts (push) Successful in 10m19s
Environments without a controlling terminal (containers, CI agents)
don't have /dev/tty. Try it first for the sudo password prompt, fall
back to plain stdin so the script works in both contexts.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 15:51:04 +10:00
beatzaplentyandClaude Sonnet 4.6 e021b49412 fix(push-host-keys): prompt sudo password once, pass via sudo -S
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Instead of ssh -t (requires PTY on both sides), prompt for the sudo
password once at startup and pipe it to each remote invocation via
sudo -S. This works from any context -- interactive terminal, background
agent, or script -- with no PTY needed on either end.

Also accepts SUDO_PASS from the environment for non-interactive callers.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 15:49:27 +10:00
beatzaplentyandClaude Sonnet 4.6 0c29c6a93d fix(push-host-keys): fix sudo PTY allocation failure
Check NixOS configurations / eval-hosts (push) Successful in 10m28s
ssh -t won't allocate a PTY when its own stdin is redirected (by a
heredoc). Replaced the heredoc-fed 'sudo bash -s' with commands passed
as an argument string so stdin stays free and -t can properly allocate
a PTY for the sudo password prompt.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 15:39:06 +10:00
beatzaplenty c329988cdd secrets: update recipients and re-encrypt for host key changes
Check NixOS configurations / eval-hosts (push) Failing after 12m10s
2026-07-25 15:31:22 +10:00
beatzaplenty 7a2b5ecf71 Merge pull request 'feat(secrets): add push-host-keys.sh; integrate into sync/recover scripts' (#53) from worktree-push-host-keys into main
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
Reviewed-on: #53
2026-07-25 05:30:32 +00:00
beatzaplentyandClaude Sonnet 4.6 d74efd9f66 feat(secrets): add push-host-keys.sh; integrate into sync/recover scripts
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m18s
New script: scripts/secrets/push-host-keys.sh
- Pushes newly-generated SSH host keys from host-keys/ to already-running
  NixOS hosts after sync-host-keys.sh --regenerate-all-keys.
- Before pushing any key, checks that .sops.yaml and secrets/*.yaml are
  committed and pushed to the remote Gitea flake (hosts rebuild from there,
  so recipient changes must land first); offers to auto-commit/push if not.
- Reads /etc/flake-target from each host to confirm which key to install,
  handling the case where multiple flake targets share a hostname.
- Deduplicates by hostname in --all mode; skips hand-registered targets
  that have no host-keys/ entry.
- --dry-run, --skip-git-check, SSH_USER override (default: nixos).

sync-host-keys.sh --regenerate-all-keys:
- Updated pre-confirmation warning to distinguish already-running hosts
  (need push-host-keys.sh) from not-yet-deployed hosts (need installer
  image rebuild).
- Added next-steps block after regeneration completes pointing to
  push-host-keys.sh --all.

recover-hosts.sh:
- Header and SSH host key mismatch warn now cross-reference
  push-host-keys.sh as the proactive (pre-drift) alternative.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 15:29:16 +10:00
beatzaplenty 63a8c627f5 Merge pull request 'refactor(tailscale): rename exit-node to subnet-router; drop --advertise-exit-node' (#52) from worktree-tailscale-subnet-router-rename into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #52
2026-07-25 04:59:26 +00:00
beatzaplentyandClaude Sonnet 4.6 e4b335be23 refactor(tailscale): rename exit-node to subnet-router; drop --advertise-exit-node
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s
The host was always intended as a LAN subnet router (--advertise-routes),
not a full exit node (--advertise-exit-node). Rename every trace of
"exit-node" to "subnet-router" and remove the --advertise-exit-node flag
from extraSetFlags; the operator supplies --advertise-routes at first
tailscale up and Tailscale persists it in state across reboots.

Routing sysctls (useRoutingFeatures = "server"), openFirewall, and
trustedInterfaces = ["tailscale0"] are still required for subnet routing
to work, so the module is kept — just correctly named.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TF2dsuKZAiyZWQ1D7CuHJm
2026-07-25 14:53:28 +10:00
beatzaplenty 0bf99c56cc Merge pull request 'tor-relay: wire beszel-agent with token secret and fix sops key' (#51) from worktree-tor-relay-beszel into main
Check NixOS configurations / eval-hosts (push) Successful in 11m0s
Reviewed-on: #51
2026-07-23 23:53:01 +00:00
beatzaplentyandClaude Sonnet 4.6 e368f68ad7 tor-relay: wire beszel-agent with token secret and fix sops key
Check NixOS configurations / eval-hosts (pull_request) Successful in 11m2s
- Add hosts/tor-relay/host.nix import of host-token.nix so the agent
  gets its TOKEN from a sops-managed environment file
- Add secrets/tor-relay.yaml (encrypted beszel token for this host)
- Add creation_rules entry for secrets/tor-relay.yaml in .sops.yaml
- Update &lxc-tor-relay age key to the host's actual current key
  (old key was from a prior LXC incarnation; new key extracted from
  Switch-nix output: age1gl5ujmhd2pe37...)
- Re-encrypt secrets/common.yaml via sops updatekeys to swap in the
  new key, so the host can decrypt its password hash on next boot

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 09:36:03 +10:00
beatzaplenty fa52c2849a Merge pull request 'fix(lxc): prevent SSH host key deletion on every rebuild; add recovery script' (#50) from worktree-rippling-riding-snail into main
Check NixOS configurations / eval-hosts (push) Successful in 10m34s
Reviewed-on: #50
2026-07-23 23:20:30 +00:00
beatzaplentyandClaude Sonnet 4.6 dce3788499 fix(lxc): prevent SSH host key deletion on every rebuild; add recovery script
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
NixOS's etc activation removes files that were in a previous generation's
environment.etc but absent from the current one -- even real copies, not
only symlinks.  LXC tarballs bake the host key into environment.etc (via
NIXOS_HOST_KEYS_DIR), but every subsequent nixos-rebuild switch lacks that
env var, so the key is removed as "obsolete".  sops-nix derives its age
decryption key from /etc/ssh/ssh_host_ed25519_key, so deletion cascades
into "Error getting data key: 0 successful groups required, got 0" for
every sops secret on the host.

Fix: two activation scripts bracket the etc step.
  preserveSshHostKey (no deps, runs before etc): copies the live key to
    /run (tmpfs) before etc can delete it.
  restoreSshHostKey (deps=[etc], runs after etc): reinstalls via `install`
    if etc removed the key.  The resulting file is not tracked in either
    generation's environment.etc, so subsequent rebuilds leave it alone.

scripts/recover-hosts.sh: restore both private and public key files (not
just the private key), use install(1) for atomic mode setting, and add a
post-rebuild sops-nix verification step to confirm success.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014zT1L6hmsq6i1evAEH7dmi
2026-07-24 09:17:58 +10:00
beatzaplenty e00be5d2da added sops
Check NixOS configurations / eval-hosts (push) Failing after 17m21s
2026-07-24 08:55:21 +10:00
beatzaplenty 82eea7f088 synced all keys
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
2026-07-24 08:17:34 +10:00
beatzaplenty 955a443b36 updated github token
Check NixOS configurations / eval-hosts (push) Failing after 15m35s
2026-07-24 07:17:12 +10:00
beatzaplenty 4800aebf43 enable beszel agent
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
2026-07-24 07:03:01 +10:00
beatzaplenty cb737642e5 updated sops keys
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
2026-07-24 06:50:18 +10:00
beatzaplenty 6d5670c8d2 Merge pull request 'server: auto-create tank ZFS pool on first boot if data disk is blank' (#49) from server-boot-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 11m14s
Reviewed-on: #49
2026-07-23 20:26:36 +00:00
beatzaplenty c911a605e9 updated sops
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
2026-07-24 06:23:52 +10:00
133 changed files with 7390 additions and 657 deletions
+2
View File
@@ -23,3 +23,5 @@ host-keys/
# Temporary Milestone 1 audit checklist (remove-sensetive-info-refactor.md)
# - working notes only, never committed, deleted once every row is rotated.
secrets-inventory.md
.claude/worktrees/
.claude/settings.local.json
+179 -28
View File
@@ -1,18 +1,29 @@
keys:
- &admin age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- &docker age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e
- &server age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
- &nix-cache age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
- &nix-minimal age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu
- &proxmox-minimal age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg
- &lxc-nix-cache age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
- &lxc-docker age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs
- &lxc-minimal age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep
- &lxc-pxe-boot age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp
- &lxc-gui age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05
- &proxmox-server age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
- &vm-server age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
- &baremetal-gui age1ehkswwz2pqaz4svzh7ela5tdnssl8kn6d4vwwxd6zwg8exfpd43syyrrjp
- &proxmox-minimal age19m0m7vdfg86yqy8l5mmle5jdd0unrn3f55t232w8h5ey42cqw34sfpt32n
- &lxc-gui age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39
- &baremetal-gui age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy
- &linode-docker age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt
- &linode-gui age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs
- &linode-minimal age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx
- &linode-nix-cache age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e
- &linode-server age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
- &linode-tailscale-router age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- &lxc-docker age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th
- &lxc-minimal age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k
- &lxc-nix-cache age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7
- &lxc-pxe-boot age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt
- &lxc-server age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
- &lxc-tailscale-router age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- &lxc-tor-relay age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
- &proxmox-docker age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c
- &proxmox-gui age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn
- &proxmox-nix-cache age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
- &proxmox-pxe-boot age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
- &proxmox-server age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
- &proxmox-tailscale-router age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
- &proxmox-ha-server-1 age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
- &proxmox-ha-server-2 age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
creation_rules:
# Shared across every currently-deployed host: root/nixos password hash,
@@ -23,40 +34,90 @@ creation_rules:
key_groups:
- age:
- *admin
- *docker
- *server
- *nix-cache
- *lxc-minimal
- *nix-minimal
- *lxc-nix-cache
- *proxmox-minimal
- *lxc-docker
- *lxc-pxe-boot
- *lxc-gui
- *proxmox-server
- *vm-server
- *baremetal-gui
- *linode-docker
- *linode-gui
- *linode-minimal
- *linode-nix-cache
- *linode-server
- *linode-tailscale-router
- *lxc-docker
- *lxc-minimal
- *lxc-nix-cache
- *lxc-pxe-boot
- *lxc-server
- *lxc-tailscale-router
- *lxc-tor-relay
- *proxmox-docker
- *proxmox-gui
- *proxmox-nix-cache
- *proxmox-pxe-boot
- *proxmox-server
- *proxmox-tailscale-router
- *proxmox-ha-server-1
- *proxmox-ha-server-2
- path_regex: secrets/nix-cache\.yaml$
key_groups:
- age:
- *admin
- *nix-cache
- *linode-nix-cache
- *lxc-nix-cache
- *proxmox-nix-cache
- path_regex: secrets/server\.yaml$
key_groups:
- age:
- *admin
- *server
- *linode-server
- *lxc-server
- *proxmox-server
- *vm-server
- path_regex: secrets/docker\.yaml$
- path_regex: secrets/tor-relay\.yaml$
key_groups:
- age:
- *admin
- *docker
- *lxc-tor-relay
- path_regex: secrets/tailscale-router\.yaml$
key_groups:
- age:
- *admin
- *linode-tailscale-router
- *lxc-tailscale-router
- *proxmox-tailscale-router
# HA file server per-node secrets (beszel-token).
# proxmox-ha-server-1 / proxmox-ha-server-2 keys are added automatically
# by scripts/secrets/sync-host-keys.sh once the hosts are provisioned;
# until then only the admin key can decrypt these files.
- path_regex: secrets/ha-server-1\.yaml$
key_groups:
- age:
- *admin
- *proxmox-ha-server-1
# proxmox-ha-server-1 added by sync-host-keys.sh
- path_regex: secrets/ha-server-2\.yaml$
key_groups:
- age:
- *admin
- *proxmox-ha-server-2
# proxmox-ha-server-2 added by sync-host-keys.sh
# Shared HA cluster corosync authkey (binary sops file).
# Encrypted for both HA nodes so either can decrypt on boot.
# Both host keys added by sync-host-keys.sh; admin key allows initial creation.
- path_regex: secrets/ha-corosync-authkey$
key_groups:
- age:
- *admin
- *proxmox-ha-server-1
- *proxmox-ha-server-2
# proxmox-ha-server-1 added by sync-host-keys.sh
# proxmox-ha-server-2 added by sync-host-keys.sh
# gui-host-specific secrets (currently: wifi-password, see
# modules/networking/wifi.nix). Only *lxc-gui has a registered key today
@@ -70,3 +131,93 @@ creation_rules:
- *admin
- *lxc-gui
- *baremetal-gui
- *linode-gui
- *proxmox-gui
# IPA host keytabs (binary sops files).
# Each keytab is encrypted for all platform variants of that host so any
# deployed variant can decrypt it at boot. Run
# scripts/ipa/create-nixos-ipa-host-account.sh <hostname> to enroll a new
# host and produce the keytab; this section is updated by that script.
- path_regex: secrets/nix-cache\.keytab$
key_groups:
- age:
- *admin
- *linode-nix-cache
- *lxc-nix-cache
- *proxmox-nix-cache
- path_regex: secrets/tailscale-router\.keytab$
key_groups:
- age:
- *admin
- *linode-tailscale-router
- *lxc-tailscale-router
- *proxmox-tailscale-router
- path_regex: secrets/pxe-boot\.keytab$
key_groups:
- age:
- *admin
- *lxc-pxe-boot
- *proxmox-pxe-boot
# nixos = the workstation (hosts/nixos/host.nix). All gui platform variants
# share the hostname "nixos" and must be able to decrypt at boot.
- path_regex: secrets/nixos\.keytab$
key_groups:
- age:
- *admin
- *baremetal-gui
- *lxc-gui
- *proxmox-gui
- *linode-gui
- path_regex: secrets/server\.keytab$
key_groups:
- age:
- *admin
- *linode-server
- *lxc-server
- *proxmox-server
- path_regex: secrets/docker\.keytab$
key_groups:
- age:
- *admin
- *linode-docker
- *lxc-docker
- *proxmox-docker
- path_regex: secrets/tor-relay\.keytab$
key_groups:
- age:
- *admin
- *lxc-tor-relay
- path_regex: secrets/nix-minimal\.keytab$
key_groups:
- age:
- *admin
- *lxc-minimal
- *proxmox-minimal
- *linode-minimal
# Host keytab for ha-server-1 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/ha-server-1\.keytab$
key_groups:
- age:
- *admin
- *proxmox-ha-server-1
# proxmox-ha-server-1 added by sync-host-keys.sh
# Host keytab for ha-server-2 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/ha-server-2\.keytab$
key_groups:
- age:
- *admin
- *proxmox-ha-server-2
# proxmox-ha-server-2 added by sync-host-keys.sh
+8 -6
View File
@@ -6,12 +6,14 @@ This repository contains flake-based NixOS configurations for Wayne's LAN
servers and workstation.
The flake exposes NixOS configurations named `<platform>-<buildtype>`
(platforms: `linode`, `proxmox`, `lxc`; build types: `minimal`, `nix-cache`,
`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`, `tor-relay`), generated from `modules/platforms/*`
and `modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not
every combination is built — `pxe-boot` has no `linode` variant. See
`README.md` for the full current target list; treat `flake.nix` as the
source of truth since this list can drift.
(platforms: `linode`, `proxmox`, `lxc`, `baremetal`; build types: `minimal`,
`nix-cache`, `server`, `docker`, `gui`, `pxe-boot`, `tailscale-router`,
`tor-relay`, `ha-server`), generated from `modules/platforms/*` and
`modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not every
combination is built — `pxe-boot` has no `linode` variant, `ha-server` only
exists on `proxmox`, and `tor-relay` only exists on `lxc`. See `README.md`
for the full current target list; treat `flake.nix` as the source of truth
since this list can drift.
Do not deploy, switch, reboot, repartition, format disks, or run destructive
install commands from this repository unless explicitly asked.
+19 -15
View File
@@ -21,15 +21,17 @@ machines when deployed.
`modules/installer/common.nix` (the auto-installer's own root/nixos login —
a deliberate, documented choice, see `docs/auto-installer.md`, not
accidental tech debt) and **SSH public keys** in `variables.nix`
(`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`) plus a couple of
per-host `KEY` values for beszel-agent auth (`hosts/server/host.nix`,
`hosts/nix-cache/host.nix`). Don't use the installer's hardcoded hash as a
(`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`, `vars.beszelHubKey`). Don't use the installer's hardcoded hash as a
template for a *real* host — every other host uses sops-nix
(`hashedPasswordFile`, see "Security Notes" in `README.md`). Flag any *new*
secret-like string you encounter instead of committing it.
- `host-keys/` is gitignored — locally-generated *private* SSH host keys for
the auto-installer (see `docs/auto-installer.md`). Never commit its
contents; if `git status` ever shows it as trackable, something is wrong.
- `host-keys/` is gitignored — used only by the auto-installer's own
environment for pre-seeding non-LXC host keys before first boot (see
`docs/auto-installer.md`). Never commit its contents; if `git status`
ever shows it as trackable, something is wrong. All deployed hosts use
clan vars (`vars/per-machine/<target>/openssh/`, committed and
sops-encrypted) for their SSH host keys — those ARE tracked by git and
belong in the repo.
### Two Proxmox nodes: `pve1.sweet.home` (production) and `pve-test.sweet.home` (sandbox)
@@ -203,8 +205,11 @@ instead of copying it.
- `scripts/secrets/sync-host-keys.sh` — generates/registers SSH host keys
and their `.sops.yaml`/`secrets/*.yaml` recipients for flake targets,
idempotently (`--all`, `<target>`, `--remove`, `--regenerate-all-keys`,
all with `--dry-run`). The primary tool for provisioning a new host's
secrets access — see "Creating a new machine" in `docs/auto-installer.md`.
all with `--dry-run`). Stores keys as clan vars
(`vars/per-machine/<target>/openssh/`, committed and sops-encrypted) for
all flake targets. The primary tool for provisioning a new host's
secrets access — see "Creating a new machine" in
`docs/auto-installer.md`.
- `scripts/secrets/prepare-host-key.sh` — narrower predecessor: generates a
key by an arbitrary name without touching `.sops.yaml`. Still useful to
pre-generate a key before its flake target exists yet, since
@@ -368,9 +373,8 @@ removing a host.
- `hosts/<name>/host.nix` — per-machine identity **only**: hostname, hostId,
per-machine secrets, `system.stateVersion`. These files carry no `imports`
of their own beyond narrow parameterized helpers (see
`modules/beszel/host-token.nix` below) — all shared behavior comes from the
platform/build-type modules composed in `flake.nix`, not from the host file.
of their own — all shared behavior comes from the platform/build-type modules
composed in `flake.nix`, not from the host file.
- `modules/platforms/{linode,proxmox,lxc,baremetal}.nix` — platform-specific
config: boot method, guest tooling, and the hardware config, imported
directly by the platform module itself — **not** wired in from
@@ -427,10 +431,10 @@ removing a host.
substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the
full design (per-host local stores, no shared `/nix/store`, and how the
`nixremote` signing/SSH keys fit together).
- `modules/beszel/host-token.nix` — parameterized helper module
(`{ name, sopsFile }`) that wires a host's beszel-agent sops secret/template
and `environmentFile`; used by `hosts/server/host.nix` and
`hosts/nix-cache/host.nix` to avoid duplicating that boilerplate.
- `modules/beszel/enable-agent.nix` — enables beszel-agent, sets `HUB_URL`,
fixes the upstream `StateDirectory` bug, and wires the universal
`beszel-token` sops secret (from `secrets/common.yaml`) into the agent's
`environmentFile`; see `docs/beszel.md` for the full setup guide.
- `modules/tailscale/`, `modules/docker/`, `modules/networking/`,
`modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose,
single-host
+15 -13
View File
@@ -10,13 +10,13 @@ pieces composed in `flake.nix`:
- **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`, `baremetal`
- **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
`gui`, `pxe-boot`, `tailscale-exit-node`, `tor-relay`
`gui`, `pxe-boot`, `tailscale-router`, `tor-relay`, `ha-server`
Not every combination exists — `pxe-boot` has no `linode` variant, since
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have,
`tor-relay` currently only exists as `lxc-tor-relay`, and `baremetal`
currently only exists as `baremetal-gui` (the real gui-host hardware). The
full list:
`tor-relay` and `ha-server` currently only exist on `lxc`/`proxmox`, and
`baremetal` currently only exists as `baremetal-gui` (the real gui-host
hardware). The full list:
| Target | Purpose |
| --- | --- |
@@ -29,8 +29,9 @@ full list:
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target |
| `baremetal-gui` | Same Cinnamon desktop workstation, on the real gui-host hardware — ZFS RAID0 root, systemd-boot |
| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host — previously the flat `pxe-boot` target |
| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node |
| `linode-tailscale-router` / `proxmox-tailscale-router` / `lxc-tailscale-router` | Tailscale subnet router + MagicDNS forwarder for the LAN |
| `lxc-tor-relay` | Tor middle relay |
| `proxmox-ha-server-1` / `proxmox-ha-server-2` | HA file-server cluster nodes — DRBD + XFS + iSCSI + NFS, managed by Corosync + Pacemaker |
Which variant of a given buildtype is actually deployed isn't tracked
anywhere in this repo — that's live infrastructure state, not something a
@@ -47,8 +48,7 @@ section for which is which.
Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
(hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay
fixed regardless of which platform it's built for — see
`flake-target-refactor-spec.md` for the full rationale. Every deployed host
fixed regardless of which platform it's built for. Every deployed host
stamps its own active target name into `/etc/flake-target` at build time, so
`nixos-rebuild switch --flake .#$(cat /etc/flake-target)` always picks up the
right one even after a platform migration changes the flake attribute name.
@@ -72,7 +72,8 @@ nix eval --json .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]'
| `modules/common/` | Shared NixOS config, Home Manager, aliases imported by every host |
| `modules/nix-cache/` | Binary cache and remote builder client/server modules |
| `modules/installer/` | Auto-installer environment (ISO, also served as PXE netboot) — see `docs/auto-installer.md` |
| `host-keys/` | Gitignored, locally-generated SSH host keys for the auto-installer — see `docs/auto-installer.md` |
| `host-keys/` | Gitignored; only used by the auto-installer environment for pre-seeding SSH host keys before first boot — see `docs/auto-installer.md`. All deployed hosts use clan vars (`vars/per-machine/<target>/openssh/`) instead |
| `vars/per-machine/` | Clan vars: committed, sops-encrypted SSH host keys for all deployed hosts; read by `create-proxmox-resource.sh` at deploy time |
| `docs/` | Operational notes for cache, builders, lock updates, boot services, the auto-installer, and Proxmox image builds |
| `scripts/` | Codex setup, validation, host-key, release-bump, and Proxmox resource helpers |
@@ -162,9 +163,10 @@ sops-nix-everywhere: it has a hardcoded login password instead (no stable
per-boot host key for sops-nix to derive from on ephemeral media) — see
"Host keys" in `docs/auto-installer.md` for why, and how the private keys it
*does* pre-seed for target hosts stay out of git via the gitignored
`host-keys/` directory.
`host-keys/` directory. All deployed hosts use clan vars
(`vars/per-machine/<target>/openssh/`, committed and sops-encrypted) for
their SSH host keys.
This repository's git *history* still contains secrets committed before this
migration (see `remove-sensetive-info-refactor.md`) — those are being
scrubbed and rotated separately; don't treat the repo as safe to make public
until that's finished.
This repository's git *history* still contains secrets committed before the
sops-nix migration — those are being scrubbed and rotated separately; don't
treat the repo as safe to make public until that's finished.
+25
View File
@@ -0,0 +1,25 @@
-----BEGIN CERTIFICATE-----
MIIESDCCArCgAwIBAgIBATANBgkqhkiG9w0BAQsFADA1MRMwEQYDVQQKDApTV0VF
VC5IT01FMR4wHAYDVQQDDBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMjYwNzI2
MjExMzQxWhcNNDYwNzI2MjExMzQxWjA1MRMwEQYDVQQKDApTV0VFVC5IT01FMR4w
HAYDVQQDDBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwggGiMA0GCSqGSIb3DQEBAQUA
A4IBjwAwggGKAoIBgQCzljYktbHdMGVJ6Wq0XQJuHLN6dkCSOgtoIzQtriPQkkNI
uo28LwobaiQQ8sX4kGRH/BTKnH8QlId/jug4Uc+sDHnABYu++AiOhPbBX8gCpRQ0
hebBjZiktHSBUEJR31siWOVdBoKBDJEoxehx7XUXvcxIJcaRN+LHYjO86nJN55HB
VwFU2JcYDk98c+144dFJxXdr++MjWe4Z/oVVU8JHIOtNtKhVhvij6oOSWxcYoJO/
S80LRj1vx/o6o/3G6bYug7PjY7JjZk/Oj61whijZkcsoO1MXSYI6UywJZGflv+ZB
7HyufdYAsK3WhE8O2FX3/kq64Ol83HNtoR8Dt68rTg1xpW6K45jS6iDPKueYGkb0
oSx7e++90VAW2PDhj6QQ3JJ4O5VQwrrecekJzUrAean0FOEbmgyi4PsEp1Vk6LDQ
SsIn1x0euyxVivQMlzNX2XrZL3urn1BNPqAdntXQMkR0Wl8sbUiJPe0kxG52CGXs
6yfNEXbPmVGcC0TBdGECAwEAAaNjMGEwHQYDVR0OBBYEFLh5QbI1UWMH0WR4z8bG
lhrOX3X5MB8GA1UdIwQYMBaAFLh5QbI1UWMH0WR4z8bGlhrOX3X5MA8GA1UdEwEB
/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgHGMA0GCSqGSIb3DQEBCwUAA4IBgQCVodVN
owwo53OQe02QhtEbIur2PL7zIfvhvCTRD4J8gwpbMIqT7JQK0tV6Mvsg2L8yTb2O
KjrWeLKHGWaZZlhGSPTbkMFdb/Ls8M9FSnkc2bwcdWW3Z1lOiCjBYYqwLCG6JhvB
5SXVwWNJwXeasL2m7oFTSwhsqPpARJ2t25u2N35o+tqIoCjijKwkmEOT66N9EAbu
2VQjtYZWPkBtP4YCe0Ey6u4oy7sy8ThNAjOylZok+J4JW7QEFjK4Q/emhA4aQq5H
gg9qgMuG+5oi6D1g2Wy+fMTRBaukJtLYZbBpQMQhMYWg44uPp/2bbNPTID/nV1KB
GcPyHaskcVxPdYWxAPMwk3AeJXWyOq7atAPTF5sbk0kQQf2m+vyOqcli5CxRMUgV
rcyi9l6+dZW4U+38Q0ET5M3OuxNI4hA7kVY2cfTakXWNqh97+TIHnstblDhAxECK
6ZLMJQYUy7LqJTX84H27CBWLexEMjXwdr5HCV88Fj6mAK0fRufnIw5FeneA=
-----END CERTIFICATE-----
+5 -3
View File
@@ -22,7 +22,7 @@ see "LXC hosts" immediately below for why those are different.**
## LXC hosts
`lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`,
`lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-exit-node`, `lxc-tor-relay`) are **not** installed via `auto-install.sh` — the
`lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-router`, `lxc-tor-relay`) are **not** installed via `auto-install.sh` — the
interactive menu deliberately excludes them. Don't try to select one there;
`nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw
disk to partition) and then refuse to touch the filesystem it's currently
@@ -133,13 +133,15 @@ Flake outputs:
```nix
nixosConfigurations.installer # ISO/netboot installer image
packages.x86_64-linux.iso # installer ISO/netboot image
packages.x86_64-linux.pxe # netboot-ipxe + netboot-initrd + netboot-kernel, bundled
packages.x86_64-linux.iso # installer ISO/netboot image
packages.x86_64-linux.pxe # auto-installer netboot bundle (kernel + initrd + ipxe script)
packages.x86_64-linux.pxe-minimal # vanilla NixOS minimal netboot bundle (no installer wiring)
```
```sh
nix build .#iso
nix build .#pxe
nix build .#pxe-minimal
```
There's no `nixosConfigurations.proxmox-lxc` (installer-boots-as-an-LXC-
+113
View File
@@ -0,0 +1,113 @@
# Beszel agent
[Beszel](https://github.com/henrygd/beszel) is the monitoring dashboard used
in this LAN. The hub runs as a Docker container on `docker.sweet.home` (port
`vars.ports.beszelHub`, 8090). Each monitored NixOS host runs a
`beszel-agent` that connects back to the hub.
---
## How it works
Everything is handled by a single module:
**`modules/beszel/enable-agent.nix`** — imported by a build type. It:
- Enables `beszel-agent`
- Sets `HUB_URL` to `docker.sweet.home:8090`
- Sets `KEY` from `vars.beszelHubKey` (`variables.nix`) — the hub's SSH
public key, shared by every agent. Update `beszelHubKey` if the docker
host is ever rebuilt and the hub generates a new keypair.
- Reads the universal `beszel-token` from `secrets/common.yaml` via sops
and passes it to the agent as `TOKEN` in an env file
- Fixes an upstream bug where the agent couldn't persist its hub-pairing
fingerprint across restarts (adds a real `StateDirectory`)
A host file needs no beszel configuration at all — just import the module
in the build type and add the system in the hub UI.
---
## Adding beszel to a new build type
Add `../beszel/enable-agent.nix` to the `imports` list in
`modules/build-types/<type>.nix`:
```nix
imports = [
../beszel/enable-agent.nix
# ... other imports
];
```
That's the only change required. The host file needs nothing.
---
## Adding a new system to the hub
1. Rebuild and deploy the host with its build type importing `enable-agent.nix`.
2. Open the beszel hub (`http://docker.sweet.home:8090`).
3. Go to **Systems → Add system**, enter the host's IP and the default port
(45876). The agent will connect and the system will appear as active.
---
## One-time setup: add the token to `secrets/common.yaml`
The universal token is stored once in the common secrets file, shared by all
agents. Only needed once, not per-host:
```sh
sops secrets/common.yaml
```
Add:
```yaml
beszel-token: <token from the beszel hub Settings → Keys>
```
`secrets/common.yaml` is already a sops recipient for every host via their
SSH host keys, so no additional sops recipient setup is needed.
---
## Optional: monitoring extra filesystems
To report disk usage for a mount beyond the root filesystem, add
`EXTRA_FILESYSTEMS` in the host file:
```nix
services.beszel.agent.environment = {
EXTRA_FILESYSTEMS = "/mnt/data"; # colon-separated for multiple paths
};
```
The `server` host uses this to expose its ZFS data pool:
```nix
services.beszel.agent.environment = {
EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
LOG_LEVEL = "debug";
};
```
---
## Optional: monitoring Docker containers
`enable-agent.nix` has a commented-out line for Docker monitoring:
```nix
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
```
Uncomment it if the host runs docker-socket-proxy and you want per-container
stats. Hosts without Docker should leave it commented out.
---
## If the hub key changes
If the docker host is ever rebuilt and beszel generates a new SSH keypair,
update `beszelHubKey` in `variables.nix` and rebuild all beszel-enabled hosts.
The new key is visible in the beszel hub under **Settings → Keys**.
+128
View File
@@ -0,0 +1,128 @@
# IP Addressing Scheme
## Subnets
| Subnet | CIDR | Purpose | Routed? |
|---|---|---|---|
| LAN | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
| Storage | `192.168.4.0/29` | HA file server DRBD replication | No — internal `vmbr1` only, no uplink |
The storage subnet never leaves pve1. `vmbr1` is a Proxmox Linux bridge with no physical port
attached; traffic between the two HA file server VMs stays in-kernel.
The host octet is consistent across subnets for any host that has multiple interfaces — e.g.
ha-node1 is always `.228` (LAN: `192.168.2.228`, storage: `192.168.4.228`).
---
## LAN — 192.168.2.0/24
### Address map
| Range | Purpose |
|---|---|
| .1.9 | Reserved, never assign |
| .10.59 | Client DHCP pool (router-assigned) |
| .60.219 | Unallocated buffer |
| .220.229 | Virtual nodes (VMs / LXC containers) |
| .230.239 | Expansion buffer (reserved, unallocated) |
| .240.249 | Physical nodes (bare-metal hosts) |
| .250.253 | Network services |
| .254 | Router / gateway |
### Network services (.250.253)
| IP | Hostname | Role |
|---|---|---|
| `192.168.2.254` | router | Gateway (TP-Link) |
| `192.168.2.253` | domain-controller | FreeIPA — authoritative DNS for `sweet.home`, Kerberos, LDAP |
| `192.168.2.250``.252` | — | Reserved for future network services |
### Physical nodes (.240.249)
| IP | Hostname | Role |
|---|---|---|
| `192.168.2.245` | pve1 | Proxmox VE hypervisor |
| `192.168.2.244` | pbs | Proxmox Backup Server |
| `192.168.2.243` | nixos | Bare-metal workstation (`baremetal-gui`) |
| `192.168.2.246``.249` | — | Reserved — second Proxmox node and associated services |
| `192.168.2.240``.242` | — | Reserved |
pve1 sits mid-range deliberately so a second Proxmox node can slot in on either side.
### Virtual nodes (.220.229)
All VMs and LXC containers run on pve1.
| IP | Hostname | Role | Status |
|---|---|---|---|
| `192.168.2.229` | ha-vip | HA file server iSCSI floating VIP (Pacemaker) | Future |
| `192.168.2.228` | ha-node1 | HA file server node 1 (DRBD + XFS + iSCSI) | Future |
| `192.168.2.227` | ha-node2 | HA file server node 2 (DRBD + XFS + iSCSI) | Future |
| `192.168.2.226` | server | Current NFS/ZFS file server — retires when HA is live | Retiring |
| `192.168.2.225` | docker | Docker / Traefik stack | Active |
| `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
| `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active |
| `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active |
| `192.168.2.221` | tor-relay | Tor relay | Active |
| `192.168.2.220` | pdm | Proxmox Deploy Manager | Active |
### Client DHCP pool (.10.59)
Assigned by the router. DNS option points to `192.168.2.253` (domain-controller).
Devices in this range: phones, laptops, IoT, Canon printer, any non-infrastructure host.
No static reservations for infrastructure hosts — all infra uses static IP configuration
on the guest itself (not DHCP reservations), so IPs survive VM recreation regardless of
MAC address churn.
---
## Storage network — 192.168.4.0/29
Internal to pve1 only. Proxmox bridge `vmbr1`, no physical NIC attached.
| IP | Hostname | Interface role |
|---|---|---|
| `192.168.4.228` | ha-node1 | DRBD replication NIC |
| `192.168.4.227` | ha-node2 | DRBD replication NIC |
| — | no gateway | Isolated — not routed to LAN or internet |
---
## Migration reference
Current → target IP for every host being renumbered.
| Host | Current IP | New IP | Config location |
|---|---|---|---|
| router | `192.168.2.254` | `192.168.2.254` | unchanged |
| domain-controller | `192.168.2.138` | `192.168.2.253` | `/etc/sysconfig/network-scripts/ifcfg-eth0` on guest |
| pve1 | `192.168.2.250` | `192.168.2.245` | `/etc/network/interfaces` on Proxmox host |
| pbs | `192.168.2.108` | `192.168.2.244` | static config on PBS host |
| nixos workstation | `192.168.2.119` | `192.168.2.243` | `networking.interfaces` / NetworkManager on guest |
| ha-node1 | — | `192.168.2.228` | future |
| ha-node2 | — | `192.168.2.227` | future |
| ha-vip | — | `192.168.2.229` | future (Pacemaker resource) |
| server | `192.168.2.252` | `192.168.2.226` | static config on guest |
| docker | `192.168.2.249` | `192.168.2.225` | static config on guest |
| nix-cache | `192.168.2.120` | `192.168.2.224` | static config on guest |
| pxe-boot | `192.168.2.247` | `192.168.2.223` | static config on guest; update `vars.pxeServerIp` in `variables.nix` ✓ |
| tailscale-router | `192.168.2.121` | `192.168.2.222` | static config on guest |
| tor-relay | `192.168.2.107` | `192.168.2.221` | static config on guest |
| pdm | `192.168.2.248` | `192.168.2.220` | static config on guest |
### Cutover notes
- **Do domain-controller first** — it becomes the DNS server; everything else depends on it
having its new IP and FreeIPA DNS configured before Pi-hole is retired.
- **pve1 last among physical hosts** — changing the Proxmox management IP drops the web UI
briefly; all guests keep running.
- **Update Pi-hole custom.list / FreeIPA DNS A records** to new IPs before flipping any host,
so name resolution stays valid throughout the migration.
- **variables.nix already updated** for `pxeServerIp` (.247→.223), `pbsIp` (.108→.244), and
new `domainControllerIp` (.253). Rebuild affected hosts after renumbering.
- **Router DHCP**: once domain-controller is at .253 and FreeIPA DNS is serving `sweet.home`,
switch router DHCP on with pool .10.59 and DNS option pointing to .253; retire Pi-hole CT.
- **Pi-hole's iPXE dnsmasq config** (`99-ipxe-chainload.conf`) moves to the pxe-boot CT as a
dnsmasq proxy-mode config before Pi-hole is decommissioned.
+366
View File
@@ -0,0 +1,366 @@
# Network Cutover Plan
Moves the LAN from the current flat/Pi-hole-managed state to the new IP scheme
defined in `docs/ip-addressing.md`. Works in five independent stages — each
stage is safe to pause after and resume later. Rollback steps are given at
every point where something can break.
**Before starting anything:** confirm you have
- SSH access to `192.168.2.138` (domain-controller, current IP)
- SSH access to `192.168.2.250` (pve1)
- Browser access to Pi-hole admin at `http://192.168.2.253`
- Browser access to router admin at `http://192.168.2.254`
- The FreeIPA `admin` password to hand
---
## Stage 1 — Prepare FreeIPA DNS (zero downtime)
Everything here is additive. Pi-hole keeps running. Nothing breaks if you stop
mid-stage.
### 1a. Add NextDNS forwarders
```bash
ssh wayne@192.168.2.138
kinit admin # enter FreeIPA admin password when prompted
ipa dnsconfig-mod \
--forwarder=45.90.28.142 \
--forwarder=45.90.30.142 \
--forward-policy=only
```
**Verify external resolution works through FreeIPA before continuing:**
```bash
dig @127.0.0.1 google.com +short # must return an IP, not SERVFAIL
```
### 1b. Add A records for every host at their CURRENT IPs
These represent the live state now. You'll update each record to the new IP
when you renumber that host in Stage 5.
```bash
ipa dnsrecord-add sweet.home pve1 --a-rec 192.168.2.250
ipa dnsrecord-add sweet.home pbs --a-rec 192.168.2.108
ipa dnsrecord-add sweet.home nixos --a-rec 192.168.2.119
ipa dnsrecord-add sweet.home server --a-rec 192.168.2.252
ipa dnsrecord-add sweet.home docker --a-rec 192.168.2.249
ipa dnsrecord-add sweet.home nix-cache --a-rec 192.168.2.120
ipa dnsrecord-add sweet.home pxe-boot --a-rec 192.168.2.247
ipa dnsrecord-add sweet.home tailscale-router --a-rec 192.168.2.121
ipa dnsrecord-add sweet.home tor-relay --a-rec 192.168.2.107
ipa dnsrecord-add sweet.home pdm --a-rec 192.168.2.248
ipa dnsrecord-add sweet.home router --a-rec 192.168.2.254
```
### 1c. Clean up stale reverse-zone PTR records
FreeIPA already has PTR records from an earlier import but some are wrong.
Fix them now so reverse DNS is accurate from day one.
```bash
# Remove stale "win11" entry at .250 (should be pve1)
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec win11.
ipa dnsrecord-add 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
# Fix unqualified PTR records (missing .sweet.home. suffix)
ipa dnsrecord-mod 2.168.192.in-addr.arpa 108 --ptr-rec pbs.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 248 --ptr-rec pdm.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 249 --ptr-rec docker.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 252 --ptr-rec server.sweet.home.
# Add any missing PTR records
ipa dnsrecord-add 2.168.192.in-addr.arpa 119 --ptr-rec nixos.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 120 --ptr-rec nix-cache.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 121 --ptr-rec tailscale-router.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 247 --ptr-rec pxe-boot.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 254 --ptr-rec router.sweet.home.
```
### 1d. Point domain-controller's own DNS at itself
```bash
sudo nmcli connection modify "System eth0" ipv4.dns "127.0.0.1"
sudo nmcli connection up "System eth0"
```
**Verify:**
```bash
dig pve1.sweet.home +short # must return 192.168.2.250
dig google.com +short # must return an IP (NextDNS forwarding)
```
**Rollback 1d:** `sudo nmcli connection modify "System eth0" ipv4.dns "192.168.2.253" && sudo nmcli connection up "System eth0"`
---
## Stage 2 — Move pxe-boot DHCP options off Pi-hole (zero downtime)
Pi-hole's dnsmasq currently serves the iPXE boot options via
`99-ipxe-chainload.conf`. Before Pi-hole is retired, that config must move to
the pxe-boot CT running dnsmasq in proxy mode so PXE boot keeps working.
### 2a. Add dnsmasq proxy config to the pxe-boot NixOS module
In `modules/build-types/pxe-boot.nix`, add:
```nix
services.dnsmasq = {
enable = true;
settings = {
# Proxy mode: respond only to PXE DHCP requests, leave normal leases to router
dhcp-range = [ "192.168.2.0,proxy" ];
# iPXE client detection
dhcp-match = [
"set:ipxe,175"
"set:efi64,option:client-arch,7"
"set:efi64,option:client-arch,9"
];
dhcp-userclass = "set:ipxe,iPXE";
# Boot file selection
dhcp-boot = [
"tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
"tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
"tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
"undionly.kpxe,,${vars.pxeServerIp}"
];
};
};
```
### 2b. Rebuild and deploy the pxe-boot CT
```bash
# On pve1 — build the new tarball
nix build .#lxc-pxe-boot.config.system.build.tarball
# Verify dnsmasq starts correctly in the CT after deploy
ssh nixos@192.168.2.247 systemctl status dnsmasq
```
### 2c. Remove the iPXE config from Pi-hole
In the Pi-hole CT, remove `/etc/dnsmasq.d/99-ipxe-chainload.conf` and
restart the FTL service:
```bash
ssh wayne@pve1.sweet.home \
"sudo pct exec 100 -- bash -c 'rm /etc/dnsmasq.d/99-ipxe-chainload.conf && systemctl restart pihole-FTL'"
```
**Verify:** PXE boot a test machine — it should still get an iPXE response and
reach the boot menu.
**Rollback 2c:** restore the file from the Pi-hole config backup at
`/etc/pihole/config_backups/` and restart pihole-FTL.
---
## Stage 3 — DHCP migration: Pi-hole → router (brief maintenance window)
**Do this in the evening.** Existing DHCP leases stay valid during the
switchover so connected devices don't drop — only new lease requests fail
during the gap, which is under 60 seconds if you follow the steps in order.
The key: configure the router's DHCP DNS option to point at `.253` (Pi-hole's
current IP). This way, all new leases issued by the router still get the same
DNS server address — clients never need to change their DNS config. When Pi-hole
is retired and the DC takes `.253` in Stage 4, `.253` just starts answering
differently. No client reconfiguration.
### 3a. Pre-configure router DHCP (do not enable yet)
Log into `http://192.168.2.254`, find the DHCP settings and fill in — but
leave DHCP **disabled** until step 3b:
| Setting | Value |
|---|---|
| Start IP | 192.168.2.10 |
| End IP | 192.168.2.59 |
| Subnet mask | 255.255.255.0 |
| Gateway | 192.168.2.254 |
| Primary DNS | 192.168.2.253 |
| Secondary DNS | *(leave blank)* |
| Lease time | 24h |
Save without enabling.
### 3b. Switchover (do steps in quick succession)
1. **Disable Pi-hole DHCP:** Pi-hole admin UI → Settings → DHCP → uncheck
"DHCP server enabled" → Save
2. **Enable router DHCP** immediately after step 1
### 3c. Verify router DHCP is working
On a phone or laptop, disconnect from WiFi and reconnect (or run
`sudo dhclient -r && sudo dhclient` on a Linux host):
```bash
ip addr show # IP should be in 192.168.2.1059 range
dig google.com # should resolve (Pi-hole DNS still running at .253)
dig pve1.sweet.home # should resolve via FreeIPA at .138 (relayed via Pi-hole)
```
Wait 1015 minutes for the most active devices to renew their leases. There's
no need to wait for all leases to expire before proceeding.
**Rollback 3b:** Re-enable Pi-hole DHCP. Disable router DHCP. Done — existing
leases remain valid so most devices are unaffected.
---
## Stage 4 — Move domain-controller from .138 to .253
Pi-hole lives at `.253`. The DC must take `.253` the moment Pi-hole stops so
clients that still have `.253` as their DNS server don't notice the change.
Script these commands in advance and run them in rapid succession.
**Pre-stage: have this SSH command ready before running step 4a:**
```bash
ssh wayne@192.168.2.138 "
sudo nmcli connection modify 'System eth0' \
ipv4.addresses '192.168.2.253/24' \
ipv4.gateway '192.168.2.254' \
ipv4.dns '127.0.0.1' \
ipv4.method manual && \
sudo nmcli connection up 'System eth0'
"
```
**Also update the Proxmox VM config to match (run from pve1):**
```bash
sudo qm set 108 \
--ipconfig0 ip=192.168.2.253/24,gw=192.168.2.254 \
--nameserver 192.168.2.253
```
### 4a. Stop Pi-hole
```bash
ssh wayne@pve1.sweet.home "sudo pct stop 100"
```
### 4b. Immediately: change DC's IP to .253
Run the pre-staged SSH command from above. You have ~30 seconds before any
client notices Pi-hole is gone. If SSH to `.138` refuses (the IP is already
changing), open a Proxmox console to VM 108 and run the `nmcli` commands
there.
### 4c. Update Proxmox VM config
Run the pre-staged `qm set 108` command from above.
### 4d. Verify
```bash
ssh wayne@192.168.2.253 # must connect (new DC IP)
dig @192.168.2.253 pve1.sweet.home +short # must return 192.168.2.250
dig @192.168.2.253 google.com +short # must return an IP
```
From a client device that renewed its DHCP lease in Stage 3:
```bash
cat /etc/resolv.conf # should show 192.168.2.253
dig pve1.sweet.home # should resolve
```
**Rollback 4:** `ssh wayne@pve1.sweet.home "sudo pct start 100"`. Change DC IP
back to .138 via Proxmox console. This restores full Pi-hole DNS/DHCP service.
Leave Pi-hole CT stopped-but-intact for 48 hours before deleting it.
---
## Stage 5 — Host renumbering (one at a time, any order)
For each host:
1. Update FreeIPA DNS A record and PTR record to the new IP
2. Change the static IP on the host itself
3. Verify SSH to new IP
4. Update `variables.nix` if that host has an IP variable (pxe-boot, pbs — already done in this PR)
**FreeIPA record update template** (run as admin on domain-controller):
```bash
ipa dnsrecord-mod sweet.home <hostname> --a-rec <new-ip>
ipa dnsrecord-del 2.168.192.in-addr.arpa <old-last-octet> --ptr-rec <hostname>.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa <new-last-octet> --ptr-rec <hostname>.sweet.home.
```
### Renumbering order
| # | Host | Old IP | New IP | How to change IP |
|---|---|---|---|---|
| 1 | nixos workstation | .119 | .243 | NetworkManager on guest; or `nmcli connection modify` |
| 2 | nix-cache | .120 | .224 | `pct set 102 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.224/24,gw=192.168.2.254` then `pct reboot 102` |
| 3 | tailscale-router | .121 | .222 | Static config on guest; check Tailscale ACLs if IP is referenced there |
| 4 | tor-relay | .107 | .221 | `pct set 104 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.221/24,gw=192.168.2.254` then `pct reboot 104` |
| 5 | pdm | .248 | .220 | `pct set 106 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.220/24,gw=192.168.2.254` then `pct reboot 106` |
| 6 | pxe-boot | .247 | .223 | `pct set 103 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.223/24,gw=192.168.2.254` then rebuild NixOS (already updated in variables.nix) |
| 7 | server | .252 | .226 | Static config on guest; NFS clients (docker) lose mounts briefly — they remount automatically |
| 8 | docker | .249 | .225 | Static config on guest; do this after server is at .226 |
| 9 | pbs | .108 | .244 | Static config on PBS host itself; update in `pbsIp` already done in variables.nix |
| 10 | pve1 | .250 | .245 | Edit `/etc/network/interfaces` on the Proxmox host — see below |
### pve1 renumber (step 10 — do last)
All guests keep running; only the Proxmox web UI is briefly unreachable.
```bash
ssh wayne@pve1.sweet.home
# Edit /etc/network/interfaces: change address from .250 to .245
sudo nano /etc/network/interfaces
# Change: address 192.168.2.250/24
# To: address 192.168.2.245/24
sudo systemctl restart networking
# SSH will drop here — reconnect to new IP
```
```bash
ssh wayne@192.168.2.245 # verify
```
Update FreeIPA DNS:
```bash
ipa dnsrecord-mod sweet.home pve1 --a-rec 192.168.2.245
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 245 --ptr-rec pve1.sweet.home.
```
**Rollback any step 5 host:** change the IP back on the guest and update the
FreeIPA record back to the old IP. The old IP is unoccupied so you can
temporarily use either.
---
## Stage 6 — Final cleanup
Once all hosts are at their new IPs and verified:
```bash
# Delete the Pi-hole CT (already stopped since Stage 4)
ssh wayne@pve1.sweet.home "sudo pct destroy 100"
# Remove stale FreeIPA records for retired addresses
ipa dnsrecord-del sweet.home pihole --del-all
ipa dnsrecord-del 2.168.192.in-addr.arpa 253 --ptr-rec pihole.sweet.home.
# Rebuild any NixOS hosts that reference pbsIp or pxeServerIp to pick up
# the updated variables.nix values (pxe-boot mandatory; others as convenient)
```
---
## Rollback summary
| What broke | How to roll back |
|---|---|
| FreeIPA DNS not resolving | Check `systemctl status named` on DC; restart if failed |
| FreeIPA DNS unreachable | `pct start 100` on pve1 (restores Pi-hole) |
| Router DHCP not handing out leases | Re-enable Pi-hole DHCP; disable router DHCP |
| DC unreachable after IP change | Proxmox console on VM 108 → `nmcli connection up "System eth0"` with old IP |
| Host unreachable after renumber | Proxmox console → revert IP; or `pct set <id> --net0 ...` old IP and reboot CT |
| pve1 web UI gone after renumber | SSH to .245 and check `/etc/network/interfaces`; if wrong, fix and restart networking |
+97 -7
View File
@@ -15,6 +15,7 @@ rescue/inspection use.
- TFTP root for first-stage bootloaders: `/srv/pxe/tftp`
- iPXE entry script: `/srv/pxe/http/boot.ipxe`
- Generated iPXE menu: `/srv/pxe/http/menu.ipxe`
- Debian Minimal iPXE script: `/srv/pxe/http/debian.ipxe`
- SystemRescue iPXE script: `/srv/pxe/http/systemrescue.ipxe`
- TFTP fallback script: `/srv/pxe/tftp/autoexec.ipxe`
- Boot binaries copied from the Nix `ipxe` package:
@@ -28,32 +29,51 @@ The host creates these directories with systemd tmpfiles:
```text
/srv/pxe
/srv/pxe/http
/srv/pxe/http/images
/srv/pxe/http/images -> /mnt/pxe-images (symlink to NFS share)
/srv/pxe/http/auto-installer
/srv/pxe/http/nixos-minimal
/srv/pxe/http/debian
/srv/pxe/http/systemrescue
/srv/pxe/http/ubuntu
/srv/pxe/http/rescue
/srv/pxe/tftp
```
Mount shared image storage under `/srv/pxe/http`, preferably
`/srv/pxe/http/images` unless a menu entry expects files in a specific
directory such as `/srv/pxe/http/auto-installer`.
`/srv/pxe/http/images` is a symlink to `/mnt/pxe-images`, which is an NFS
mount of `server.sweet.home:/tank/pxe-boot/images`
(`modules/pxe-boot/mount-pxe-images.nix`). Place large images there (ISOs,
disk images) rather than on the pxe-boot host's own root disk. For an LXC
pxe-boot container the mount uses NFSv3+nolock with `nofail` (eager,
non-blocking on server unavailability); for a Proxmox VM it uses NFSv4.2
with `x-systemd.automount` (lazy, triggered on first access).
When running as `lxc-pxe-boot`, the Proxmox container must have
`features: nesting=1,mount=nfs` (at minimum) in its Proxmox config. `nesting=1`
is required by systemd 260+ for credential isolation (user namespace creation
and internal move-mounts); without it, AppArmor denies both, and every
systemd service that uses `PrivateUsers`, `PrivateDevices`, or credential
passing fails on boot. `mount=nfs` allows the NFSv3 mount. Both are set
automatically by `scripts/proxmox/create-proxmox-resource.sh` (via
`PROXMOX_DEFAULT_LXC_FEATURES` in `scripts/env.sh` which defaults to
`nesting=1,keyctl=1,mount=nfs;nfs4`). If you ever change these features
manually via `pct set`, be sure to include both — `pct set` replaces the
entire features string, it does not append to it.
The HTTP iPXE chain is:
```text
undionly.kpxe or ipxe.efi
-> autoexec.ipxe from the TFTP root, when iPXE requests it
-> http://192.168.2.247/boot.ipxe
-> http://192.168.2.247/menu.ipxe
-> http://192.168.2.223/boot.ipxe
-> http://192.168.2.223/menu.ipxe
```
The generated menu currently exposes entries for:
- NixOS Auto-Installer
- NixOS Minimal
- Debian Minimal
- FreeIPA Server (Rocky Linux 9)
- SystemRescue environment
- iPXE shell
- Reboot
@@ -84,19 +104,82 @@ directory name (`auto-installer` / `nixos-minimal`), so each one's
generated system name (`nixos-system-<name>-*`) is self-describing rather
than the nixpkgs default of `nixos-system-nixos-*` for both.
The Debian Minimal entry chains `http://<pxeServerIp>/debian.ipxe`, which loads
the Debian bookworm netboot kernel and initrd from `/srv/pxe/http/debian/`. The
`fetch-debian-netboot.service` oneshot downloads these files from
`deb.debian.org` on first boot (idempotent — skips if files are already
present):
```text
/srv/pxe/http/debian/linux (Debian bookworm netboot kernel)
/srv/pxe/http/debian/initrd.gz (Debian bookworm netboot initrd)
```
The service requires outbound internet access on the pxe-boot host. To
re-download (e.g. after a Debian point release), delete the files and restart
the service:
```bash
rm /srv/pxe/http/debian/linux /srv/pxe/http/debian/initrd.gz
systemctl restart fetch-debian-netboot.service
```
To update to a different Debian release, change `debianRelease` in
`modules/build-types/pxe-boot.nix` and redeploy.
The **FreeIPA Server (Rocky Linux 9)** entry chains
`http://<pxeServerIp>/rocky-freeipa.ipxe`, which boots the Rocky Linux 9
Anaconda installer with a Kickstart file (`rocky-freeipa.ks`) hosted on the
same server. The `fetch-rocky-pxeboot.service` oneshot downloads the pxeboot
kernel and initrd from the Rocky Linux mirror on first boot (idempotent):
```text
/srv/pxe/http/rocky/vmlinuz (Rocky Linux 9 Anaconda pxeboot kernel)
/srv/pxe/http/rocky/initrd.img (Rocky Linux 9 Anaconda pxeboot initrd)
```
The Kickstart file is generated from the NixOS module and staged at
`/srv/pxe/http/rocky-freeipa.ks`. It performs a fully unattended install:
1. Installs Rocky Linux 9 with `ipa-server` + `ipa-server-dns` packages
2. Configures static IP `192.168.2.138`, hostname `domain-controller.sweet.home`
3. Creates user `wayne` with the `adminSshKey` from `variables.nix`
4. Generates random IPA passwords and writes them to `/root/ipa-credentials.txt`
5. Creates a `freeipa-first-boot.service` oneshot that runs `ipa-server-install`
on first reboot (~20 minutes)
After the install completes:
- SSH in as `wayne@domain-controller` using the admin key
- Monitor FreeIPA install progress: `sudo tail -f /root/freeipa-install.log`
- Retrieve credentials: `sudo cat /root/ipa-credentials.txt` (save to password manager)
- Configure Pi-hole: `server=/sweet.home/192.168.2.138` in dnsmasq
To refresh the pxeboot files (e.g. after a Rocky point release):
```bash
rm /srv/pxe/http/rocky/vmlinuz /srv/pxe/http/rocky/initrd.img
systemctl restart fetch-rocky-pxeboot.service
```
To update to a different Rocky release, change `rockyRelease` in
`modules/build-types/pxe-boot.nix` and redeploy.
The SystemRescue entry expects the source ISO at:
```text
/srv/pxe/http/images/systemrescue.iso
```
Since `/srv/pxe/http/images` is the NFS-backed symlink, place the ISO on the
NFS share at `server.sweet.home:/tank/pxe-boot/images/systemrescue.iso`.
The `stage-systemrescue.service` oneshot extracts that ISO into:
```text
/srv/pxe/http/systemrescue
```
The rescue menu entry then chains `http://192.168.2.247/systemrescue.ipxe`,
The rescue menu entry then chains `http://192.168.2.223/systemrescue.ipxe`,
which loads the SystemRescue kernel and initramfs from the extracted tree and
uses `archiso_http_srv` to fetch the squashfs payload over HTTP.
@@ -113,6 +196,13 @@ After deployment by an operator, basic service checks are:
```bash
curl http://pxe-boot/boot.ipxe
curl http://pxe-boot/menu.ipxe
curl http://pxe-boot/debian.ipxe
curl -I http://pxe-boot/debian/linux
curl -I http://pxe-boot/debian/initrd.gz
curl http://pxe-boot/rocky-freeipa.ipxe
curl http://pxe-boot/rocky-freeipa.ks
curl -I http://pxe-boot/rocky/vmlinuz
curl -I http://pxe-boot/rocky/initrd.img
curl http://pxe-boot/systemrescue.ipxe
curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/vmlinuz
curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/sysresccd.img
Generated
+157 -7
View File
@@ -1,5 +1,62 @@
{
"nodes": {
"clan-core": {
"inputs": {
"data-mesher": "data-mesher",
"disko": [
"disko"
],
"flake-parts": "flake-parts",
"nix-darwin": "nix-darwin",
"nix-select": "nix-select",
"nixpkgs": [
"nixpkgs"
],
"sops-nix": [
"sops-nix"
],
"systems": "systems",
"treefmt-nix": "treefmt-nix"
},
"locked": {
"lastModified": 1783497933,
"narHash": "sha256-TxmwEews6URFPqOWEHNychtXbFDgLZjbOfEXtvtOm6U=",
"rev": "3dc0221ca09033599fe98055e9bbc81bdf32732a",
"type": "tarball",
"url": "https://git.clan.lol/api/v1/repos/clan/clan-core/archive/3dc0221ca09033599fe98055e9bbc81bdf32732a.tar.gz"
},
"original": {
"type": "tarball",
"url": "https://git.clan.lol/clan/clan-core/archive/26.05.tar.gz"
}
},
"data-mesher": {
"inputs": {
"flake-parts": [
"clan-core",
"flake-parts"
],
"nixpkgs": [
"clan-core",
"nixpkgs"
],
"treefmt-nix": [
"clan-core",
"treefmt-nix"
]
},
"locked": {
"lastModified": 1778718524,
"narHash": "sha256-pXLoI6Ax0EnUK6r34UM1vibVC7CfTu6j72R2692ZzPs=",
"rev": "12c552ad547d87254f33f33bddd1a2cdbeac754d",
"type": "tarball",
"url": "https://git.clan.lol/api/v1/repos/clan/data-mesher/archive/12c552ad547d87254f33f33bddd1a2cdbeac754d.tar.gz"
},
"original": {
"type": "tarball",
"url": "https://git.clan.lol/clan/data-mesher/archive/main.tar.gz"
}
},
"disko": {
"inputs": {
"nixpkgs": [
@@ -51,9 +108,30 @@
"type": "github"
}
},
"flake-parts": {
"inputs": {
"nixpkgs-lib": [
"clan-core",
"nixpkgs"
]
},
"locked": {
"lastModified": 1778716662,
"narHash": "sha256-m1Yf0wZ8j1OHjTc2UwHwyQRSnNeSgLJOd7q5Y45hzi4=",
"owner": "hercules-ci",
"repo": "flake-parts",
"rev": "f7c1a2d347e4c52d5fb8d10cb4d94b5884e546fb",
"type": "github"
},
"original": {
"owner": "hercules-ci",
"repo": "flake-parts",
"type": "github"
}
},
"flake-utils": {
"inputs": {
"systems": "systems"
"systems": "systems_2"
},
"locked": {
"lastModified": 1694529238,
@@ -95,11 +173,11 @@
]
},
"locked": {
"lastModified": 1784350909,
"narHash": "sha256-ZWyzLbS1yKUTeFJLmdVuWNnHttL333/ldJbEE+KzCrM=",
"lastModified": 1785119570,
"narHash": "sha256-Rgs2xKnGLFWQscxUaXX07oyZeuMDOHEbqDOsgliLFGM=",
"owner": "nix-community",
"repo": "home-manager",
"rev": "4ce190229c73d44536caa7072f6308fb2d8feeb3",
"rev": "d4fd24667c8cbef124bb70a20380cab75ec8474d",
"type": "github"
},
"original": {
@@ -109,6 +187,40 @@
"type": "github"
}
},
"nix-darwin": {
"inputs": {
"nixpkgs": [
"clan-core",
"nixpkgs"
]
},
"locked": {
"lastModified": 1779036909,
"narHash": "sha256-zXcwYQGCT6pzinK+1dBB2ekTVtfxGZAapb3Evdcu4fY=",
"owner": "nix-darwin",
"repo": "nix-darwin",
"rev": "56c666e108467d87d13508936aade6d567f2a501",
"type": "github"
},
"original": {
"owner": "nix-darwin",
"repo": "nix-darwin",
"type": "github"
}
},
"nix-select": {
"locked": {
"lastModified": 1763303120,
"narHash": "sha256-yxcNOha7Cfv2nhVpz9ZXSNKk0R7wt4AiBklJ8D24rVg=",
"rev": "3d1e3860bef36857a01a2ddecba7cdb0a14c35a9",
"type": "tarball",
"url": "https://git.clan.lol/api/v1/repos/clan/nix-select/archive/3d1e3860bef36857a01a2ddecba7cdb0a14c35a9.tar.gz"
},
"original": {
"type": "tarball",
"url": "https://git.clan.lol/clan/nix-select/archive/main.tar.gz"
}
},
"nixos-conf-editor": {
"inputs": {
"flake-compat": "flake-compat",
@@ -147,11 +259,11 @@
},
"nixpkgs_2": {
"locked": {
"lastModified": 1784432872,
"narHash": "sha256-n3gKTBIV4ZA5VQpUakffBe3KGu4+mhPoA34rrqS0GkA=",
"lastModified": 1785133411,
"narHash": "sha256-Yjv0WEg39KRYS0rBdTbu6Fc/or/ihAKk13W9sQ6VWd0=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "fd1462031fdee08f65fd0b4c6b64e22239a77870",
"rev": "2f5a153c270b70cb0f8c11f46d96d6d3bc39f4e3",
"type": "github"
},
"original": {
@@ -163,6 +275,7 @@
},
"root": {
"inputs": {
"clan-core": "clan-core",
"disko": "disko",
"home-manager": "home-manager",
"nixos-conf-editor": "nixos-conf-editor",
@@ -214,6 +327,22 @@
}
},
"systems": {
"locked": {
"lastModified": 1774449309,
"narHash": "sha256-brhZ8DmuGtzkCYHJg4HEd602amKm89Y9ytsFZ5uWD1w=",
"owner": "nix-systems",
"repo": "default",
"rev": "c29398b59d2048c4ab79345812849c9bd15e9150",
"type": "github"
},
"original": {
"owner": "nix-systems",
"ref": "future-26.11",
"repo": "default",
"type": "github"
}
},
"systems_2": {
"locked": {
"lastModified": 1681028828,
"narHash": "sha256-Vy1rq5AaRuLzOxct8nz4T6wlgyUR7zLU309k9mBC768=",
@@ -227,6 +356,27 @@
"repo": "default",
"type": "github"
}
},
"treefmt-nix": {
"inputs": {
"nixpkgs": [
"clan-core",
"nixpkgs"
]
},
"locked": {
"lastModified": 1780220602,
"narHash": "sha256-eynAfOmbmxJnkp7YewvCEbShNnnYJ9gLLqkzsYtBPeM=",
"owner": "numtide",
"repo": "treefmt-nix",
"rev": "db947814a175b7ca6ded66e21383d938df01c227",
"type": "github"
},
"original": {
"owner": "numtide",
"repo": "treefmt-nix",
"type": "github"
}
}
},
"root": "root",
+35 -3
View File
@@ -16,6 +16,19 @@
url = "github:Mic92/sops-nix";
inputs.nixpkgs.follows = "nixpkgs";
};
clan-core = {
url = "https://git.clan.lol/clan/clan-core/archive/26.05.tar.gz";
# Deduplicate modules: clan-core bundles its own disko and sops-nix
# (both imported by nixosModules.clanCore). Without follows, we'd get
# two different versions of each, and disko's _module.args.diskoLib
# unique option would conflict. With follows, clan-core uses the same
# store paths as us, so NixOS deduplicates the imports.
inputs = {
nixpkgs.follows = "nixpkgs";
disko.follows = "disko";
sops-nix.follows = "sops-nix";
};
};
};
outputs = { self, nixpkgs, nixos-conf-editor, home-manager, sops-nix, ... } @ inputs:
@@ -41,6 +54,22 @@
modules = [
inputs.disko.nixosModules.disko
sops-nix.nixosModules.sops
inputs.clan-core.nixosModules.clanCore
{
# Required clan settings. directory is the flake root (where
# vars/ and sops/ directories live); machine.name is the flake
# target name (matches what clan vars generate uses as the key
# under vars/per-machine/). enableRecommendedDefaults = false
# is mandatory: without it, clan unconditionally enables
# networking.useNetworkd, adds packages, and tweaks nix settings
# -- none of which belong here.
clan.core = {
settings.directory = self;
settings.machine.name = flakeTarget;
enableRecommendedDefaults = false;
};
}
./modules/clan/ssh-host-key.nix
./modules/common/configuration.nix
./modules/platforms/${platform}.nix
./modules/build-types/${buildType}.nix
@@ -96,11 +125,14 @@
proxmox-pxe-boot = mkTarget { platform = "proxmox"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
lxc-pxe-boot = mkTarget { platform = "lxc"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
linode-tailscale-exit-node = mkTarget { platform = "linode"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
proxmox-tailscale-exit-node = mkTarget { platform = "proxmox"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
lxc-tailscale-exit-node = mkTarget { platform = "lxc"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
linode-tailscale-router = mkTarget { platform = "linode"; buildType = "tailscale-router"; hostPath = ./hosts/tailscale-router/host.nix; };
proxmox-tailscale-router = mkTarget { platform = "proxmox"; buildType = "tailscale-router"; hostPath = ./hosts/tailscale-router/host.nix; };
lxc-tailscale-router = mkTarget { platform = "lxc"; buildType = "tailscale-router"; hostPath = ./hosts/tailscale-router/host.nix; };
lxc-tor-relay = mkTarget { platform = "lxc"; buildType = "tor-relay"; hostPath = ./hosts/tor-relay/host.nix; };
proxmox-ha-server-1 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-1/host.nix; };
proxmox-ha-server-2 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-2/host.nix; };
};
# Auto-install environments (migrated from the former nix-auto-installer
+17 -3
View File
@@ -1,10 +1,24 @@
_:
{ vars, ... }:
{
networking.hostName = "docker";
networking.hostId = "007f0200";
networking = {
hostName = "docker";
hostId = "007f0200";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.dockerIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
boot.zfs.forceImportRoot = false;
# Only advertise the LAN interface to IPA DNS. Without this, SSSD registers
# every Docker bridge (172.x.x.x) as an A record for docker.sweet.home —
# the default dyndns.interface = "*" catches them all.
security.ipa.dyndns.interface = vars.lxcLanInterface; # eth0
# Preserved from the pre-refactor `docker` target — stateVersion must never
# be bumped on an already-installed machine.
system.stateVersion = "25.05";
+20
View File
@@ -0,0 +1,20 @@
{ vars, ... }:
{
networking = {
hostName = vars.haServer1Host;
hostId = "3a4b5c6d";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.haServer1Ip;
prefixLength = vars.lanPrefixLength;
}];
interfaces.${vars.vmStorageInterface}.ipv4.addresses = [{
address = vars.haServer1StorageIp;
prefixLength = vars.haStoragePrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
system.stateVersion = "26.05";
}
+20
View File
@@ -0,0 +1,20 @@
{ vars, ... }:
{
networking = {
hostName = vars.haServer2Host;
hostId = "7e8f9a0b";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.haServer2Ip;
prefixLength = vars.lanPrefixLength;
}];
interfaces.${vars.vmStorageInterface}.ipv4.addresses = [{
address = vars.haServer2StorageIp;
prefixLength = vars.haStoragePrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
system.stateVersion = "26.05";
}
+9 -12
View File
@@ -1,18 +1,15 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "nix-cache";
sopsFile = ../../secrets/nix-cache.yaml;
})
];
networking.hostName = vars.nixCacheHost;
services.beszel.agent.environment = {
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
networking = {
hostName = vars.nixCacheHost;
useDHCP = false;
interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
address = vars.nixCacheIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# Preserved from the pre-refactor `nix-cache` target — stateVersion must
+3 -2
View File
@@ -21,12 +21,13 @@
chromium
claude-code
fish
sops
];
# Optional: set environment vars
sessionVariables = {
EDITOR = "vim";
SOPS_AGE_KEY_FILE = "/home/nixos/Nextcloud/Filing Cabinet/keys/nixos-sops-age-key-txt";
EDITOR = "nano";
SOPS_AGE_KEY_FILE = "${config.home.homeDirectory}/.config/sops/age/keys.txt";
};
file = {
+14 -3
View File
@@ -1,8 +1,19 @@
_:
{ vars, ... }:
{
networking.hostName = "pxe-boot";
networking = {
hostName = "pxe-boot";
useDHCP = false;
interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
address = vars.pxeServerIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
services.beszel.agent.environment = {
# KEY = "";
};
# Preserved from the pre-refactor `pxe-boot` target — stateVersion must
# never be bumped on an already-installed machine.
system.stateVersion = "25.05";
+11 -11
View File
@@ -1,19 +1,19 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "server";
sopsFile = ../../secrets/server.yaml;
})
];
networking.hostName = vars.nfsServerHost;
networking.hostId = "6689f93e";
networking = {
hostName = vars.nfsServerHost;
hostId = "6689f93e";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.serverIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
services.beszel.agent.environment = {
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
LOG_LEVEL = "debug";
};
-12
View File
@@ -1,12 +0,0 @@
_:
{
networking.hostName = "exit-node";
# No networking.hostId: only ZFS-touching hosts (server, docker) need one
# for pool-import safety, and this host does neither.
# A genuinely new host (not a pre-refactor carry-over), so it tracks the
# flake's current nixpkgs release rather than being pinned to an older one.
system.stateVersion = "26.05";
}
+19
View File
@@ -0,0 +1,19 @@
{ vars, ... }:
{
networking = {
hostName = "tailscale-router";
useDHCP = false;
interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
address = vars.tailscaleRouterIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# No networking.hostId: only ZFS-touching hosts (server, docker) need one
# for pool-import safety, and this host does neither.
system.stateVersion = "26.05";
}
+13 -4
View File
@@ -1,10 +1,19 @@
_:
{ vars, ... }:
{
networking.hostName = "tor-relay";
networking = {
hostName = "tor-relay";
useDHCP = false;
interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
address = vars.torRelayIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# No networking.hostId: only ZFS-touching hosts (server, docker) need one
# for pool-import safety, and this host does neither.
# No networking.hostId: only ZFS-touching hosts need one for pool-import
# safety, and this host does neither.
# A genuinely new host (not a pre-refactor carry-over), so it tracks the
# flake's current nixpkgs release rather than being pinned to an older one.
+18 -5
View File
@@ -1,10 +1,23 @@
{ vars, ... }:
{ config, vars, ... }:
{
services.beszel.agent.enable = true;
services.beszel.agent.environment = {
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
# Universal token shared by all beszel agents. Add to secrets/common.yaml:
# sops secrets/common.yaml
# beszel-token: <value from the beszel hub UI>
sops.secrets."beszel-token" = { };
sops.templates."beszel.env".content = ''
TOKEN=${config.sops.placeholder."beszel-token"}
'';
services.beszel.agent = {
enable = true;
environmentFile = config.sops.templates."beszel.env".path;
environment = {
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
KEY = vars.beszelHubKey;
};
};
# The upstream module runs beszel-agent under DynamicUser with
-11
View File
@@ -1,11 +0,0 @@
{ name, sopsFile }:
{ config, ... }:
{
sops.secrets."beszel-token".sopsFile = sopsFile;
sops.templates."${name}-beszel.env".content = ''
TOKEN=${config.sops.placeholder."beszel-token"}
'';
services.beszel.agent.environmentFile = config.sops.templates."${name}-beszel.env".path;
}
-1
View File
@@ -15,7 +15,6 @@
../docker/enable-service.nix
../docker/nextcloud-cron-job.nix
../docker/docker-health-to-gotify.nix
../tailscale/enable-service.nix
../traefik/rotate-logs.nix
../raspi/mount-data.nix
../services/enable-rpcbind.nix
+78 -1
View File
@@ -1,6 +1,17 @@
{ config, pkgs, lib, inputs, vars, ... }:
{
imports = [
../docker/enable-service.nix
];
nixpkgs.overlays = [
(final: prev: {
docker = prev.docker_29;
docker_cli = prev.docker_29;
})
];
environment.systemPackages = with pkgs; [
inputs.nixos-conf-editor.packages.${pkgs.stdenv.hostPlatform.system}.nixos-conf-editor
nodejs
@@ -18,7 +29,7 @@
];
boot.loader.grub.useOSProber = true;
programs.direnv.enable = true;
programs.direnv.enable = true;
services = {
xserver = {
enable = true;
@@ -70,4 +81,70 @@ programs.direnv.enable = true;
programs.firefox.enable = true;
nixpkgs.config.allowUnfree = true;
# GUI-specific Home Manager additions for the IPA primary user, extending
# the baseline in modules/ipa/client.nix with desktop apps and services
# that only make sense on a graphical workstation.
home-manager.users.${vars.ipaUser} = { pkgs, ... }: {
home = {
packages = with pkgs; [
git
vim
nextcloud-client
chromium
claude-code
fish
sops
];
sessionVariables = {
EDITOR = "vim";
SOPS_AGE_KEY_FILE = "/home/${vars.ipaUser}/.config/sops/age/keys.txt";
};
file = {
".local/share/applications/proxmox-chromium-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox (Chromium)
Exec=chromium --app=https://pve.${vars.homeDomain}:${toString vars.ports.pveWeb} --window-size=1920,1080 --window-position=0,0
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=Hypervisor;
StartupWMClass=PVE
'';
".local/share/applications/pbs-chromium-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox Backup Server (Chromium)
Exec=chromium --app=https://${vars.pbsIp}:${toString vars.ports.pbsWeb} --window-size=1920,1080 --window-position=0,0
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=backup;
'';
".local/share/applications/proxmox-firefox-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox (Firefox)
Exec=firefox --new-instance https://pve.${vars.homeDomain}:${toString vars.ports.pveWeb} --profile ProxmoxWebApp --window-size=1920,1080 --class ProxmoxWebApp
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=Hypervisor;
StartupWMClass=PVE
'';
".local/share/applications/pbs-firefox-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox Backup Server (Firefox)
Exec=firefox --new-window https://${vars.pbsIp}:${toString vars.ports.pbsWeb} --profile PbsWebApp --window-size=1920,1080 --class PbsWebApp
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=backup;
StartupWMClass=PBS
'';
};
};
services.nextcloud-client = {
enable = true;
startInBackground = true;
};
};
}
+51
View File
@@ -0,0 +1,51 @@
# HA file server build type: DRBD + XFS + LIO iSCSI + NFS, managed by
# Corosync + Pacemaker. Both ha-server-1 and ha-server-2 use this type.
#
# NFS start/stop:
# services.nfs.server.enable = true configures /etc/exports, wires up
# rpcbind, and loads kernel modules — but nfs-server.service.wantedBy is
# force-cleared so systemd does NOT auto-start it at boot. Pacemaker's
# ha-group resource group (configured by scripts/ha/cluster-init.sh)
# starts and stops nfs-server as part of the failover sequence after the
# XFS mount and iSCSI target are brought up on the new Active node.
#
# Beszel agent:
# Enabled here via enable-agent.nix. The agent KEY (used to pair with
# the Beszel hub) is not set yet — add it to hosts/ha-server-{1,2}/host.nix
# under services.beszel.agent.environment.KEY once the hub accepts the
# new agents, following the pattern in hosts/server/host.nix.
{ lib, pkgs, vars, ... }:
let
# Generates /etc/exports lines for all nfsShares data entries. Shared
# pattern with modules/build-types/server.nix — both export the same
# set of shares, differing only in the storage root they serve from.
mkNfsExports = storageRoot:
lib.concatMapStrings
(share: " ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n")
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares));
in
{
imports = [
../ha/pacemaker-stack.nix
../ha/iscsi-target.nix
../ha/cluster-config.nix
../beszel/enable-agent.nix
];
# xfsprogs: mkfs.xfs/xfs_info needed by cluster-init.sh.
# openiscsi: iscsiadm needed by acceptance-tests.sh T4 (iSCSI discovery check).
environment.systemPackages = [ pkgs.xfsprogs pkgs.openiscsi ];
services.nfs.server = {
enable = true;
exports = mkNfsExports vars.haStorageRoot;
};
# Pacemaker controls nfs-server — prevent systemd from starting it at boot
# on both nodes (only the Active node should be serving NFS).
systemd.services.nfs-server.wantedBy = lib.mkForce [ ];
# Same reason as server.nix: exports use standard auth, not Kerberos.
systemd.services.rpc-svcgssd.enable = false;
}
+317 -33
View File
@@ -21,6 +21,216 @@ let
chain ${pxeBaseUrl}/boot.ipxe
'';
debianRelease = "bookworm";
debianMirror = "https://deb.debian.org/debian";
debianNetbootBase = "${debianMirror}/dists/${debianRelease}/main/installer-amd64/current/images/netboot/debian-installer/amd64";
rockyRelease = "9";
rockyArch = "x86_64";
rockyMirror = "https://dl.rockylinux.org/pub/rocky/${rockyRelease}";
rockyPxebootBase = "${rockyMirror}/BaseOS/${rockyArch}/os/images/pxeboot";
debianIpxe = pkgs.writeText "debian.ipxe" ''
#!ipxe
set base ${pxeBaseUrl}
kernel ''${base}/debian/linux
initrd ''${base}/debian/initrd.gz
boot
'';
fetchDebianNetboot = pkgs.writeShellScript "fetch-debian-netboot" ''
set -eu
dir="${httpRoot}/debian"
mirror="${debianNetbootBase}"
if [ -f "$dir/linux" ] && [ -f "$dir/initrd.gz" ]; then
echo "Debian ${debianRelease} netboot files already present; skipping download."
exit 0
fi
echo "Downloading Debian ${debianRelease} netboot kernel and initrd from $mirror ..."
${pkgs.curl}/bin/curl -fsSL -o "$dir/linux.tmp" "$mirror/linux"
${pkgs.curl}/bin/curl -fsSL -o "$dir/initrd.gz.tmp" "$mirror/initrd.gz"
mv "$dir/linux.tmp" "$dir/linux"
mv "$dir/initrd.gz.tmp" "$dir/initrd.gz"
echo "Debian ${debianRelease} netboot files staged."
'';
# Rocky Linux 9 iPXE script — boots vmlinuz+initrd.img from the staged
# /rocky/ directory and hands Anaconda the hosted Kickstart URL.
# net.ifnames=0 biosdevname=0 ensures the NIC is eth0 in both the
# installer and the installed system (matches the Kickstart NM config).
rockyFreeIpaIpxe = pkgs.writeText "rocky-freeipa.ipxe" ''
#!ipxe
set base ${pxeBaseUrl}
kernel ''${base}/rocky/vmlinuz inst.ks=''${base}/rocky-freeipa.ks inst.repo=${rockyMirror}/BaseOS/${rockyArch}/os/ net.ifnames=0 biosdevname=0 ip=dhcp quiet
initrd ''${base}/rocky/initrd.img
boot
'';
# Kickstart file for domain-controller.sweet.home.
# Installs Rocky Linux 9, sets a static IP, creates wayne with the
# admin SSH key, then on first reboot runs ipa-server-install via a
# systemd oneshot service. Passwords are generated at %post time,
# written to /root/ipa-credentials.txt (chmod 600), and read back by
# the first-boot script — never hardcoded here or in the repo.
rockyFreeIpaKs = pkgs.writeText "rocky-freeipa.ks" ''
#version=RHEL9
# Unattended Rocky Linux 9 + FreeIPA install
# Target: domain-controller.${vars.homeDomain} ${vars.domainControllerIp}
url --url=${rockyMirror}/BaseOS/${rockyArch}/os/
repo --name=appstream --baseurl=${rockyMirror}/AppStream/${rockyArch}/os/
lang en_US.UTF-8
keyboard us
timezone UTC --utc
# DHCP during install; static IP configured in %post via NM config file
network --bootproto=dhcp --device=link --activate
network --hostname=domain-controller.sweet.home
selinux --enforcing
firewall --enabled --service=ssh
rootpw --lock
user --name=wayne --groups=wheel --shell=/bin/bash
sshkey --username=wayne "${vars.adminSshKey}"
zerombr
clearpart --all --initlabel --drives=sda
# Keep net.ifnames=0 biosdevname=0 in the installed GRUB so the NIC
# stays eth0 after reboot (matches the NM connection file below).
bootloader --location=mbr --boot-drive=sda --append="net.ifnames=0 biosdevname=0"
part /boot --fstype=xfs --size=1024 --ondisk=sda
part swap --fstype=swap --size=2048 --ondisk=sda
part / --fstype=xfs --grow --size=1 --ondisk=sda --asprimary
%packages
@^minimal-environment
ipa-server
ipa-server-dns
%end
reboot
%post --log=/root/ks-post.log
set -euo pipefail
# -- Static IP: write NM connection file directly (NM not running in chroot) --
mkdir -p /etc/NetworkManager/system-connections
cat > /etc/NetworkManager/system-connections/eth0.nmconnection << 'NMCONN'
[connection]
id=eth0
type=ethernet
interface-name=eth0
autoconnect=true
[ethernet]
[ipv4]
method=manual
addresses=${vars.domainControllerIp}/${toString vars.lanPrefixLength}
gateway=${vars.lanGateway}
dns=${vars.domainControllerIp};
dns-search=${vars.homeDomain};
[ipv6]
method=auto
NMCONN
chmod 600 /etc/NetworkManager/system-connections/eth0.nmconnection
# -- /etc/hosts: FQDN must resolve to the real IP (not loopback) for IPA --
sed -i '/domain-controller/d' /etc/hosts
echo '${vars.domainControllerIp} domain-controller.${vars.homeDomain} domain-controller' >> /etc/hosts
# -- Generate IPA passwords and store securely --
DM_PASS=$(openssl rand -base64 24 | tr -dc 'A-Za-z0-9' | head -c 24)
ADMIN_PASS=$(openssl rand -base64 24 | tr -dc 'A-Za-z0-9' | head -c 24)
printf 'Directory Manager: %s\nIPA Admin: %s\n' "$DM_PASS" "$ADMIN_PASS" \
> /root/ipa-credentials.txt
chmod 600 /root/ipa-credentials.txt
# -- First-boot script: reads passwords back, runs ipa-server-install --
cat > /usr/local/sbin/freeipa-first-boot.sh << 'FIRSTBOOT'
#!/bin/bash
set -euo pipefail
exec >> /root/freeipa-install.log 2>&1
echo "=== FreeIPA first-boot install started at $(date) ==="
DM_PASS=$(grep '^Directory Manager:' /root/ipa-credentials.txt | awk '{print $NF}')
ADMIN_PASS=$(grep '^IPA Admin:' /root/ipa-credentials.txt | awk '{print $NF}')
ipa-server-install \
--realm=SWEET.HOME \
--domain=sweet.home \
--hostname=domain-controller.sweet.home \
--ds-password="$DM_PASS" \
--admin-password="$ADMIN_PASS" \
--setup-dns \
--forwarder=192.168.2.253 \
--no-dnssec-validation \
--no-ntp \
--unattended
echo "=== FreeIPA install complete at $(date) ==="
echo "Credentials: /root/ipa-credentials.txt (save to password manager)"
echo "CA backup: /root/cacert.p12 (encrypted with Directory Manager password)"
systemctl disable freeipa-first-boot.service
FIRSTBOOT
chmod 700 /usr/local/sbin/freeipa-first-boot.sh
# -- Systemd oneshot service: runs freeipa-first-boot.sh on first real boot --
cat > /etc/systemd/system/freeipa-first-boot.service << 'UNIT'
[Unit]
Description=FreeIPA first-boot installation
After=network-online.target
Wants=network-online.target
ConditionPathExists=/root/ipa-credentials.txt
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/freeipa-first-boot.sh
TimeoutStartSec=1800
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
UNIT
mkdir -p /etc/systemd/system/multi-user.target.wants
ln -sf /etc/systemd/system/freeipa-first-boot.service \
/etc/systemd/system/multi-user.target.wants/freeipa-first-boot.service
echo "Kickstart %post complete. FreeIPA installs on first reboot (~20 min)."
%end
'';
fetchRockyPxeboot = pkgs.writeShellScript "fetch-rocky-pxeboot" ''
set -eu
dir="${httpRoot}/rocky"
base="${rockyPxebootBase}"
if [ -f "$dir/vmlinuz" ] && [ -f "$dir/initrd.img" ]; then
echo "Rocky Linux ${rockyRelease} pxeboot files already present; skipping download."
exit 0
fi
echo "Downloading Rocky Linux ${rockyRelease} pxeboot kernel and initrd from $base ..."
${pkgs.curl}/bin/curl -fsSL -o "$dir/vmlinuz.tmp" "$base/vmlinuz"
${pkgs.curl}/bin/curl -fsSL -o "$dir/initrd.img.tmp" "$base/initrd.img"
mv "$dir/vmlinuz.tmp" "$dir/vmlinuz"
mv "$dir/initrd.img.tmp" "$dir/initrd.img"
echo "Rocky Linux ${rockyRelease} pxeboot files staged."
'';
systemRescueIpxe = pkgs.writeText "systemrescue.ipxe" ''
#!ipxe
@@ -68,11 +278,13 @@ let
set base ${pxeBaseUrl}
menu PXE Boot Menu
item auto-installer NixOS Auto-Installer
item nixos-minimal NixOS Minimal
item rescue Rescue Environment
item shell iPXE Shell
item reboot Reboot
item auto-installer NixOS Auto-Installer
item nixos-minimal NixOS Minimal
item debian Debian Minimal
item rocky-freeipa FreeIPA Server (Rocky Linux 9)
item rescue Rescue Environment
item shell iPXE Shell
item reboot Reboot
choose target && goto ''${target}
@@ -82,6 +294,12 @@ let
:nixos-minimal
chain ''${base}/nixos-minimal/netboot.ipxe
:debian
chain ''${base}/debian.ipxe
:rocky-freeipa
chain ''${base}/rocky-freeipa.ipxe
:rescue
chain ''${base}/systemrescue.ipxe
@@ -95,6 +313,8 @@ in
{
imports = [
../pxe-boot/stage-installer-artifacts.nix
../pxe-boot/mount-pxe-images.nix
../beszel/enable-agent.nix
];
environment.systemPackages = with pkgs; [
@@ -129,37 +349,101 @@ in
openssh.settings.PermitRootLogin = "yes";
};
systemd.tmpfiles.rules = [
"d ${pxeRoot} 0755 root root -"
"d ${httpRoot} 0755 root root -"
"d ${httpRoot}/images 0755 root root -"
"d ${httpRoot}/auto-installer 0755 root root -"
"d ${httpRoot}/nixos-minimal 0755 root root -"
"d ${httpRoot}/systemrescue 0755 root root -"
"d ${httpRoot}/ubuntu 0755 root root -"
"d ${httpRoot}/rescue 0755 root root -"
"d ${tftpRoot} 0755 root root -"
"C+ ${httpRoot}/boot.ipxe 0644 root root - ${bootIpxe}"
"C+ ${httpRoot}/menu.ipxe 0644 root root - ${menuIpxe}"
"C+ ${httpRoot}/systemrescue.ipxe 0644 root root - ${systemRescueIpxe}"
"C+ ${tftpRoot}/autoexec.ipxe 0644 root root - ${autoexecIpxe}"
"C+ ${tftpRoot}/ipxe.efi 0644 root root - ${pkgs.ipxe}/ipxe.efi"
"C+ ${tftpRoot}/undionly.kpxe 0644 root root - ${pkgs.ipxe}/undionly.kpxe"
];
systemd.services.stage-systemrescue = {
description = "Stage SystemRescue ISO contents for HTTP PXE boot";
after = [
"local-fs.target"
"systemd-tmpfiles-setup.service"
systemd = {
tmpfiles.rules = [
"d ${pxeRoot} 0755 root root -"
"d ${httpRoot} 0755 root root -"
"L+ ${httpRoot}/images - - - - ${vars.nfsShares.pxebootImages.mountpoint}"
"d ${httpRoot}/auto-installer 0755 root root -"
"d ${httpRoot}/nixos-minimal 0755 root root -"
"d ${httpRoot}/systemrescue 0755 root root -"
"d ${httpRoot}/debian 0755 root root -"
"d ${httpRoot}/ubuntu 0755 root root -"
"d ${httpRoot}/rescue 0755 root root -"
"d ${httpRoot}/rocky 0755 root root -"
"d ${tftpRoot} 0755 root root -"
"C+ ${httpRoot}/boot.ipxe 0644 root root - ${bootIpxe}"
"C+ ${httpRoot}/menu.ipxe 0644 root root - ${menuIpxe}"
"C+ ${httpRoot}/debian.ipxe 0644 root root - ${debianIpxe}"
"C+ ${httpRoot}/rocky-freeipa.ipxe 0644 root root - ${rockyFreeIpaIpxe}"
"C+ ${httpRoot}/rocky-freeipa.ks 0644 root root - ${rockyFreeIpaKs}"
"C+ ${httpRoot}/systemrescue.ipxe 0644 root root - ${systemRescueIpxe}"
"C+ ${tftpRoot}/autoexec.ipxe 0644 root root - ${autoexecIpxe}"
"C+ ${tftpRoot}/ipxe.efi 0644 root root - ${pkgs.ipxe}/ipxe.efi"
"C+ ${tftpRoot}/undionly.kpxe 0644 root root - ${pkgs.ipxe}/undionly.kpxe"
];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
ExecStart = stageSystemRescue;
services = {
fetch-debian-netboot = {
description = "Download Debian ${debianRelease} netboot kernel and initrd for HTTP PXE boot";
after = [
"local-fs.target"
"systemd-tmpfiles-setup.service"
"network-online.target"
];
wants = [ "network-online.target" ];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
ExecStart = fetchDebianNetboot;
RemainAfterExit = true;
};
};
fetch-rocky-pxeboot = {
description = "Download Rocky Linux ${rockyRelease} pxeboot kernel and initrd for HTTP PXE boot";
after = [
"local-fs.target"
"systemd-tmpfiles-setup.service"
"network-online.target"
];
wants = [ "network-online.target" ];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
ExecStart = fetchRockyPxeboot;
RemainAfterExit = true;
};
};
stage-systemrescue = {
description = "Stage SystemRescue ISO contents for HTTP PXE boot";
after = [
"local-fs.target"
"systemd-tmpfiles-setup.service"
];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
ExecStart = stageSystemRescue;
};
};
};
};
services.dnsmasq = {
enable = true;
settings = {
# Disable DNS listener — only proxy DHCP is needed here.
# Without this dnsmasq tries to bind port 53 which systemd-resolved
# already owns, causing startup failure.
port = 0;
dhcp-range = [ "192.168.2.0,proxy" ];
dhcp-match = [
"set:ipxe,175"
"set:efi64,option:client-arch,7"
"set:efi64,option:client-arch,9"
];
dhcp-userclass = "set:ipxe,iPXE";
dhcp-boot = [
"tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
"tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
"tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
"undionly.kpxe,,${vars.pxeServerIp}"
];
};
};
networking.firewall.allowedTCPPorts = [ vars.ports.pxeBootHttp ];
networking.firewall.allowedUDPPorts = [ vars.ports.pxeBootTftp ];
networking.firewall.allowedUDPPorts = [ vars.ports.pxeBootTftp 67 ];
}
+49 -18
View File
@@ -10,8 +10,18 @@ let
in lib.imap1 (i: _: lib.concatStringsSep "/" (lib.take i parts)) parts;
poolDatasets = lib.unique (
lib.concatMap (share: ancestors share.subpath) (lib.attrValues vars.nfsShares)
lib.concatMap (share: ancestors share.subpath)
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares))
);
# Generates /etc/exports lines for all nfsShares data entries (every
# attrset value — excludes the bare `options` string). Both server and
# ha-server export the same share set from different storage roots, so
# this helper is the single source of truth for the export line format.
mkNfsExports = storageRoot:
lib.concatMapStrings
(share: " ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n")
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares));
in
{
imports = [
@@ -43,14 +53,8 @@ in
exit 0
fi
# Pool exists on a device but not yet imported let the standard
# zfs-import-${poolName}.service handle it normally.
if zpool import -d /dev/disk/by-id -N "${poolName}" 2>/dev/null; then
exit 0
fi
# No pool found at all. Create it on the Proxmox data disk (scsi1),
# which appears as /dev/disk/by-id/scsi-*drive-scsi1 inside the VM.
# Locate the data disk first used for both the fallback import
# attempt and, only if the disk is genuinely blank, pool creation.
DATA_DISK=""
for candidate in /dev/disk/by-id/scsi-*drive-scsi1; do
[[ "$candidate" == *-part* ]] && continue
@@ -62,8 +66,31 @@ in
exit 1
fi
# Try importing via the by-id symlink directory first (normal path),
# then fall back to scanning the disk directly. The two-step exists
# because of a udev race: systemd-udev-settle.service can clear before
# /dev/disk/by-id/ entries are fully populated, causing the first
# import to fail even when the pool is intact on the disk.
if zpool import -d /dev/disk/by-id -N "${poolName}" 2>/dev/null; then
exit 0
fi
if zpool import -d "$DATA_DISK" -N "${poolName}" 2>/dev/null; then
exit 0
fi
# Both import attempts failed. Before creating a new pool, verify the
# disk is genuinely blank if ZFS label metadata is present the import
# failed for some other reason and we must not clobber existing data.
if zdb -l "$DATA_DISK" 2>/dev/null | grep -q "name: '${poolName}'"; then
echo "zfs-init-${poolName}: $DATA_DISK has ZFS pool '${poolName}' metadata but import failed refusing to overwrite existing data. Run 'zpool import -d $DATA_DISK ${poolName}' manually to investigate." >&2
exit 1
fi
# Disk is genuinely blank: create the pool. -f is intentionally
# omitted so that if we somehow reach this point with an existing pool
# on the disk, zpool refuses rather than silently destroying data.
echo "zfs-init-${poolName}: creating pool on $DATA_DISK"
zpool create -f "${poolName}" "$DATA_DISK"
zpool create "${poolName}" "$DATA_DISK"
${lib.concatMapStrings (ds: ''
zfs create "${poolName}/${ds}"
'') poolDatasets}
@@ -75,16 +102,20 @@ in
requires = [ "zfs-mount.service" ];
};
# rpc-svcgssd handles Kerberos/GSS-API for NFS. Not needed: exports use
# standard auth, not sec=krb5. On IPA-joined hosts the keytab exists (host/
# principal only) but has no nfs/ principal, causing spurious failure.
# Mask it so nfs-server's Wants= can't pull it in.
systemd.services.rpc-svcgssd.enable = false;
services.nfs.server = {
enable = true;
exports = ''
${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath} ${vars.lanCidr}(rw,sync,no_subtree_check,no_root_squash)
${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath} ${vars.lanCidr}(rw,sync,no_subtree_check,no_root_squash)
${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath} ${vars.lanCidr}(rw,sync,no_subtree_check,no_root_squash)
${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath} ${vars.lanCidr}(rw,sync,no_subtree_check,no_root_squash)
${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath} ${vars.lanCidr}(rw,sync,no_subtree_check,no_root_squash)
'';
exports = mkNfsExports vars.storageRoot;
};
networking.firewall.allowedTCPPorts = [ vars.ports.nfsRpcbind vars.ports.nfsd ];
# mountd (20048) is needed for showmount/NFSv3 mount protocol — without it
# clients can reach portmapper (111) and get the mountd port back, then
# time out trying to connect to it. All three ports need TCP and UDP.
networking.firewall.allowedTCPPorts = [ vars.ports.nfsRpcbind vars.ports.nfsd vars.ports.nfsMountd ];
networking.firewall.allowedUDPPorts = [ vars.ports.nfsRpcbind vars.ports.nfsd vars.ports.nfsMountd ];
}
@@ -1,21 +0,0 @@
{ ... }:
{
imports = [
../tailscale/exit-node.nix
];
# "server", not "both": this build type only ever advertises itself as an
# exit node (see ../tailscale/exit-node.nix) -- it doesn't advertise LAN
# subnet routes, so it doesn't need the "client"-side loose reverse-path
# filtering that "both" would also turn on. Deliberately left unbundled
# from LAN-subnet-route advertisement so this build type stays valid on
# every platform, including linode (a remote VPS with no network path to
# the home LAN at all).
services.tailscale.useRoutingFeatures = "server";
# Forwarded exit-node traffic arrives on tailscale0 already
# tailscale-authenticated -- the firewall's normal per-port allow-list
# would otherwise drop it. Standard NixOS/Tailscale exit-node guidance.
networking.firewall.trustedInterfaces = [ "tailscale0" ];
}
+44
View File
@@ -0,0 +1,44 @@
{ vars, ... }:
{
imports = [
../tailscale/subnet-router.nix
../tailscale/ts-dns-forwarder.nix
../beszel/enable-agent.nix
];
# "server", not "both": this build type advertises LAN subnet routes but
# doesn't use another tailscale exit node itself, so it doesn't need the
# "client"-side loose reverse-path filtering that "both" would also enable.
# Deliberately kept explicit here (not just relying on subnet-router.nix's
# own setting) so the intent is clear at the build-type level.
services.tailscale.useRoutingFeatures = "server";
# Advertise the LAN subnet so Tailscale peers can route back to LAN machines.
# Must also be approved in the Tailscale admin console (Machines → Edit route settings).
services.tailscale.extraUpFlags = [ "--advertise-routes=${vars.lanCidr}" ];
networking.firewall = {
# Forwarded subnet-router traffic arrives on tailscale0 already
# tailscale-authenticated -- the firewall's normal per-port allow-list
# would otherwise drop it. Standard NixOS/Tailscale subnet-router guidance.
trustedInterfaces = [ "tailscale0" ];
# SNAT LAN traffic going into Tailscale so the remote peer sees it as
# coming from this router's Tailscale IP rather than a raw LAN IP.
# Without this, Tailscale drops forwarded packets whose source is not a
# recognised Tailscale address.
#
# We target POSTROUTING directly (always-existing built-in chain) rather
# than nixos-nat-post: extraCommands runs after the old nixos-nat-post is
# deleted but before the new one is created, so -A nixos-nat-post silently
# fails. The -C check makes the rule idempotent across firewall reloads.
extraCommands = ''
iptables -t nat -C POSTROUTING -s ${vars.lanCidr} -o tailscale0 -j MASQUERADE 2>/dev/null || \
iptables -t nat -A POSTROUTING -s ${vars.lanCidr} -o tailscale0 -j MASQUERADE
'';
extraStopCommands = ''
iptables -t nat -D POSTROUTING -s ${vars.lanCidr} -o tailscale0 -j MASQUERADE 2>/dev/null || true
'';
};
}
+1
View File
@@ -3,5 +3,6 @@
{
imports = [
../tor/enable-relay.nix
../beszel/enable-agent.nix
];
}
+30
View File
@@ -0,0 +1,30 @@
{ pkgs, ... }: {
# Defines the SSH host key as a clan vars generator so that:
# - `clan vars generate <target>` creates and encrypts the key pair
# - The private key lives at vars/per-machine/<target>/openssh/ssh_host_ed25519_key/secret
# (sops binary-encrypted, admin-key-only; decrypted by the build script)
# - The public key lives at vars/per-machine/<target>/openssh/ssh_host_ed25519_key.pub/value
# (plaintext; used by sync-host-keys.sh to derive the sops age fingerprint)
#
# neededFor = "activation" means clan's deployment tool would upload this
# before running nixos-rebuild/nixos-install (for VM/baremetal via
# nixos-anywhere). For lxc-* hosts, the build script bakes it into the
# tarball directly via NIXOS_HOST_KEYS_DIR -- the neededFor value here
# simply ensures it is NOT mapped to sops.secrets (which would try to
# decrypt it at runtime as a regular service secret, which is wrong: the
# SSH host key reaches the container via the tarball, not sops).
clan.core.vars.generators.openssh = {
files."ssh_host_ed25519_key" = {
secret = true;
neededFor = "activation";
};
files."ssh_host_ed25519_key.pub" = {
secret = false;
neededFor = "activation";
};
runtimeInputs = [ pkgs.openssh ];
script = ''
ssh-keygen -t ed25519 -N "" -C "" -f "$out/ssh_host_ed25519_key"
'';
};
}
+4 -47
View File
@@ -1,50 +1,7 @@
{ config, pkgs, lib, vars, ... }:
_:
let
# Flake attribute names are now <platform>-<buildtype> (e.g. proxmox-docker)
# and no longer match networking.hostName, since a host's hostname stays
# fixed while the platform backing it can change. Each nixosConfiguration
# stamps its own active target name into /etc/flake-target at build time.
mySwitchCmd = ''
sudo nixos-rebuild switch \
--no-write-lock-file \
--refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
'';
myTestCmd = ''
sudo nixos-rebuild test \
--no-write-lock-file \
--refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
'';
# lxc-* hosts pre-seed their SSH host key at build time (see
# modules/platforms/lxc.nix) so sops-nix's .sops.yaml recipient matches on
# first boot -- without it, secrets permanently fail to decrypt (see that
# file's comment for the confirmed failure). That requires --impure plus
# NIXOS_HOST_KEYS_DIR pointing at the repo's host-keys/ dir, same pattern
# docs/auto-installer.md uses for the installer ISO. A function, not a
# shellAlias, since the target name has to interpolate into the middle of
# the flake attribute path, not just append after it. Must be run from the
# repo root, same as every other host-keys/ command in this repo.
buildImageFn = ''
buildImage() {
if [ -z "$1" ]; then
echo "usage: buildImage <flake-target> (e.g. lxc-docker)" >&2
return 1
fi
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
".#nixosConfigurations.$1.config.system.build.tarball"
}
'';
in
{
programs.bash = {
enable = true;
shellAliases = {
"Switch-nix" = mySwitchCmd;
"Test-nix" = myTestCmd;
};
initExtra = buildImageFn;
};
# Switch-nix, Test-nix, and buildImage are defined system-wide in
# modules/common/configuration.nix so all users (including IPA accounts)
# get them. Add any Home-Manager-only per-user shell config here.
}
+63 -27
View File
@@ -1,17 +1,45 @@
{ config, lib, pkgs, vars, ... }:
let
switchCmd = ''
sudo nixos-rebuild switch \
--no-write-lock-file \
--refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
'';
testCmd = ''
sudo nixos-rebuild test \
--no-write-lock-file \
--refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
'';
buildImageFn = ''
buildImage() {
if [ -z "$1" ]; then
echo "usage: buildImage <flake-target> (e.g. lxc-docker)" >&2
return 1
fi
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
".#nixosConfigurations.$1.config.system.build.tarball"
}
'';
in
{
imports =
[
# Include the results of the hardware scan.
# ./hardware-configuration.nix
./set-locale.nix
];
# Use the GRUB 2 boot loader.
# boot.loader.grub.enable = true;
#boot.loader.grub.device = "/dev/sda"; # or "nodev" for efi only
imports = [
./set-locale.nix
../ipa/client.nix
];
networking.networkmanager.enable = true; # Easiest to use and most distros use this by default.
# System-wide shell config so all users (including IPA accounts) get the
# same management aliases as the local nixos user's Home Manager provides.
programs.bash = {
shellAliases = {
"Switch-nix" = switchCmd;
"Test-nix" = testCmd;
};
interactiveShellInit = buildImageFn;
};
networking.networkmanager.enable = true;
# Recommended over the true default (bypasses ZFS's own import safeguards)
# per the option's own docs; matches hosts/docker/host.nix and
@@ -61,24 +89,32 @@
!include ${config.sops.templates."nix-github-token.conf".path}
'';
#Set root password
users.users.root = {
hashedPasswordFile = config.sops.secrets."root-hashedPassword".path;
};
users = {
# With mutableUsers = false, update-users-groups.pl enforces hashedPasswordFile
# on every activation regardless of whether the account already exists in
# /etc/shadow. The default (true) only applies hashedPasswordFile to newly-
# created accounts — which means a freshly-built proxmox disk image (where
# activation runs without a usable sops key, so both accounts land in shadow
# with !) will never have its passwords fixed by subsequent boots.
mutableUsers = false;
# Define a user account. Don't forget to set a password with passwd.
users.users.${vars.primaryUser} = {
isNormalUser = true;
extraGroups = [ "wheel" ]; # Enable sudo for the user.
packages = with pkgs; [
tree
];
hashedPasswordFile = config.sops.secrets."nixos-hashedPassword".path;
openssh.authorizedKeys.keys = [
vars.adminSshKey
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGygkCljN6uKpdJbHTOQtn8ZnH+wKXDLAwrDFbLrE/65 nixos@nixos"
];
users.root = {
hashedPasswordFile = config.sops.secrets."root-hashedPassword".path;
};
users.${vars.primaryUser} = {
isNormalUser = true;
extraGroups = [ "wheel" ]; # Enable sudo for the user.
packages = with pkgs; [
tree
];
hashedPasswordFile = config.sops.secrets."nixos-hashedPassword".path;
openssh.authorizedKeys.keys = [
vars.adminSshKey
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGygkCljN6uKpdJbHTOQtn8ZnH+wKXDLAwrDFbLrE/65 nixos@nixos"
];
};
};
+35
View File
@@ -0,0 +1,35 @@
# Shared activation-script logic to preserve the SSH host key across
# nixos-rebuild on platforms that embed the key via environment.etc (lxc and
# proxmox). When NIXOS_HOST_KEYS_DIR is not set the key is absent from
# environment.etc, and NixOS's etc activation removes any /etc file not in
# the new generation — which would destroy the live key and break sops-nix
# decryption permanently. These scripts save the key to /run before etc
# removes it, then restore it afterward.
#
# Explicit deps enforce the correct ordering: without them the topological
# sort places preserveSshHostKey after etc (confirmed live on lxc-tor-relay:
# position 7 vs etc's position 5), so the key is gone before it can be saved.
_: {
system.activationScripts = {
preserveSshHostKey = ''
if [ -f /etc/ssh/ssh_host_ed25519_key ]; then
cp /etc/ssh/ssh_host_ed25519_key /run/sshd-host-key-preserve.tmp
cp /etc/ssh/ssh_host_ed25519_key.pub /run/sshd-host-key-preserve.pub.tmp
fi
'';
restoreSshHostKey = {
deps = [ "etc" ];
text = ''
if [ ! -f /etc/ssh/ssh_host_ed25519_key ] && [ -f /run/sshd-host-key-preserve.tmp ]; then
install -m 0600 /run/sshd-host-key-preserve.tmp /etc/ssh/ssh_host_ed25519_key
install -m 0644 /run/sshd-host-key-preserve.pub.tmp /etc/ssh/ssh_host_ed25519_key.pub
fi
rm -f /run/sshd-host-key-preserve.tmp /run/sshd-host-key-preserve.pub.tmp
'';
};
etc = { deps = [ "preserveSshHostKey" ]; };
setupSecrets = { deps = [ "restoreSshHostKey" ]; };
};
}
+34 -12
View File
@@ -1,23 +1,45 @@
{ pkgs, ... }:
{ lib, pkgs, vars, ... }:
let
gid = toString vars.dockerAccessGid;
in
{
# virtualisation.docker.enable = true;
virtualisation.docker = {
enable = true;
package = pkgs.docker;
# listenOptions = [
# "unix:///var/run/docker.sock"
# "tcp://0.0.0.0:2375"
#];
# daemon.settings = {
# metrics-addr = "0.0.0.0:9323";
# experimental = true;
# };
};
# Pin the docker group GID to match the IPA "docker-access" group so that
# IPA group membership alone grants access to the Docker socket. Any user
# whose supplementary groups (resolved by SSSD from IPA) include GID
# vars.dockerAccessGid will pass the socket group-permission check without
# any per-host users.groups.docker.members entry.
users.groups.docker.gid = lib.mkForce vars.dockerAccessGid;
users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
environment.systemPackages = with pkgs; [
docker-compose
docker-buildx
];
# NixOS's group activation uses plain `groupmod` without --non-unique.
# When SSSD is active it exposes the IPA "docker-access" group at
# vars.dockerAccessGid via NSS, so groupmod sees that GID as already in
# use and silently skips the change (warning: "not applying GID change").
# This script runs after the normal "groups" step and applies the change
# with --non-unique (which lets the local docker group share the GID with
# the SSSD-provided IPA group). If the GID actually changed it also
# restarts docker.socket so the socket is recreated with the new GID.
system.activationScripts.docker-group-gid = {
deps = [ "groups" ];
text = ''
current=$(grep "^docker:" /etc/group | cut -d: -f3)
if [ "$current" != "${gid}" ]; then
${pkgs.shadow}/bin/groupmod --non-unique -g ${gid} docker
if ${pkgs.systemd}/bin/systemctl is-active --quiet docker.socket; then
${pkgs.systemd}/bin/systemctl stop docker.service docker.socket
rm -f /var/run/docker.sock
${pkgs.systemd}/bin/systemctl start docker.socket docker.service
fi
fi
'';
};
}
+154
View File
@@ -0,0 +1,154 @@
# Cluster-wide HA config shared by both ha-server nodes.
#
# Covers everything that is identical on both nodes and references cluster
# topology (node IPs, hostnames, DRBD resource). Per-node identity
# (hostname, static IP, stateVersion) lives in hosts/ha-server-{1,2}/host.nix.
#
# Corosync authkey:
# /etc/corosync/authkey (mode 0400) is managed by sops-nix below.
# Bootstrap: run scripts/ha/cluster-init.sh on node1 to generate the key,
# then encrypt it with: sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey
# Both host keys must be registered via sync-host-keys.sh first so both nodes can decrypt it.
#
# DRBD fencing:
# resource-only with crm-fence-peer.sh: DRBD calls the Pacemaker-aware
# crm-fence-peer.sh handler before promoting. The handler checks the CIB
# to confirm the peer's DRBD resource is stopped and returns 7 (successfully
# fenced), allowing safe promotion without requiring power-fencing (STONITH).
# The unfence handler crm-unfence-peer.sh clears the outdate flag when the
# peer reconnects. This is the correct setting for Pacemaker+DRBD clusters
# with STONITH disabled; crm-fence-peer.sh replaces the need for a separate
# STONITH device during the testing phase. Switch to resource-and-stonith
# once the fence_pve_ssh STONITH resource is active (see
# scripts/ha/cluster-enable-stonith.sh).
#
# PATH wrapper: when the DRBD kernel module invokes the fence-peer handler
# via the UMH (User Mode Helper) mechanism it provides a minimal PATH that
# omits /run/current-system/sw/bin. crm-fence-peer.sh calls cibadmin,
# crm_mon etc.; if those aren't found a pipeline in the script breaks with
# SIGPIPE. A process killed by signal has WEXITSTATUS() == 0, so the kernel
# sees exit code 0 and logs "fence-peer helper broken, returned 0", looping
# forever. The writeShellScript wrappers below prepend the NixOS sw path
# before exec-ing the real handler, giving it a working Pacemaker toolchain.
{ lib, pkgs, vars, ... }:
let
fencePeerWrapper = pkgs.writeShellScript "drbd-fence-peer" ''
export PATH="/run/current-system/sw/bin:/run/current-system/sw/sbin:$PATH"
exec /run/current-system/sw/lib/drbd/crm-fence-peer.sh "$@"
'';
unfencePeerWrapper = pkgs.writeShellScript "drbd-unfence-peer" ''
export PATH="/run/current-system/sw/bin:/run/current-system/sw/sbin:$PATH"
exec /run/current-system/sw/lib/drbd/crm-unfence-peer.sh "$@"
'';
in
{
# Root SSH access — same key set as nixos user so all admin keys can reach root.
users.users.root.openssh.authorizedKeys.keys = [
vars.adminSshKey
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGygkCljN6uKpdJbHTOQtn8ZnH+wKXDLAwrDFbLrE/65 nixos@nixos"
];
# Passwordless sudo for wheel — operator SSHes as nixos and uses sudo for
# cluster management commands (drbdadm, crm*, pcs, etc.)
security.sudo.wheelNeedsPassword = lib.mkForce false;
# DRBD lock-file directory (drbd-utils checks for it; missing → harmless but noisy warnings).
systemd.tmpfiles.rules = [ "d /var/lib/drbd 0750 root root -" ];
# Prevent drbd.service from auto-starting at boot / nixos-rebuild switch.
# Pacemaker's OCF drbd agent calls drbdadm up/down directly when managing
# the resource. If drbd.service also runs drbdadm up all while DRBD is
# already Primary under Pacemaker, apply-al fails with "device busy" (exit 20).
systemd.services.drbd.wantedBy = lib.mkForce [];
services.drbd = {
enable = true;
config = ''
global {
usage-count yes;
}
common {
net {
protocol C;
ping-int 1;
verify-alg sha256;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
}
disk {
fencing resource-only;
}
handlers {
fence-peer "${fencePeerWrapper}";
unfence-peer "${unfencePeerWrapper}";
}
}
resource ha-data {
volume 0 {
device /dev/drbd0;
disk ${vars.haServerDrbdDisk};
meta-disk internal;
}
on ${vars.haServer1Host} {
address ${vars.haServer1StorageIp}:${toString vars.ports.haServerDrbd};
}
on ${vars.haServer2Host} {
address ${vars.haServer2StorageIp}:${toString vars.ports.haServerDrbd};
}
}
'';
};
# /etc/corosync/authkey — sops binary secret, identical on both nodes.
# Decryptable by both ha-server host keys (added by sync-host-keys.sh).
sops.secrets.corosync_authkey = {
sopsFile = ../../secrets/ha-corosync-authkey;
format = "binary";
path = "/etc/corosync/authkey";
mode = "0400";
restartUnits = [ "corosync.service" ];
};
# NixOS common config enables NetworkManager by default; HA cluster nodes
# need stable static IPs with predictable interface names — NM is not suitable.
networking.networkmanager.enable = lib.mkForce false;
# services.corosync.enable is set by modules/ha/pacemaker-stack.nix.
services.corosync = {
clusterName = "ha-cluster";
nodelist = [
{ nodeid = 1; name = vars.haServer1Host; ring_addrs = [ vars.haServer1StorageIp ]; }
{ nodeid = 2; name = vars.haServer2Host; ring_addrs = [ vars.haServer2StorageIp ]; }
];
};
networking.firewall = {
allowedTCPPorts = [
vars.ports.haServerIscsi
vars.ports.haServerPacemakerRemoted
vars.ports.haServerPcsd
vars.ports.haServerDrbd
vars.ports.nfsRpcbind
vars.ports.nfsd
vars.ports.nfsMountd
];
allowedUDPPorts = [
vars.ports.haServerCorosync1
vars.ports.haServerCorosync2
vars.ports.haServerCorosyncCrypto
vars.ports.nfsRpcbind
vars.ports.nfsd
vars.ports.nfsMountd
];
extraCommands = ''
iptables -A INPUT -s ${vars.haServer1Ip}/32 -j ACCEPT
iptables -A INPUT -s ${vars.haServer2Ip}/32 -j ACCEPT
iptables -A INPUT -s ${vars.haStorageCidr} -j ACCEPT
'';
};
}
+99
View File
@@ -0,0 +1,99 @@
# LIO iSCSI target service (targetctl) for NixOS HA clusters.
#
# Provides the targetctl.service that saves/restores LIO configuration from
# /etc/target/saveconfig.json. Pacemaker manages this service via its
# systemd resource agent (class="systemd" type="targetctl").
#
# Why ExecStop is not simply "targetctl save":
# targetctl save writes the LIO config to JSON but does NOT remove the LIO
# target from the kernel's configfs. As a result, any fileio backing store
# that LIO has open (e.g. iscsi-lun.img on an XFS-over-DRBD filesystem)
# stays referenced in the kernel. The subsequent XFS umount from the
# Filesystem OCF resource then returns EBUSY and either hangs for the full
# op-stop timeout or fails outright, blocking the entire failover.
#
# The ExecStop script here additionally tears down the kernel LIO state
# via rtslib_fb after saving, so the backing-store file descriptor is
# released and umount succeeds immediately.
#
# Empty-config guard:
# The save step is skipped when no iSCSI targets are currently active.
# This prevents the secondary node (where LIO was never started) from
# overwriting a valid saveconfig.json with an empty one when Pacemaker
# stops the iscsi-target resource as part of a failover or cleanup.
{ pkgs, ... }:
let
python3 = pkgs.python3.withPackages (ps: [ ps.rtslib-fb ]);
targetctl = "${python3}/bin/targetctl";
targetctlStop = pkgs.writeScript "targetctl-stop" ''
#!${python3}/bin/python3
import subprocess, sys
import rtslib_fb
root = rtslib_fb.RTSRoot()
targets = list(root.targets)
if targets:
subprocess.run(
["${targetctl}", "save", "/etc/target/saveconfig.json"],
capture_output=True,
)
print(f"saved {len(targets)} iSCSI target(s)")
else:
print("no active LIO targets saveconfig.json unchanged")
for target in targets:
try:
for tpg in list(target.tpgs):
tpg.enable = False
target.delete()
except Exception as e:
print(f"warn (target): {e}", file=sys.stderr)
for so in list(root.storage_objects):
try:
so.delete()
except Exception as e:
print(f"warn (backstore): {e}", file=sys.stderr)
print("LIO kernel target cleared")
'';
in
{
boot.kernelModules = [
"target_core_mod"
"iscsi_target_mod"
"target_core_file"
"target_core_pscsi"
"target_core_user"
"configfs"
];
systemd = {
mounts = [{
where = "/sys/kernel/config";
what = "configfs";
type = "configfs";
wantedBy = [ "multi-user.target" ];
before = [ "targetctl.service" ];
}];
services.targetctl = {
description = "LIO iSCSI target config save/restore";
wantedBy = [ "multi-user.target" ];
after = [ "sys-kernel-config.mount" "network.target" ];
requires = [ "sys-kernel-config.mount" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
ExecStart = "${targetctl} restore /etc/target/saveconfig.json";
ExecStop = "${targetctlStop}";
};
unitConfig.ConditionFileNotEmpty = "/etc/target/saveconfig.json";
};
tmpfiles.rules = [
"d /etc/target 0750 root root -"
"f /etc/target/saveconfig.json 0640 root root -"
];
};
environment.systemPackages = [ pkgs.targetcli-fb ];
}
+94
View File
@@ -0,0 +1,94 @@
# Pacemaker + Corosync HA stack for NixOS with known-good workarounds.
#
# Issues fixed here (confirmed through live testing on NixOS 25.11):
#
# 1. StateDirectory ownership reset: systemd's StateDirectory=pacemaker
# creates /var/lib/pacemaker owned root:root. pacemaker-based (the CIB
# daemon) runs as the hacluster user and calls pcmk__daemon_can_write,
# which requires the CIB directory to be owned by hacluster or be
# group-writable by haclient. Workaround: remove StateDirectory and let
# ExecStartPre create every required subdirectory with correct ownership.
#
# 2. HA_SBIN_DIR wrong path: ocf-shellfuncs sets HA_SBIN_DIR to the Nix
# store path of the resource-agents derivation's /sbin, which doesn't
# exist. The DRBD OCF agent uses ${HA_SBIN_DIR}/crm_master, so it exits
# 127 without this override. Fix: export HA_SBIN_DIR=/run/current-system/sw/bin.
#
# 3. Broad PATH for OCF agents: the resource executor (pacemaker-execd) runs
# OCF agent scripts as children. NixOS provides no implicit PATH for
# system services; without an explicit PATH the agents can't find ip, ss,
# mount, umount, drbdadm, etc.
#
# 4. FUSER=true: the Filesystem OCF agent calls check_binary $FUSER (default:
# fuser from psmisc), which is not installed. Setting FUSER=true makes
# check_binary succeed (true is always in PATH) and the subsequent
# "$FUSER -km $mountpoint" becomes a no-op. Pair with force_unmount=false
# on each Filesystem resource unless you want lazy unmount behaviour.
{ lib, pkgs, ... }:
let
ocfBinPath = lib.concatStringsSep ":" [
"${pkgs.iproute2}/bin"
"${pkgs.iproute2}/sbin"
"${pkgs.iputils}/bin"
"${pkgs.util-linux}/bin"
"${pkgs.util-linux}/sbin"
"${pkgs.gawk}/bin"
"${pkgs.gnugrep}/bin"
"${pkgs.gnused}/bin"
"${pkgs.coreutils}/bin"
"${pkgs.bash}/bin"
"${pkgs.procps}/bin"
"${pkgs.xfsprogs}/bin"
"${pkgs.drbd}/bin"
"${pkgs.python3}/bin"
"/run/current-system/sw/bin"
"/run/current-system/sw/sbin"
"/usr/local/sbin"
"/usr/local/bin"
"/usr/sbin"
"/usr/bin"
"/sbin"
"/bin"
];
# Single pre-start script: schemas symlink + directory ownership.
# Runs before pacemakerd so pacemaker-based finds hacluster-owned dirs.
preStartCmd = "${pkgs.bash}/bin/bash -c '"
+ "ln -sfn ${pkgs.pacemaker}/share/pacemaker /var/lib/pacemaker/schemas; "
+ "for d in /var/lib/pacemaker /var/lib/pacemaker/cib /var/lib/pacemaker/cores "
+ "/var/lib/pacemaker/pengine /var/lib/pacemaker/blackbox "
+ "/var/lib/pacemaker/hostcache; do "
+ "mkdir -p \"\\$d\" && chown hacluster:pacemaker \"\\$d\" && chmod 2770 \"\\$d\"; "
+ "done'";
ocfEnv = {
PATH = lib.mkForce ocfBinPath;
OCF_ROOT = "${pkgs.ocf-resource-agents}/usr/lib/ocf";
HA_SBIN_DIR = "/run/current-system/sw/bin";
FUSER = "true";
};
in
{
users.groups.haclient = { };
services.corosync.enable = true;
services.pacemaker.enable = true;
systemd.services = {
pacemaker = {
serviceConfig = {
StateDirectory = lib.mkForce "";
ExecStartPre = lib.mkBefore [ preStartCmd ];
};
environment = ocfEnv;
};
pacemaker-execd.environment = ocfEnv;
};
environment.systemPackages = with pkgs; [
corosync
pacemaker
ocf-resource-agents
];
}
+210
View File
@@ -0,0 +1,210 @@
# Fully declarative FreeIPA domain membership.
#
# Imported by modules/common/configuration.nix — no per-host wiring needed.
# Enables itself automatically on any host that has a sops-encrypted keytab
# at secrets/<hostname>.keytab; is a no-op for all other hosts.
#
# To enroll a new host:
# 0. scripts/secrets/sync-host-keys.sh <flake-target>
# 1. scripts/ipa/create-nixos-ipa-host-account.sh [--ip <addr>] <hostname>
# (adds .sops.yaml rule, runs ipa host-add, encrypts keytab in one step)
# 2. git add secrets/<hostname>.keytab .sops.yaml && git commit
# 3. Deploy — no further steps required.
#
# Manual fallback (if the script isn't usable):
# a. On the FreeIPA server: ipa host-add <fqdn> [--ip-address=<ip>] --force
# b. On the FreeIPA server: ipa-getkeytab -s <ipa-server> -p host/<fqdn> -k /tmp/<host>.keytab
# c. From the repo root (path must match for sops creation rule to apply):
# cp /tmp/<host>.keytab secrets/<host>.keytab
# sops -e --input-type binary -i secrets/<host>.keytab
# d. Commit secrets/<host>.keytab and the updated .sops.yaml, then deploy.
#
# vars dependencies: homeDomain, ipaServer, domainControllerIp, ipaUser
{ config, lib, pkgs, vars, ... }:
let
keytabPath = ../../secrets + "/${config.networking.hostName}.keytab";
enabled = builtins.pathExists keytabPath;
realm = lib.strings.toUpper vars.homeDomain;
fqdn = "${config.networking.hostName}.${vars.homeDomain}";
# "sweet.home" -> "dc=sweet,dc=home"
basedn = lib.strings.concatMapStringsSep "," (c: "dc=${c}") (lib.strings.splitString "." vars.homeDomain);
# security.ipa.certificate expects a derivation (package), not a raw path.
caCertPkg = pkgs.writeText "ipa-ca.crt" (builtins.readFile ../../certs/ipa-ca.crt);
in
lib.mkIf enabled {
networking.domain = lib.mkDefault vars.homeDomain;
networking.nameservers = lib.mkDefault [ vars.domainControllerIp ];
security = {
ipa = {
enable = true;
domain = vars.homeDomain;
inherit realm;
server = vars.ipaServer;
certificate = caCertPkg;
inherit basedn;
ipaHostname = fqdn;
offlinePasswords = true;
cacheCredentials = true;
};
# Create the home directory on first login if it doesn't exist yet.
# IPA users have no pre-created home on the host; without this sshd
# opens a session to a non-existent directory and resets the connection.
# lightdm also needs this so the GUI login path can create the home dir
# if it was not pre-seeded by the tmpfiles rule above (e.g. on first boot
# before SSSD has resolved the user).
pam.services = {
sshd.makeHomeDir = true;
lightdm.makeHomeDir = true;
# pam_unix returns PAM_AUTHINFO_UNAVAIL without prompting when the local
# stub has "!" in shadow (account locked), so PAM_AUTHTOK is never set
# and pam_sss's use_first_pass fails with "No authentication token".
# Changing to try_first_pass makes pam_sss prompt independently when no
# prior module has set the token, restoring IPA password login via
# LightDM and su.
login.rules.auth.sss.settings = lib.mkForce { try_first_pass = true; };
su.rules.auth.sss.settings = lib.mkForce { try_first_pass = true; };
};
# HM with useUserPackages = true (flake.nix) sets users.users.${ipaUser}.packages,
# which forces the stub into /etc/passwd. pam_sss.so with the "localusers" flag
# (added by NixOS when SSSD is enabled) then skips SSSD for any user it finds in
# local /etc/passwd — including this stub — falling through to pam_unix, which has
# no password for the stub → sudo auth always fails.
#
# Fix: NOPASSWD for the IPA user. The IPA user already authenticated to reach a
# shell (SSH public key from IPA or Kerberos), so re-prompting via a broken PAM
# path is security theater on a single-admin homelab.
sudo.extraRules = [{
users = [ vars.ipaUser ];
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
}];
};
systemd = {
# Fetch SSH public keys from IPA so users can log in with the key stored
# in their IPA profile rather than needing ~/.ssh/authorized_keys on every
# host. sss_ssh_authorizedkeys queries SSSD (which queries IPA LDAP).
#
# /nix/store is 1775 (group-writable by nixbld). OpenSSH 10.0+ rejects
# AuthorizedKeysCommand binaries whose path contains any group-writable
# component, silently skipping the command. Copy to /usr/local/bin (all
# components root-owned, 755) so the path passes sshd's safety check.
tmpfiles.rules = [
"d /usr/local 0755 root root - -"
"d /usr/local/bin 0755 root root - -"
"C+ /usr/local/bin/sss_ssh_authorizedkeys 0555 root root - ${pkgs.sssd}/bin/sss_ssh_authorizedkeys"
# Pre-create the IPA user's home dir so Home Manager activation succeeds
# even before their first login. On a fresh system SSSD may not have
# resolved the user yet — tmpfiles warns and skips in that case (non-fatal),
# and pam_mkhomedir covers the first-login path as a fallback.
"d /home/${vars.ipaUser} 0700 ${vars.ipaUser} ${vars.ipaUser} - -"
];
# security.ipa enables Kerberos (security.krb5) which causes systemd to
# start auth-rpcgss-module.service and rpc-gssd.service for Kerberos NFS
# authentication. LXC containers can't load the auth_rpcgss kernel module
# and don't have /var/lib/nfs/rpc_pipefs, so both services fail.
#
# The NixOS IPA module already adds a drop-in for auth-rpcgss-module.service
# with ConditionPathExists=/etc/krb5.keytab. We use lib.mkForce to win the
# text conflict and add ConditionVirtualization=!container alongside it so
# the service is skipped (not failed) in containers that do have a keytab.
# Same fix for rpc-gssd.service which also fails in containers.
units = lib.mkIf config.boot.isContainer {
"auth-rpcgss-module.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
# rpc-gssd also has ConditionPathExists from the NixOS IPA module (and an
# X-Restart-Triggers store path from systemd.nix). Use mkForce to win;
# omit X-Restart-Triggers since this service is skipped in containers anyway.
"rpc-gssd.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
};
# home-manager-<user>.service fails on first enrollment because /home/wayne
# doesn't exist until the user's first login (pam_mkhomedir creates it then).
# ConditionPathExists makes systemd skip the service (exit 0, condition not
# met) instead of failing. After first login the dir exists and subsequent
# rebuilds activate HM normally.
services."home-manager-${vars.ipaUser}".unitConfig.ConditionPathExists =
"/home/${vars.ipaUser}";
};
services.openssh.extraConfig = ''
AuthorizedKeysCommand /usr/local/bin/sss_ssh_authorizedkeys %u
AuthorizedKeysCommandUser nobody
'';
# Host keytab: pre-provisioned on the IPA server, sops-encrypted binary.
# Placed at /etc/krb5.keytab before SSSD starts so the host authenticates
# to IPA without running ipa-client-install.
sops.secrets."ipa-host-keytab" = {
sopsFile = keytabPath;
format = "binary";
path = "/etc/krb5.keytab";
owner = "root";
group = "root";
mode = "0600";
restartUnits = [ "sssd.service" ];
};
# NixOS requires isNormalUser/isSystemUser + group on any entry in
# users.users. HM with useUserPackages = true (set in flake.nix) adds a stub
# entry for each HM user so it can install packages to
# /etc/profiles/per-user/<name>/. This definition satisfies those assertions.
# With security.ipa setting "passwd: sss files" in nsswitch, SSSD's IPA entry
# takes priority for NSS lookups — this local stub is only a fallback when
# SSSD is unreachable (at which point auth fails anyway).
users.users.${vars.ipaUser} = {
isNormalUser = true;
group = "users";
extraGroups = [ "wheel" ];
createHome = false;
# "!" is not a password hash — it is the standard "account locked" marker.
# It cannot authenticate anyone locally. It exists solely so NixOS generates
# a shadow entry for this stub user; without one pam_unix returns
# PAM_AUTHINFO_UNAVAIL before prompting, which means PAM_AUTHTOK is never
# set and the subsequent pam_sss use_first_pass call has nothing to work
# with — blocking LightDM and su logins even when IPA/SSSD auth succeeds.
hashedPassword = "!";
};
# Home Manager config for the IPA primary user, applied on every enrolled
# host. Manages what IPA doesn't: dotfiles, user-scoped packages, session
# variables. Switch-nix/Test-nix/buildImage are system-wide (configuration.nix)
# so they don't need to be repeated here.
#
# homeDirectory uses mkForce because HM's NixOS integration module sets it to
# "/var/empty" for users not found in config.users.users at eval time (SSSD
# users aren't visible there).
home-manager.users.${vars.ipaUser} = { pkgs, ... }: {
home = {
username = vars.ipaUser;
homeDirectory = lib.mkForce "/home/${vars.ipaUser}";
stateVersion = "26.05";
packages = with pkgs; [ tmux sshfs ];
sessionVariables.EDITOR = lib.mkDefault "nano";
};
programs.home-manager.enable = true;
programs.bash.enable = true;
};
}
+1 -1
View File
@@ -3,7 +3,7 @@
{
nix.settings = {
substituters = [
"http://${vars.nixCacheHost}"
"http://${vars.nixCacheHost}.${vars.homeDomain}"
"https://cache.nixos.org/"
];
trusted-public-keys = [
+4 -4
View File
@@ -8,12 +8,12 @@
# dedicated keypair). If this host doesn't have one yet:
# sudo -u root ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
# # then add its .pub to vars.remoteBuilderAuthorizedKeys and rebuild nix-cache
# sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
# sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache.sweet.home nix-store --version
# Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
# can connect the first time without a manual ssh-keyscan/known_hosts
# step on every new client.
programs.ssh.knownHosts.${vars.nixCacheHost} = {
hostNames = [ vars.nixCacheHost ];
programs.ssh.knownHosts."${vars.nixCacheHost}.${vars.homeDomain}" = {
hostNames = [ "${vars.nixCacheHost}.${vars.homeDomain}" ];
publicKey = vars.nixCacheHostKey;
};
@@ -22,7 +22,7 @@
buildMachines = [
{
hostName = vars.nixCacheHost;
hostName = "${vars.nixCacheHost}.${vars.homeDomain}";
sshUser = vars.remoteBuilderUser;
sshKey = "/root/.ssh/id_ed25519";
inherit (pkgs.stdenv.hostPlatform) system;
+1 -1
View File
@@ -20,7 +20,7 @@
nginx = {
enable = true;
recommendedProxySettings = true;
virtualHosts.${vars.nixCacheHost} = {
virtualHosts."${vars.nixCacheHost}.${vars.homeDomain}" = {
locations."/" = {
proxyPass = "http://${config.services.nix-serve.bindAddress}:${toString config.services.nix-serve.port}";
};
+60 -20
View File
@@ -52,6 +52,7 @@ in
# LXC container does).
imports = [
(modulesPath + "/virtualisation/proxmox-lxc.nix")
../common/preserve-ssh-host-key.nix
];
proxmoxLXC = {
@@ -63,23 +64,22 @@ in
# back to decide `pct create`'s --unprivileged flag, so the two stay
# in sync).
#
# lxc-docker is the one exception: the kernel's NFS client doesn't set
# FS_USERNS_MOUNT, so mounting NFS from inside *any* non-init user
# namespace -- which is exactly what an unprivileged container's
# UID-mapped root runs in -- is rejected at the VFS layer with EPERM,
# no matter what Proxmox's own `mount=nfs;nfs4` container feature
# allows at the AppArmor layer (confirmed live: TCP to the NFS server
# succeeds, the server's export table matches the container's IP, and
# `mount.nfs: Operation not permitted` still fires immediately with no
# corresponding denial anywhere in the server's logs -- a kernel-level
# rejection, not a network or export-permission one). Keying off
# hostName rather than something docker-build-type-specific because
# modules/build-types/docker.nix is also composed for linode-docker/
# proxmox-docker, which don't import proxmox-lxc.nix at all --setting
# this option there would break their eval with "option does not
# exist" regardless of any mkIf guard, since mkIf only makes a value
# conditional, not whether the option needs to exist somewhere.
privileged = config.networking.hostName == "docker";
# Any lxc-* host with an NFS fileSystem must be privileged: the kernel's
# NFS client doesn't set FS_USERNS_MOUNT, so mounting NFS from inside
# *any* non-init user namespace -- which is exactly what an unprivileged
# container's UID-mapped root runs in -- is rejected at the VFS layer
# with EPERM, no matter what Proxmox's own `mount=nfs;nfs4` container
# feature allows at the AppArmor layer (confirmed live: TCP to the NFS
# server succeeds, the server's export table matches the container's IP,
# and `mount.nfs: Operation not permitted` still fires immediately with
# no corresponding denial anywhere in the server's logs -- a kernel-level
# rejection, not a network or export-permission one). Deriving this from
# fileSystems rather than a per-host override keeps it self-consistent:
# any new lxc-* host that declares an NFS mount automatically gets the
# privilege level it needs without a separate manual flag.
privileged = builtins.any
(fs: fs.fsType == "nfs" || fs.fsType == "nfs4")
(builtins.attrValues config.fileSystems);
};
boot.loader = {
@@ -112,9 +112,12 @@ in
# sops-nix's "for users" secrets (password hashes -- installed by the
# activation script itself, not a systemd service, since they need to
# exist *before* user creation) nor the user-creation step that
# consumes them ever run on a real lxc-* boot. Regular secrets
# (nix-serve's key, beszel's token, etc.) work anyway because sops-nix
# provides its own systemd service for those.
# consumes them ever run on a real lxc-* boot. In this config sops-nix
# does NOT generate its own boot-time service (confirmed live: no
# sops-nix.service in systemctl list-unit-files on a deployed
# lxc-tor-relay container); /run/secrets is a tmpfs cleared on every
# reboot, so secrets must be reinstalled on each non-first boot by
# nixos-lxc-sops-reinstall (below).
#
# A systemd service, not boot.postBootCommands: tried that first (it's
# a genuine, generally-invoked hook -- nixos/modules/system/boot/stage-2-init.sh,
@@ -165,4 +168,41 @@ in
touch /var/lib/nixos-lxc-first-boot-activated
'';
};
# Reinstalls sops secrets on every non-first boot. /run/secrets is a
# tmpfs that is cleared on each reboot; without this service, secrets
# are permanently absent after the first boot and every service that
# reads from /run/secrets fails on start.
#
# wantedBy/before network.target: switch-to-configuration test requires
# D-Bus to restart systemd targets after running activation scripts. D-Bus
# is available once basic.target completes (the default After=basic.target
# that DefaultDependencies would otherwise add). Placing the service before
# network.target ensures secrets are ready before any network-dependent
# service (including beszel-agent and nix-serve) starts, while running late
# enough that D-Bus is already up.
#
# ConditionPathExists=... skips this service on the genuine first boot
# (the marker doesn't exist yet); nixos-lxc-first-boot-activate handles
# that case. On every subsequent boot the condition passes and secrets
# are reinstalled before user services start.
#
# SuccessExitStatus=11: switch-to-configuration exits 11 when it cannot
# acquire the activation lock (another switch is already in progress).
# During a nixos-rebuild switch the activation already installs secrets, so
# treating the lock-held case as success is correct.
systemd.services.nixos-lxc-sops-reinstall = {
description = "Reinstall sops secrets on each non-first boot (LXC, /run is tmpfs)";
wantedBy = [ "network.target" ];
before = [ "network.target" ];
unitConfig.ConditionPathExists = "/var/lib/nixos-lxc-first-boot-activated";
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
SuccessExitStatus = "11";
};
script = ''
/run/current-system/bin/switch-to-configuration test
'';
};
}
+42 -1
View File
@@ -1,9 +1,50 @@
{ ... }:
{ lib, flakeTarget, ... }:
let
# Bakes this exact flake target's pre-generated SSH host key straight
# into /etc/ssh/ -- mirrors lxc.nix's builtins.getEnv pattern (impure
# and empty under normal `nix build`/`nix eval`, so this is a no-op
# unless explicitly opted into with NIXOS_HOST_KEYS_DIR=... --impure).
#
# Unlike --pre-format-files (which places files on the QEMU builder VM's
# rootfs, not the target disk), embedding via environment.etc here means
# nixos-install's own activation step installs the key onto the target
# disk. sshd-keygen then finds it already present and skips generation,
# so the disk image boots with the clan-registered key and sops can
# decrypt on first boot.
#
# Without this, nixos-install's sshd-keygen activation generates a fresh
# key (unregistered in .sops.yaml), sops decryption fails permanently,
# and password hashes are never applied -- confirmed live: passwords
# stayed '!' even with mutableUsers = false because hashedPasswordFile
# pointed to a path that sops never wrote.
hostKeysDirStr = builtins.getEnv "NIXOS_HOST_KEYS_DIR";
hasHostKeysDir = hostKeysDirStr != "" && builtins.pathExists hostKeysDirStr;
hostKeysDir = /. + hostKeysDirStr;
privKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key";
pubKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key.pub";
hasKeyForThisTarget =
hasHostKeysDir
&& builtins.pathExists privKeyFile
&& builtins.pathExists pubKeyFile;
in
{
imports = [
../hardware-configuration/vm/proxmox.nix
../boot/efi.nix
../disko/proxmox.nix
../common/preserve-ssh-host-key.nix
];
environment.etc = lib.mkIf hasKeyForThisTarget {
"ssh/ssh_host_ed25519_key" = {
source = privKeyFile;
mode = "0600";
};
"ssh/ssh_host_ed25519_key.pub" = {
source = pubKeyFile;
mode = "0644";
};
};
}
+42
View File
@@ -0,0 +1,42 @@
{ config, lib, vars, ... }:
let
# Use the same FQDN approach as docker/mount-data.nix — a bare hostname is
# unreliable: systemd-resolved only tries LLMNR for single-label names, and
# a global search domain causes it to skip the interface-scoped LAN DNS.
nfsServer = "${vars.nfsServerHost}.${vars.homeDomain}";
in
{
fileSystems.${vars.nfsShares.pxebootImages.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.pxebootImages.subpath}";
fsType = "nfs";
options = [
"_netdev"
"noatime"
] ++ (if config.boot.isContainer
# NFSv4 requires rpc_pipefs (sunrpc filesystem), which Proxmox LXC
# containers block unless `features: mount=nfs` is set. Use NFSv3+nolock
# instead: no rpc_pipefs dependency at the protocol level, and rpcbind
# on the server handles port resolution without needing client-side
# sunrpc infrastructure. nofail keeps boot clean if server is unreachable.
then [ "nfsvers=3" "proto=tcp" "nolock" "nofail" ]
else [ "nfsvers=4.2" "x-systemd.automount" ]);
};
# NixOS pulls var-lib-nfs-rpc_pipefs.mount (the sunrpc filesystem) into
# nfs-client.target for any nfs fileSystems entry. In LXC containers the
# sunrpc mount is blocked by Proxmox's AppArmor profile, causing it to fail
# and the activation to report an error even though our mount uses nofail.
# Add ConditionVirtualization=!container via drop-in so systemd skips the
# unit entirely in containers (skip = inactive, not failed), which keeps
# nfs-client.target green and activation clean.
systemd.units = lib.mkIf config.boot.isContainer {
"var-lib-nfs-rpc_pipefs.mount" = {
overrideStrategy = "asDropin";
text = ''
[Unit]
ConditionVirtualization=!container
'';
};
};
}
-27
View File
@@ -1,27 +0,0 @@
_:
{
imports = [ ./enable-service.nix ];
services.tailscale = {
# Enables the sysctl forwarding settings exit nodes/subnet routers need;
# without this, --advertise-exit-node has no effect.
useRoutingFeatures = "server";
# Lets peers reach this node directly over the tailscale UDP port
# instead of relaying through DERP.
openFirewall = true;
# extraSetFlags (tailscale set, via the always-on tailscaled-set
# service), not extraUpFlags -- extraUpFlags is only ever applied by
# tailscaled-autoconnect, which itself only runs when
# services.tailscale.authKeyFile is set (nothing in this repo sets one,
# so tailscale up is a manual, one-time operator step on every host that
# uses this service). extraSetFlags has no such gate, so
# --advertise-exit-node self-reapplies on every boot once the operator
# has authenticated the node once.
extraSetFlags = [
"--advertise-exit-node"
];
};
}
+35
View File
@@ -0,0 +1,35 @@
{ pkgs, ... }:
{
imports = [ ./enable-service.nix ];
services.tailscale = {
# Enables the sysctl forwarding settings subnet routers need;
# without this, --advertise-routes has no effect.
useRoutingFeatures = "server";
# Lets peers reach this node directly over the tailscale UDP port
# instead of relaying through DERP.
openFirewall = true;
};
# Tailscale recommends these ethtool flags on the uplink interface to get
# full UDP GRO throughput on subnet routers (https://tailscale.com/s/ethtool-config-udp-gro).
# The interface is derived from the default route so it works regardless of
# what the NIC is named on a given host.
systemd.services.tailscale-udp-gro = {
description = "Enable UDP GRO forwarding on uplink for Tailscale subnet router";
after = [ "network-online.target" ];
wants = [ "network-online.target" ];
wantedBy = [ "multi-user.target" ];
path = [ pkgs.ethtool pkgs.iproute2 ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
ExecStart = pkgs.writeShellScript "tailscale-udp-gro" ''
NETDEV=$(ip -o route get 8.8.8.8 | cut -f 5 -d " ")
ethtool -K "$NETDEV" rx-udp-gro-forwarding on rx-gro-list off
'';
};
};
}
+60
View File
@@ -0,0 +1,60 @@
{ vars, ... }:
{
# Run dnsmasq on the LAN interface as a forwarding-only resolver for
# *.ts.net (Tailscale MagicDNS names). FreeIPA's bind-dyndb-ldap
# cannot reach 100.100.100.100 (Tailscale's internal resolver) directly
# because the DC is not a Tailscale node. This host IS a Tailscale node
# and can reach 100.100.100.100 via its tailscale0 interface, so it
# acts as an intermediary: FreeIPA has a conditional forward zone for
# ts.net pointing here (vars.tailscaleRouterIp), and this dnsmasq
# instance forwards those queries onward to Tailscale's resolver.
#
# Configure FreeIPA once after deploying this host:
# kinit admin
# ipa dnsforwardzone-add ${vars.tailnetDomain} \
# --forwarder=${vars.tailscaleRouterIp} \
# --forward-policy=only
# Note: IPA refuses to shadow ts.net (a real public TLD); use the
# tailnet-specific subdomain (vars.tailnetDomain) instead.
services.dnsmasq = {
enable = true;
# NixOS's dnsmasq module defaults resolveLocalQueries to true, which adds
# 127.0.0.1 to networking.nameservers and makes dnsmasq bind to
# listen-address=127.0.0.1. This instance is not the host's local
# resolver — it only serves IPA's conditional forwarder for tailnet names.
# The host uses domainControllerIp directly (networking.nameservers in
# host.nix). Without this, all host DNS goes through dnsmasq, which has
# no upstream for general queries (no-resolv=true), breaking resolution.
resolveLocalQueries = false;
settings = {
# Listen only on the LAN interface — not tailscale0 or loopback.
# bind-interfaces prevents dnsmasq from binding to 0.0.0.0 and
# then filtering by interface later; combined with `interface` this
# ensures it genuinely listens only on eth0.
bind-interfaces = true;
interface = [ vars.lxcLanInterface ];
# Forward-only: no local /etc/hosts or /etc/resolv.conf reading,
# no negative caching of NXDOMAIN for names this instance doesn't
# serve. All ts.net queries come from FreeIPA's conditional forwarder
# and must be answered by Tailscale's resolver.
no-hosts = true;
no-resolv = true;
# Tailscale's internal "Quad100" resolver — reachable from any
# Tailscale node via the tailscale0 interface. Scoped to the
# specific tailnet subdomain (vars.tailnetDomain) rather than
# all of ts.net: FreeIPA refuses to shadow ts.net (a real public
# TLD with DNSimple nameservers) so the conditional forward zone
# in FreeIPA must use the tailnet-specific subdomain instead:
# ipa dnsforwardzone-add ${vars.tailnetDomain} \
# --forwarder=${vars.tailscaleRouterIp} \
# --forward-policy=only
server = [ "/${vars.tailnetDomain}/100.100.100.100" ];
};
};
networking.firewall.allowedUDPPorts = [ 53 ];
networking.firewall.allowedTCPPorts = [ 53 ];
}
Binary file not shown.
+11
View File
@@ -88,6 +88,17 @@ export NIX_CACHE_HOST
: "${LAN_DOMAIN:=gitea.lan.ddnsgeek.com}"
export LAN_DOMAIN
# Matches variables.nix's homeDomain -- the base LAN domain for service
# subdomains, FreeIPA Kerberos realm, and host FQDNs.
: "${HOME_DOMAIN:=sweet.home}"
export HOME_DOMAIN
# Matches variables.nix's ipaServer -- the FreeIPA server hostname.
# scripts/ipa/create-nixos-ipa-host-account.sh SSHes here to run
# ipa host-add and ipa-getkeytab.
: "${IPA_SERVER:=domain-controller.sweet.home}"
export IPA_SERVER
# nix_extra_opts: call as a plain statement (NOT inside $(...)/<(...) --
# that forks a subshell, and the whole point is exporting a decision back
# into *this* shell) to populate the global NIX_OPTS array with whatever
+232
View File
@@ -0,0 +1,232 @@
#!/usr/bin/env bash
# gc-hosts.sh — Run nix-collect-garbage -d on all live NixOS hosts.
#
# The host list is rebuilt on every run:
# 1. This workstation (nixos) — always first
# 2. pve1 — always second (non-NixOS Proxmox node with Nix installed)
# 3. Every NixOS guest currently running on pve1 (discovered via pct/qm list)
#
# nix-cache is excluded: gc-ing the shared binary cache evicts store paths
# that other hosts depend on for substitution.
#
# NixOS hosts: tries "sudo -n nix-collect-garbage -d" first (works when
# wheelNeedsPassword = false, e.g. the HA cluster). Falls back to user-level
# "nix-collect-garbage -d" if sudo needs a password — still collects
# unreferenced store paths and old nixos-user profile generations, but leaves
# old system generations in place.
# pve1: runs "bash -l -c nix-collect-garbage -d" as the login user so
# /etc/profile is sourced and the Nix daemon's PATH is set up automatically.
#
# Usage (from repo root):
# bash scripts/gc-hosts.sh [--dry-run]
set -euo pipefail
cd "$(dirname "$0")/.."
source scripts/env.sh 2>/dev/null || true
source scripts/lib/nix-eval.sh 2>/dev/null || true
# ── config ────────────────────────────────────────────────────────────────────
: "${MAX_JOBS:=8}"
: "${NIXOS_USER:=nixos}"
: "${PVE1_SSH_USER:=${PROXMOX_SSH_USER:-wayne}}"
# GC connections use BatchMode — no interactive prompts, just succeed or fail.
SSH_OPTS=(-o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=10)
# Discovery connections do NOT use BatchMode so that sudo can prompt if needed
# (pct/qm list require root access on Proxmox).
SSH_QUERY_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=10)
DRY_RUN=0
for arg in "$@"; do
case "$arg" in
--dry-run) DRY_RUN=1 ;;
*) echo "Unknown option: $arg" >&2; exit 1 ;;
esac
done
# ── build the host list ───────────────────────────────────────────────────────
# ORDERED_HOSTS: names in display/execution order.
# HOST_TARGET[name]: SSH target string (user@host).
# HOST_TYPE[name]: "nixos" (try sudo gc, fallback user) | "nix" (login-shell gc).
declare -a ORDERED_HOSTS=()
declare -A HOST_TARGET=()
declare -A HOST_TYPE=()
declare -A _SEEN_HOSTNAMES=() # dedup tracker
_add_host() {
local name="$1" target="$2" type="$3"
if [[ -n "${_SEEN_HOSTNAMES[$name]+_}" ]]; then return; fi
_SEEN_HOSTNAMES[$name]=1
ORDERED_HOSTS+=("$name")
HOST_TARGET[$name]="$target"
HOST_TYPE[$name]="$type"
}
# 1. Workstation (hard-wired first)
_add_host "nixos" "${NIXOS_USER}@nixos" "nixos"
# 2. pve1 (hard-wired second; non-NixOS, no system generations)
_add_host "pve1" "${PVE1_SSH_USER}@${PVE1_HOST}" "nix"
# 3. Dynamically discover running NixOS guests on pve1
echo "Discovering running guests on ${PVE1_HOST}..."
# Evaluate the full flake hostname map in one shot.
hostname_map="{}"
if ! hostname_map="$(
nix eval --json "${NIX_EVAL_FLAGS[@]}" .#nixosConfigurations \
--apply 'cfgs: builtins.mapAttrs (_: cfg: cfg.config.networking.hostName) cfgs' \
2>/dev/null
)"; then
echo " warning: flake eval failed — skipping dynamic host discovery" >&2
fi
# SSH_QUERY_OPTS (no BatchMode) so sudo can prompt if wayne's sudo needs a password.
if ssh "${SSH_QUERY_OPTS[@]}" "${PVE1_SSH_USER}@${PVE1_HOST}" "true" 2>/dev/null; then
running_guests="$(
ssh "${SSH_QUERY_OPTS[@]}" "${PVE1_SSH_USER}@${PVE1_HOST}" bash <<'REMOTE'
{ sudo pct list 2>/dev/null | awk 'NR>1 && $2=="running" { print $NF }';
sudo qm list 2>/dev/null | awk 'NR>1 && $3=="running" { print $2 }'; } | sort -u
REMOTE
)" || running_guests=""
while IFS= read -r guest; do
[[ -z "$guest" ]] && continue
# Resolve flake target name → NixOS hostname.
hostname="$(printf '%s' "$hostname_map" \
| jq -r --arg g "$guest" '.[$g] // empty' 2>/dev/null || true)"
[[ -z "$hostname" ]] && continue
# Exclude nix-cache and any hostname already in the list.
case "$hostname" in nix-cache) continue ;; esac
if [[ -n "${_SEEN_HOSTNAMES[$hostname]+_}" ]]; then continue; fi
echo " + $guest$hostname"
_add_host "$hostname" "${NIXOS_USER}@${hostname}" "nixos"
done <<< "$running_guests"
else
echo " warning: ${PVE1_HOST} unreachable — skipping dynamic host discovery" >&2
fi
echo ""
echo "Hosts: ${ORDERED_HOSTS[*]}"
echo ""
# ── dry-run ───────────────────────────────────────────────────────────────────
if [[ "$DRY_RUN" -eq 1 ]]; then
echo "[dry-run] commands that would run:"
for host in "${ORDERED_HOSTS[@]}"; do
target="${HOST_TARGET[$host]}"
type="${HOST_TYPE[$host]}"
if [[ "$type" == "nixos" ]]; then
echo " ssh ${SSH_OPTS[*]} $target 'sudo -n nix-collect-garbage -d'"
echo " # fallback: ssh ... $target 'nix-collect-garbage -d'"
else
echo " ssh ${SSH_OPTS[*]} $target 'bash -l -c nix-collect-garbage -d'"
fi
done
exit 0
fi
# ── gc worker ─────────────────────────────────────────────────────────────────
gc_one() {
local host="$1" target="${HOST_TARGET[$1]}" type="${HOST_TYPE[$1]}" logfile="$2"
if ! ssh "${SSH_OPTS[@]}" "$target" "true" 2>>"$logfile"; then
echo "unreachable"; return
fi
if [[ "$type" == "nixos" ]]; then
if ssh "${SSH_OPTS[@]}" "$target" "sudo -n nix-collect-garbage -d" \
>>"$logfile" 2>>"$logfile"; then
echo "ok(sudo)"; return
fi
echo "[sudo needs password — falling back to user-level gc]" >>"$logfile"
if ssh "${SSH_OPTS[@]}" "$target" "nix-collect-garbage -d" \
>>"$logfile" 2>>"$logfile"; then
echo "ok(user)"; return
fi
else
# Non-NixOS node: use a login shell so /etc/profile is sourced and the
# Nix daemon's bin dir is on PATH (set up by /etc/profile.d/nix-daemon.sh
# which the Nix installer adds to /etc/profile).
if ssh "${SSH_OPTS[@]}" "$target" "bash -l -c 'nix-collect-garbage -d'" \
>>"$logfile" 2>>"$logfile"; then
echo "ok"; return
fi
fi
echo "failed:$?"
}
# ── parallel execution ────────────────────────────────────────────────────────
echo "Running gc on ${#ORDERED_HOSTS[@]} hosts (up to ${MAX_JOBS} parallel)..."
echo ""
TMPDIR_GC="$(mktemp -d)"
trap 'rm -rf "$TMPDIR_GC"' EXIT
declare -A LOGS=()
job_count=0
for host in "${ORDERED_HOSTS[@]}"; do
logfile="${TMPDIR_GC}/${host}.log"
resultfile="${TMPDIR_GC}/${host}.result"
LOGS[$host]="$logfile"
: > "$logfile"
( result="$(gc_one "$host" "$logfile")"; echo "$result" > "$resultfile" ) &
(( job_count++ )) || true
if [[ "$job_count" -ge "$MAX_JOBS" ]]; then
wait -n 2>/dev/null || wait
(( job_count-- )) || true
fi
done
wait
# ── summary ───────────────────────────────────────────────────────────────────
echo "Results:"
echo "──────────────────────────────"
ok_hosts=()
warn_hosts=()
fail_hosts=()
for host in "${ORDERED_HOSTS[@]}"; do
result="$(cat "${TMPDIR_GC}/${host}.result" 2>/dev/null || echo "failed:missing")"
case "$result" in
ok|"ok(sudo)"|"ok(user)")
printf " %-22s %s\n" "$host" "$result"
ok_hosts+=("$host") ;;
unreachable)
printf " %-22s UNREACHABLE\n" "$host"
warn_hosts+=("$host") ;;
*)
printf " %-22s FAILED (%s)\n" "$host" "$result"
fail_hosts+=("$host") ;;
esac
done
echo ""
echo " ${#ok_hosts[@]} succeeded, ${#warn_hosts[@]} unreachable, ${#fail_hosts[@]} failed"
for host in "${warn_hosts[@]+"${warn_hosts[@]}"}" "${fail_hosts[@]+"${fail_hosts[@]}"}"; do
logfile="${LOGS[$host]}"
if [[ -s "$logfile" ]]; then
echo ""
echo "── $host ──"
cat "$logfile"
fi
done
echo ""
[[ "${#fail_hosts[@]}" -eq 0 ]]
+231
View File
@@ -0,0 +1,231 @@
#!/usr/bin/env bash
# acceptance-tests.sh — HA cluster acceptance tests (T1T7)
#
# Run from a host with SSH access to both HA nodes (or from node1 itself).
# All 7 tests must pass before considering the cluster production-ready.
# Test values below must match variables.nix haServer* values.
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
# All values override-able via environment variables; defaults match variables.nix.
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.2.229}" # vars.haServerVip
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
ISCSI_IQN="${ISCSI_IQN:-iqn.2026-01.home.sweet:ha-storage}" # vars.haIscsiIqn
# ──────────────────────────────────────────────────────────────────────────
PASS=0
FAIL=0
RESULTS=()
# Use PASS=$((PASS+1)) instead of ((PASS++)) — the latter evaluates to 0 when
# PASS=0, which triggers set -e and kills the script after the very first PASS.
pass() { echo " PASS: $1"; PASS=$((PASS+1)); RESULTS+=("PASS $1"); }
fail() { echo " FAIL: $1"; FAIL=$((FAIL+1)); RESULTS+=("FAIL $1"); }
HA_USER="nixos"
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
echo "════════════════════════════════════════════════════"
echo " HA Cluster Acceptance Tests — $(date '+%Y-%m-%d %H:%M:%S')"
echo "════════════════════════════════════════════════════"
# ── Pre-flight: DRBD sync must be complete ────────────────────────────────
# Tests that check disk state, XFS mount, and iSCSI will fail or give false
# results while the initial full sync is in progress. Block until done.
echo ""
echo "Pre-flight: verifying DRBD sync is complete..."
DRBD_PREFLIGHT=$(n1 "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_PREFLIGHT" | grep -q "^UpToDate/UpToDate$"; then
echo ""
echo " ERROR: DRBD initial sync not complete."
echo " Current dstate on $NODE1: $DRBD_PREFLIGHT"
echo ""
echo " Monitor progress:"
echo " ssh nixos@$NODE1_IP 'sudo watch -n3 cat /proc/drbd'"
echo ""
echo " Re-run this script once dstate shows UpToDate/UpToDate."
exit 1
fi
echo " dstate: $DRBD_PREFLIGHT — ready."
# ── Detect Active/Standby nodes ────────────────────────────────────────────
# Use crm_mon to detect which node holds the Promoted (Primary) DRBD resource.
# Pacemaker is authoritative; DRBD role can briefly read as Secondary while
# Pacemaker is mid-transition, giving a false Active/Standby swap.
# Wait up to 90 s for Pacemaker to settle before giving up.
echo ""
echo "Detecting Active/Standby nodes (waiting for Pacemaker to settle)..."
ACTIVE_NODE=""
for i in $(seq 1 30); do
# crm_mon -1 output contains "Promoted: [ <node> ]" for the DRBD master.
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || n2 "crm_mon -1" 2>/dev/null || true)
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" — the * bullet
# means ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | grep -E '(Promoted|Masters):' | grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
[[ -n "$ACTIVE_NODE" ]] && break
sleep 3
done
if [[ -z "$ACTIVE_NODE" ]]; then
echo " WARNING: could not determine Active node from crm_mon after 90 s — defaulting to $NODE1"
ACTIVE_NODE="$NODE1"
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
STANDBY_NODE="$NODE2"; STANDBY_IP="$NODE2_IP"
na() { n1 "$@"; }
ns() { n2 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
STANDBY_NODE="$NODE1"; STANDBY_IP="$NODE1_IP"
na() { n2 "$@"; }
ns() { n1 "$@"; }
fi
echo " Active: $ACTIVE_NODE ($ACTIVE_IP)"
echo " Standby: $STANDBY_NODE ($STANDBY_IP)"
# ── T1: Corosync quorum established ──────────────────────────────────────
echo ""
echo "[T1] Corosync quorum"
if na "corosync-quorumtool -s" 2>/dev/null | grep -q "Quorate:.*Yes"; then
pass "cluster has quorum"
else
fail "cluster does not have quorum — check corosync on both nodes"
fi
# ── T2: DRBD Primary on Active node, Secondary on Standby ────────────────
echo ""
echo "[T2] DRBD roles"
DRBD_ROLE=$(na "drbdadm role ha-data" 2>/dev/null || echo "unknown")
if [[ "$DRBD_ROLE" == "Primary/Secondary" || "$DRBD_ROLE" == "Primary" ]]; then
pass "DRBD Primary on $ACTIVE_NODE ($DRBD_ROLE)"
else
fail "unexpected DRBD role on $ACTIVE_NODE: $DRBD_ROLE (expected Primary/Secondary)"
fi
DRBD_DSTATE=$(na "drbdadm dstate ha-data" 2>/dev/null || echo "unknown")
if echo "$DRBD_DSTATE" | grep -q "UpToDate"; then
pass "DRBD disk state UpToDate ($DRBD_DSTATE)"
else
fail "DRBD disk not UpToDate: $DRBD_DSTATE"
fi
# ── T3: XFS mounted at haStorageRoot on the Active node ──────────────────
echo ""
echo "[T3] XFS mount"
if na "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
pass "XFS mounted at ${XFS_MOUNT} on $ACTIVE_NODE"
else
fail "XFS not mounted at ${XFS_MOUNT} on $ACTIVE_NODE"
fi
if ns "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
fail "XFS unexpectedly mounted on $STANDBY_NODE (should only be on Active node)"
else
pass "XFS not mounted on $STANDBY_NODE (correct — Standby)"
fi
# ── T4: iSCSI target visible on Active node ───────────────────────────────
echo ""
echo "[T4] iSCSI target"
IQN_COUNT=$(na "bash -c 'ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn || true'" 2>/dev/null || echo "0")
if [[ "$IQN_COUNT" -ge 1 ]]; then
pass "iSCSI IQN active on $ACTIVE_NODE ($IQN_COUNT target(s))"
else
fail "no iSCSI IQN active on $ACTIVE_NODE"
fi
# iSCSI port reachable from Standby node via VIP.
# Use bash TCP probe (no iscsiadm needed — just checks port 3260 is open).
if ns "bash -c 'echo >/dev/tcp/${VIP}/3260' 2>/dev/null"; then
pass "iSCSI port 3260 reachable from $STANDBY_NODE via VIP ${VIP}"
else
fail "iSCSI port 3260 not reachable from $STANDBY_NODE via ${VIP}"
fi
# ── T5: Failover — standby Active node, verify resources move to Standby ──
echo ""
echo "[T5] Failover (standby $ACTIVE_NODE)"
ACTIVE_CRMD_NAME=$(na "crm_node -n" 2>/dev/null || echo "")
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v on" 2>/dev/null || true
echo " Waiting up to 120 s for resources to move to $STANDBY_NODE..."
MOVED=false
for i in $(seq 1 120); do
if ns "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
MOVED=true
echo " Resources moved in ${i}s"
break
fi
sleep 1
done
if $MOVED; then
pass "XFS mounted on $STANDBY_NODE after failover"
IQN_ON_STANDBY=$(ns "bash -c 'ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn || true'" 2>/dev/null || echo "0")
[[ "$IQN_ON_STANDBY" -ge 1 ]] \
&& pass "iSCSI target active on $STANDBY_NODE after failover" \
|| fail "iSCSI target NOT active on $STANDBY_NODE after failover"
else
fail "XFS did not mount on $STANDBY_NODE within 120 s — failover incomplete"
fi
# ── T6: Data integrity — file written post-failover readable ─────────────
echo ""
echo "[T6] Data integrity"
# Write a test file on the new Active (former Standby) and verify it.
# Use `echo | sudo tee` for the write: "echo ... > file" via bash -c has the
# redirect interpreted by the remote nixos shell (not sudo), so the file open
# runs as nixos and fails with EACCES on the root-owned XFS mount. Piping
# through sudo tee lets tee (running as root) open the file instead.
TEST_FILE="${XFS_MOUNT}/.acceptance-test-$$"
TEST_CONTENT="ha-acceptance-test-$(date +%s)"
echo "${TEST_CONTENT}" | ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${STANDBY_IP}" sudo tee "${TEST_FILE}" > /dev/null 2>/dev/null || true
READBACK=$(ns cat "${TEST_FILE}" 2>/dev/null || echo "")
if [[ "$READBACK" == "$TEST_CONTENT" ]]; then
pass "test file written and read back correctly on $STANDBY_NODE"
else
fail "data integrity check failed (wrote: '$TEST_CONTENT', read: '$READBACK')"
fi
ns rm -f "${TEST_FILE}" 2>/dev/null || true
# ── T7: Node rejoin — un-standby original Active, verify cluster is healthy ─
echo ""
echo "[T7] Node rejoin"
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
na "crm_resource --cleanup" 2>/dev/null || true
sleep 5
if na "corosync-quorumtool -s 2>/dev/null | grep -q 'Quorate:.*Yes'"; then
pass "$ACTIVE_NODE rejoined — cluster has quorum"
else
fail "$ACTIVE_NODE did not rejoin with quorum"
fi
DRBD_ROLE_AFTER=$(na "drbdadm role ha-data" 2>/dev/null || echo "unknown")
if echo "$DRBD_ROLE_AFTER" | grep -q "Secondary"; then
pass "$ACTIVE_NODE is DRBD Secondary after rejoin ($DRBD_ROLE_AFTER)"
else
fail "unexpected DRBD role on $ACTIVE_NODE after rejoin: $DRBD_ROLE_AFTER"
fi
# ── Summary ───────────────────────────────────────────────────────────────
echo ""
echo "════════════════════════════════════════════════════"
echo " Results: ${PASS} PASS, ${FAIL} FAIL"
echo "════════════════════════════════════════════════════"
for r in "${RESULTS[@]}"; do echo " $r"; done
echo ""
if [[ "$FAIL" -eq 0 ]]; then
echo "ALL PASS — cluster is production-ready."
exit 0
else
echo "SOME TESTS FAILED — investigate before deploying."
exit 1
fi
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env bash
# cluster-enable-stonith.sh — enable STONITH fence agent after the fence SSH
# key is deployed to both nodes and authorised on the Proxmox host.
#
# Run from ha-server-1 as root AFTER:
# - /etc/pacemaker/fence_pve_ssh exists on both nodes (chmod +x)
# (copy from scripts/ha/fence-pve-ssh.py)
# - /etc/fence-pve-ssh-key (SSH private key) exists on both nodes
# - The corresponding public key is in authorized_keys on PVE_HOST
# - VMID_NODE1 / VMID_NODE2 filled in below
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="ha-server-1"
NODE2="ha-server-2"
VMID_NODE1="" # FILL IN: Proxmox VMID for ha-server-1
VMID_NODE2="" # FILL IN: Proxmox VMID for ha-server-2
PVE_HOST="pve1.sweet.home"
PVE_USER="wayne"
FENCE_KEY="/etc/fence-pve-ssh-key"
FENCE_SCRIPT="/etc/pacemaker/fence_pve_ssh"
# ──────────────────────────────────────────────────────────────────────────
log() { echo "[stonith-setup] $*"; }
die() { echo "[stonith-setup] ERROR: $*" >&2; exit 1; }
[[ $(id -u) -eq 0 ]] || die "must run as root"
[[ -n "$VMID_NODE1" ]] || die "VMID_NODE1 not set — edit this script"
[[ -n "$VMID_NODE2" ]] || die "VMID_NODE2 not set — edit this script"
[[ -f "$FENCE_KEY" ]] || die "fence key not found at $FENCE_KEY"
[[ -f "$FENCE_SCRIPT" ]] || die "fence script not found at $FENCE_SCRIPT"
log "Verifying fence agent can reach ${PVE_HOST}..."
ssh -i "$FENCE_KEY" -o BatchMode=yes -o ConnectTimeout=10 \
-o StrictHostKeyChecking=no "${PVE_USER}@${PVE_HOST}" \
"sudo /usr/sbin/qm list" &>/dev/null \
|| die "Cannot SSH to ${PVE_USER}@${PVE_HOST} — check authorized_keys and sudo"
log "Fence agent SSH connectivity confirmed"
log "Creating Pacemaker STONITH resources..."
cibadmin --create --scope resources --xml-text "
<primitive id=\"stonith-${NODE1}\" class=\"stonith\" type=\"external/fence_pve_ssh\">
<instance_attributes id=\"stonith-${NODE1}-attrs\">
<nvpair id=\"stonith-${NODE1}-plug\" name=\"plug\" value=\"${NODE1}\"/>
<nvpair id=\"stonith-${NODE1}-pve-host\" name=\"pve_host\" value=\"${PVE_HOST}\"/>
<nvpair id=\"stonith-${NODE1}-pve-user\" name=\"pve_user\" value=\"${PVE_USER}\"/>
<nvpair id=\"stonith-${NODE1}-key-file\" name=\"key_file\" value=\"${FENCE_KEY}\"/>
<nvpair id=\"stonith-${NODE1}-vmid1\" name=\"vmid_node1\" value=\"${VMID_NODE1}\"/>
<nvpair id=\"stonith-${NODE1}-vmid2\" name=\"vmid_node2\" value=\"${VMID_NODE2}\"/>
<nvpair id=\"stonith-${NODE1}-host-list\" name=\"pcmk_host_list\" value=\"${NODE1}\"/>
</instance_attributes>
<operations>
<op id=\"stonith-${NODE1}-monitor\" name=\"monitor\" interval=\"30s\" timeout=\"30s\"/>
</operations>
</primitive>
" 2>/dev/null || true
cibadmin --create --scope resources --xml-text "
<primitive id=\"stonith-${NODE2}\" class=\"stonith\" type=\"external/fence_pve_ssh\">
<instance_attributes id=\"stonith-${NODE2}-attrs\">
<nvpair id=\"stonith-${NODE2}-plug\" name=\"plug\" value=\"${NODE2}\"/>
<nvpair id=\"stonith-${NODE2}-pve-host\" name=\"pve_host\" value=\"${PVE_HOST}\"/>
<nvpair id=\"stonith-${NODE2}-pve-user\" name=\"pve_user\" value=\"${PVE_USER}\"/>
<nvpair id=\"stonith-${NODE2}-key-file\" name=\"key_file\" value=\"${FENCE_KEY}\"/>
<nvpair id=\"stonith-${NODE2}-vmid1\" name=\"vmid_node1\" value=\"${VMID_NODE1}\"/>
<nvpair id=\"stonith-${NODE2}-vmid2\" name=\"vmid_node2\" value=\"${VMID_NODE2}\"/>
<nvpair id=\"stonith-${NODE2}-host-list\" name=\"pcmk_host_list\" value=\"${NODE2}\"/>
</instance_attributes>
<operations>
<op id=\"stonith-${NODE2}-monitor\" name=\"monitor\" interval=\"30s\" timeout=\"30s\"/>
</operations>
</primitive>
" 2>/dev/null || true
log "Enabling STONITH and restoring quorum policy..."
crm_attribute -t crm_config -n stonith-enabled -v true
crm_attribute -t crm_config -n no-quorum-policy -v stop
log "DRBD fencing mode must also be updated to resource-only (already the"
log "default in cluster-config.nix; confirm with: cat /etc/drbd.d/ha-data.conf)"
log "Testing fence agent..."
stonith_admin --list-devices && log "Fence devices listed successfully." \
|| warn "stonith_admin --list-devices failed — check config"
log "STONITH enabled. Cluster is now fully HA."
+467
View File
@@ -0,0 +1,467 @@
#!/usr/bin/env bash
# cluster-init.sh — one-time HA cluster initialisation script
#
# Run ONCE from ha-server-1 as root AFTER both VMs are booted and have SSH
# access. It:
# 1. Generates and distributes the corosync authkey
# 2. Waits for corosync quorum and pacemaker
# 3. Initialises DRBD metadata, promotes node1 to primary
# 4. Creates XFS on /dev/drbd0 and mounts it
# 5. Creates the directory tree and iSCSI LUN backing file
# 6. Configures LIO iSCSI target (file-backed LUN)
# 7. Configures Pacemaker resources: DRBD → XFS → iSCSI → NFS → VIP
#
# Prerequisites:
# - Both VMs booted with the ha-server config (nixos-rebuild done)
# - SSH key access from node1 to root@NODE2_IP
# - VMID_NODE1 / VMID_NODE2 filled in below (needed for STONITH setup;
# cluster starts without STONITH, which you enable separately via
# scripts/ha/cluster-enable-stonith.sh)
# - Run as root on ha-server-1
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
# All values override-able via environment variables; defaults match variables.nix.
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.2.229}" # vars.haServerVip
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
ISCSI_IQN="${ISCSI_IQN:-iqn.2026-01.home.sweet:ha-storage}" # vars.haIscsiIqn
ISCSI_LUN_FILE="${XFS_MOUNT}/iscsi-lun.img"
ISCSI_LUN_SIZE="10G"
DRBD_DEVICE="/dev/drbd0"
# DRBD backing disk — by-id path that resolves correctly on both nodes
# regardless of whether the OS-level name is sda or sdb (Proxmox VM disk
# ordering is not guaranteed). Matches haServerDrbdDisk in variables.nix.
# Override DRBD_DISK if your hardware uses a different controller/slot path.
DRBD_DISK="${DRBD_DISK:-/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1}"
VMID_NODE1="${VMID_NODE1:-}" # set by deploy.sh; needed for STONITH
VMID_NODE2="${VMID_NODE2:-}"
PVE_HOST="${PVE_HOST:-pve1.sweet.home}"
PVE_USER="${PVE_USER:-wayne}"
# Inter-node SSH: HA_USER is the user to SSH as on NODE2; HA_KEY is the private
# key to use. Default is root-to-root (no key arg). deploy.sh sets HA_USER=nixos
# and HA_KEY=/tmp/cluster-init-key so the script works even when root-to-root SSH
# is not available.
HA_USER="${HA_USER:-root}"
HA_KEY="${HA_KEY:-}"
# NFS dataset subdirectories to create under XFS_MOUNT.
# Must mirror vars.nfsShares subpath values in variables.nix.
NFS_SUBDIRS=(
"docker/config"
"docker/volumes"
"docker/databases"
"docker/nextcloud-data"
"raspi/volumes"
"proxmox/iso"
"proxmox/lxc"
"pxe-boot/images"
)
# ──────────────────────────────────────────────────────────────────────────
log() { echo "[cluster-init] $*"; }
die() { echo "[cluster-init] ERROR: $*" >&2; exit 1; }
warn() { echo "[cluster-init] WARNING: $*" >&2; }
[[ $(id -u) -eq 0 ]] || die "must run as root"
[[ "$(hostname)" == "$NODE1" ]] || die "must run on $NODE1"
# NixOS may not include xfsprogs in root's PATH even when it's in the store.
# If mkfs.xfs is missing, search the Nix store for it.
if ! command -v mkfs.xfs &>/dev/null; then
_xfs_bin=$(find /nix/store -maxdepth 3 -name mkfs.xfs 2>/dev/null | head -1 | xargs dirname 2>/dev/null || true)
[[ -n "$_xfs_bin" ]] && export PATH="$_xfs_bin:$PATH" \
|| die "mkfs.xfs not found — add xfsprogs to ha-server.nix environment.systemPackages and rebuild"
fi
# drbdmeta lives alongside drbdadm but may not be in PATH when run via sudo.
if ! command -v drbdmeta &>/dev/null; then
_drbd_bin=$(dirname "$(command -v drbdadm)" 2>/dev/null || true)
[[ -n "$_drbd_bin" ]] && export PATH="$_drbd_bin:$PATH" \
|| die "drbdmeta not found — is drbd-utils in ha-server environment.systemPackages?"
fi
# Portable 16-hex-char UUID generator (no openssl required).
_rand_uuid() {
cat /proc/sys/kernel/random/uuid 2>/dev/null | tr -d '-' | cut -c1-16 | tr '[:lower:]' '[:upper:]'
}
# Inter-node SSH/SCP helpers — abstract over root-to-root vs nixos+sudo.
_SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=10"
[[ -n "$HA_KEY" ]] && _SSH_OPTS="-i $HA_KEY $_SSH_OPTS"
if [[ "$HA_USER" == "root" ]]; then
n2_ssh() { ssh $_SSH_OPTS "root@${NODE2_IP}" "$@"; }
n2_scp() { scp $_SSH_OPTS "$1" "root@${NODE2_IP}:$2"; }
else
# Non-root user with passwordless sudo; wrap each command with sudo.
n2_ssh() { ssh $_SSH_OPTS "${HA_USER}@${NODE2_IP}" sudo "$@"; }
n2_scp() {
# SCP to a tmp path, then sudo-move to the real destination as the remote user.
local src="$1" dst="$2"
local tmp="/tmp/_cluster_init_scp_$$"
scp $_SSH_OPTS "$src" "${HA_USER}@${NODE2_IP}:${tmp}"
ssh $_SSH_OPTS "${HA_USER}@${NODE2_IP}" sudo mv "${tmp}" "${dst}"
}
fi
# ── 0. Corosync authkey ───────────────────────────────────────────────────
AUTHKEY="/etc/corosync/authkey"
mkdir -p /etc/corosync
if [[ ! -f "$AUTHKEY" ]]; then
log "Generating corosync authkey..."
corosync-keygen -k "$AUTHKEY"
chmod 0400 "$AUTHKEY"
fi
log "Distributing authkey to $NODE2..."
n2_ssh "mkdir -p /etc/corosync"
n2_scp "$AUTHKEY" "$AUTHKEY"
n2_ssh "chmod 0400 '${AUTHKEY}'"
log "Restarting corosync and pacemaker on both nodes..."
systemctl restart corosync
n2_ssh "systemctl restart corosync"
sleep 3
log "Starting pacemaker on both nodes (may have failed at boot before authkey was placed)..."
systemctl start pacemaker 2>/dev/null || systemctl restart pacemaker 2>/dev/null || true
n2_ssh "systemctl start pacemaker 2>/dev/null || systemctl restart pacemaker 2>/dev/null || true"
sleep 2
# ── 1. Corosync quorum ────────────────────────────────────────────────────
log "Waiting for corosync quorum..."
for i in $(seq 1 30); do
if corosync-quorumtool -s 2>/dev/null | grep -q 'Quorate:.*Yes'; then
log "Quorum established"
break
fi
[[ $i -eq 30 ]] && die "corosync quorum not established after 60 s"
sleep 2
done
log "Waiting for pacemaker..."
for i in $(seq 1 30); do
if crm_mon -1 &>/dev/null; then
log "Pacemaker running"
break
fi
[[ $i -eq 30 ]] && die "pacemaker not running after 60 s"
sleep 2
done
# ── 2. DRBD initialisation ────────────────────────────────────────────────
# Put both nodes in Pacemaker standby first so it stops managed resources
# cleanly, then enable maintenance-mode so Pacemaker's monitor operations are
# suspended. Without maintenance-mode, Pacemaker keeps monitoring: when it
# sees DRBD Primary on a standby node (that it didn't start), it triggers a
# stop action — killing the initial sync after ~10 s. Maintenance-mode
# disables all start/stop/monitor actions for the duration of the sync; it is
# cleared after UpToDate/UpToDate is confirmed.
log "Setting both nodes to Pacemaker standby for DRBD metadata init..."
crm_standby -N "$NODE1" -v on 2>/dev/null || true
crm_standby -N "$NODE2" -v on 2>/dev/null || true
# Wait for Pacemaker to actually stop DRBD (if it was managing it).
log "Waiting for DRBD to stop under Pacemaker control..."
for i in $(seq 1 30); do
n1_role=$(drbdadm role ha-data 2>/dev/null || echo "Unconfigured")
n2_role=$(n2_ssh "drbdadm role ha-data 2>/dev/null" 2>/dev/null || echo "Unconfigured")
if [[ "$n1_role" == "Unconfigured" ]] && [[ "$n2_role" == "Unconfigured" ]]; then
log "DRBD stopped on both nodes"
break
fi
[[ $i -eq 30 ]] && warn "DRBD still active after 60s standby — forcing down anyway"
sleep 2
done
log "Enabling Pacemaker maintenance-mode (suspends monitor/start/stop during sync)..."
crm_attribute -t crm_config -n maintenance-mode -v true 2>/dev/null || true
log "Detaching DRBD on $NODE1 (belt-and-suspenders after standby)..."
drbdadm down ha-data 2>/dev/null || true
log "Detaching DRBD on $NODE2..."
n2_ssh "drbdadm down ha-data 2>/dev/null || true"
sleep 2
# Ensure /etc/drbd.conf on both nodes points to DRBD_DISK (the stable by-id
# path). VMs built before this fix may have /dev/sda or /dev/sdb hardcoded.
# NixOS makes /etc/drbd.conf a symlink into the read-only Nix store, so
# sed -i on the symlink target would fail — we break the symlink first with
# cp --remove-destination, creating a regular writable copy.
# Rebuild+redeploy (--force-rebuild) to make this permanent.
_PATCH_DRBD=$(mktemp)
cat > "$_PATCH_DRBD" << 'PATCHEOF'
#!/bin/bash
WANT="$1"
conf=/etc/drbd.conf
if [[ -L "$conf" ]]; then
cp --remove-destination "$(readlink -f "$conf")" "$conf"
fi
cur=$(drbdadm sh-ll-dev ha-data 2>/dev/null | head -1 || true)
if [[ -n "$cur" && "$cur" != "$WANT" ]]; then
echo "[cluster-init] WARNING: patching $conf: $cur → $WANT (rebuild to make permanent)"
sed -i "s,${cur},${WANT},g" "$conf"
fi
PATCHEOF
chmod +x "$_PATCH_DRBD"
bash "$_PATCH_DRBD" "$DRBD_DISK"
n2_scp "$_PATCH_DRBD" "/tmp/patch-drbd-disk.sh"
n2_ssh "bash /tmp/patch-drbd-disk.sh ${DRBD_DISK}"
n2_ssh "rm -f /tmp/patch-drbd-disk.sh"
rm -f "$_PATCH_DRBD"
log "Initialising DRBD metadata on $NODE1..."
# Use drbdmeta --force directly for BOTH create-md and write-dev-uuid.
# drbdadm create-md --force passes --force to drbdmeta create-md but NOT to
# the write-dev-uuid sub-call it makes internally, so write-dev-uuid fails when
# the backing disk is still busy and stdin is not a TTY:
# "stdin not a TTY, not waiting for confirmation" → exit 20.
# Calling drbdmeta --force directly bypasses the exclusive-open confirmation on
# both steps without needing a TTY, regardless of whether the device is busy.
# Skip metadata creation only if DRBD is UP and fully synced (UpToDate/UpToDate).
# When the resource is down, drbdadm dstate reads metadata and returns just
# "UpToDate" (no slash) — that must not be treated as "already synced".
# Mismatched UUIDs from an interrupted sync cause instant WFConnection→StandAlone,
# so we always recreate metadata unless the sync is genuinely complete.
if [[ "$(drbdadm dstate ha-data 2>/dev/null)" != "UpToDate/UpToDate" ]]; then
UUID1=$(_rand_uuid)
drbdmeta --force 0 v08 "${DRBD_DISK}" internal create-md
drbdmeta --force 0 v08 "${DRBD_DISK}" internal write-dev-uuid "$UUID1"
fi
log "Initialising DRBD metadata on $NODE2..."
if [[ "$(n2_ssh "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null)" != "UpToDate/UpToDate" ]]; then
UUID2=$(n2_ssh "cat /proc/sys/kernel/random/uuid 2>/dev/null | tr -d '-' | cut -c1-16 | tr '[:lower:]' '[:upper:]'")
n2_ssh "drbdmeta --force 0 v08 ${DRBD_DISK} internal create-md"
n2_ssh "drbdmeta --force 0 v08 ${DRBD_DISK} internal write-dev-uuid ${UUID2}"
fi
log "Bringing up DRBD on both nodes..."
drbdadm up ha-data 2>/dev/null || true
n2_ssh "drbdadm up ha-data" 2>/dev/null || true
log "Forcing $NODE1 to DRBD Primary for initial sync..."
drbdadm primary ha-data --force
# NOTE: Pacemaker standby is intentionally kept ON until after the sync
# completes. Clearing it here races with the OCF DRBD agent: Pacemaker
# sees DRBD in WFConnection/SyncSource and may call drbdadm-down thinking
# something went wrong, killing the sync. Standby is cleared below, after
# UpToDate/UpToDate is confirmed.
log "Waiting for DRBD initial sync to complete (32 GB may take 1020 min)..."
log " (monitor with: watch -n3 cat /proc/drbd)"
_sync_chars=('|' '/' '-' $'\\')
_sync_iter=0
while true; do
_dstate=$(drbdadm dstate ha-data 2>/dev/null || echo "unknown")
if echo "$_dstate" | grep -q "UpToDate/UpToDate"; then
printf "\r%-80s\r" ""
log "DRBD initial sync complete (dstate: $_dstate)"
break
fi
# Parse connection state from /proc/drbd (cs:SyncSource, cs:Connected, cs:StandAlone …)
_cs=$(grep -oE 'cs:[A-Za-z]+' /proc/drbd 2>/dev/null | head -1 | sed 's/cs://' || echo "unknown")
# /proc/drbd uses variable whitespace: "sync'ed: 5.2%" (two spaces).
_pct=$(grep -oE "sync'ed:[[:space:]]+[0-9.]+" /proc/drbd 2>/dev/null | grep -oE "[0-9.]+" | head -1 || echo "")
_eta=$(grep -oE "finish:[[:space:]]+[0-9:]+" /proc/drbd 2>/dev/null | grep -oE "[0-9:]+$" | head -1 || echo "")
_spd=$(grep -oE "speed:[[:space:]]+[0-9,]+" /proc/drbd 2>/dev/null | grep -oE "[0-9,]+$" | head -1 || echo "")
_sync_iter=$(( _sync_iter + 1 ))
_sc="${_sync_chars[$_sync_iter % 4]}"
if [[ "$_cs" == "StandAlone" && $_sync_iter -gt 5 ]]; then
printf "\r%-80s\r" ""
die "DRBD is StandAlone after 15 s — peer connection lost (dstate: $_dstate). " \
"Check corosync/network and re-run cluster-init."
elif [[ -n "$_pct" ]]; then
printf "\r [%s] syncing: %s%% done — ETA %s @ %s K/s " \
"$_sc" "$_pct" "${_eta:-??:??:??}" "${_spd:-?}"
else
printf "\r [%s] cs:%s dstate:%s — waiting for sync to start " "$_sc" "$_cs" "$_dstate"
fi
sleep 3
done
log "Disabling Pacemaker maintenance-mode and clearing standby — handing DRBD back to Pacemaker..."
crm_attribute -t crm_config -n maintenance-mode -v false 2>/dev/null || true
crm_standby -N "$NODE1" -v off 2>/dev/null || true
crm_standby -N "$NODE2" -v off 2>/dev/null || true
# ── 3. XFS filesystem ─────────────────────────────────────────────────────
log "Creating XFS on ${DRBD_DEVICE}..."
if ! xfs_info "${DRBD_DEVICE}" &>/dev/null; then
mkfs.xfs -f "${DRBD_DEVICE}"
fi
log "Mounting ${DRBD_DEVICE} at ${XFS_MOUNT}..."
mkdir -p "${XFS_MOUNT}"
mountpoint -q "${XFS_MOUNT}" || mount "${DRBD_DEVICE}" "${XFS_MOUNT}"
# ── 4. NFS dataset directories ────────────────────────────────────────────
log "Creating NFS dataset directories..."
for subdir in "${NFS_SUBDIRS[@]}"; do
mkdir -p "${XFS_MOUNT}/${subdir}"
done
# ── 5. iSCSI LUN backing file ─────────────────────────────────────────────
log "Creating iSCSI LUN backing file ${ISCSI_LUN_FILE} (${ISCSI_LUN_SIZE})..."
if [[ ! -f "${ISCSI_LUN_FILE}" ]]; then
fallocate -l "${ISCSI_LUN_SIZE}" "${ISCSI_LUN_FILE}"
fi
# ── 6. LIO iSCSI target ───────────────────────────────────────────────────
log "Configuring LIO iSCSI target via targetcli..."
# Note: do NOT bind portal to ${VIP} here — the VIP isn't assigned yet (Pacemaker
# creates it). The default portal (all IPs, port 3260) is correct; Pacemaker's
# VIP resource will make the target reachable at the VIP address.
#
# Clear any existing LIO state first (idempotent: re-run after a partial failure).
# Use specific delete commands — clearconfig does not reliably clear kernel state.
if ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -q "${ISCSI_IQN}"; then
log "Clearing existing LIO target ${ISCSI_IQN} before reconfiguration..."
targetcli "/iscsi delete ${ISCSI_IQN}" 2>/dev/null || true
fi
if ls /sys/kernel/config/target/core/ 2>/dev/null | grep -q "fileio"; then
log "Clearing existing LIO backstore ha-lun0 before reconfiguration..."
targetcli "/backstores/fileio delete ha-lun0" 2>/dev/null || true
fi
targetcli <<EOF
/backstores/fileio create name=ha-lun0 file_or_dev=${ISCSI_LUN_FILE} size=0 write_back=false
/iscsi create ${ISCSI_IQN}
/iscsi/${ISCSI_IQN}/tpg1/luns create /backstores/fileio/ha-lun0
/iscsi/${ISCSI_IQN}/tpg1 set attribute authentication=0
/iscsi/${ISCSI_IQN}/tpg1 set attribute demo_mode_write_protect=0
saveconfig /etc/target/saveconfig.json
EOF
log "Tearing down LIO kernel objects — Pacemaker will restore via targetctl on the Active node..."
# LIO holds the backing file open; clear kernel state now so the XFS unmount succeeds.
# Use specific delete commands (clearconfig does not reliably clear kernel configfs state).
targetcli "/iscsi delete ${ISCSI_IQN}" 2>/dev/null || warn "LIO iscsi delete failed — umount may fail"
targetcli "/backstores/fileio delete ha-lun0" 2>/dev/null || warn "LIO backstore delete failed"
log "Distributing iSCSI saveconfig to $NODE2..."
n2_scp /etc/target/saveconfig.json /etc/target/saveconfig.json
log "Unmounting ${XFS_MOUNT} — Pacemaker manages it..."
umount "${XFS_MOUNT}" || { sync; umount -l "${XFS_MOUNT}"; }
log "Demoting DRBD to Secondary — Pacemaker manages primary role..."
drbdadm role ha-data 2>/dev/null | grep -q "^Primary" && drbdadm secondary ha-data || true
# ── 7. Pacemaker resources ────────────────────────────────────────────────
log "Configuring Pacemaker cluster properties..."
crm_attribute -t crm_config -n stonith-enabled -v false
crm_attribute -t crm_config -n no-quorum-policy -v ignore
log "Creating Pacemaker resources via cibadmin..."
# Use cibadmin --replace with pacemaker-4.0-compatible XML.
# Key schema rules for pacemaker-4.0:
# - globally-unique must be in <meta_attributes>, not a direct <clone> attribute
# - promoted-max / promoted-node-max (not master-max / master-node-max)
# - constraint with-rsc-role="Promoted" (not "Master")
cibadmin --replace --scope resources --xml-text '<resources>
<clone id="ms-drbd0">
<meta_attributes id="ms-drbd0-meta">
<nvpair id="ms-drbd0-globally-unique" name="globally-unique" value="false"/>
<nvpair id="ms-drbd0-promotable" name="promotable" value="true"/>
<nvpair id="ms-drbd0-promoted-max" name="promoted-max" value="1"/>
<nvpair id="ms-drbd0-promoted-node-max" name="promoted-node-max" value="1"/>
<nvpair id="ms-drbd0-clone-max" name="clone-max" value="2"/>
<nvpair id="ms-drbd0-clone-node-max" name="clone-node-max" value="1"/>
<nvpair id="ms-drbd0-notify" name="notify" value="true"/>
<nvpair id="ms-drbd0-interleave" name="interleave" value="true"/>
</meta_attributes>
<primitive id="drbd0" class="ocf" type="drbd" provider="linbit">
<instance_attributes id="drbd0-attrs">
<nvpair id="drbd0-resource" name="drbd_resource" value="ha-data"/>
</instance_attributes>
<operations>
<op id="drbd0-start" name="start" interval="0" timeout="240s"/>
<op id="drbd0-stop" name="stop" interval="0" timeout="120s"/>
<op id="drbd0-promote" name="promote" interval="0" timeout="240s"/>
<op id="drbd0-demote" name="demote" interval="0" timeout="90s"/>
<op id="drbd0-monitor-promoted" name="monitor" interval="20s" timeout="20s" role="Promoted"/>
<op id="drbd0-monitor-unpromoted" name="monitor" interval="30s" timeout="20s" role="Unpromoted"/>
</operations>
</primitive>
</clone>
<group id="ha-group">
<primitive id="xfs-data" class="ocf" type="Filesystem" provider="heartbeat">
<instance_attributes id="xfs-data-attrs">
<nvpair id="xfs-data-device" name="device" value="/dev/drbd0"/>
<nvpair id="xfs-data-directory" name="directory" value="/srv/ha-data"/>
<nvpair id="xfs-data-fstype" name="fstype" value="xfs"/>
<nvpair id="xfs-data-options" name="options" value="defaults"/>
<nvpair id="xfs-data-force_unmount" name="force_unmount" value="true"/>
</instance_attributes>
<operations>
<op id="xfs-data-start" name="start" interval="0" timeout="60s"/>
<op id="xfs-data-stop" name="stop" interval="0" timeout="60s"/>
<op id="xfs-data-monitor" name="monitor" interval="20s" timeout="40s"/>
</operations>
</primitive>
<primitive id="iscsi-target" class="systemd" type="targetctl">
<operations>
<op id="iscsi-start" name="start" interval="0" timeout="60s"/>
<op id="iscsi-stop" name="stop" interval="0" timeout="60s"/>
<op id="iscsi-monitor" name="monitor" interval="20s" timeout="40s"/>
</operations>
</primitive>
<primitive id="nfs-server" class="systemd" type="nfs-server">
<operations>
<op id="nfs-start" name="start" interval="0" timeout="60s"/>
<op id="nfs-stop" name="stop" interval="0" timeout="60s"/>
<op id="nfs-monitor" name="monitor" interval="30s" timeout="40s"/>
</operations>
</primitive>
<primitive id="vip" class="ocf" type="IPaddr2" provider="heartbeat">
<instance_attributes id="vip-attrs">
<nvpair id="vip-ip" name="ip" value="192.168.2.229"/>
<nvpair id="vip-cidr" name="cidr_netmask" value="24"/>
</instance_attributes>
<operations>
<op id="vip-start" name="start" interval="0" timeout="20s"/>
<op id="vip-stop" name="stop" interval="0" timeout="20s"/>
<op id="vip-monitor" name="monitor" interval="10s" timeout="20s"/>
</operations>
</primitive>
</group>
</resources>'
log "Adding Pacemaker ordering and colocation constraints..."
cibadmin --replace --scope constraints --xml-text '<constraints>
<rsc_order id="order-drbd-group" first="ms-drbd0" first-action="promote" then="ha-group" then-action="start" kind="Mandatory"/>
<rsc_colocation id="coloc-group-with-drbd" score="INFINITY" rsc="ha-group" with-rsc="ms-drbd0" with-rsc-role="Promoted"/>
</constraints>'
log "Clearing stale Pacemaker failure history..."
crm_resource --cleanup 2>/dev/null || true
log "Waiting for resources to start..."
for i in $(seq 1 60); do
if crm_resource -r vip --locate 2>/dev/null | grep -q "running on"; then
log "VIP is up: $(crm_resource -r vip --locate)"
break
fi
[[ $i -eq 60 ]] && { warn "VIP not up after 120 s — check: crm_mon -1"; break; }
sleep 2
done
log ""
log "═══════════════════════════════════════════════════════════════"
log " HA cluster initialised."
log ""
log " crm_mon -1 — cluster status"
log " iscsiadm -m discovery -t st -p ${VIP} — verify iSCSI target"
log " showmount -e ${VIP} — verify NFS exports"
log ""
log " To enable STONITH (after deploying fence SSH key):"
log " 1. Fill in VMID_NODE1 / VMID_NODE2 in cluster-enable-stonith.sh"
log " 2. Copy scripts/ha/fence-pve-ssh.py to /etc/pacemaker/fence_pve_ssh"
log " on both nodes (chmod +x)"
log " 3. Generate and distribute the fence SSH key"
log " (see docs or cluster-enable-stonith.sh header)"
log " 4. bash scripts/ha/cluster-enable-stonith.sh"
log "═══════════════════════════════════════════════════════════════"
+499
View File
@@ -0,0 +1,499 @@
#!/usr/bin/env bash
# deploy.sh — Full lifecycle management for the HA file-server cluster.
#
# Handles everything from zero (no VMs, no secrets) through a running,
# tested cluster, and optionally tears it back down.
#
# Usage:
# scripts/ha/deploy.sh [options]
# scripts/ha/deploy.sh --destroy [options]
#
# Phases (all run by default; skip any with --skip-*):
# 1. ensure-bridge Create storage bridge (vmbr1) on the Proxmox node if absent.
# 2. sync-keys Generate SSH host keys and register age keys for both nodes.
# 3. create-vms Build disk images and create both VMs via create-proxmox-resource.sh.
# 4. add-hardware Attach storage NIC (vmbr1) and DRBD data disk to each VM.
# 5. boot-wait Start VMs, wait for SSH on both nodes.
# 6. cluster-init Form the cluster: DRBD, corosync, Pacemaker, NFS, VIP.
# Also encrypts the generated corosync authkey into the repo.
# 7. run-tests Run acceptance tests (T1T7).
#
# Options:
# --node <host> Proxmox host to deploy on (default: pve1.sweet.home)
# --vmid1 <n> VMID for ha-server-1 (default: 200)
# --vmid2 <n> VMID for ha-server-2 (default: 201)
# --storage <pool> Proxmox storage pool (default: local-zfs)
# --storage-bridge <br> Bridge for HA storage network (default: vmbr1)
# --drbd-disk-gb <n> DRBD data disk size in GB (default: 32)
# --memory <MB> RAM per node (default: 4096)
# --cores <n> vCPUs per node (default: 4)
# --skip-ensure-bridge Skip storage bridge creation/check
# --skip-sync-keys Skip sync-host-keys.sh (clan vars already exist)
# --skip-create-vms Skip VM creation (VMs already exist)
# --skip-add-hardware Skip net1/scsi1 attachment (already attached)
# --skip-boot-wait Skip boot/SSH wait (VMs already running)
# --skip-refresh-sops-keys Skip scanning running VMs for fresh SSH host keys
# --skip-cluster-init Skip cluster formation (cluster already configured)
# --skip-tests Skip acceptance tests
# --force-rebuild Pass --force-rebuild to create-proxmox-resource.sh
# --destroy Stop and delete both VMs (skip all other phases)
# --dry-run Print what would run without executing
# -h|--help Show this message
#
# Prerequisites:
# - SSH access to the Proxmox node as $PROXMOX_SSH_USER (wayne).
# - sops age key in the standard location (used by sync-host-keys.sh).
# - For --skip-sync-keys: clan vars already in vars/per-machine/proxmox-ha-server-{1,2}/.
# - For full tests: secrets/common.yaml decryptable on both nodes (run
# `sops updatekeys secrets/common.yaml` after sync-keys).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# shellcheck source=../env.sh
source "${REPO_ROOT}/scripts/env.sh"
# ── Defaults ──────────────────────────────────────────────────────────────────
NODE="${PROXMOX_HOST:-$PVE1_HOST}"
VMID1=200
VMID2=201
STORAGE="${PROXMOX_STORAGE:-local-zfs}"
STORAGE_BRIDGE="vmbr1"
DRBD_DISK_GB=32
MEMORY_MB=4096
CORES=4
SKIP_ENSURE_BRIDGE=false
SKIP_SYNC_KEYS=false
SKIP_CREATE_VMS=false
SKIP_ADD_HARDWARE=false
SKIP_BOOT_WAIT=false
SKIP_REFRESH_SOPS_KEYS=false
SKIP_CLUSTER_INIT=false
SKIP_TESTS=false
FORCE_REBUILD=false
DESTROY=false
DRY_RUN=false
# ── Variables from repo ───────────────────────────────────────────────────────
NODE1_HOST="ha-server-1"
NODE2_HOST="ha-server-2"
NODE1_IP="192.168.2.228"
NODE2_IP="192.168.2.227"
STORAGE_IP1="192.168.4.228"
STORAGE_IP2="192.168.4.227"
STORAGE_CIDR="192.168.4.0/29"
SSH_USER="${PROXMOX_SSH_USER:-wayne}"
# ── Argument parsing ──────────────────────────────────────────────────────────
usage() {
sed -n '/^# Usage:/,/^[^#]/{ /^#/{ s/^# \?//; p } }' "$0"
exit "${1:-0}"
}
while [[ $# -gt 0 ]]; do
case "$1" in
--node) NODE="$2"; shift 2 ;;
--vmid1) VMID1="$2"; shift 2 ;;
--vmid2) VMID2="$2"; shift 2 ;;
--storage) STORAGE="$2"; shift 2 ;;
--storage-bridge) STORAGE_BRIDGE="$2"; shift 2 ;;
--drbd-disk-gb) DRBD_DISK_GB="$2"; shift 2 ;;
--memory) MEMORY_MB="$2"; shift 2 ;;
--cores) CORES="$2"; shift 2 ;;
--skip-ensure-bridge) SKIP_ENSURE_BRIDGE=true; shift ;;
--skip-sync-keys) SKIP_SYNC_KEYS=true; shift ;;
--skip-create-vms) SKIP_CREATE_VMS=true; shift ;;
--skip-add-hardware) SKIP_ADD_HARDWARE=true; shift ;;
--skip-boot-wait) SKIP_BOOT_WAIT=true; shift ;;
--skip-refresh-sops-keys) SKIP_REFRESH_SOPS_KEYS=true; shift ;;
--skip-cluster-init) SKIP_CLUSTER_INIT=true; shift ;;
--skip-tests) SKIP_TESTS=true; shift ;;
--force-rebuild) FORCE_REBUILD=true; shift ;;
--destroy) DESTROY=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage 0 ;;
*) echo "Unknown option: $1" >&2; usage 1 ;;
esac
done
# ── Helpers ───────────────────────────────────────────────────────────────────
log() { echo "==> $*"; }
logn() { echo " $*"; }
err() { echo "ERROR: $*" >&2; exit 1; }
run() {
if $DRY_RUN; then
echo "[dry-run] $*"
else
"$@"
fi
}
pve() {
# Run a command on the Proxmox node via SSH.
if $DRY_RUN; then
echo "[dry-run] ssh ${SSH_USER}@${NODE} sudo $*"
else
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
fi
}
pve_check() {
# Run a read-only probe on the Proxmox node — always executes even in dry-run.
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
}
HA_USER="nixos"
n1() {
# Run a command on ha-server-1 via SSH as nixos with sudo.
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null
}
n2() {
# Run a command on ha-server-2 via SSH as nixos with sudo.
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null
}
wait_for_ssh() {
local ip="$1" label="$2"
if $DRY_RUN; then
logn "[dry-run] Skipping SSH wait for ${label} (${ip})"
return 0
fi
local deadline=$(( $(date +%s) + 300 ))
log "Waiting for SSH on ${label} (${ip}) — up to 5 min..."
while [[ $(date +%s) -lt $deadline ]]; do
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=3 \
-o BatchMode=yes "${HA_USER}@${ip}" true 2>/dev/null; then
logn "${label} is up."
return 0
fi
sleep 5
done
err "Timed out waiting for SSH on ${label} (${ip})"
}
# ── Destroy mode ─────────────────────────────────────────────────────────────
if $DESTROY; then
log "Destroying HA cluster VMs (${VMID1}=${NODE1_HOST}, ${VMID2}=${NODE2_HOST}) on ${NODE}"
for vmid in "$VMID1" "$VMID2"; do
STATUS=$(pve "qm status ${vmid} 2>/dev/null" 2>/dev/null || true)
if echo "$STATUS" | grep -q "running"; then
log "Stopping VMID ${vmid}..."
pve "qm stop ${vmid} --skiplock 1"
sleep 5
fi
if $DRY_RUN || pve "qm config ${vmid} >/dev/null 2>&1"; then
log "Deleting VMID ${vmid}..."
run pve "qm destroy ${vmid} --purge 1"
else
logn "VMID ${vmid} not found — already gone."
fi
done
log "Done — cluster VMs destroyed."
exit 0
fi
# ── Phase 1: Storage bridge ───────────────────────────────────────────────────
if ! $SKIP_ENSURE_BRIDGE; then
log "Phase 1: Ensuring storage bridge ${STORAGE_BRIDGE} on ${NODE}"
if pve_check "test -d /sys/class/net/${STORAGE_BRIDGE}" &>/dev/null; then
logn "${STORAGE_BRIDGE} already exists — skipping."
else
logn "Creating isolated internal bridge ${STORAGE_BRIDGE} (no upstream port, ${STORAGE_CIDR})"
BRIDGE_CONF="auto ${STORAGE_BRIDGE}
iface ${STORAGE_BRIDGE} inet manual
bridge-ports none
bridge-stp off
bridge-fd 0"
if $DRY_RUN; then
echo "[dry-run] Would write /etc/network/interfaces.d/${STORAGE_BRIDGE}.conf and ifup it"
else
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"echo '${BRIDGE_CONF}' | sudo tee /etc/network/interfaces.d/${STORAGE_BRIDGE}.conf > /dev/null && sudo ifup ${STORAGE_BRIDGE}"
logn "${STORAGE_BRIDGE} created and brought up."
fi
fi
fi
# ── Phase 2: Sync host keys ───────────────────────────────────────────────────
if ! $SKIP_SYNC_KEYS; then
log "Phase 2: Syncing SSH host keys for both HA targets"
for target in proxmox-ha-server-1 proxmox-ha-server-2; do
CLAN_DIR="${REPO_ROOT}/vars/per-machine/${target}/openssh"
if [[ -d "$CLAN_DIR" ]]; then
logn "Clan vars for ${target} already exist — skipping."
else
logn "Generating host keys for ${target}..."
run bash "${REPO_ROOT}/scripts/secrets/sync-host-keys.sh" "$target"
fi
done
fi
# ── Phase 2.5: Prepare Proxmox node for building ─────────────────────────────
if ! $SKIP_CREATE_VMS && ! $DRY_RUN; then
CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"
# Fix /nix ownership if it exists but belongs to a different UID.
# pve1's IPA-enrolled wayne (UID 50002) can't write to a store created by
# another UID — passwordless sudo corrects it once.
# Use direct SSH (no sudo) for the writability check so we test wayne's own
# access, not root's.
local_ssh() { ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "$*"; }
if local_ssh "test -d /nix" &>/dev/null && ! local_ssh "test -w /nix" &>/dev/null; then
logn "/nix exists but not writable by ${SSH_USER} — fixing ownership with sudo (one-time)..."
local_ssh "sudo chown -R ${SSH_USER} /nix"
logn "Done."
fi
unset -f local_ssh
# Ensure the remote clone is on the correct branch so create-proxmox-resource.sh
# builds from the same commits we're deploying.
REMOTE_REPO="/home/${SSH_USER}/nixos"
if pve_check "test -d ${REMOTE_REPO}/.git" &>/dev/null; then
REMOTE_BRANCH=$(ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"cd ${REMOTE_REPO} && git rev-parse --abbrev-ref HEAD 2>/dev/null")
if [[ "$REMOTE_BRANCH" != "$CURRENT_BRANCH" ]]; then
logn "Remote clone is on '${REMOTE_BRANCH}', switching to '${CURRENT_BRANCH}'..."
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"cd ${REMOTE_REPO} && git fetch origin && git checkout '${CURRENT_BRANCH}' && git pull --ff-only"
logn "Done."
fi
fi
fi
# ── Phase 3: Create VMs ───────────────────────────────────────────────────────
if ! $SKIP_CREATE_VMS; then
log "Phase 3: Building and creating VMs on ${NODE}"
CREATE="${REPO_ROOT}/scripts/proxmox/create-proxmox-resource.sh"
for spec in "${VMID1}:ha-server-1:proxmox-ha-server-1" "${VMID2}:ha-server-2:proxmox-ha-server-2"; do
IFS=: read -r vmid host_name flake_target <<< "$spec"
log "Creating ${flake_target} (VMID ${vmid}) on ${NODE}..."
# Always --force-rebuild: create-proxmox-resource.sh only calls
# sync_remote_host_keys (which populates host-keys/ for proxmox.nix to
# bake the clan-var SSH key into the disko image) when it actually builds.
# Reusing a cached image skips that step, so destroy+recreate would reuse
# an image with a stale/random key baked in → sops fails on first boot.
run bash "$CREATE" \
--type vm \
--host "$host_name" \
--vmid "$vmid" \
--node "$NODE" \
--storage "$STORAGE" \
--memory "$MEMORY_MB" \
--cores "$CORES" \
--force-rebuild
done
fi
# ── Phase 4: Add storage NIC and DRBD disk ────────────────────────────────────
if ! $SKIP_ADD_HARDWARE; then
log "Phase 4: Attaching storage NIC (${STORAGE_BRIDGE}) and DRBD disk (${DRBD_DISK_GB}G) to each VM"
for vmid in "$VMID1" "$VMID2"; do
log " VMID ${vmid}: stopping to add hardware..."
pve "qm stop ${vmid} --skiplock 1 2>/dev/null; sleep 3" || true
logn "Adding net1 (${STORAGE_BRIDGE})..."
pve "qm set ${vmid} --net1 virtio,bridge=${STORAGE_BRIDGE},firewall=0"
logn "Adding scsi1 (${STORAGE}:${DRBD_DISK_GB}G for DRBD)..."
pve "qm set ${vmid} --scsi1 ${STORAGE}:${DRBD_DISK_GB},format=raw"
logn "Starting VMID ${vmid}..."
pve "qm start ${vmid}"
done
fi
# ── Phase 5: Wait for SSH ─────────────────────────────────────────────────────
if ! $SKIP_BOOT_WAIT; then
log "Phase 5: Waiting for both nodes to come up"
wait_for_ssh "$NODE1_IP" "$NODE1_HOST"
wait_for_ssh "$NODE2_IP" "$NODE2_HOST"
logn "Both nodes are SSHable."
# Give systemd a few seconds to settle after activation
sleep 10
fi
# ── Phase 5.5: Refresh sops host-key registrations ───────────────────────────
#
# Disko builds raw disk images; each new VM boots with a freshly-generated SSH
# host key rather than the one pre-seeded in clan vars. This phase scans the
# actual running VMs, and if their ed25519 host keys differ from what clan vars
# record: updates the clan var pub-key files, rewrites the .sops.yaml age-key
# anchors, and re-encrypts all affected sops files so the nodes can decrypt
# secrets on the next nixos-rebuild. Safe no-op when keys haven't changed.
if ! $SKIP_REFRESH_SOPS_KEYS; then
if $DRY_RUN; then
logn "[dry-run] Would scan VM host keys and refresh .sops.yaml / secrets if needed"
else
log "Phase 5.5: Refreshing sops host-key registrations (disko key drift fix)"
SOPS_UPDATED=false
for spec in \
"${NODE1_IP}:proxmox-ha-server-1:${NODE1_HOST}" \
"${NODE2_IP}:proxmox-ha-server-2:${NODE2_HOST}"; do
IFS=: read -r node_ip flake_target host_name <<< "$spec"
CLAN_PUB="${REPO_ROOT}/vars/per-machine/${flake_target}/openssh/ssh_host_ed25519_key.pub/value"
logn "Scanning ed25519 host key from ${host_name} (${node_ip})..."
RAW=$(ssh-keyscan -t ed25519 "${node_ip}" 2>/dev/null | grep -v "^#") || true
if [[ -z "$RAW" ]]; then
logn "WARNING: no ed25519 key returned by ssh-keyscan for ${node_ip} — skipping"
continue
fi
# ssh-keyscan returns: <ip> ssh-ed25519 <b64key>
SCANNED_TYPE=$(awk '{print $2}' <<< "$RAW")
SCANNED_KEY=$(awk '{print $3}' <<< "$RAW")
SCANNED_PUBKEY="${SCANNED_TYPE} ${SCANNED_KEY} ${host_name}"
CURRENT=$(tr -d '\n' < "$CLAN_PUB" 2>/dev/null || true)
if [[ "$SCANNED_PUBKEY" == "$CURRENT" ]]; then
logn "${host_name}: clan var matches running key — no update needed"
continue
fi
logn "${host_name}: key drift detected — updating clan var"
logn " old: ${CURRENT}"
logn " new: ${SCANNED_PUBKEY}"
echo "$SCANNED_PUBKEY" > "$CLAN_PUB"
SOPS_UPDATED=true
# Rewrite the .sops.yaml anchor for this host with the new age key.
ANCHOR="${flake_target}" # e.g. proxmox-ha-server-1
NEW_AGE=$(echo "$SCANNED_PUBKEY" | \
nix run --quiet --no-warn-dirty nixpkgs#ssh-to-age 2>/dev/null)
if [[ -z "$NEW_AGE" ]]; then
err "ssh-to-age produced no output for ${host_name} — check nixpkgs#ssh-to-age"
fi
logn " new age key: ${NEW_AGE}"
sed -i "/&${ANCHOR} /s| age[a-z0-9]*$| ${NEW_AGE}|" "${REPO_ROOT}/.sops.yaml"
done
if $SOPS_UPDATED; then
logn "Running sops updatekeys on affected secrets..."
SOPS="nix run --quiet --no-warn-dirty nixpkgs#sops --"
(cd "${REPO_ROOT}" && \
$SOPS updatekeys -y secrets/common.yaml && \
$SOPS updatekeys -y secrets/ha-server-1.yaml && \
$SOPS updatekeys -y secrets/ha-server-2.yaml && \
$SOPS updatekeys -y secrets/ha-server-1.keytab && \
$SOPS updatekeys -y secrets/ha-server-2.keytab)
# Note: ha-corosync-authkey is re-generated and re-encrypted by cluster-init below.
logn "Committing refreshed host keys and re-encrypted secrets..."
(cd "${REPO_ROOT}" && \
git add \
vars/per-machine/proxmox-ha-server-1/openssh/ssh_host_ed25519_key.pub/value \
vars/per-machine/proxmox-ha-server-2/openssh/ssh_host_ed25519_key.pub/value \
.sops.yaml \
secrets/common.yaml \
secrets/ha-server-1.yaml \
secrets/ha-server-2.yaml \
secrets/ha-server-1.keytab \
secrets/ha-server-2.keytab && \
git commit -m "secrets(ha): refresh sops host-key registrations for new VM instances" || true)
logn "Sops keys refreshed and committed."
fi
fi
fi
# ── Phase 6: Cluster init ─────────────────────────────────────────────────────
if ! $SKIP_CLUSTER_INIT; then
log "Phase 6: Initialising HA cluster"
CLUSTER_INIT="${REPO_ROOT}/scripts/ha/cluster-init.sh"
[[ -x "$CLUSTER_INIT" ]] || chmod +x "$CLUSTER_INIT"
if $DRY_RUN; then
logn "[dry-run] Would generate temp key, authorise on ${NODE2_HOST}, scp cluster-init.sh to ${NODE1_HOST}, and run it as root via sudo"
else
# Generate a temp keypair so cluster-init.sh can SSH node1→node2 as ${HA_USER}.
# Root on node1 has no keys; a temp key bridging node1→node2 nixos solves this.
TEMP_KEY="${REPO_ROOT}/.tmp-cluster-init-key"
TEMP_KEY_PUB="${TEMP_KEY}.pub"
rm -f "$TEMP_KEY" "$TEMP_KEY_PUB"
ssh-keygen -t ed25519 -f "$TEMP_KEY" -N "" -C "cluster-init-temp-$(date +%s)" -q
TEMP_PUBKEY=$(cat "$TEMP_KEY_PUB")
logn "Authorising temp key on ${NODE2_HOST} for ${HA_USER}..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE2_IP}" \
"mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo '${TEMP_PUBKEY}' >> ~/.ssh/authorized_keys"
logn "Placing temp key on ${NODE1_HOST} for root..."
scp -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"$TEMP_KEY" "${HA_USER}@${NODE1_IP}:/tmp/cluster-init-key"
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo mkdir -p /root/.ssh && sudo cp /tmp/cluster-init-key /root/.ssh/cluster-init-key && \
sudo chmod 600 /root/.ssh/cluster-init-key && rm -f /tmp/cluster-init-key"
logn "Uploading cluster-init.sh to ${NODE1_HOST}..."
scp -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"$CLUSTER_INIT" "${HA_USER}@${NODE1_IP}:/tmp/cluster-init.sh"
logn "Running cluster-init.sh on ${NODE1_HOST}..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo env NODE1=${NODE1_HOST} NODE2=${NODE2_HOST} \
NODE1_IP=${NODE1_IP} NODE2_IP=${NODE2_IP} \
VIP=192.168.2.229 XFS_MOUNT=/srv/ha-data \
ISCSI_IQN=iqn.2026-01.home.sweet:ha-storage \
VMID_NODE1=${VMID1} VMID_NODE2=${VMID2} \
HA_USER=${HA_USER} HA_KEY=/root/.ssh/cluster-init-key \
bash /tmp/cluster-init.sh"
logn "Cleaning up temp key from both nodes..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE2_IP}" \
"sed -i '/cluster-init-temp/d' ~/.ssh/authorized_keys" 2>/dev/null || true
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo rm -f /root/.ssh/cluster-init-key" 2>/dev/null || true
rm -f "$TEMP_KEY" "$TEMP_KEY_PUB"
# Encrypt the corosync authkey generated by cluster-init and commit it.
log " Encrypting corosync authkey into secrets/ha-corosync-authkey..."
AUTHKEY_TMP="${REPO_ROOT}/secrets/ha-corosync-authkey.tmp"
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo cat /etc/corosync/authkey" > "$AUTHKEY_TMP"
if [[ ! -s "$AUTHKEY_TMP" ]]; then
err "corosync authkey on node1 is empty — cluster-init may have failed."
fi
mv "$AUTHKEY_TMP" "${REPO_ROOT}/secrets/ha-corosync-authkey"
(cd "${REPO_ROOT}" && nix run nixpkgs#sops -- -e --input-type binary -i secrets/ha-corosync-authkey)
logn "Authkey encrypted. Committing..."
(cd "${REPO_ROOT}" && git add secrets/ha-corosync-authkey && \
git commit -m "secrets(ha): encrypt corosync authkey generated by cluster-init")
logn "Committed."
fi
fi
# ── Phase 7: Acceptance tests ─────────────────────────────────────────────────
if ! $SKIP_TESTS; then
log "Phase 7: Running acceptance tests (T1T7)"
if $DRY_RUN; then
logn "[dry-run] Would run acceptance-tests.sh against ${NODE1_HOST}/${NODE2_HOST}"
else
NODE1="$NODE1_HOST" NODE2="$NODE2_HOST" \
NODE1_IP="$NODE1_IP" NODE2_IP="$NODE2_IP" \
VIP="192.168.2.229" \
bash "${REPO_ROOT}/scripts/ha/acceptance-tests.sh"
fi
fi
log "Deploy complete."
+256
View File
@@ -0,0 +1,256 @@
#!/usr/bin/env bash
# failover.sh — graceful HA cluster failover
#
# Detects which node is active and moves all resources to the other node by
# putting the active node into Pacemaker standby. Waits for the XFS mount to
# appear on the target before returning.
#
# Usage:
# scripts/ha/failover.sh [--to node1|node2] [--force] [--timeout <s>] [--dry-run]
#
# --to node1|node2 target node (default: the node that is NOT currently active)
# --force skip the interactive confirmation prompt
# --timeout <s> seconds to wait for resources to move (default: 120)
# --dry-run show what would be done without changing anything
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}"
NODE2_IP="${NODE2_IP:-192.168.2.227}"
VIP="${VIP:-192.168.2.229}"
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"
HA_USER="${HA_USER:-nixos}"
# ──────────────────────────────────────────────────────────────────────────
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
# ── Argument parsing ───────────────────────────────────────────────────────
TARGET_NODE=""
FORCE=false
DRY_RUN=false
TIMEOUT=120
while [[ $# -gt 0 ]]; do
case "$1" in
--to)
shift
case "${1:-}" in
node1|ha-server-1) TARGET_NODE="$NODE1" ;;
node2|ha-server-2) TARGET_NODE="$NODE2" ;;
*) echo "ERROR: --to must be node1, node2, ha-server-1, or ha-server-2"; exit 1 ;;
esac
;;
--force) FORCE=true ;;
--dry-run) DRY_RUN=true ;;
--timeout) shift; TIMEOUT="${1:?--timeout requires a value}" ;;
*) echo "Unknown argument: $1"; echo "Usage: $0 [--to node1|node2] [--force] [--timeout <s>] [--dry-run]"; exit 1 ;;
esac
shift
done
DRY_PREFIX=""
$DRY_RUN && DRY_PREFIX="[dry-run] "
echo "════════════════════════════════════════════════════"
echo " HA Cluster Failover — $(date '+%Y-%m-%d %H:%M:%S')"
$DRY_RUN && echo " MODE: dry-run — no changes will be made"
echo "════════════════════════════════════════════════════"
# ── Detect active node ─────────────────────────────────────────────────────
echo ""
echo "Detecting active node..."
CRM_OUT=""
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" true 2>/dev/null; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
fi
if [[ -z "$CRM_OUT" ]]; then
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" true 2>/dev/null; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
fi
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" — the * bullet
# means ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -z "$ACTIVE_NODE" ]]; then
echo ""
echo "ERROR: could not determine active node from crm_mon."
echo " Is Pacemaker still settling? Try running scripts/ha/health.sh first."
echo " If Pacemaker is down on both nodes, manual recovery is required."
exit 1
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
STANDBY_NODE="$NODE2"
STANDBY_IP="$NODE2_IP"
na() { n1 "$@"; }
ns() { n2 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
STANDBY_NODE="$NODE1"
STANDBY_IP="$NODE1_IP"
na() { n2 "$@"; }
ns() { n1 "$@"; }
fi
echo " Active: $ACTIVE_NODE ($ACTIVE_IP)"
echo " Standby: $STANDBY_NODE ($STANDBY_IP)"
# ── Validate target ────────────────────────────────────────────────────────
if [[ -n "$TARGET_NODE" ]]; then
if [[ "$TARGET_NODE" == "$ACTIVE_NODE" ]]; then
echo ""
echo "ERROR: $TARGET_NODE is already the active node — nothing to do."
exit 1
fi
echo " Target: $TARGET_NODE (as requested)"
else
echo " Target: $STANDBY_NODE (auto — the other node)"
fi
# ── Pre-checks ─────────────────────────────────────────────────────────────
echo ""
echo "Pre-checks..."
DRBD_DSTATE=$(na "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_DSTATE" | grep -q "UpToDate/UpToDate"; then
echo ""
echo " WARNING: DRBD dstate is '$DRBD_DSTATE' (not UpToDate/UpToDate)."
echo " Failing over with a partially-synced disk risks split-brain."
if ! $FORCE; then
echo " Use --force to proceed anyway (not recommended)."
exit 1
fi
echo " --force specified — proceeding despite non-ideal DRBD state."
else
echo " DRBD dstate: $DRBD_DSTATE — OK"
fi
QUORUM_OK=$(na "corosync-quorumtool -s 2>/dev/null | grep -c 'Quorate:.*Yes'" 2>/dev/null || echo "0")
if [[ "$QUORUM_OK" -lt 1 ]]; then
echo " ERROR: cluster does not have quorum — failover would be unsafe."
exit 1
fi
echo " Quorum: OK"
# ── Confirm ────────────────────────────────────────────────────────────────
if ! $FORCE && ! $DRY_RUN; then
echo ""
echo " This will move all resources from $ACTIVE_NODE$STANDBY_NODE."
echo " VIP and services will be unreachable for ~1030 seconds."
printf " Proceed? [y/N] "
read -r ANSWER
[[ "${ANSWER,,}" == "y" || "${ANSWER,,}" == "yes" ]] || { echo "Aborted."; exit 0; }
fi
# ── Capture active node's crm_node name ───────────────────────────────────
# crm_node -n returns the node name as registered in Pacemaker (may differ
# from hostname if Pacemaker was configured with explicit node names).
ACTIVE_CRMD_NAME=$(na "crm_node -n 2>/dev/null" 2>/dev/null || echo "$ACTIVE_NODE")
# ── Perform failover ───────────────────────────────────────────────────────
echo ""
echo "${DRY_PREFIX}Putting $ACTIVE_NODE into standby (resources will migrate to $STANDBY_NODE)..."
if ! $DRY_RUN; then
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v on" 2>/dev/null || true
fi
# ── Wait for resources to move ─────────────────────────────────────────────
echo "${DRY_PREFIX}Waiting up to ${TIMEOUT}s for XFS to mount on $STANDBY_NODE..."
MOVED=false
SPIN_CHARS=('|' '/' '-' '\')
SPIN_I=0
if $DRY_RUN; then
echo " [dry-run] would wait for mountpoint $XFS_MOUNT on $STANDBY_NODE"
MOVED=true
else
for i in $(seq 1 "$TIMEOUT"); do
if ns "mountpoint -q '${XFS_MOUNT}' 2>/dev/null" 2>/dev/null; then
printf "\r%-80s\r" ""
echo " Resources moved in ${i}s"
MOVED=true
break
fi
SPIN_I=$(( SPIN_I + 1 ))
SC="${SPIN_CHARS[$((SPIN_I % 4))]}"
printf "\r [%s] waiting... (%ds) " "$SC" "$i"
sleep 1
done
fi
if ! $MOVED; then
echo ""
echo "ERROR: XFS did not mount on $STANDBY_NODE within ${TIMEOUT}s."
echo ""
echo " Current resource state:"
na "crm_mon -1 2>/dev/null" 2>/dev/null | grep -E 'Started|Stopped|Promoted|Unpromoted|FAILED' | sed 's/^/ /' || true
echo ""
echo " Clearing standby to restore $ACTIVE_NODE (undo the failover attempt)..."
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
na "crm_resource --cleanup" 2>/dev/null || true
exit 1
fi
# ── Clear failure history ──────────────────────────────────────────────────
echo "${DRY_PREFIX}Clearing Pacemaker failure history..."
if ! $DRY_RUN; then
ns "crm_resource --cleanup 2>/dev/null" 2>/dev/null || true
fi
# ── Re-enable original active node as standby ─────────────────────────────
echo "${DRY_PREFIX}Re-enabling $ACTIVE_NODE (now standby — will not claim resources)..."
if ! $DRY_RUN; then
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
fi
# ── Wait briefly for DRBD resync to begin ─────────────────────────────────
if ! $DRY_RUN; then
sleep 5
fi
# ── Final state ────────────────────────────────────────────────────────────
echo ""
echo "Failover complete. Final state:"
echo ""
CRM_OUT_AFTER=""
if ! $DRY_RUN; then
CRM_OUT_AFTER=$(ns "crm_mon -1" 2>/dev/null || na "crm_mon -1" 2>/dev/null || true)
else
CRM_OUT_AFTER="$CRM_OUT"
fi
NEW_ACTIVE=$(echo "$CRM_OUT_AFTER" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -n "$NEW_ACTIVE" ]]; then
if [[ "$NEW_ACTIVE" == "$ACTIVE_NODE" ]]; then
echo " WARNING: $ACTIVE_NODE is still showing as active in crm_mon."
echo " Pacemaker may still be settling — check again in a few seconds."
else
echo " Active: $NEW_ACTIVE"
echo " Standby: $ACTIVE_NODE"
fi
fi
echo ""
RESOURCES_AFTER=$(echo "$CRM_OUT_AFTER" | awk '/Full List of Resources/,0' | tail -n +2 || true)
[[ -z "$RESOURCES_AFTER" ]] && RESOURCES_AFTER=$(echo "$CRM_OUT_AFTER" | \
grep -E 'Started|Stopped|Promoted|Unpromoted|FAILED|Master|Slave' || true)
[[ -n "$RESOURCES_AFTER" ]] && echo "$RESOURCES_AFTER" | sed 's/^/ /'
echo ""
echo " (DRBD resync of $ACTIVE_NODE may take a moment; monitor with:"
echo " ssh nixos@${ACTIVE_IP} 'sudo watch -n3 cat /proc/drbd')"
echo ""
echo "════════════════════════════════════════════════════"
+179
View File
@@ -0,0 +1,179 @@
#!/usr/bin/env python3
"""
fence_pve_ssh - Proxmox VE SSH fence agent for Pacemaker.
Uses SSH to reach the Proxmox host and run 'qm stop/start <vmid>'.
Deploy to /etc/pacemaker/fence_pve_ssh on both HA nodes (chmod +x).
Configuration (as pacemaker stonith resource attributes):
pve_host Proxmox host to SSH to (default: pve1.sweet.home)
pve_user SSH user (default: wayne)
key_file SSH private key path (default: /etc/fence-pve-ssh-key)
vmid_node1 VMID for ha-server-1
vmid_node2 VMID for ha-server-2
plug Node name to act on (set by pacemaker: ha-server-1 or ha-server-2)
action Action: off|on|reboot|status|list|metadata
"""
import argparse
import subprocess
import sys
import os
METADATA = """<?xml version="1.0" ?>
<resource-agent name="fence_pve_ssh" shortdesc="Proxmox VE SSH fence agent (test lab)">
<longdesc>Fences a VM on a Proxmox VE host by SSHing to the PVE host and
running qm stop/start. For test use only.</longdesc>
<vendor-url>https://proxmox.com</vendor-url>
<parameters>
<parameter name="action" required="1" unique="0">
<getopt mixed="-a, --action=[action]"/>
<content type="string" default="reboot"/>
<shortdesc lang="en">Fencing action: off|on|reboot|status|list</shortdesc>
</parameter>
<parameter name="plug" required="0" unique="0">
<getopt mixed="-n, --plug=[nodename]"/>
<content type="string"/>
<shortdesc lang="en">Cluster node name to fence</shortdesc>
</parameter>
<parameter name="pve_host" required="0" unique="0">
<getopt mixed="--pve-host=[host]"/>
<content type="string" default="pve1.sweet.home"/>
<shortdesc lang="en">Proxmox VE host to SSH to</shortdesc>
</parameter>
<parameter name="pve_user" required="0" unique="0">
<getopt mixed="--pve-user=[user]"/>
<content type="string" default="wayne"/>
<shortdesc lang="en">SSH user on the Proxmox host</shortdesc>
</parameter>
<parameter name="key_file" required="0" unique="0">
<getopt mixed="--key-file=[path]"/>
<content type="string" default="/etc/fence-pve-ssh-key"/>
<shortdesc lang="en">SSH private key file path</shortdesc>
</parameter>
<parameter name="vmid_node1" required="1" unique="0">
<getopt mixed="--vmid-node1=[vmid]"/>
<content type="string"/>
<shortdesc lang="en">VMID for ha-test-node1</shortdesc>
</parameter>
<parameter name="vmid_node2" required="1" unique="0">
<getopt mixed="--vmid-node2=[vmid]"/>
<content type="string"/>
<shortdesc lang="en">VMID for ha-test-node2</shortdesc>
</parameter>
</parameters>
<actions>
<action name="off" timeout="60s"/>
<action name="on" timeout="60s"/>
<action name="reboot" timeout="60s"/>
<action name="status" timeout="30s"/>
<action name="list" timeout="10s"/>
<action name="metadata" timeout="5s"/>
</actions>
</resource-agent>
"""
def parse_args():
p = argparse.ArgumentParser(add_help=False)
p.add_argument("-a", "--action", default="reboot")
p.add_argument("-n", "--plug")
p.add_argument("--pve-host", default="pve1.sweet.home")
p.add_argument("--pve-user", default="wayne")
p.add_argument("--key-file", default="/etc/fence-pve-ssh-key")
p.add_argument("--vmid-node1")
p.add_argument("--vmid-node2")
# Allow remaining unknown args (pacemaker may pass extra ones)
return p.parse_known_args()[0]
def ssh(pve_host, pve_user, key_file, cmd):
result = subprocess.run(
[
"ssh",
"-i", key_file,
"-o", "StrictHostKeyChecking=no",
"-o", "BatchMode=yes",
"-o", "ConnectTimeout=10",
f"{pve_user}@{pve_host}",
cmd,
],
capture_output=True,
text=True,
timeout=30,
)
return result
def get_vmid(args):
node = args.plug
if not node:
print("ERROR: --plug not specified", file=sys.stderr)
sys.exit(1)
mapping = {
"ha-server-1": args.vmid_node1,
"ha-server-2": args.vmid_node2,
}
vmid = mapping.get(node)
if not vmid:
print(f"ERROR: unknown node '{node}'", file=sys.stderr)
sys.exit(1)
return vmid
def main():
args = parse_args()
action = args.action.lower()
if action == "metadata":
print(METADATA)
sys.exit(0)
if action == "list":
if args.vmid_node1:
print("ha-server-1")
if args.vmid_node2:
print("ha-server-2")
sys.exit(0)
vmid = get_vmid(args)
if not os.path.exists(args.key_file):
print(f"ERROR: SSH key not found at {args.key_file}", file=sys.stderr)
sys.exit(1)
if action in ("off", "reboot"):
print(f"Stopping VM {vmid} ({args.plug}) on {args.pve_host}...")
r = ssh(args.pve_host, args.pve_user, args.key_file,
f"sudo /usr/sbin/qm stop {vmid}")
if r.returncode != 0:
print(f"ERROR stopping VM: {r.stderr}", file=sys.stderr)
sys.exit(1)
print(f"VM {vmid} stopped")
if action in ("on", "reboot"):
print(f"Starting VM {vmid} ({args.plug}) on {args.pve_host}...")
r = ssh(args.pve_host, args.pve_user, args.key_file,
f"sudo /usr/sbin/qm start {vmid}")
if r.returncode != 0:
print(f"ERROR starting VM: {r.stderr}", file=sys.stderr)
sys.exit(1)
print(f"VM {vmid} started")
if action == "status":
r = ssh(args.pve_host, args.pve_user, args.key_file,
f"sudo /usr/sbin/qm status {vmid}")
if r.returncode != 0:
print(f"ERROR querying VM status: {r.stderr}", file=sys.stderr)
sys.exit(1)
# qm status returns "status: running" or "status: stopped"
status_line = r.stdout.strip()
print(status_line)
if "stopped" in status_line:
sys.exit(2) # pacemaker interprets exit 2 as "off"
sys.exit(0) # running = exit 0
if __name__ == "__main__":
main()
+180
View File
@@ -0,0 +1,180 @@
#!/usr/bin/env bash
# health.sh — HA cluster health snapshot (read-only, non-destructive)
#
# Prints a compact status panel across both nodes: SSH reachability, quorum,
# DRBD state, Pacemaker resources, and service ports via the VIP.
# Run from any host with SSH access to the HA nodes.
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.2.229}" # vars.haServerVip
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
HA_USER="${HA_USER:-nixos}"
# ──────────────────────────────────────────────────────────────────────────
REACHABLE_1=false
REACHABLE_2=false
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
probe_node() {
local ip=$1
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${ip}" true 2>/dev/null && echo "ONLINE" || echo "OFFLINE"
}
section() { echo ""; echo "── $* ──"; }
echo "════════════════════════════════════════════════════"
echo " HA Cluster Health — $(date '+%Y-%m-%d %H:%M:%S')"
echo "════════════════════════════════════════════════════"
# ── Node reachability ──────────────────────────────────────────────────────
section "Nodes"
N1_STATUS=$(probe_node "$NODE1_IP")
N2_STATUS=$(probe_node "$NODE2_IP")
[[ "$N1_STATUS" == "ONLINE" ]] && REACHABLE_1=true
[[ "$N2_STATUS" == "ONLINE" ]] && REACHABLE_2=true
if ! $REACHABLE_1 && ! $REACHABLE_2; then
echo " ERROR: both nodes unreachable — cannot continue."
exit 1
fi
# ── Detect active node ─────────────────────────────────────────────────────
# crm_mon 2.x formats the Promoted line as " * Promoted: [ node ]" — the
# bullet * means ^\s*(Promoted|Masters): never matches. Filter out Unpromoted
# first, then match anywhere on the line.
ACTIVE_NODE=""
CRM_OUT=""
if $REACHABLE_1; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
elif $REACHABLE_2; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
ACTIVE_NODE=$(echo "${CRM_OUT:-}" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
STANDBY_NODE=""
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
STANDBY_NODE="$NODE2"
elif [[ "$ACTIVE_NODE" == "$NODE2" ]]; then
STANDBY_NODE="$NODE1"
fi
n1_tag=""; n2_tag=""
[[ "$ACTIVE_NODE" == "$NODE1" ]] && n1_tag=" [ACTIVE]" || n1_tag=" [STANDBY]"
[[ "$ACTIVE_NODE" == "$NODE2" ]] && n2_tag=" [ACTIVE]" || n2_tag=" [STANDBY]"
[[ -z "$ACTIVE_NODE" ]] && { n1_tag=""; n2_tag=""; }
printf " %-14s [%s]%s\n" "$NODE1" "$N1_STATUS" "$n1_tag"
printf " %-14s [%s]%s\n" "$NODE2" "$N2_STATUS" "$n2_tag"
if [[ -z "$ACTIVE_NODE" ]]; then
echo ""
echo " WARNING: could not determine active node from crm_mon."
echo " Pacemaker may still be settling, or both nodes may be in standby."
fi
# ── Quorum ─────────────────────────────────────────────────────────────────
section "Quorum"
if $REACHABLE_1; then
QUORUM=$(n1 "corosync-quorumtool -s 2>/dev/null" 2>/dev/null || echo "")
elif $REACHABLE_2; then
QUORUM=$(n2 "corosync-quorumtool -s 2>/dev/null" 2>/dev/null || echo "")
fi
if [[ -z "${QUORUM:-}" ]]; then
echo " corosync-quorumtool: unavailable"
else
QUORATE=$(echo "$QUORUM" | grep "Quorate:" | awk '{print $2}' || echo "?")
VOTES=$(echo "$QUORUM" | grep "Total votes:" | awk '{print $3}' || echo "?")
NEEDED=$(echo "$QUORUM" | grep "Quorum votes:" | awk '{print $3}' || echo "?")
echo " Quorate: $QUORATE Votes: $VOTES / Expected: $NEEDED"
fi
# ── DRBD ───────────────────────────────────────────────────────────────────
section "DRBD (ha-data)"
drbd_info_from() {
local node=$1 run=$2
local role dstate cs pct
role=$($run "drbdadm role ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
dstate=$($run "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
cs=$($run "grep -oE 'cs:[A-Za-z]+' /proc/drbd 2>/dev/null | head -1 | sed 's/cs://'" 2>/dev/null || echo "unknown")
pct=$($run "grep -oE \"sync'ed:[[:space:]]+[0-9.]+\" /proc/drbd 2>/dev/null | grep -oE '[0-9.]+$' | head -1" 2>/dev/null || echo "")
printf " %-14s role: %-22s dstate: %-25s cs: %s" \
"$node" "${role:-unknown}" "${dstate:-unknown}" "${cs:-unknown}"
[[ -n "$pct" ]] && printf " syncing: %s%%" "$pct"
echo ""
}
$REACHABLE_1 && drbd_info_from "$NODE1" n1 || echo " $NODE1 [OFFLINE]"
$REACHABLE_2 && drbd_info_from "$NODE2" n2 || echo " $NODE2 [OFFLINE]"
# ── Pacemaker (full crm_mon output) ───────────────────────────────────────
section "Pacemaker"
if [[ -n "${CRM_OUT:-}" ]]; then
echo "$CRM_OUT" | sed 's/^/ /'
else
echo " crm_mon returned no output — trying again without suppression:"
if $REACHABLE_1; then
n1 "crm_mon -1" || true
elif $REACHABLE_2; then
n2 "crm_mon -1" || true
fi
fi
# ── XFS mount ─────────────────────────────────────────────────────────────
section "XFS Mount ($XFS_MOUNT)"
check_mount() {
local node=$1 run=$2
local status
if $run "mountpoint -q '$XFS_MOUNT' 2>/dev/null" 2>/dev/null; then
local usage
usage=$($run "df -h '$XFS_MOUNT' 2>/dev/null | tail -1 | awk '{print \$3\"/\"\$2\" used (\"\$5\")\";}'" 2>/dev/null || echo "")
status="mounted"
[[ -n "$usage" ]] && status="mounted $usage"
else
status="not mounted"
fi
printf " %-14s %s\n" "$node" "$status"
}
$REACHABLE_1 && check_mount "$NODE1" n1 || echo " $NODE1 [OFFLINE]"
$REACHABLE_2 && check_mount "$NODE2" n2 || echo " $NODE2 [OFFLINE]"
# ── Service ports via VIP ──────────────────────────────────────────────────
section "Services via VIP ($VIP)"
check_port() {
local name=$1 port=$2
if bash -c "echo >/dev/tcp/${VIP}/${port}" 2>/dev/null; then
printf " %-10s port %-5s OK\n" "$name" "$port"
else
printf " %-10s port %-5s UNREACHABLE\n" "$name" "$port"
fi
}
if ping -c1 -W2 "$VIP" >/dev/null 2>&1; then
echo " Ping OK"
else
echo " Ping UNREACHABLE"
fi
check_port "NFS" 2049
check_port "iSCSI" 3260
echo ""
echo "════════════════════════════════════════════════════"
if [[ -n "$ACTIVE_NODE" ]]; then
echo " Active: $ACTIVE_NODE Standby: $STANDBY_NODE"
else
echo " Active: unknown (Pacemaker not settled)"
fi
echo "════════════════════════════════════════════════════"
echo ""
+274
View File
@@ -0,0 +1,274 @@
#!/usr/bin/env bash
# resize-data-disk.sh — online resize of the HA cluster data disk
#
# Three-phase process (all online-safe, no downtime required):
# 1. Proxmox: grow scsi1 on both HA VMs (qm resize)
# 2. Guest: rescan block device on both nodes so the kernel sees the new size
# 3. DRBD + XFS: drbdadm resize, then xfs_growfs — active node only
#
# Usage:
# scripts/ha/resize-data-disk.sh --size +20G [--force] [--dry-run]
#
# --size +NNg amount to grow scsi1 by, e.g. +20G, +50G (required)
# XFS and DRBD cannot shrink; only positive deltas accepted
# --force skip the interactive confirmation prompt
# --dry-run show what would be done without changing anything
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=../env.sh
source "${SCRIPT_DIR}/../env.sh"
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}"
NODE2_IP="${NODE2_IP:-192.168.2.227}"
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"
DRBD_RESOURCE="${DRBD_RESOURCE:-ha-data}"
DATA_DISK_SLOT="${DATA_DISK_SLOT:-scsi1}" # Proxmox disk name (scsi1 = data disk)
HA_USER="${HA_USER:-nixos}"
PVE_HOST="${PVE_HOST:-${PVE1_HOST}}"
PVE_SSH_USER="${PVE_SSH_USER:-${PROXMOX_SSH_USER}}"
PVE_SUDO=""
[[ "$PVE_SSH_USER" != "root" ]] && PVE_SUDO="sudo"
# By-id symlink for the data disk; basename resolves to the raw block device.
# matches variables.nix's haServerDrbdDisk.
DATA_DISK_BYID="${DATA_DISK_BYID:-scsi-0QEMU_QEMU_HARDDISK_drive-${DATA_DISK_SLOT}}"
# ──────────────────────────────────────────────────────────────────────────
pve() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
"${PVE_SSH_USER}@${PVE_HOST}" "$@"; }
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
# ── Argument parsing ───────────────────────────────────────────────────────
SIZE=""
FORCE=false
DRY_RUN=false
while [[ $# -gt 0 ]]; do
case "$1" in
--size) shift; SIZE="${1:?--size requires a value (e.g. +20G)}" ;;
--force) FORCE=true ;;
--dry-run) DRY_RUN=true ;;
*) echo "Unknown argument: $1"
echo "Usage: $0 --size +NNg [--force] [--dry-run]"
exit 1 ;;
esac
shift
done
if [[ -z "$SIZE" ]]; then
echo "ERROR: --size is required (e.g. --size +20G)"
echo "Usage: $0 --size +NNg [--force] [--dry-run]"
exit 1
fi
# Only positive deltas — qm resize, DRBD, and XFS all refuse to shrink.
if [[ ! "$SIZE" =~ ^\+[0-9]+(G|M|T)$ ]]; then
echo "ERROR: --size must be a positive delta like +20G, +50G, +500M, +2T"
echo " (qm resize, drbdadm resize, and xfs_growfs can only grow, not shrink)"
exit 1
fi
DRY_PREFIX=""
$DRY_RUN && DRY_PREFIX="[dry-run] "
echo "════════════════════════════════════════════════════"
echo " HA Data Disk Resize — $(date '+%Y-%m-%d %H:%M:%S')"
echo " Size delta: $SIZE • Proxmox: $PVE_HOST"
$DRY_RUN && echo " MODE: dry-run — no changes will be made"
echo "════════════════════════════════════════════════════"
# ── Detect active node ─────────────────────────────────────────────────────
echo ""
echo "Detecting active node..."
CRM_OUT=""
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE1_IP}" true 2>/dev/null; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
fi
if [[ -z "$CRM_OUT" ]]; then
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE2_IP}" true 2>/dev/null; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
fi
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" (bullet *),
# so ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -z "$ACTIVE_NODE" ]]; then
echo "ERROR: could not determine active node from crm_mon."
echo " Is Pacemaker still settling? Try running scripts/ha/health.sh first."
exit 1
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
na() { n1 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
na() { n2 "$@"; }
fi
echo " Active node: $ACTIVE_NODE ($ACTIVE_IP)"
# ── Pre-check DRBD state ───────────────────────────────────────────────────
echo ""
echo "Pre-checks..."
DRBD_DSTATE=$(na "drbdadm dstate ${DRBD_RESOURCE} 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_DSTATE" | grep -q "UpToDate/UpToDate"; then
echo " WARNING: DRBD dstate is '$DRBD_DSTATE' (expected UpToDate/UpToDate)."
echo " Resizing with a partially-synced disk may cause issues."
if ! $FORCE; then
echo " Use --force to proceed anyway."
exit 1
fi
echo " --force specified — proceeding despite non-ideal DRBD state."
else
echo " DRBD dstate: $DRBD_DSTATE — OK"
fi
# ── Find VMIDs on Proxmox ─────────────────────────────────────────────────
echo ""
echo "Looking up VM IDs on ${PVE_HOST}..."
QM_LIST=$(pve "$PVE_SUDO qm list 2>/dev/null" || true)
VMID1=$(echo "$QM_LIST" | awk -v name="$NODE1" '$0 ~ name {print $1}' | head -1)
VMID2=$(echo "$QM_LIST" | awk -v name="$NODE2" '$0 ~ name {print $1}' | head -1)
if [[ -z "$VMID1" ]]; then
echo " ERROR: could not find VMID for $NODE1 on $PVE_HOST"
echo " qm list output:"
echo "$QM_LIST" | sed 's/^/ /'
exit 1
fi
if [[ -z "$VMID2" ]]; then
echo " ERROR: could not find VMID for $NODE2 on $PVE_HOST"
echo " qm list output:"
echo "$QM_LIST" | sed 's/^/ /'
exit 1
fi
echo " $NODE1: VMID $VMID1"
echo " $NODE2: VMID $VMID2"
# ── Confirm ────────────────────────────────────────────────────────────────
if ! $FORCE && ! $DRY_RUN; then
echo ""
echo " Plan:"
echo " Phase 1 — qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE} (on $PVE_HOST)"
echo " qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE} (on $PVE_HOST)"
echo " Phase 2 — block device rescan on $NODE1 and $NODE2"
echo " Phase 3 — drbdadm resize + xfs_growfs on $ACTIVE_NODE"
echo " No downtime required (all operations are online-safe)."
printf " Proceed? [y/N] "
read -r ANSWER
[[ "${ANSWER,,}" == "y" || "${ANSWER,,}" == "yes" ]] || { echo "Aborted."; exit 0; }
fi
# ═══════════════════════════════════════════════════════════════
# Phase 1 — Resize both VM data disks in Proxmox
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 1 — Proxmox disk resize (${DATA_DISK_SLOT} ${SIZE} on both VMs) ──"
echo " ${DRY_PREFIX}qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE} ($NODE1 on ${PVE_HOST})"
if ! $DRY_RUN; then
pve "$PVE_SUDO qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE}"
fi
echo " ${DRY_PREFIX}qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE} ($NODE2 on ${PVE_HOST})"
if ! $DRY_RUN; then
pve "$PVE_SUDO qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE}"
fi
echo " Phase 1 done."
# ═══════════════════════════════════════════════════════════════
# Phase 2 — Rescan block device on both guest nodes
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 2 — Block device rescan (both nodes) ──"
rescan_node() {
local node_name=$1 run_fn=$2
# Resolve block device name from the stable by-id symlink on the guest.
# Read-only lookup — safe to run even in dry-run so we show the real device.
local blk_dev=""
blk_dev=$($run_fn "bash -c 'basename \$(readlink -f /dev/disk/by-id/${DATA_DISK_BYID})'" 2>/dev/null || true)
if [[ -z "$blk_dev" ]]; then
echo " ERROR: /dev/disk/by-id/${DATA_DISK_BYID} not found on $node_name" >&2
echo " Check DATA_DISK_BYID or DATA_DISK_SLOT configuration." >&2
exit 1
fi
echo " ${DRY_PREFIX}Rescanning /dev/${blk_dev} on ${node_name}..."
if ! $DRY_RUN; then
$run_fn "bash -c 'echo 1 > /sys/block/${blk_dev}/device/rescan'" 2>/dev/null
local new_size
new_size=$($run_fn "lsblk -nd -o SIZE /dev/${blk_dev} 2>/dev/null" 2>/dev/null || echo "unknown")
echo " /dev/${blk_dev} on $node_name now reports: $new_size"
fi
}
rescan_node "$NODE1" n1
rescan_node "$NODE2" n2
echo " Phase 2 done."
# ═══════════════════════════════════════════════════════════════
# Phase 3 — Grow DRBD metadata, then XFS (active node only)
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 3 — DRBD resize + XFS grow (on active node: $ACTIVE_NODE) ──"
echo " ${DRY_PREFIX}drbdadm resize ${DRBD_RESOURCE}"
if ! $DRY_RUN; then
na "drbdadm resize ${DRBD_RESOURCE}"
fi
echo " ${DRY_PREFIX}xfs_growfs ${XFS_MOUNT}"
if ! $DRY_RUN; then
na "xfs_growfs ${XFS_MOUNT}"
fi
echo " Phase 3 done."
# ── Verify ────────────────────────────────────────────────────────────────
echo ""
echo "── Verify ──"
if ! $DRY_RUN; then
DF_OUT=$(na "df -h '${XFS_MOUNT}' 2>/dev/null" 2>/dev/null || echo "")
if [[ -n "$DF_OUT" ]]; then
echo " ${XFS_MOUNT}:"
echo "$DF_OUT" | sed 's/^/ /'
fi
DRBD_DSTATE_AFTER=$(na "drbdadm dstate ${DRBD_RESOURCE} 2>/dev/null" 2>/dev/null || echo "unknown")
echo " DRBD dstate: $DRBD_DSTATE_AFTER"
if ! echo "$DRBD_DSTATE_AFTER" | grep -q "UpToDate/UpToDate"; then
echo " NOTE: DRBD is resyncing — normal immediately after resize."
echo " Monitor: ssh nixos@${ACTIVE_IP} 'sudo watch -n3 cat /proc/drbd'"
fi
else
echo " [dry-run] would verify df -h ${XFS_MOUNT} and drbdadm dstate on $ACTIVE_NODE"
fi
echo ""
echo "════════════════════════════════════════════════════"
echo " Resize complete."
echo " Active node: $ACTIVE_NODE"
echo "════════════════════════════════════════════════════"
echo ""
+303
View File
@@ -0,0 +1,303 @@
#!/usr/bin/env bash
# Add a NixOS host to the FreeIPA domain and produce a sops-encrypted keytab
# at secrets/<hostname>.keytab, ready for modules/ipa/client.nix.
#
# One command replaces three error-prone manual steps:
# 1. ipa host-add on the domain controller
# 2. ipa-getkeytab on the domain controller + SCP back
# 3. sops encrypt in-place (must be at secrets/<hostname>.keytab for
# the creation rule to match -- the common mistake that breaks sops)
#
# Usage:
# scripts/ipa/create-nixos-ipa-host-account.sh [options] <hostname>
#
# Arguments:
# <hostname> Short hostname, e.g. "tailscale-router". The FQDN is
# derived as <hostname>.<HOME_DOMAIN>.
#
# Options:
# --ip <addr> Register this IP with the IPA host record (optional).
# --dc <host> SSH to this host to run IPA commands.
# Default: $IPA_SERVER (from env.sh / environment).
# --dc-user <u> SSH user on the domain controller. Default: wayne.
# --dry-run Print what would be done without making any changes.
# -h, --help Show this message.
#
# Prereqs:
# 1. Run from the repo root (so .sops.yaml and secrets/ are found).
# 2. SSH access to the domain controller as --dc-user (default: wayne)
# with passwordless sudo (or sudo cached). IPA commands and kinit run
# as root via sudo so the Kerberos ticket is in root's cache where all
# ipa tools expect it. If there's no valid ticket, the script runs
# `sudo kinit admin` interactively — you'll be prompted for the IPA
# admin password once. The password never touches this script.
# 3. The host's age key(s) must already be in .sops.yaml. Run
# scripts/secrets/sync-host-keys.sh <flake-target> first so the host
# can decrypt its own keytab on boot. This script adds the .sops.yaml
# creation rule for secrets/<hostname>.keytab automatically, but the
# host age key anchor (&lxc-<hostname> etc.) must already exist —
# otherwise only the admin key can decrypt the keytab and the deployed
# host will fail to read it.
# 4. sops in PATH, or Nix available to run it via `nix run`.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# shellcheck source=../env.sh
source "${SCRIPT_DIR}/../env.sh"
# --- Argument parsing ---
DC_HOST="${IPA_SERVER}"
DC_USER="wayne"
IP_ADDR=""
DRY_RUN=false
TARGET=""
usage() {
sed -n '/^# Usage:/,/^[^#]/{ /^#/{ s/^# \?//; p } }' "$0"
exit "${1:-0}"
}
while [[ $# -gt 0 ]]; do
case "$1" in
--ip) IP_ADDR="$2"; shift 2 ;;
--dc) DC_HOST="$2"; shift 2 ;;
--dc-user) DC_USER="$2"; shift 2 ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage 0 ;;
-*) echo "Unknown flag: $1" >&2; usage 1 ;;
*)
if [[ -n "${TARGET}" ]]; then echo "Unexpected argument: $1" >&2; usage 1; fi
TARGET="$1"; shift
;;
esac
done
if [[ -z "${TARGET}" ]]; then
echo "Error: hostname required." >&2
usage 1
fi
# Reject FQDNs passed by mistake — the script appends HOME_DOMAIN itself.
# "nixos.sweet.home" → FQDN would become "nixos.sweet.home.sweet.home".
if [[ "${TARGET}" == *"."* ]]; then
echo "Error: <hostname> must be the short name (e.g. 'nixos'), not a FQDN." >&2
echo " The FQDN is derived automatically as ${TARGET}.${HOME_DOMAIN}." >&2
exit 1
fi
FQDN="${TARGET}.${HOME_DOMAIN}"
KEYTAB_SECRET="${REPO_ROOT}/secrets/${TARGET}.keytab"
# Temp path on the domain controller — use a name that won't collide.
DC_TMP="/tmp/nixos-keytab-${TARGET}-$$.keytab"
# --- Helpers ---
log() { echo "==> $*"; }
logn() { echo " $*"; }
run() {
if $DRY_RUN; then
echo "[dry-run] $*"
else
"$@"
fi
}
dc_run() {
# Run a command string on the domain controller via SSH.
if $DRY_RUN; then
echo "[dry-run] ssh ${DC_USER}@${DC_HOST} $*"
else
ssh "${DC_USER}@${DC_HOST}" "$@"
fi
}
# --- Locate sops ---
if command -v sops &>/dev/null; then
SOPS_CMD=(sops)
else
log "sops not in PATH — will use 'nix run nixpkgs#sops'"
SOPS_CMD=(nix run "nixpkgs#sops" --)
fi
# --- Preflight checks ---
cd "${REPO_ROOT}"
[[ -f .sops.yaml ]] || { echo "Error: .sops.yaml not found — run from repo root." >&2; exit 1; }
[[ -d secrets ]] || { echo "Error: secrets/ not found — run from repo root." >&2; exit 1; }
# --- Step 1: Ensure .sops.yaml has a creation rule for this keytab ---
#
# sops matches creation rules against the PATH of the file being encrypted,
# not the output path. To match secrets/<hostname>.keytab, the file must
# already be at that path when sops -e -i is called. The creation rule must
# also exist at that point or sops will refuse with "no matching creation
# rules found."
log "Checking .sops.yaml for creation rule: secrets/${TARGET}.keytab"
RULE_EXISTS=false
# Match "path_regex: secrets/<hostname>...keytab" — using .*keytab rather
# than \.keytab because the file stores the regex verbatim (\.keytab = two
# chars: backslash + dot), which a BRE \. (= escaped literal dot) won't span.
if grep -q "path_regex: secrets/${TARGET}.*keytab" .sops.yaml 2>/dev/null; then
RULE_EXISTS=true
logn "Rule already exists — skipping addition."
fi
if ! $RULE_EXISTS; then
# Collect which platform-variant age anchors exist in .sops.yaml for this
# hostname. The keytab is platform-agnostic (same FQDN regardless of
# whether lxc/proxmox/linode variant is deployed), so all platform anchors
# that have been registered get added as recipients.
RECIPIENTS=("*admin")
for platform in lxc proxmox linode; do
anchor="${platform}-${TARGET}"
if grep -q "^ - &${anchor} " .sops.yaml; then
RECIPIENTS+=("*${anchor}")
fi
done
if [[ ${#RECIPIENTS[@]} -eq 1 ]]; then
echo "Warning: no platform age keys found for '${TARGET}' in .sops.yaml." >&2
echo " Run scripts/secrets/sync-host-keys.sh <flake-target> first," >&2
echo " otherwise only the admin key can decrypt the keytab and the" >&2
echo " deployed host won't be able to read it at boot." >&2
echo " Continuing with admin-only encryption..." >&2
fi
# Build the indented recipient list for the YAML block.
RECIPIENT_YAML=""
for r in "${RECIPIENTS[@]}"; do
RECIPIENT_YAML+=" - ${r}"$'\n'
done
RECIPIENT_YAML="${RECIPIENT_YAML%$'\n'}" # strip trailing newline
NEW_RULE="
# Host keytab for ${TARGET} FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/${TARGET}\\.keytab\$
key_groups:
- age:
${RECIPIENT_YAML}"
if $DRY_RUN; then
echo "[dry-run] Would append to .sops.yaml:"
echo "${NEW_RULE}"
else
logn "Adding creation rule (recipients: ${RECIPIENTS[*]})"
printf '%s\n' "${NEW_RULE}" >> .sops.yaml
logn "Added."
fi
fi
# --- Step 2: Add IPA host account (idempotent) ---
log "Adding FreeIPA host account: ${FQDN}"
# Ensure there's a valid admin Kerberos ticket on the DC.
# ipa host-add and ipa-getkeytab both need one. All IPA commands run via
# sudo so the ticket must be in root's cache — check and refresh as root.
# ssh -t allocates a PTY so kinit (and sudo if needed) can prompt normally;
# no password ever touches this script or the shell history.
if ! $DRY_RUN; then
if ! ssh "${DC_USER}@${DC_HOST}" "sudo klist -s" &>/dev/null; then
log "No valid Kerberos ticket on ${DC_HOST} — running sudo kinit admin"
ssh -t "${DC_USER}@${DC_HOST}" "sudo kinit admin"
if ! ssh "${DC_USER}@${DC_HOST}" "sudo klist -s" &>/dev/null; then
echo "Error: kinit admin failed or produced no valid ticket." >&2
exit 1
fi
else
logn "Kerberos ticket on ${DC_HOST} is valid."
fi
fi
IP_FLAG=""
[[ -n "${IP_ADDR}" ]] && IP_FLAG="--ip-address=${IP_ADDR}"
# --force: create the host record even if DNS doesn't resolve it yet.
if $DRY_RUN; then
echo "[dry-run] ssh ${DC_USER}@${DC_HOST} sudo ipa host-add '${FQDN}' ${IP_FLAG} --force"
else
HOST_ADD_OUT=$(ssh "${DC_USER}@${DC_HOST}" "sudo ipa host-add '${FQDN}' ${IP_FLAG} --force 2>&1") \
&& HOST_ADD_RC=0 || HOST_ADD_RC=$?
if [[ $HOST_ADD_RC -eq 0 ]]; then
echo "${HOST_ADD_OUT}"
elif echo "${HOST_ADD_OUT}" | grep -q "already exists"; then
logn "(host already registered)"
else
echo "Error: ipa host-add failed (exit ${HOST_ADD_RC}):" >&2
echo "${HOST_ADD_OUT}" >&2
exit 1
fi
fi
# --- Step 3: Fetch the keytab from the domain controller ---
log "Fetching keytab for host/${FQDN}"
# Remove the plaintext keytab if the script aborts before encryption completes.
# The trap is cleared at the end of step 4 once sops has encrypted it in-place.
trap 'rm -f "${KEYTAB_SECRET}"' EXIT
dc_run "sudo ipa-getkeytab -s '${IPA_SERVER}' -p 'host/${FQDN}' -k '${DC_TMP}'"
if $DRY_RUN; then
echo "[dry-run] Would stream ${DC_USER}@${DC_HOST}:${DC_TMP} → secrets/${TARGET}.keytab"
else
logn "Streaming keytab from ${DC_HOST}:${DC_TMP} → secrets/${TARGET}.keytab"
# scp can't read a root-owned temp file as ${DC_USER}; pipe through sudo cat instead.
ssh "${DC_USER}@${DC_HOST}" "sudo cat '${DC_TMP}'" > "${KEYTAB_SECRET}"
logn "Removing temp file on ${DC_HOST}"
dc_run "sudo rm -f '${DC_TMP}'"
fi
# --- Step 4: Encrypt in-place ---
#
# The file must already be at secrets/<hostname>.keytab (done above) so
# sops matches the creation rule by path. Using -i (in-place) rather than
# stdout redirect keeps the path intact through the encrypt call.
log "Encrypting secrets/${TARGET}.keytab in-place with sops"
run "${SOPS_CMD[@]}" -e --input-type binary -i "${KEYTAB_SECRET}"
# Encryption succeeded — the file is now sops-encrypted; cancel the cleanup trap.
trap - EXIT
# --- Done ---
if ! $DRY_RUN; then
echo ""
echo "Done. secrets/${TARGET}.keytab is sops-encrypted and ready."
echo ""
echo "Next steps:"
echo " 1. Verify: grep '\"data\": \"ENC' secrets/${TARGET}.keytab"
echo " 2. Stage and commit:"
echo " git add secrets/${TARGET}.keytab .sops.yaml"
echo " git commit -m 'secrets: add IPA keytab for ${TARGET}'"
echo " 3. Add to hosts/${TARGET}/host.nix (networking block and imports):"
echo ""
echo " networking = {"
echo " hostName = \"${TARGET}\";"
echo " domain = vars.homeDomain; # required for Kerberos FQDN"
echo " nameservers = [ vars.domainControllerIp ]; # IPA DNS"
echo " ..."
echo " };"
echo ""
echo " imports = ["
echo " (import ../../modules/ipa/client.nix {"
echo " keytabSopsFile = ../../secrets/${TARGET}.keytab;"
echo " caCertFile = ../../certs/ipa-ca.crt;"
echo " })"
echo " ];"
echo ""
echo " 4. Deploy: nixos-rebuild switch (or create-proxmox-resource.sh)"
fi
+106
View File
@@ -0,0 +1,106 @@
#!/usr/bin/env bash
# Clan vars helpers: manage SSH host keys stored as clan vars (sops-encrypted
# binary files under vars/per-machine/<target>/openssh/) instead of the
# gitignored host-keys/ directory.
#
# Layout (per clan's convention):
# vars/per-machine/<target>/openssh/ssh_host_ed25519_key/secret -- sops binary (admin-encrypted)
# vars/per-machine/<target>/openssh/ssh_host_ed25519_key.pub/value -- plaintext SSH pubkey
#
# Sourced by create-proxmox-resource.sh and sync-host-keys.sh.
# Depends on sops-age.sh and ssh-host-keys.sh being sourced first (for
# sops_yaml_admin_pubkey, ssh_pubkey_to_age, and NIX_OPTS).
if ! declare -p NIX_OPTS >/dev/null 2>&1; then
declare -a NIX_OPTS=()
fi
# clan_ssh_key_exists <target> <repo_root>
# Returns 0 if clan vars hold a SSH host key for <target>, 1 otherwise.
clan_ssh_key_exists() {
local target="$1" repo_root="$2"
[[ -f "${repo_root}/vars/per-machine/${target}/openssh/ssh_host_ed25519_key/secret" ]]
}
# clan_ssh_pubkey_path <target> <repo_root>
# Prints the path to the plaintext SSH public key value file.
clan_ssh_pubkey_path() {
local target="$1" repo_root="$2"
echo "${repo_root}/vars/per-machine/${target}/openssh/ssh_host_ed25519_key.pub/value"
}
# clan_decrypt_ssh_key <target> <repo_root> <dest_dir>
# Decrypts the sops-encrypted SSH host private key for <target> into <dest_dir>,
# naming it <target>_ssh_host_ed25519_key (to match NIXOS_HOST_KEYS_DIR
# conventions that lxc.nix and the disko build already expect). Also copies
# the plaintext public key. The caller is responsible for protecting and
# cleaning up <dest_dir>.
clan_decrypt_ssh_key() {
local target="$1" repo_root="$2" dest_dir="$3"
local secret="${repo_root}/vars/per-machine/${target}/openssh/ssh_host_ed25519_key/secret"
local pubval="${repo_root}/vars/per-machine/${target}/openssh/ssh_host_ed25519_key.pub/value"
local dest_priv="${dest_dir}/${target}_ssh_host_ed25519_key"
local dest_pub="${dest_dir}/${target}_ssh_host_ed25519_key.pub"
nix-shell "${NIX_OPTS[@]}" -p sops --run \
"sops -d --output-type binary '${secret}'" > "$dest_priv"
chmod 0600 "$dest_priv"
cp "$pubval" "$dest_pub"
}
# clan_generate_ssh_key <target> <repo_root>
# Generates a new SSH host key pair and stores it in clan vars format:
# - private key: sops binary-encrypted for the admin age key
# - public key: plaintext value file
# Idempotent: if the secret already exists, prints a note and returns 0.
# Requires sops_yaml_admin_pubkey (from sops-age.sh) to be available.
clan_generate_ssh_key() {
local target="$1" repo_root="$2"
local var_base="${repo_root}/vars/per-machine/${target}/openssh"
local secret_dir="${var_base}/ssh_host_ed25519_key"
local pubval_dir="${var_base}/ssh_host_ed25519_key.pub"
if [[ -f "${secret_dir}/secret" ]]; then
echo "Clan SSH host key for ${target} already exists -- skipping generation."
return 0
fi
# Resolve admin age public key from .sops.yaml
local admin_pubkey
admin_pubkey="$(sops_yaml_admin_pubkey "${repo_root}/.sops.yaml")"
if [[ -z "$admin_pubkey" ]]; then
echo "ERROR: Could not find &admin age key in ${repo_root}/.sops.yaml" >&2
return 1
fi
# Generate the SSH key pair in a secure temp directory
local tmpdir
tmpdir="$(mktemp -d)"
local priv_tmp="${tmpdir}/ssh_host_ed25519_key"
# shellcheck disable=SC2064
trap "rm -rf '${tmpdir}'" RETURN
nix-shell "${NIX_OPTS[@]}" -p openssh --run \
"ssh-keygen -t ed25519 -N '' -C '${target}' -f '${priv_tmp}'" >/dev/null
# Create a minimal sops config that uses only the admin age key -- this
# prevents sops from merging in ALL recipients from .sops.yaml (which
# would unnecessarily encrypt for every host's key, not just admin).
local sops_cfg="${tmpdir}/sops-config.json"
printf '{"creation_rules":[{"key_groups":[{"age":["%s"]}]}]}\n' \
"$admin_pubkey" > "$sops_cfg"
# Encrypt the private key in sops binary format (admin-only recipient)
mkdir -p "$secret_dir" "$pubval_dir"
nix-shell "${NIX_OPTS[@]}" -p sops --run \
"sops -e --config '${sops_cfg}' --input-type binary '${priv_tmp}'" \
> "${secret_dir}/secret"
# Store the public key as a plaintext value file
cp "${priv_tmp}.pub" "${pubval_dir}/value"
echo "Generated and stored clan SSH host key for ${target}."
echo " Private key: ${secret_dir}/secret (sops binary, admin-key encrypted)"
echo " Public key: ${pubval_dir}/value"
}
@@ -38,7 +38,7 @@
set -euo pipefail
: "${NIX_CACHE_HOST:=nix-cache}"
: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache}"
: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICuHUxGNH6ei3BZD+EfZs3l4X8uJNcjQiOsM/G4yo4O/ lxc-nix-cache}"
: "${REMOTE_BUILDER_USER:=nixremote}"
CACHE_PUB_KEY="cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4="
+164 -53
View File
@@ -8,9 +8,12 @@
# script -- there's no multi-gigabyte image to transfer afterward. The first
# time a node doesn't have that repo path yet, it's bootstrapped: cloned from
# this checkout's own `origin` remote, then scripts/codex-setup.sh installs
# the build tooling (Nix, etc.). Every run after that just `git pull`s it and
# copies over the locally-managed host-keys/ (gitignored, so a git pull
# alone wouldn't carry it) before building.
# the build tooling (Nix, etc.). Every run after that just `git pull`s it.
# SSH host keys are stored as clan vars (vars/per-machine/<target>/openssh/,
# committed and sops-encrypted) -- the script decrypts them locally and
# copies only the two files for this target to the node's host-keys/ before
# building. A target with no clan var is an error (generate one first with
# scripts/secrets/sync-host-keys.sh <target>).
#
# --node (default: $PROXMOX_HOST, see scripts/env.sh) picks which of the two
# LAN Proxmox nodes this runs against: production, pve1.sweet.home
@@ -59,6 +62,10 @@ source "${repo_root}/scripts/env.sh"
source "${repo_root}/scripts/lib/nix-eval.sh"
# shellcheck source=../lib/confirm.sh
source "${repo_root}/scripts/lib/confirm.sh"
# shellcheck source=../lib/sops-age.sh
source "${repo_root}/scripts/lib/sops-age.sh"
# shellcheck source=../lib/clan-vars.sh
source "${repo_root}/scripts/lib/clan-vars.sh"
sync_keys="${repo_root}/scripts/secrets/sync-host-keys.sh"
@@ -295,6 +302,12 @@ platform_prefix="lxc"
[[ -z "$cores" ]] && cores="$PROXMOX_DEFAULT_CORES"
[[ -z "$memory" ]] && memory="$PROXMOX_DEFAULT_MEMORY_MB"
if [[ "$type" == "vm" && -n "$disk_size" ]]; then
echo "WARNING: --disk-size is LXC-only for create mode and is ignored for VMs." >&2
echo " VM disk size comes from proxmoxImageSize in variables.nix (currently ${disk_size}G was requested)." >&2
echo " To expand after creation, use: --modify --vmid <n> --grow-disk <GB>" >&2
fi
# --- discover / resolve the flake target from --host --------------------
# Emits "<target>\t<hostName>" pairs for every ${platform_prefix}-* flake
# target -- the one source both --list and the --host lookup below read
@@ -347,6 +360,14 @@ fi
# feeds straight into the guest's real hostname) disagree with host.nix.
[[ -z "$name" ]] && name="$host"
# For VM builds: the diskoImagesScript (run via QEMU on the node) writes the
# raw disk image as <hostname>.raw into the CWD it was called from (the remote
# repo dir), not to /var/lib/vz/import/ or anywhere else. Import directly from
# there -- no intermediate mv that can fail crossing filesystem boundaries or
# leave a stale file on error.
vm_built_raw=""
[[ "$type" == "vm" ]] && vm_built_raw="${remote_repo_dir}/${host}.raw"
# --- refuse to duplicate a host that's already live on the node ---------
# Queries the node itself (qm/pct's own name/hostname config), not any
# static list in this repo -- a file can't track whether a resource still
@@ -485,12 +506,37 @@ echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '
nix_extra_opts
# --- make sure this target has a registered host key --------------------
# sync-host-keys.sh is idempotent and generates the key (via clan vars) if
# no key exists yet -- the old inline prepare-host-key.sh call is gone.
echo
echo "==> Ensuring host key exists and is registered..."
sync_args=("$flake_target")
[[ "$dry_run" -eq 1 ]] && sync_args+=(--dry-run)
bash "$sync_keys" "${sync_args[@]}"
# If sync-host-keys.sh changed .sops.yaml, secrets/, or vars/per-machine/,
# those changes must be committed and pushed before the remote `git pull`
# below picks them up -- the PVE node builds from whatever HEAD is checked
# out there, not the local working tree. Uncommitted clan vars or sops
# recipients mean the image builds fine but the host cannot decrypt its
# secrets on first boot. Block until the operator confirms they've pushed.
if [[ "$dry_run" -eq 0 ]]; then
_dirty="$(git -C "$repo_root" status --porcelain -- .sops.yaml secrets/ vars/per-machine/ 2>/dev/null || true)"
if [[ -n "$_dirty" ]]; then
echo
echo "==> COMMIT + PUSH REQUIRED before the remote build can succeed:"
echo " Uncommitted changes in .sops.yaml, secrets/, or vars/per-machine/."
echo " The PVE node builds from the git-tracked flake, so these changes"
echo " must be committed and pushed first -- otherwise the image build will"
echo " succeed but the host cannot decrypt its secrets on first boot."
echo
git -C "$repo_root" status --short -- .sops.yaml secrets/ vars/per-machine/ || true
echo
read -rp " Commit and push those changes, then press Enter to continue (Ctrl-C to abort): "
fi
unset _dirty
fi
# --- VMID: pick one, and refuse to touch anything that already exists ---
echo
if [[ -z "$vmid" ]]; then
@@ -613,21 +659,35 @@ ensure_remote_repo() {
fi
}
# --- sync locally-managed host-keys/ to the node ---------------------------
# Gitignored (see .gitignore), so `git pull` above never carries it -- both
# build paths need it present as NIXOS_HOST_KEYS_DIR / --pre-format-files
# input on the node itself now that the build runs there. scp (not rsync,
# not already a dependency anywhere else in this repo) mirrors how this
# script already transfers the --image case below.
# --- sync host key to the node ---------------------------------------------
# SSH host keys are stored as clan vars (vars/per-machine/<target>/openssh/).
# Decrypt locally and scp just the two files for this target to the node's
# host-keys/ directory, where the remote build script picks them up via
# NIXOS_HOST_KEYS_DIR (LXC) or --pre-format-files (VM). A target with no
# clan var is an error -- generate one first with sync-host-keys.sh.
sync_remote_host_keys() {
echo
echo "==> Syncing host-keys/ to ${node}..."
echo "==> Syncing host key for ${flake_target} to ${node}..."
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would copy ${repo_root}/host-keys/ to ${ssh_target}:${remote_repo_dir}/host-keys/"
echo "[dry-run] would decrypt clan SSH key for ${flake_target} and copy to ${ssh_target}:${remote_repo_dir}/host-keys/"
return
fi
if ! clan_ssh_key_exists "$flake_target" "$repo_root"; then
echo "ERROR: no clan SSH key found for ${flake_target}" >&2
echo " (expected: ${repo_root}/vars/per-machine/${flake_target}/openssh/ssh_host_ed25519_key/secret)" >&2
echo " Generate one first: bash scripts/secrets/sync-host-keys.sh ${flake_target}" >&2
exit 1
fi
local tmpdir
tmpdir="$(mktemp -d)"
# shellcheck disable=SC2064
trap "rm -rf '${tmpdir}'" RETURN
echo " Decrypting clan SSH key for ${flake_target}..."
clan_decrypt_ssh_key "$flake_target" "$repo_root" "$tmpdir"
ssh "$ssh_target" "mkdir -p '${remote_repo_dir}/host-keys'"
scp -pr "${repo_root}/host-keys/." "${ssh_target}:${remote_repo_dir}/host-keys/"
scp -p "${tmpdir}/${flake_target}_ssh_host_ed25519_key" \
"${tmpdir}/${flake_target}_ssh_host_ed25519_key.pub" \
"${ssh_target}:${remote_repo_dir}/host-keys/"
}
# --- build (or reuse an image already on the node) ------------------------
@@ -642,10 +702,14 @@ if [[ -n "$image" ]]; then
elif [[ "$force_rebuild" -eq 1 ]]; then
echo "--force-rebuild: skipping the existing-image check on ${node}."
else
echo "==> Checking whether ${node} already has ${remote_path}..."
# VMs: check for the raw image in the remote repo dir (where disko writes it).
# LXC: check for the tarball in iso_storage (where the LXC build stages it).
_check_path="$remote_path"
[[ "$type" == "vm" ]] && _check_path="$vm_built_raw"
echo "==> Checking whether ${node} already has ${_check_path}..."
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would check: ssh ${ssh_target} -- test -f ${remote_path}"
elif ssh "$ssh_target" "test -f '${remote_path}'" 2>/dev/null; then
echo "[dry-run] would check: ssh ${ssh_target} -- test -f ${_check_path}"
elif ssh "$ssh_target" "test -f '${_check_path}'" 2>/dev/null; then
echo "Found it -- reusing, skipping build (use --force-rebuild to override)."
image_already_remote=1
else
@@ -696,6 +760,12 @@ cd "$repo_dir"
# right after a successful install.
. scripts/lib/nix-bootstrap.sh
ensure_nix_profile
if [[ ! -f "host-keys/${target}_ssh_host_ed25519_key" ]]; then
echo "ERROR: host-keys/${target}_ssh_host_ed25519_key not found in ${repo_dir}." >&2
echo "Generate the key locally (scripts/secrets/sync-host-keys.sh ${target})" >&2
echo "and ensure it was synced here before starting the build." >&2
exit 1
fi
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
--no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \
".#nixosConfigurations.${target}.config.system.build.tarball" \
@@ -714,46 +784,60 @@ REMOTE_SCRIPT
fi
else
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would build on ${node}: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
echo "[dry-run] would build on ${node}: NIXOS_HOST_KEYS_DIR=\$(pwd)/host-keys nix build --impure --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
echo "[dry-run] would run: ${sudo_display}./result-${flake_target} \\"
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\"
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\"
echo "[dry-run] --build-memory 2048"
echo "[dry-run] would stage the result at ${remote_path}"
echo "[dry-run] would run: ${sudo_display}./result-${flake_target} --build-memory 2048"
echo "[dry-run] image will be at ${vm_built_raw} (imported from there; no mv to /var/lib/vz/import/)"
local_image="<built-image>.raw"
else
echo "==> Building Disko image for ${flake_target} on ${node}..."
# See the LXC branch above for why this is one %q-quoted command
# string rather than separate ssh argv elements.
printf -v remote_cmd 'bash -s -- %q %q %q %q %q %q' \
"$remote_repo_dir" "$flake_target" "$remote_dir" "$remote_filename" "$NIX_EXTRA_OPTS" "$sudo_prefix"
# $7 = image_name (hostname, the diskoImagesScript's own output filename).
#
# NIXOS_HOST_KEYS_DIR + --impure: modules/platforms/proxmox.nix reads
# this env var at eval time (like lxc.nix) to embed the clan SSH host
# key in environment.etc. nixos-install's own activation then places the
# key on the target disk, so sshd-keygen finds it already present and
# skips generation. --pre-format-files put the key on the QEMU builder
# VM's rootfs (not the target disk), so sshd-keygen regenerated a fresh
# key -- one not registered in .sops.yaml -- and sops could never decrypt.
printf -v remote_cmd 'bash -s -- %q %q %q %q %q %q %q' \
"$remote_repo_dir" "$flake_target" "$remote_dir" "$remote_filename" "$NIX_EXTRA_OPTS" "$sudo_prefix" "$host"
ssh "$ssh_target" "$remote_cmd" <<'REMOTE_SCRIPT'
set -euo pipefail
repo_dir="$1"; target="$2"; dest_dir="$3"; dest_name="$4"; nix_extra_opts_str="$5"; sudo_pfx="$6"
repo_dir="$1"; target="$2"; dest_dir="$3"; dest_name="$4"; nix_extra_opts_str="$5"; sudo_pfx="$6"; image_name="$7"
declare -a NIX_OPTS=()
[[ -n "$nix_extra_opts_str" ]] && eval "NIX_OPTS=(${nix_extra_opts_str})"
cd "$repo_dir"
. scripts/lib/nix-bootstrap.sh
ensure_nix_profile
nix build --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \
".#nixosConfigurations.${target}.config.system.build.diskoImagesScript" \
--out-link "result-${target}"
$sudo_pfx "./result-${target}" \
--pre-format-files "host-keys/${target}_ssh_host_ed25519_key" /etc/ssh/ssh_host_ed25519_key \
--pre-format-files "host-keys/${target}_ssh_host_ed25519_key.pub" /etc/ssh/ssh_host_ed25519_key.pub \
--build-memory 2048
built="$(find . -maxdepth 1 -name '*.raw' -newer "result-${target}" | head -1)"
if [[ -z "$built" ]]; then
echo "ERROR: no .raw image found in ${repo_dir} after build." >&2
if [[ ! -f "host-keys/${target}_ssh_host_ed25519_key" ]]; then
echo "ERROR: host-keys/${target}_ssh_host_ed25519_key not found in ${repo_dir}." >&2
echo "Generate the key locally (scripts/secrets/sync-host-keys.sh ${target})" >&2
echo "and ensure it was synced here before starting the build." >&2
exit 1
fi
$sudo_pfx mkdir -p "$dest_dir"
$sudo_pfx mv "$built" "${dest_dir}/${dest_name}"
echo "Built and staged: ${dest_dir}/${dest_name}"
# Build diskoImagesScript with NIXOS_HOST_KEYS_DIR so proxmox.nix embeds the
# clan SSH key in environment.etc (same as lxc.nix). This causes nixos-install
# to place the key on the target disk, so sshd-keygen finds it and skips
# generation -- the disk image boots with the registered key, sops decrypts.
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
--no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \
".#nixosConfigurations.${target}.config.system.build.diskoImagesScript" \
--out-link "result-${target}"
# Remove any stale .raw from a previous failed build so the post-build check
# below is unambiguous (diskoImagesScript writes to CWD as ${image_name}.raw).
$sudo_pfx rm -f "${image_name}.raw" 2>/dev/null || true
$sudo_pfx "./result-${target}" --build-memory 2048
if [[ ! -f "${image_name}.raw" ]]; then
echo "ERROR: ${image_name}.raw not found in ${repo_dir} after build -- disko/QEMU may have failed." >&2
exit 1
fi
echo "Built image: ${repo_dir}/${image_name}.raw"
REMOTE_SCRIPT
local_image="$remote_path"
echo "Built on ${node}: ${remote_path}"
local_image="$vm_built_raw"
echo "Built on ${node}: ${vm_built_raw}"
fi
fi
fi
@@ -783,17 +867,17 @@ if [[ "$type" == "lxc" ]]; then
local_swap="${swap:-$memory}"
# --unprivileged: read back from modules/platforms/lxc.nix's own
# proxmoxLXC.privileged (via flake_target_lxc_privileged) rather than
# hardcoded, since that's no longer the same for every lxc-* target --
# lxc-docker sets it true so the container's NFS mounts work at all (the
# kernel's NFS client can't mount from inside any unprivileged
# container's user namespace, no matter what AppArmor allows -- see that
# option's own comment). The NixOS config inside the image bakes in
# cgroup/capability/mount expectations matching whichever value it was
# built with, so this must stay in sync with it -- `pct create`'s own
# CLI default for this flag is privileged (unlike the web UI, which
# defaults its checkbox the other way), so leaving it unset would create
# a privileged container running a NixOS config that assumes
# unprivileged for every target except lxc-docker, a real mismatch.
# hardcoded. lxc.nix derives this automatically: any lxc-* host whose
# config.fileSystems has an NFS entry gets privileged=true, because the
# kernel's NFS client (FS_USERNS_MOUNT not set) rejects NFS mounts from
# inside any non-init user namespace -- exactly what an unprivileged
# container's UID-mapped root lives in -- with EPERM at the VFS layer,
# regardless of AppArmor (see lxc.nix's own comment). The NixOS config
# bakes in cgroup/capability/mount expectations matching whichever value
# it was built with, so this must stay in sync -- `pct create`'s CLI
# default is privileged (unlike the web UI, which defaults the other
# way), so leaving it unset would create a privileged container running
# a NixOS config that assumes unprivileged, a real mismatch.
privileged_eval="$(flake_target_lxc_privileged "$repo_root" "$flake_target")"
unprivileged_flag=1
[[ "$privileged_eval" == "true" ]] && unprivileged_flag=0
@@ -831,14 +915,35 @@ else
--net0 virtio,bridge=${bridge} --bios ovmf --machine q35 --scsihw virtio-scsi-pci \
--efidisk0 ${storage}:1,efitype=4m,pre-enrolled-keys=0 --agent enabled=1"
# VMs built on the node: import from the repo dir (where disko/QEMU wrote it).
# VMs from --image: import from remote_path (where scp uploaded it).
_import_path="${remote_path}"
[[ -z "$image" ]] && _import_path="${vm_built_raw}"
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] ssh ${ssh_target} -- ${sudo_display}qm importdisk ${vmid} ${remote_path} ${storage}"
echo "[dry-run] ssh ${ssh_target} -- ${sudo_display}qm importdisk ${vmid} ${_import_path} ${storage}"
echo "[dry-run] (would parse the resulting disk identifier from that output)"
echo "[dry-run] ssh ${ssh_target} -- ${sudo_display}qm set ${vmid} --scsi0 ${storage}:<parsed-disk-id>"
else
importdisk_output="$(ssh "$ssh_target" "${sudo_prefix} qm importdisk ${vmid} ${remote_path} ${storage}")"
if ! importdisk_output="$(ssh "$ssh_target" "${sudo_prefix} qm importdisk ${vmid} ${_import_path} ${storage}" 2>&1)"; then
echo "ERROR: qm importdisk failed:" >&2
echo "${importdisk_output}" >&2
exit 1
fi
echo "$importdisk_output"
disk_id="$(echo "$importdisk_output" | grep -oP "(?<=Successfully imported disk as ')[^']+" | sed 's/^unused[0-9]*://')"
# PVE output format: "unusedN: successfully imported disk '<storage>:<vol>'"
# (lowercase "successfully", no "as"; the primary regex targets this form; the
# || true inside the substitution prevents set -e from aborting when grep finds
# no match -- without it the script would silently exit before reaching the
# fallback whenever the PVE format doesn't match).
disk_id="$(echo "$importdisk_output" | grep -oP "successfully imported disk '\\K[^']+" || true)"
if [[ -z "$disk_id" ]]; then
# Fallback for other PVE output variants: read qm config directly.
unused_line="$(ssh "$ssh_target" "${sudo_prefix} qm config ${vmid}" | grep '^unused[0-9]*:' | head -1 || true)"
if [[ -n "$unused_line" ]]; then
disk_id="${unused_line#*: }"
echo "Note: disk ID resolved from qm config: ${disk_id}"
fi
fi
if [[ -z "$disk_id" ]]; then
echo "ERROR: couldn't parse the imported disk identifier from qm importdisk's output above." >&2
echo "The VM shell (${vmid}) and imported disk both exist -- finish attaching it by hand:" >&2
@@ -847,6 +952,12 @@ else
exit 1
fi
remote "${sudo_prefix} qm set ${vmid} --scsi0 ${disk_id}"
# The disk data is now in ZFS; remove the source raw file (only for images
# we built on the node -- --image uploads are the operator's to manage).
if [[ -z "$image" ]]; then
ssh "$ssh_target" "${sudo_prefix} rm -f '${_import_path}'" 2>/dev/null || \
echo "Warning: couldn't remove ${_import_path} from ${node} -- you can delete it manually" >&2
fi
fi
remote "${sudo_prefix} qm set ${vmid} --boot order=scsi0"
remote "${sudo_prefix} qm start ${vmid}"
+253
View File
@@ -0,0 +1,253 @@
#!/usr/bin/env bash
# recover-hosts.sh — Fix sops/SSH-key/GitHub-token issues on deployed NixOS hosts
# and trigger a Switch-nix rebuild on each.
#
# Run from the repo root on the workstation (nixos@nixos):
# bash scripts/recover-hosts.sh [<hostname> ...]
#
# With no args it discovers and checks every known hostname.
# With args it checks only those hostnames:
# bash scripts/recover-hosts.sh tor-relay
#
# Fixes applied automatically (then prompts before rebuilding):
# 1. SSH host key drift — live key no longer matches host-keys/<target>_ssh_host_ed25519_key
# Fix: scp the registered key back and restore it (needs sudo once per host).
# To push new keys proactively (before drift, e.g. right after
# sync-host-keys.sh --regenerate-all-keys), use instead:
# scripts/secrets/push-host-keys.sh --all
# 2. Stale/invalid GitHub access token — the rendered nix-github-token.conf has
# a token GitHub rejects (401), blocking any rebuild that fetches disko or
# other public GitHub flake inputs.
# Fix: empty the rendered file so nix makes unauthenticated requests instead.
# Public repos (disko, nixpkgs, etc.) work fine without auth. sops-nix
# re-renders the correct new token automatically after the first successful
# rebuild.
#
# Both fixes need one interactive sudo session per host. The script opens a
# single ssh -t per broken host so you enter the password once and all steps
# run in sequence.
set -euo pipefail
cd "$(dirname "$0")/.."
source scripts/env.sh 2>/dev/null || true
SSH_OPTS=(-o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=5)
SSH_USER=nixos
# Known flake-target → ssh hostname map for all currently-defined hosts.
# Add new hosts here as they are deployed.
declare -A TARGET_HOST=(
[lxc-docker]=docker
[lxc-nix-cache]=nix-cache
[lxc-pxe-boot]=pxe-boot
[lxc-tor-relay]=tor-relay
[lxc-minimal]=nix-minimal
[proxmox-server]=server
[baremetal-gui]=nixos
)
# ── helpers ───────────────────────────────────────────────────────────────────
info() { echo " [✓] $*"; }
warn() { echo " [!] $*"; }
step() { echo "==> $*"; }
ssh_host_age() {
ssh-keyscan -t ed25519 "$1" 2>/dev/null \
| nix shell nixpkgs#ssh-to-age --command ssh-to-age 2>/dev/null \
| head -1 || true
}
registered_age() {
local keyfile="host-keys/${1}_ssh_host_ed25519_key.pub"
[ -f "$keyfile" ] || return 0
nix shell nixpkgs#ssh-to-age --command ssh-to-age < "$keyfile" 2>/dev/null \
| head -1 || true
}
github_token_valid() {
local host=$1
local raw token code
raw=$(ssh "${SSH_OPTS[@]}" "$SSH_USER@$host" \
"cat /run/secrets/rendered/nix-github-token.conf 2>/dev/null || true")
token=$(echo "$raw" | grep -oP '(?<=github\.com=)\S+' || true)
if [ -z "$token" ]; then
return 0 # no token = unauthenticated, works for public repos
fi
code=$(curl -s -o /dev/null -w "%{http_code}" \
-H "Authorization: token $token" \
"https://api.github.com/repos/nix-community/disko" 2>/dev/null || echo 000)
[ "$code" = "200" ]
}
# ── discover hosts ────────────────────────────────────────────────────────────
if [ $# -gt 0 ]; then
HOSTNAMES=("$@")
else
HOSTNAMES=()
seen=()
for target in "${!TARGET_HOST[@]}"; do
h="${TARGET_HOST[$target]}"
# deduplicate (e.g. proxmox-server and lxc-server both map to "server")
if [[ ! " ${seen[*]:-} " =~ " $h " ]]; then
seen+=("$h")
if ssh "${SSH_OPTS[@]}" "$SSH_USER@$h" "true" 2>/dev/null; then
HOSTNAMES+=("$h")
fi
fi
done
fi
if [ ${#HOSTNAMES[@]} -eq 0 ]; then
echo "No reachable hosts found. Pass hostnames explicitly or check SSH."
exit 1
fi
echo ""
echo "Hosts to check: ${HOSTNAMES[*]}"
echo ""
# ── check phase ───────────────────────────────────────────────────────────────
NEEDS_FIX=()
for host in "${HOSTNAMES[@]}"; do
step "$host"
if ! ssh "${SSH_OPTS[@]}" "$SSH_USER@$host" "true" 2>/dev/null; then
warn "SSH unreachable — clearing stale known_hosts entry"
ssh-keygen -R "$host" 2>/dev/null || true
continue
fi
flake_target=$(ssh "${SSH_OPTS[@]}" "$SSH_USER@$host" \
"cat /etc/flake-target 2>/dev/null || true")
echo " flake-target: ${flake_target:-unknown}"
host_broken=false
# SSH host key
if [ -n "$flake_target" ] && [ -f "host-keys/${flake_target}_ssh_host_ed25519_key.pub" ]; then
live=$(ssh_host_age "$host")
want=$(registered_age "$flake_target")
if [ "$live" = "$want" ]; then
info "SSH host key OK"
else
warn "SSH host key MISMATCH (live ≠ host-keys/) -- use push-host-keys.sh proactively next time"
echo " live: $live"
echo " registered: $want"
host_broken=true
fi
else
echo " [~] No host-keys/ entry for ${flake_target:-unknown} — skipping key check"
fi
# GitHub token
if github_token_valid "$host"; then
info "GitHub token OK"
else
warn "GitHub token invalid (rebuild will fail with 401)"
host_broken=true
fi
# sops-nix result
sops_result=$(ssh "${SSH_OPTS[@]}" "$SSH_USER@$host" \
"systemctl show sops-nix --property=Result --value 2>/dev/null || echo unknown")
if [ "$sops_result" = "success" ]; then
info "sops-nix: success"
else
warn "sops-nix: $sops_result"
fi
$host_broken && NEEDS_FIX+=("$host")
echo ""
done
# ── fix phase ─────────────────────────────────────────────────────────────────
if [ ${#NEEDS_FIX[@]} -eq 0 ]; then
echo "All hosts healthy — nothing to fix."
exit 0
fi
echo "Hosts needing fixes: ${NEEDS_FIX[*]}"
echo ""
echo "Each fix requires one sudo session per host. You will be prompted for"
echo "the nixos sudo password once per host; all steps run in that session."
echo ""
read -r -p "Proceed with fixes + Switch-nix on each broken host? [y/N] " confirm
[[ "$confirm" =~ ^[Yy]$ ]] || { echo "Aborted."; exit 0; }
echo ""
for host in "${NEEDS_FIX[@]}"; do
step "Fixing $host"
flake_target=$(ssh "${SSH_OPTS[@]}" "$SSH_USER@$host" \
"cat /etc/flake-target 2>/dev/null || true")
fix_script=""
# Fix 1: restore SSH host key
live=$(ssh_host_age "$host")
want=$(registered_age "${flake_target:-}")
if [ -n "$want" ] && [ "$live" != "$want" ]; then
echo " Uploading registered SSH host key (private + public)..."
scp -o StrictHostKeyChecking=no \
"host-keys/${flake_target}_ssh_host_ed25519_key" \
"$SSH_USER@$host:/tmp/recover_ed25519_key"
scp -o StrictHostKeyChecking=no \
"host-keys/${flake_target}_ssh_host_ed25519_key.pub" \
"$SSH_USER@$host:/tmp/recover_ed25519_key.pub"
fix_script+='
echo "[fix] Restoring SSH host key..."
install -m 0600 /tmp/recover_ed25519_key /etc/ssh/ssh_host_ed25519_key
install -m 0644 /tmp/recover_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub
rm -f /tmp/recover_ed25519_key /tmp/recover_ed25519_key.pub
echo " Done."
'
ssh-keygen -R "$host" 2>/dev/null || true
fi
# Fix 2: clear invalid GitHub token
if ! github_token_valid "$host"; then
fix_script+='
echo "[fix] Clearing stale GitHub token (nix will use unauthenticated access)..."
echo "" > /run/secrets/rendered/nix-github-token.conf
systemctl restart nix-daemon 2>/dev/null || true
echo " Done."
'
fi
# Fix 3: rebuild
fix_script+='
echo "[fix] Running nixos-rebuild switch..."
nixos-rebuild switch \
--no-write-lock-file \
--refresh \
--flake "git+https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos.git#$(cat /etc/flake-target)"
echo "[fix] Rebuild complete."
'
echo " Opening SSH session (enter sudo password when prompted)..."
if ssh -t -o StrictHostKeyChecking=no "$SSH_USER@$host" \
"sudo bash -s" <<< "$fix_script"; then
echo ""
info "$host fixed and rebuilt"
else
rc=$?
echo ""
warn "$host: rebuild exited with code $rc (may still have succeeded — check sops-nix below)"
fi
# Verify: re-check sops-nix result post-rebuild
sops_result_after=$(ssh "${SSH_OPTS[@]}" "$SSH_USER@$host" \
"systemctl show sops-nix --property=Result --value 2>/dev/null || echo unknown" 2>/dev/null || echo "ssh-failed")
if [ "$sops_result_after" = "success" ]; then
info "$host sops-nix: success post-rebuild"
else
warn "$host sops-nix: $sops_result_after post-rebuild (may need another pass)"
fi
echo ""
done
echo "Recovery complete."
+3 -2
View File
@@ -41,8 +41,9 @@ mkdir -p "$keydir"
keyfile="${keydir}/${hostname}_ssh_host_ed25519_key"
if [[ -f "$keyfile" ]]; then
echo "ERROR: $keyfile already exists. Remove it first if you want to regenerate." >&2
exit 1
echo "Key already exists: ${keyfile}"
echo "Reusing the existing key. Remove it first if you want to regenerate."
exit 0
fi
nix_extra_opts
+318
View File
@@ -0,0 +1,318 @@
#!/usr/bin/env bash
# Pushes newly-generated SSH host keys from host-keys/ to already-running
# NixOS hosts, so they can decrypt sops secrets after a nixos-rebuild
# following scripts/secrets/sync-host-keys.sh --regenerate-all-keys.
#
# Before pushing any key, verifies that .sops.yaml and secrets/*.yaml are
# committed and pushed to the remote -- hosts rebuild from the remote Gitea
# flake, so recipient changes must land there before any rebuild, not just
# before the key push.
#
# push-host-keys.sh --all [--dry-run] [--skip-git-check]
# push-host-keys.sh <target> [--dry-run] [--skip-git-check]
#
# --all Push to every reachable managed host. Default when no
# target is given.
# <target> Push to one flake target only (e.g. lxc-server).
# --dry-run Print what would be done; write nothing.
# --skip-git-check Skip the commit/push check. Use only when the remote
# already has the current .sops.yaml/secrets/*.yaml.
#
# SSH: connects as SSH_USER@<hostname> (default: nixos, the user with the
# admin authorized key), then installs files via sudo -S (reads the sudo
# password from stdin). The password is prompted once at startup and reused
# for every host -- no PTY or terminal required on the remote side.
# Hosts are reached at their bare hostname (relies on LAN DNS/mDNS).
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
keydir="${repo_root}/host-keys"
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=../lib/nix-eval.sh
source "${repo_root}/scripts/lib/nix-eval.sh"
: "${SSH_USER:=nixos}"
SSH_OPTS=(-o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=5)
dry_run=0
skip_git_check=0
sudo_password=""
usage() {
cat <<EOF
Usage: $0 [--all | <target>] [--dry-run] [--skip-git-check]
--all Push to every reachable managed host. Default when no
target is given.
<target> Push to one flake target only (e.g. lxc-server).
--dry-run Print what would be done; write nothing.
--skip-git-check Skip the check that .sops.yaml/secrets/*.yaml are
committed and pushed to the remote repo.
Environment:
SSH_USER SSH username (default: nixos).
SUDO_PASS Sudo password (skips the interactive prompt; useful
when calling from another script).
EOF
}
# Prompt for the sudo password once; store it for all _do_push calls.
# Accepts SUDO_PASS from the environment to allow non-interactive callers.
prompt_sudo_password() {
[[ "$dry_run" -eq 1 ]] && return
if [[ -n "${SUDO_PASS:-}" ]]; then
sudo_password="$SUDO_PASS"
return
fi
# read exits non-zero when stdin is not a terminal (e.g. CI, background
# agents). Catch that and give a clear message rather than a silent exit.
if ! read -r -s -p "sudo password for ${SSH_USER} on remote hosts: " sudo_password; then
echo >&2
echo "ERROR: stdin is not a terminal -- cannot prompt for sudo password." >&2
echo " Set SUDO_PASS=<password> in the environment and re-run." >&2
exit 1
fi
echo >&2
}
locally_managed_hosts() {
for f in "${keydir}"/*_ssh_host_ed25519_key.pub; do
[[ -e "$f" ]] || continue
basename "$f" _ssh_host_ed25519_key.pub
done
}
# --- git state check/fix --------------------------------------------------
# Hosts rebuild from the remote Gitea flake:
# nixos-rebuild switch --flake "git+https://<gitea>/nixos.git#<target>"
# so .sops.yaml (updated recipients) and secrets/*.yaml (re-encrypted DEKs)
# must be committed and pushed before any rebuild can succeed. This check
# catches the common case where --regenerate-all-keys was just run but the
# resulting diff hasn't been committed/pushed yet.
ensure_remote_current() {
[[ "$skip_git_check" -eq 1 ]] && return
cd "$repo_root"
local dirty_unstaged dirty_staged
dirty_unstaged="$(git diff --name-only -- .sops.yaml secrets/ 2>/dev/null || true)"
dirty_staged="$(git diff --cached --name-only -- .sops.yaml secrets/ 2>/dev/null || true)"
if [[ -n "$dirty_unstaged" || -n "$dirty_staged" ]]; then
echo "Uncommitted changes in sops-managed files:"
[[ -n "$dirty_unstaged" ]] && sed 's/^/ (unstaged) /' <<<"$dirty_unstaged"
[[ -n "$dirty_staged" ]] && sed 's/^/ (staged) /' <<<"$dirty_staged"
echo
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would prompt to commit .sops.yaml/secrets/ before continuing."
else
read -rp "Commit .sops.yaml + secrets/ now? [y/N]: " ans
if [[ "$ans" =~ ^[Yy]$ ]]; then
git add -- .sops.yaml secrets/
git commit -m "secrets: update recipients and re-encrypt for host key changes"
echo "Committed."
else
echo "Continuing with uncommitted changes -- the remote won't have the"
echo "updated recipients until you commit and push."
fi
fi
echo
fi
# Check if we're ahead of the remote tracking branch
local ahead
ahead="$(git rev-list --count '@{upstream}..HEAD' 2>/dev/null || echo "")"
if [[ -z "$ahead" ]]; then
echo "NOTE: no remote tracking branch found -- skipping push check."
echo " Ensure the remote has the current .sops.yaml/secrets/ before"
echo " triggering nixos-rebuild on any host."
echo
return
fi
if [[ "$ahead" -gt 0 ]]; then
echo "Local branch is ${ahead} commit(s) ahead of remote."
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would prompt to push before continuing."
else
read -rp "Push to remote now? [y/N]: " ans
if [[ "$ans" =~ ^[Yy]$ ]]; then
git push
echo "Pushed."
else
echo "Continuing without pushing -- remember to push before running"
echo "nixos-rebuild on any of these hosts."
fi
fi
echo
fi
}
# --- key installation (shared) -------------------------------------------
_do_push() {
local hostname="$1" target="$2"
local keyfile="${keydir}/${target}_ssh_host_ed25519_key"
local pubfile="${keyfile}.pub"
if [[ "$dry_run" -eq 1 ]]; then
echo " [dry-run] would scp host-keys/${target}_ssh_host_ed25519_key{,.pub} to /tmp/"
echo " [dry-run] would: sudo -S install -m 0600/0644 to /etc/ssh/ and rm /tmp copies"
return
fi
# Upload to /tmp (writable as nixos, no privilege needed)
scp -o StrictHostKeyChecking=no \
"$keyfile" "${SSH_USER}@${hostname}:/tmp/push_ed25519_key"
scp -o StrictHostKeyChecking=no \
"$pubfile" "${SSH_USER}@${hostname}:/tmp/push_ed25519_key.pub"
# Install via sudo -S: the password is piped via herestring so no PTY is
# needed on either side. -p '' suppresses sudo's own prompt string.
ssh -o StrictHostKeyChecking=no "${SSH_USER}@${hostname}" \
"sudo -S -p '' bash -c '
install -m 0600 /tmp/push_ed25519_key /etc/ssh/ssh_host_ed25519_key
install -m 0644 /tmp/push_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub
rm -f /tmp/push_ed25519_key /tmp/push_ed25519_key.pub
echo \" [ok] host key installed\"
'" <<< "$sudo_password"
# Drop the stale known_hosts entry for this host (public key just changed)
ssh-keygen -R "$hostname" 2>/dev/null || true
echo " Done. Run nixos-rebuild switch on ${hostname} to activate."
}
# --- single named target --------------------------------------------------
push_target() {
local target="$1"
local keyfile="${keydir}/${target}_ssh_host_ed25519_key"
if [[ ! -f "$keyfile" ]]; then
echo "ERROR: host-keys/${target}_ssh_host_ed25519_key not found." >&2
echo " This target may not be locally managed (e.g. &${target} was" >&2
echo " registered from the host's real SSH key, not generated here)." >&2
exit 1
fi
local hostname
hostname="$(flake_target_hostname "$repo_root" "$target")"
if [[ -z "$hostname" ]]; then
echo "ERROR: cannot resolve hostname for '${target}' from the flake." >&2
exit 1
fi
echo "==> ${target} (→ ${hostname})"
if ! ssh "${SSH_OPTS[@]}" "${SSH_USER}@${hostname}" true 2>/dev/null; then
echo " SKIP: ${SSH_USER}@${hostname} unreachable."
return
fi
# Sanity-check that /etc/flake-target on the host agrees
local live_target
live_target="$(ssh "${SSH_OPTS[@]}" "${SSH_USER}@${hostname}" \
"cat /etc/flake-target 2>/dev/null || true")"
if [[ -n "$live_target" && "$live_target" != "$target" ]]; then
echo " WARN: host reports /etc/flake-target='${live_target}', not '${target}'."
echo " Pushing the key you specified (${target}) anyway."
fi
_do_push "$hostname" "$target"
}
# --- all managed hosts ----------------------------------------------------
# For each unique hostname derived from managed targets, SSHes in and reads
# /etc/flake-target to determine which key to push -- handles the case where
# multiple targets share a hostname (e.g. lxc-server and proxmox-server both
# resolve to "server"; only one is actually running).
push_all() {
mapfile -t managed < <(locally_managed_hosts)
if [[ "${#managed[@]}" -eq 0 ]]; then
echo "No managed keys in host-keys/ -- nothing to push."
return
fi
echo "Pushing to all reachable managed hosts..."
echo
declare -A seen_hostnames=()
local t hostname
for t in "${managed[@]}"; do
hostname="$(flake_target_hostname "$repo_root" "$t" 2>/dev/null || true)"
[[ -z "$hostname" ]] && continue
[[ -n "${seen_hostnames[$hostname]+x}" ]] && continue
seen_hostnames["$hostname"]=1
echo "==> checking ${hostname}"
if ! ssh "${SSH_OPTS[@]}" "${SSH_USER}@${hostname}" true 2>/dev/null; then
echo " SKIP: ${SSH_USER}@${hostname} unreachable."
continue
fi
# Ask the host which flake target it actually is
local live_target
live_target="$(ssh "${SSH_OPTS[@]}" "${SSH_USER}@${hostname}" \
"cat /etc/flake-target 2>/dev/null || true")"
if [[ -z "$live_target" ]]; then
echo " SKIP: no /etc/flake-target on host -- can't determine which key to push."
continue
fi
local live_keyfile="${keydir}/${live_target}_ssh_host_ed25519_key"
if [[ ! -f "$live_keyfile" ]]; then
echo " SKIP: host is '${live_target}' but no host-keys/${live_target}_... (hand-registered key, not managed here)."
continue
fi
echo " target: ${live_target}"
_do_push "$hostname" "$live_target"
done
}
# --- main -----------------------------------------------------------------
mode="all"
target_arg=""
extra_args=()
for arg in "$@"; do
case "$arg" in
--dry-run) dry_run=1 ;;
--skip-git-check) skip_git_check=1 ;;
--all) mode="all" ;;
-h|--help) usage; exit 0 ;;
--*) echo "Unknown option: $arg" >&2; usage >&2; exit 1 ;;
*) extra_args+=("$arg") ;;
esac
done
if [[ "${#extra_args[@]}" -gt 1 ]]; then
echo "ERROR: specify at most one target (or --all)." >&2
usage >&2; exit 1
elif [[ "${#extra_args[@]}" -eq 1 ]]; then
mode="single"
target_arg="${extra_args[0]}"
fi
[[ "$dry_run" -eq 1 ]] && { echo "[dry-run] no changes will be made"; echo; }
nix_extra_opts
ensure_remote_current
prompt_sudo_password
if [[ "$mode" == "single" ]]; then
push_target "$target_arg"
else
push_all
fi
echo
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] Nothing was changed. Re-run without --dry-run to apply."
else
echo "Key push complete. For each updated host, run nixos-rebuild switch to"
echo "apply the config and let sops-nix decrypt secrets with the new key."
fi
+87 -62
View File
@@ -11,17 +11,16 @@
# sync-host-keys.sh --regenerate-all-keys Remove and freshly regenerate
# every locally-managed key.
#
# "Generate/register" is idempotent and additive only: an existing
# host-keys/ file is never touched, and .sops.yaml only ever gains an
# anchor/alias it doesn't already have -- safe to re-run any time, e.g.
# right after adding a new host to flake.nix.
# "Generate/register" is idempotent and additive only: an existing clan
# var is never overwritten, and .sops.yaml only ever gains an anchor/alias
# it doesn't already have -- safe to re-run any time, e.g. right after
# adding a new host to flake.nix.
#
# --remove and --regenerate-all-keys only ever operate on anchors that have
# a corresponding host-keys/<name>_ssh_host_ed25519_key file. Anchors
# without one (&admin, and any anchor for an already-deployed host whose
# real /etc/ssh key was registered by hand, e.g. &docker/&server/&nix-cache
# today) are never listed, removed, or regenerated -- this tooling only
# ever touches keys it itself manages.
# --remove and --regenerate-all-keys only ever operate on anchors that
# have a corresponding clan var (vars/per-machine/<name>/openssh/) or
# host-keys/ file. Anchors without either (&admin) are never listed,
# removed, or regenerated -- this tooling only ever touches keys it itself
# manages.
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
@@ -39,6 +38,8 @@ source "${repo_root}/scripts/lib/ssh-host-keys.sh"
source "${repo_root}/scripts/lib/sops-age.sh"
# shellcheck source=../lib/confirm.sh
source "${repo_root}/scripts/lib/confirm.sh"
# shellcheck source=../lib/clan-vars.sh
source "${repo_root}/scripts/lib/clan-vars.sh"
mkdir -p "$keydir"
@@ -54,13 +55,14 @@ Usage: $0 --all [--dry-run]
<flake-target> Same, for just one target (e.g. lxc-server).
Reports if it already has one.
--remove Interactively pick one locally-managed key to
remove from .sops.yaml and host-keys/.
remove from .sops.yaml and vars/per-machine/
(or host-keys/ for legacy keys).
--regenerate-all-keys Remove every locally-managed key and generate
fresh replacements for every current flake
target. Destructive -- requires typed
fresh clan-var replacements for every current
flake target. Destructive -- requires typed
confirmation.
--dry-run Combine with any of the above: print what would
change (host-keys/ files, .sops.yaml anchors and
change (clan vars, .sops.yaml anchors and
key_groups, which secrets/*.yaml would be
re-encrypted) without touching anything. No keys
generated, no files written, no sops calls,
@@ -83,6 +85,10 @@ ensure_admin_decrypt_key() {
fi
local key_file="$DEFAULT_SOPS_AGE_KEY_FILE"
# Expand a leading ~ that survived variable substitution without tilde
# expansion (happens when SOPS_AGE_KEY_FILE or XDG_CONFIG_HOME is set with
# a literal ~ in the caller's environment).
key_file="${key_file/#~\//$HOME/}"
if [[ -s "$key_file" ]]; then
echo "Found existing sops age key at ${key_file}."
@@ -91,36 +97,23 @@ ensure_admin_decrypt_key() {
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] No sops age decryption key found (checked \$SOPS_AGE_KEY, \$SOPS_AGE_KEY_FILE, ${key_file})."
echo "[dry-run] Would generate a new one here -- continuing the dry run without one; any"
echo "[dry-run] 'would re-encrypt' output below couldn't actually run for real yet."
echo "[dry-run] Continuing dry run without one -- any 'would re-encrypt' output below"
echo "[dry-run] couldn't actually run for real until a key is present."
return
fi
echo "No sops age decryption key found (checked \$SOPS_AGE_KEY, \$SOPS_AGE_KEY_FILE, ${key_file})."
echo "Generating a new one at ${key_file}..."
mkdir -p "$(dirname "$key_file")"
nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -o '${key_file}'" 2>&1 | grep -v "^Public key:" || true
local new_pub
new_pub="$(age_pubkey_from_identity_file "$key_file")"
cat >&2 <<EOF
No sops age decryption key found (checked \$SOPS_AGE_KEY, \$SOPS_AGE_KEY_FILE, ${key_file}).
cat <<EOF
Place your admin age private key at ${key_file}, or set SOPS_AGE_KEY (inline
key) or SOPS_AGE_KEY_FILE (path to a different key file) and re-run.
A brand-new age key was just generated -- it cannot decrypt anything that
already exists in secrets/*.yaml, since nothing was ever encrypted for it.
That trust can't be bootstrapped automatically (nobody can decrypt a file
for a recipient that didn't exist when it was last encrypted).
To actually use this key:
1. Have someone who currently CAN decrypt replace the &admin entry in
.sops.yaml with this public key:
${new_pub}
2. They re-encrypt every secrets/*.yaml:
sops updatekeys --yes secrets/common.yaml
sops updatekeys --yes secrets/nix-cache.yaml
sops updatekeys --yes secrets/server.yaml
3. Re-run this script.
Exiting without making any other changes.
If the key is truly missing (not just mislocated), this is a manual recovery
situation -- generating a brand-new admin key won't help, since it cannot
decrypt anything already encrypted for the old one. Each secrets/*.yaml is
also encrypted for its respective host key(s), so a running deployed host can
still decrypt what it needs -- but the admin key is required for re-encryption
(e.g. adding new recipients via sops updatekeys).
EOF
exit 1
}
@@ -133,10 +126,17 @@ discover_targets() {
}
locally_managed_hosts() {
for f in "$keydir"/*_ssh_host_ed25519_key.pub; do
[[ -e "$f" ]] || continue
basename "$f" _ssh_host_ed25519_key.pub
done
{
for f in "$keydir"/*_ssh_host_ed25519_key.pub; do
[[ -e "$f" ]] || continue
basename "$f" _ssh_host_ed25519_key.pub
done
local d
for d in "${repo_root}/vars/per-machine"/*/openssh/ssh_host_ed25519_key/secret; do
[[ -f "$d" ]] || continue
basename "$(dirname "$(dirname "$(dirname "$d")")")"
done
} | sort -u
}
add_keys_json="[]"
@@ -146,13 +146,15 @@ dry_run=0
queue_host_sync() {
local host="$1"
local keyfile="${keydir}/${host}_ssh_host_ed25519_key"
local has_local_key=0 has_anchor=0
local has_local_key=0 has_clan_key=0 has_anchor=0
[[ -f "$keyfile" ]] && has_local_key=1
clan_ssh_key_exists "$host" "$repo_root" && has_clan_key=1
grep -qE "^ - &${host} age1" "$sops_yaml" && has_anchor=1
if [[ "$has_local_key" -eq 0 && "$has_anchor" -eq 1 ]]; then
if [[ "$has_local_key" -eq 0 && "$has_clan_key" -eq 0 && "$has_anchor" -eq 1 ]]; then
echo "SKIP ${host}: .sops.yaml already has an &${host} anchor, but"
echo " host-keys/${host}_ssh_host_ed25519_key is missing locally."
echo " neither host-keys/${host}_ssh_host_ed25519_key nor"
echo " vars/per-machine/${host}/openssh/ exist locally."
echo " Not generating a replacement -- it wouldn't match whatever's"
echo " already registered (and possibly deployed). Remove the"
echo " &${host} line from .sops.yaml first if you really want a"
@@ -160,21 +162,26 @@ queue_host_sync() {
return 1
fi
if [[ "$has_local_key" -eq 0 ]]; then
if [[ "$has_local_key" -eq 0 && "$has_clan_key" -eq 0 ]]; then
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] ${host}: would generate host key"
echo "[dry-run] ${host}: would generate host key via clan vars"
else
echo "==> ${host}: generating host key"
generate_host_ed25519_key "$host" "$keyfile"
echo "==> ${host}: generating host key via clan vars"
clan_generate_ssh_key "$host" "$repo_root"
has_clan_key=1
fi
elif [[ "$has_clan_key" -eq 1 ]]; then
echo "==> ${host}: clan-managed SSH host key already present"
else
echo "==> ${host}: host key already present"
echo "==> ${host}: host key already present (host-keys/)"
fi
if [[ "$has_anchor" -eq 0 ]]; then
local age_pub
if [[ "$dry_run" -eq 1 ]]; then
age_pub="dry-run-placeholder-not-a-real-key"
elif [[ "$has_clan_key" -eq 1 ]]; then
age_pub="$(ssh_pubkey_to_age "$(clan_ssh_pubkey_path "$host" "$repo_root")")"
else
age_pub="$(ssh_pubkey_to_age "${keyfile}.pub")"
fi
@@ -299,7 +306,7 @@ cmd_remove() {
local hosts
mapfile -t hosts < <(locally_managed_hosts)
if [[ "${#hosts[@]}" -eq 0 ]]; then
echo "No locally-managed keys in host-keys/ -- nothing to remove."
echo "No locally-managed keys found (checked host-keys/ and vars/per-machine/) -- nothing to remove."
return
fi
@@ -308,7 +315,9 @@ cmd_remove() {
for host in "${hosts[@]}"; do
local registered="not registered in .sops.yaml"
grep -qE "^ - &${host} age1" "$sops_yaml" && registered="registered in .sops.yaml"
printf ' %d) %s (%s)\n' "$i" "$host" "$registered"
local where="host-keys/"
clan_ssh_key_exists "$host" "$repo_root" && where="clan-vars"
printf ' %d) %s [%s, %s]\n' "$i" "$host" "$where" "$registered"
i=$((i + 1))
done
@@ -325,7 +334,7 @@ cmd_remove() {
local target="${hosts[$((choice - 1))]}"
if [[ "$dry_run" -ne 1 ]]; then
read -rp "Really remove '${target}'? Its host-keys/ files will be deleted and it will lose access to every secrets file it can currently decrypt. (y/N): " confirm
read -rp "Really remove '${target}'? Its key files will be deleted and it will lose access to every secrets file it can currently decrypt. (y/N): " confirm
if [[ ! "$confirm" =~ ^[Yy]$ ]]; then
echo "Cancelled."
return
@@ -338,11 +347,13 @@ cmd_remove() {
apply_edit_plan "$plan"
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would delete host-keys/${target}_ssh_host_ed25519_key(.pub)."
echo "[dry-run] would delete host-keys/${target}_ssh_host_ed25519_key(.pub) if present."
echo "[dry-run] would delete vars/per-machine/${target}/openssh/ if present."
echo "[dry-run] Nothing was changed. Re-run without --dry-run to apply this."
else
rm -f "${keydir}/${target}_ssh_host_ed25519_key" "${keydir}/${target}_ssh_host_ed25519_key.pub"
echo "Removed host-keys/${target}_ssh_host_ed25519_key(.pub)."
rm -rf "${repo_root}/vars/per-machine/${target}/openssh"
echo "Removed key for ${target} (host-keys/ and/or vars/per-machine/ as applicable)."
echo
echo "Review the diff, then commit and push."
fi
@@ -352,15 +363,18 @@ cmd_regenerate_all() {
local hosts
mapfile -t hosts < <(locally_managed_hosts)
if [[ "${#hosts[@]}" -eq 0 ]]; then
echo "No locally-managed keys in host-keys/ -- nothing to regenerate."
echo "No locally-managed keys found (checked host-keys/ and vars/per-machine/) -- nothing to regenerate."
return
fi
echo "This will remove and freshly regenerate ALL locally-managed keys:"
printf ' %s\n' "${hosts[@]}"
echo
echo "Every host above will need its new key baked into a rebuilt install"
echo "image/tarball before it can decrypt secrets again."
echo "After regenerating, each host needs its new key before it can decrypt secrets:"
echo " • Already running: push the key before rebuilding:"
echo " scripts/secrets/push-host-keys.sh --all"
echo " • Not yet deployed: rebuild the install image with the new keys baked in"
echo " (see docs/auto-installer.md)."
if [[ "$dry_run" -ne 1 ]]; then
if ! confirm_typed "REGENERATE" "Type REGENERATE to confirm: "; then
@@ -378,8 +392,8 @@ cmd_regenerate_all() {
apply_edit_plan "$plan"
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would delete ${#hosts[@]} host-keys/ file pair(s)."
echo "[dry-run] would then generate fresh replacements for the same hosts"
echo "[dry-run] would delete ${#hosts[@]} key pair(s) from host-keys/ and/or vars/per-machine/."
echo "[dry-run] would then generate fresh clan vars replacements for the same hosts"
echo "[dry-run] (not simulated further here -- run without --dry-run, or"
echo "[dry-run] preview a specific target with: $0 <target> --dry-run)."
echo
@@ -391,12 +405,23 @@ cmd_regenerate_all() {
local host
for host in "${hosts[@]}"; do
rm -f "${keydir}/${host}_ssh_host_ed25519_key" "${keydir}/${host}_ssh_host_ed25519_key.pub"
rm -rf "${repo_root}/vars/per-machine/${host}/openssh"
done
echo "Removed ${#hosts[@]} host-keys/ file pair(s)."
echo "Removed ${#hosts[@]} key pair(s)."
echo
echo "Regenerating fresh keys for every current flake target..."
cmd_all
echo
echo "Next steps:"
echo " 1. Commit and push .sops.yaml + secrets/ so the remote flake is current."
echo " 2. Push the new host key to each already-running managed host:"
echo " scripts/secrets/push-host-keys.sh --all"
echo " (this also prompts to commit/push if step 1 wasn't done yet)"
echo " 3. Run nixos-rebuild switch on each updated host."
echo " 4. For hosts not yet deployed, rebuild the install image (see"
echo " docs/auto-installer.md)."
}
main() {
+187 -87
View File
@@ -1,135 +1,235 @@
root-hashedPassword: ENC[AES256_GCM,data:Kp0nOZI7vDoLhJHiOJBwJn0rQZ5yhnwapGnAcA+qh8vlDETtFs/iQdetF/2ZxmANf62SviTNd+Ag0q5JIF1996x7onZGXqxgSMCuVzZLBdUlsO5IR0BslWWz47khYGTe4WkUg4NB1itBfQ==,iv:5Sra5vJ79V8hxQT3g9qJ+dOj2W2sumIhqpitqnHjJdk=,tag:3Igu0+8GeUZHqS3fKUVwog==,type:str]
nixos-hashedPassword: ENC[AES256_GCM,data:pT7tVRN6X4a+DNUgB7fIUUE3CbnetkjxmoSL1PxSU+ktsFU+fB0mEvJjA1uujsGH5Rcztg7YM815+M0Z67ILmHaXbza5DtFacrqhi4/b277xly0SHRX4yOvBwQh6mJG1jn/0O/wvUUIYdw==,iv:bp2nfhC8nFbk6o5iWDAugvbzu7J/a1xayFnBEtkhNpE=,tag:HqWgkIpSrSM/K9OK2WO+VQ==,type:str]
nix-github-token: ENC[AES256_GCM,data:OfNRGJg16Ede6EilWUetCs9za+xk5/Lsa3SpVajsqz8PMdA1xQNeCWdX7ZAMdijHClpBhU6ETFGsXvt41O9aORS951uijeGSW7/NH35/bnPISrKdYeBx/+xEiqwH,iv:QGU3v7xOy89uzRTCb1U9ICyJ8XYIpXrUsDt12aL3g2Y=,tag:Bde2wcWNv8H4WLxSEUAodg==,type:str]
nix-github-token: ENC[AES256_GCM,data:k1vYz7SqVhzpWa6jTL6NUD8lKOCpHCgTm+HT4IcnbzbSTUZP/bJUYw==,iv:UqAULZnr/4+VcioUDfTwvOSuwM8K9JgGhiApvYQPyoc=,tag:1LKHXhWAO/AHPDIZFBb04A==,type:str]
beszel-token: ENC[AES256_GCM,data:ds7OFjIXpOe/OIiEIydK9qsYkq3rMShK+jCTRRHKLzxUV0Bl,iv:nEt5FxkQaiTmAPFbj7vGJIMAEBjXmx+XYcdqaAxGzo8=,tag:+ql/UsgBR+TPySG9VDZi1g==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIakJkWDQ2TjVIS1Q1UlRJ
elVLNmhEVUpMNjROWmN6VC8wTnZzeWhLb0hFCmxmTXp3ZVdQMUhDeGlscThSdFhP
eWNZSkFpeEg0cUUxbGFyQWIrTkJkc0EKLS0tIGRBTHdaZ2d5eHhxd3BtWVBLbFlo
OVJlYnQ0N09qWFp2TmtXb2E0Wkk2bzQKTxA4rfkF3qlGpDqaZF/J9tgTx1UZ7ZFt
W1KevMq/Klnkjb9XDuTEx7zgrjILoViVspe6eGI14myBNYtAP4nLSA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiU2IxZ056SUs3SVZiUTlQ
NXZEYmg3VzJzeWJPRnZXcUtOMGQ1MElBU0dVCm5xSURZZ0sxTjVxQnhpVVNEU092
bDBHbFAzZ3hzejc5NmM2U0FQTkRNSTQKLS0tIC9OOURPRlcvQXIxMVZXR1JjUWRX
bE5ja3FoWXFIZ2ZLTklqWHhMRHBTRWcK2Z4sEcdyvx1rdBIGrbNw7bYvEPXdobvF
h0aNihn817DvtVr8eeucj22iYCS40etElQNcO5i68t5wab4/d3rUbQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUWXYyUG5UdHdFa2NDZmtt
ZG1GSmxVbTg5Qnh4UGxJaitmTmFqK2J1cUV3CmdPSU5GTC9hWkROTkJzMk5Tcmpw
ODJzdlIvNkp0ZXdZNlRlUHBkOGVoR1kKLS0tIGsvUWxpU2hkanFNaExJWGozMTJL
bndPNkpQNktmQVNKNEhGOEtHSlRBRXMK9rr6NHf3H91GIqTmckjD1eV94FW1vk9G
h2KauyebWPyBe8hsCExiPd45ZGqKF0g8pEtWUzonMq9NU/MNVL8YVQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrN0lNNGtYbE8rRTB0U2hL
eUNOZUNFRGk1RzE3R1VjV0Q3ckszcDNHWGxJCkhFN25JbDVSaWJIZ09uNzczbkM1
ZWZ1Y2wrTHZ6VzlnVWRoT2JOdDZHM0UKLS0tIHQ2ZE9BaFRCeDNqbldRQkpLclI4
dXZqOXNaR2Q2QWJtRlZUa0hLaTh2YlkKeCh3+yIt0sjh2Ig8m/YQifhMDHnTpuZE
zHBE74SUyWFmqLgtd1b8Rx5gn9vBcS/3SMv3yKWispg4DZ3bCOeNFA==
-----END AGE ENCRYPTED FILE-----
recipient: age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e
recipient: age19m0m7vdfg86yqy8l5mmle5jdd0unrn3f55t232w8h5ey42cqw34sfpt32n
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXdnpGR2ZnaFRSUUNlOUVy
ME9pWkFJRTl6ampuZTR0QlQ2TUVEVHV3dkRFCmxML1NSV2dwanRPK3AySDJiUU9P
a2VKZ09EaktYU2xvWWpESWtrU3oxalEKLS0tIEt1aWxuNmlaV3l1OGNqS2RtMmhm
Rk9SUEpBaXY0RG9uMk53ejdJdVZ4NDQKfn4paPsHrfU3Ki2AgPBB8aLBbmD2yh1O
9rDxv/6xSsDXNTquP11smPOKsRG7mDMDHVByn6GieZrpSxUf9vu3Iw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5c2luYmpYcVVxRHROQlpk
cTM4cm9uNEtVS21pK2pDRVRySmxYS3NwdGg0CnQvS3g4Q2tSdlpVeWZxbzArcTM1
YWRMcDAxcWgySFRqRC9XSElsTVorVEEKLS0tIFd6YjdrY2ZXQVBaZ01wMTIvV2sv
ZEk0TG0ybENlTlhpZnovMXh0OGd5V1kKEbjrmKP6Su16KRDQ4PMqi0tHRS6+PfuT
0v5DEyi8EDtIYSMzzdF7Jk2C0h5XuRux45Z6E6JhKW5c788dGQHz/A==
-----END AGE ENCRYPTED FILE-----
recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
recipient: age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArNHZpMDV5a2JCN1NxY2hF
MWxtNUNVQXp6K3lWYndTRU85Wjh4YlN3UFFrCnNZbHA2Z2RUSmxITm4yeU9rYnV0
TTFkZUczcTZLU2pJRDQ3TVl5ZGErWDgKLS0tIDJGbzZZNjQ2RXUzRmUwKzdiUHlK
TEU5VERpZ1p3bU1KcnEvRDRCSVNqT2MKQPfu0lskXaEAYgecmN1a0kPHF+fGEm7R
wiY7TFNLeXM5aJqEnKARtOotJDetI+9VNssTT21X/Qaik6fqgM+b1A==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1eW5kd1FrSGJaSDkwTzdY
SnJhR2hqRkNTTGtSTWsvYmZpSlVWV0xvYkRZCmtqR1FQOTRQR3dGYU8zNFpMdkUr
SWN2KzhXaE1rYWZYNHJsOVVvRDY3R00KLS0tIG1aUldSWHlHeHkrcjF2TENvazR3
YXJLUXV3VVNSWEdseFlZQUJPQUF6OFkKOPKH52muUha+MV2leWDiqpuTaIj0ZKzV
keEdyqkIck6KKfIjphNKTW5ujiuKjGQbRliquBknfCRGMnRlMNfmkA==
-----END AGE ENCRYPTED FILE-----
recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
recipient: age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOUlp5MmcvSFRnaVdWZ1g0
NmwvS3pGa1JCd0RuQnFMeU5wZDZYK0NQR1FJClF0Z0VENThtTzZ1OUtmUjdDenV4
Mko0T2o4UzVubVRCdUlCMERyV2w5WmcKLS0tIGZUallPeUhXUGtVMmNIWWRkaFVT
TVNVTjhiQzJkdlVQN3p1bnR1aUYyQmcKVnqbCuaSYEA7stk1MyfCzRbqt7EL+E5/
jgxraiFmaZqjDI5mxG/e7eFdXfv53AtKzZm92TlzvV+4bfidj5SvYw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmTDExNFpCZHZmWnpwYis4
bFJGZXV5UksyS01MT0d3cXE2Wno2Zzh6OFFnCm9hT3MyZ0hTbnkzdXk3NmN1TUVF
OGtCZnlJRmZJaWJWaE1pMGcrTzJzZm8KLS0tIFE1L3ZxeUl6WVA2UXN2b0lIcTBl
YmtXZnZaMml6ejY1Y3lRUFFtTEVMS0UKZbcuunA9GRqctCfC0Fh2foKt7n3l8KkE
FNjjxUzL7OQjNG9M1aAtRMMO5KXL7YKXRUJWOPcpiLeLUNvDLeLXvA==
-----END AGE ENCRYPTED FILE-----
recipient: age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep
recipient: age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCWTdCWHhTamI1VHpvUzZ3
M2VnV3NFZTcrc2Jzazc0V3NsMDQ4RmtYZ0ZrCmVhV0pweTVLdzBLRnlJZURwNXVT
YzN1aHZyTmRqc0dtYjJWekRaODZoR3MKLS0tIHZPV2ppSTVwb2VHWStRY0pKSTZ5
MXVFSE1odzRMeW9LQjVQUytBTWhHUDAK+UUowhIQ3w5O4Y/m9Pq41X2l0DZQfzOT
itFkXJRnvUmkCxWYRDFJjQ6FpHHaNDqs0BlQZ5QZ+chzcKuUAG+uEg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQYlp5U0FvVkdIKzBGZVRl
UEtOOG5kVHVLQ0ptVTVpdG9Lb1Y3bzlOODNzCmNYN0Y5ZGNzTGFGYmxJcWFxaExn
NkkrT2RLSlZTdmFiNks3RVZXQlBEYnMKLS0tIDdCVlBxUG8vYU05d3c2MXhjQjhM
TTVJUGxlWUxWalBRYzFKQUQzd1R6dHcKPz1ciNICph7qnqSH5WYmOjUCNHsIAmTX
lRnbFCZdD/ZIFRK8OYAyCGfQ42UYSTf16kH2jDdTFUy7Fk2+O7lNIA==
-----END AGE ENCRYPTED FILE-----
recipient: age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu
recipient: age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2MEZxZTI3Y0xWSDdzanRC
RytNV2x0UkY0SnRzVnR0eFNvaDRJeUtpZjBjCko2T09qSTJsQm51aW9pR3JUbkRm
NWRXODg0L08xdGNLdCtORXA1VWRteTAKLS0tIC90bkZEd2pOYWFCeGg3UW54TWxH
Y1liQWVWS0t4WXBROENGNnpDVUpxRVUK57qCQ0l5Gw1ZqM50XSBNwVlXkue3QOT/
BPmPFkMNpVIcUDENj+mJAS5GHuEm1MkZcMi/wN9Hp2KE05ZywPR2Zg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiemVqWW9LdlV0RjRuRHpF
U2MwN3RDeHU5SWhuQ2l4YStRVVE2eU9XSUc4ClNGMkNXQVdkWFkwcys0aFVhZnZv
Y3J6RmxwakdaZVo5VWZhb0lVTUdTYzQKLS0tIEQ2NW0yb2lGQ0NLZm1PSWNFVkQ1
a0Y1NlYwZ0dnbHhsQVB5eS9kZVdqbmsKgFLUpZMqGywO9Ext5WRwco2bpenVNrzL
XrGCqhn/7TW9DrJlPnFe8r4fl4DFP6GWByn7axDl58Qy1OzeBZwnNg==
-----END AGE ENCRYPTED FILE-----
recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
recipient: age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCaGhvbE9EU1dMZitXYlZk
NVBEYWxHbzAwQ1FUU2JidjJPZDZLVDZLODI4CjYxaG9OSlVOeGQ5SmlGNWEzNG1n
WU9PRVpVRk9rd1BvWDNnMFcrZFNJZDAKLS0tIEUzV0E3OGE5cWM4MjBwTk9Mb3Np
cDJQZGNqNXlySjRLcWF4R0V0ZS9DVFEKS0CmXOfJ1qpUF24EJT6F5/6xzR7h593O
mLiMdOCeFYWlTtBwrD1dkweAzStiHlzTVEq3w9BFvAsr5x/NbQJUJA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLRzlzWEcwTUx6M1JNYWgr
ZWxnVXhFWkhQVjBrSndGTU9oY2MrOTQxTGhFCjhUdERPc3hzQlkzMjBEa3AvaWVX
T0RJWXFFZHdmUVBObGhpQktCQ2NPbWcKLS0tIDN2YkE4bEVtbjJlcndWVmlrQ3ZT
dlVvRmt4RklOdUJyZ25ia0ZoYjU4ZHcK5fL1yvRcty6+EMzfSkwwJM3qvrr6h46d
d7Zs+EElXrsGDQvL0fevB/zDcICAtudYnVv4nP3Jf8GQnL19zWlwbg==
-----END AGE ENCRYPTED FILE-----
recipient: age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg
recipient: age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAzOUxYa2xOYlVYd1lWclR1
bDJFTEg1SjBhNjhFT0IwY0ZxZFRFTzRUVlhZCmpwL1VBMElZNzV3UkFCV0FWb3Jr
SlN2dTNmOFRXeGJVTFgwdjA4SmdSQXMKLS0tIHZOdjFzalRkL2N5eU1iSDkzbHYw
NEU3VWxBNE9NZy9hVFJwM0VnQ2RIbjQKc62J04UtVjqiU7p7GueMicdCDRTvM9zY
IPOJSwTCatRMWeuBJIsRNkbyOLeSAesQdfAXL5GoAE8mBtBdhJKaLA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhZElRNXNLK0tVUjQ0ektQ
Tzc5NDJReldiYVFFZnRLS3l5dWw3UVQ0WWhJCm1DYVloZGpYeVJtNUNRcEJvVUZE
Q0xMcmR0WWdVVTBaWko0dGI5V1hqaUkKLS0tIFBBeXdNY1pFai9QT2t3NWFLV2sw
cWpRU2RISSs1UXh3Wkl1dWI3VEk1bXMKP6v3Lm0XZkp3Gn3iJkB0K0Y0E5BeilqR
plmPh1rKbT/wkefFwpa8e2Wwmhx80YK+bjn633MH7mLQDf6AiK83Xw==
-----END AGE ENCRYPTED FILE-----
recipient: age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs
recipient: age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1NkppWjNXRDlsVXZ6b1ND
OExFRXVOcXVSWmZXemNoK09EcXBHZ0JMYnd3CjlsaEM4MjdJT1lBZGFCZTNISHZV
MjBRTUlTVkZBQitFc1I3ekRLN0RtYzQKLS0tIC9oelk1TTRzeFNpbWQzL0pObXJq
eWxHbkF3anVPRFgyQ0kvSlJqQTdLQW8KLfZ+UIzas43ROGO51KKOmy58R2tl3jwh
M+1WXB8vIXA7DLQ6vMbzMPMhBnJ4kbZgRtxELg7fzSLQStLvkY+O9A==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMVk8yT1FtMVoxYjF2YVBs
UTFVbmhzblBCRno2V3YvaERRN0xWcHd2NVE4CmVyK3p1SWtodk5iWjdkQjJRc0dM
Vm8zZHczYlFFSG91QThmUGNaWHBoN0UKLS0tIDZMMXoxbUM0TUVaNFUzRExLc2VM
OVFrNGZNL256cjg0Q3ZBQW9xUFcxSUEKv4HqaTcTzNYRJCmNEeUQLFfEoFXyOBEC
yZrg8D+ROBn98484bxNeoyOjQdcVn7OYiCv1y6utnyeA9sdMUB0ong==
-----END AGE ENCRYPTED FILE-----
recipient: age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp
recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWVm5uZU43cFFLcC8wVkZO
NmlOMmlocGpwbXFkUFlCMEl5Tkx1QWJIdVRjCnFBTzdWT3NaM0N2eGlCbDV2NUU3
Z3FnYktsZ3NHT3Y0b2hOd0xmRjlLMjgKLS0tIFBQVXVDeHkyeVNNek5CSldSMFlY
a0Q5bHVlRHUxMWduMCtLRjdnMUw5RG8KDJX8I21+bUpkJ5wnX+kGhsSa8mKqsR25
iNdOtVtb2WRxIAgyjR72yp1tLMrop9NkZN68MGvd/LsNM+6simyosQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1Rkl0UXduS3k3WVU1RzdG
N2IybE1TZ1kwVmM4Y2lvUS9DVG0zSkdpZGhRCnFkZnJVS2FBZm9Xa0V4SjRwaGdR
TWpaaTh4VEk4cUFrb2pURERzbWtSR0UKLS0tIEsybHdvUWg3UlUvL0lhdHIrUE0y
R0pMdnI5QlpIcWVWdGRYQ3JCaElLWTgK5cTfvehj4BPU22tYiR5piZdouUdeh5Gr
x/xSbFWqYVauDB6rpzZvhGMI4KCJh2RmDdt7FrlCb/YCB3QoSgvl8w==
-----END AGE ENCRYPTED FILE-----
recipient: age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05
recipient: age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBHcmIxZVc4aGRmNEhSY3Iv
YXhzLzduWFdzVVlKZTR6WExzcDNtSWQza0FjCmlsZDVJUWpqY0pRSktQeGk2SlAy
cjZXL0JCVFRtQXpHNkNIejhyTnVtWjgKLS0tIHBSZ1JyMFBJVWZ3VWFLOUt5OHd6
YUI4ODhHc1I2UHVRQ0diSnhsSUhDelEK3hBBX1+Uwe/MusLqmt4oAy7Z6jOU96Cd
7zTe8YyLc0/DUDsyuFZ7a68riO9/My+zrFllwtKe/JPmPHLY0+3Pnw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwL0NNbGlGS3lTZjEydGJm
MWZRK0Z5YktuNVNudk84MndLUUdXR0RlRFF3ClB4MWRUTFBWYUdwbityeUU2SlR4
azZMb012YXQzWHUxZnZzeVp3MmdzcU0KLS0tIFJ2SUNyMTBEY1NWejdVQ3hyUytQ
VnlQWHRDYWZFaU1ON2RQTXBwNFNGYUEKwnRUWCdsiStia1jLHJ6UNP/kgEt4Aw8U
pA8PKKu6TZVPI8pw7svtI/uvJuW+8ARuNcyb1FdYmX+sOHnOEqHyjg==
-----END AGE ENCRYPTED FILE-----
recipient: age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
recipient: age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZSE53Yy9FOHNpeVFtTmF1
UUExbzJlZktxMGR3Ukora3RpbVN2THlyVjNJCnFpWVEyK2U1N1diOWZQWm5Bb1lD
djV0U3QxamRJY3BRN3hBTmZ4SVRiV0UKLS0tIFU2Q0o1UTRjaEFaSTRVTm5MVW5T
QXdEbVB0Y3YwM3R5dGp5d1oxeUxHNGcKRj5hNLlXtZoT3IwXHTxaReJLu8k133n/
ZoCtv470LPL1M/kTjdPc/nWMgYOHDKkO90gr3WILfHg8idkVuCLT2Q==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxM0FyaFlTR2FzMkhGTzZx
d1d3dWNGOG1ESllETEVxRGJlYWkrTUNIdkYwCmI4andLMUUvNVU3R0N2WCsvWWhV
UmlkcHdMSFA3bFp0ZG5OckE3YmJrejAKLS0tIDhEVEJYOFc5VWVHcGcvMm5yanBl
eGxaNEFxdEN4NWh3cnlGeElBUTJmUzAKvCSJF+LGf1JqEdkzT/Wk6hEzferseyWO
vAYR8Z+u6GDwQi7JdFz9iZKD21LAs0IMzd7cZqXKIXVedyXuaIhPtg==
-----END AGE ENCRYPTED FILE-----
recipient: age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
recipient: age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQazFqSWpaVGtkNFhPU3A5
cE1oVXEvSU5RNnY5NU9mbFhGOFQycVFuL1U4CnNwRnVkYmxCNFZzNVZvM2N5eVN0
clNjUWY4RHZxbVNsMFlFSnZKcmI3dm8KLS0tIFBDZTRuRlprdzdVT1ZHejNsNDBJ
OUVjclpPSjl0RGtLVlZRSkpFeG02KzAKt0rcJunZppojjijrjbXsztLwRD9pgWRb
jYsn7dCvWGFZGVgeyxqfL0jfYNz6dW0yecRE/gbP5hIooxbcPcxfag==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGZ2M0cG5nMU5nYVliRzlx
TVpDcHpwclhXNkxuc2VKanBvcnRxRW04UXh3CmNJeEl3K0FrWWVHQWRwK3Q4Tzk2
MEZQbFV1QURLZEhCdEhaMjBRUzllaVUKLS0tIFhSUTJ4SXJwRHhVbkZFc2Rwb3Fh
QzAxTnpWSHZaWlF3NUlFMlBHUkRUa2sK3Kim0mp2adSvruI6Necd3UpZHow0EFmT
JFOIpIK+3Pc06gsiqrDtDAx2Qtc10Ja4ILTutPlp4va6zzmzrPABiw==
-----END AGE ENCRYPTED FILE-----
recipient: age1ehkswwz2pqaz4svzh7ela5tdnssl8kn6d4vwwxd6zwg8exfpd43syyrrjp
lastmodified: "2026-07-19T02:30:40Z"
mac: ENC[AES256_GCM,data:UiL3VMDF6rq4Nr87KspcDx434q3tfNXeb5pwH2O+4ssNQ6xzcYDdzXBnhAY3zLBsqPMKrvHBd4Ot/gEMcq3FMIVe7Q6p9yWKpep66KZ/yWEhAlwIVhD79Oj8VS+1CHKjf25zpRdhZorp04oeFQQd9VfjJB4EE/Q1aVbwTGlpIic=,iv:i/0conaFgFia+wzNTdUL6tlSTw35HTK3Ap1Sr5RGHf8=,tag:ULbz5FllShA/JjlSRdxA0g==,type:str]
recipient: age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwTUZ1RzJKcVF5bnJGamlU
bWJvcU1sTTJuME94YjhMZWRWRWR0aGJ3UUNvCjh6U1BkaEg0TDlCeTNyUlNTRWV6
Nnd0NjE3NmplSWFjVWpSRy9qWjBUbnMKLS0tIGdMZUFzbGdMejZoMlFBVUJXSE13
TW4rUHFjZk5oVkw0UGtpaG5mOHRRbHcKFX4ZyNQ0TUeqwXEaz1qmAXQRIOctDkvz
TIwxJseAjS3qVxCslnxzgaqmCh8DzCfjrRxLUS/HVTnwOaE1YAh0vg==
-----END AGE ENCRYPTED FILE-----
recipient: age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTWStVRDhlblZDMDJ3ejIr
MkwyeHRwcnpWQVBmOXY0a3NCU1JnTjkvVGtVClVvVUpLTVhUNGNNLzY4U0o2SDRD
bGZteUU1Uzlzc0NjTW56OVNCcnVpUUEKLS0tIC8wN1V2Nkc0cVU3OHNYa1NBVlhq
YTV5QUpsQWppNHRpT1dMb2JDQnpoR3MKCoBngnNg5qpktg5j4CcuTUKiooOVI8IN
04plE+1lra8q5ZK8q4r1/K8hBM23poyfq+XushO1GZ17dPtWs9wITg==
-----END AGE ENCRYPTED FILE-----
recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUU2ZSazhZNWFZVXk0UXdx
Mjh3R1NLMThlaHUvWVFMZEQ4Z0NYOW83dWgwCkVnd2dhdzQrZkxaL1NRbDJodERQ
SkM5S0kxcVBtY3laNWp5OXpXZHhqZ0UKLS0tIHVOT2VUZU5DZjJjVW1kcU5qS3FD
YXNhRkpaREMzRDB5aWgrdVNvUk1qcWsKQnPqE+5xSBGeDMmLfKB0tEBt2eylBOxT
oi26mC+EoO5T2pCIpbD+DODAAlWKLattqCYHneUO9wRk9DnhuEXRBg==
-----END AGE ENCRYPTED FILE-----
recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZUlFlenlOVDk1NmxQVGUx
UHhVK3ZtMHdzaXdtWmE1TXhOekpsTmUwSlJ3CmdncjMxOUw0alBQZkhZZjA0YzJZ
bFNVOHUySXRuYW8wVjdHMDN4ZThuQmsKLS0tIGdvUUdHcXlpWlZkL0xBcVpMN1E1
NGZCQURYQVY5ZzdDODNNY0o3SG04eDAKiYVuKsJLaSC7/bQRB51tX282/yqpAjWT
XNnqPQBQ7H3xuK0tDVdCAUMruOSlqAe9YRZwmiou1irpg9vZYwgTPA==
-----END AGE ENCRYPTED FILE-----
recipient: age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnUUlPS3ZPS1lSVmxwV0lq
WWE2QXltb1BzU3p3cFRuMzVieGJta05nWno4Cnk1RG1CUXZjM3dNQzlJK1ZGaHpZ
b3drMTB2L3hidkpoTXdMQ05vOUtTb28KLS0tIHhyUllpQnlzL093MCs4YWx4VlVn
S3Q3QnpVNEx6ek54RGx5bmlqUk5hbVEKHVUvfk/xKtlOk2o73LodweIivSB4Oii/
6dwnE8wa+Qx/OU0vDJVelKOUE6JiLwckX/0u2YIUOybkx49n+tYs/Q==
-----END AGE ENCRYPTED FILE-----
recipient: age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3MHlrTTdxYit0MTkxa293
eFFIcmRLMUpkZjd3eWdsOXYySEljNDB2NUdNCnAyanVkS0dmenZSa2dpSWZTSkZw
Tm5LMkRoWEk3L2ExVktnZDZMM09hZ0EKLS0tIGVGcHFuMjh4OTUxdHNJQSsrR1pq
WU1jVG5UN2hMbW9iQ3JmbHlwc2VveFUK4JC32glesv/9nMWBR0lFeX7MPgIt9sIo
9HsAkOMTxPTI0NZW8q79+G5hq8qG+QvQhwO6wQtwz/KEHPXntKDX+Q==
-----END AGE ENCRYPTED FILE-----
recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRbEZGNmtlUjJVZ2dlVGZC
MGVEQXV1SFRpMjFqWks1Y2F6c29TY25oL1MwCjVFRk90Q2dJRXhUMmIrRVdHVFRD
TWgvQS91VWdFWERJWEYrL3VnRXFraUkKLS0tIExUU01VRWlENlpBSGt3V1Izc1dS
RFBhSTVKTGV2KzFidDQ3b3lmUHZ3d0UKHWvWO/Om6ahwBkONreGeENqdFG9UI6JB
fRA3gRHy98EorEiia1him6lGY+ewYkTtVNo6iEt0abncW5x5xMQywQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dFFpeDU2Y2dtQU12cytZ
WUJjKzZ5S2RiZmtvYVJScStYVHI5d2FvcmpzCm9COGZmRllyaElqb2NsczhKRjhO
ZGhwZCtoaFcrWmNZbGVHMUNLMFQ3Ym8KLS0tIHJQR1JOblZlY0cydnhNUnBCT1Fh
aHJiZVpMU3Y3NXd0ajc0MGMxSGp4MmcKky1oaS8yFydszcvZE2pAL90ZUKoN9MnJ
1TdqatDSMtmo3PWkdFeHbWxPjTizQGIcKEcsv6GmO/5KCAJDCHDNMg==
-----END AGE ENCRYPTED FILE-----
recipient: age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrd3QwWE4yaVlIakdiYkNX
bFVmR01yNjdWa0d6V29tcWh4M0hWRmQwR3pBCkFRV2gzSVdPQmQ5dXZYc0F1dW1r
OFZNTkRwSzRFR0tlUXE2UzFxeU1Ld0kKLS0tIEZ6RlZkMFFiZytPZ05DSEI1Ni9X
WEtpUlMycVhkMERsWDRGQ2lrNHdQcDAKRqVAOzQY4ZM1uH7LyZSqX1T6+1PwNhBY
5M0RxSuI43E8YKC9axcppvKFJMQI1GAGvgfQxg2J0CEpnmPSVsO5DA==
-----END AGE ENCRYPTED FILE-----
recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvb0xNMloyRW5tTUUvZGph
YWxIc0ZhSWI5RHNmaEIvdjZJNHFDM2xBMkNzCldlM2RUeFZCNE9tMVp0MFlWM2sr
Sm44MjkrWlNkY0U0eUlyNlE2ODl5SFkKLS0tIEdISElQa2VBZlduajhsV0ZaU3dZ
ci9COWE4bldUMS9lSVB1eEk4TG1sVjQKYu9zby36DP+41dTV8sRdtDil3Az40pf0
vIhqhCTxQymnJHEwEukiEzM/w3Z85R/W/MbOKyzEZmxzG0a5O5svhg==
-----END AGE ENCRYPTED FILE-----
recipient: age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZlFtOHI0NnlMeWdjZEsr
UnQ0UFJXY1pHTm4yLyszY0dBaldtUkIxdzJvCkNQbk9mN0wzK3BtN2pEa3JSWWFk
RXNoemFFbDZ1SlUxOEZMUm9mY1RaZW8KLS0tIEk3bCtyaUVGM3MwNUwxS3QyTjA4
YVpTTFlOdzljbmI2cnZUNG1nd1NrVFkKxARm3bpBL685Wky4Y06eA18VvUZRkAl3
BBYPJMn3lfDgiQPvfXINfhJI6O+bWyjt0WnfodCGFM6EJLARXiTaqA==
-----END AGE ENCRYPTED FILE-----
recipient: age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
lastmodified: "2026-07-29T01:49:58Z"
mac: ENC[AES256_GCM,data:+m1nB00cOyr5IuxUMwvumkEIPKbYSw30UQVFyLPe+4VSOniKm8zxb52DTTaZyFyxUNAQOWewdZfvPRpTHBAgqt/HE8dMcALqPLAneNzpxIYr7oUh3TkRw9Qkk8NVsZnqTiar6C0n9xsA23tfVt5FSByqYzuAirNdwbKi2pkH59c=,iv:jonNGWQP0plAL/lrpiBKwpVcXULWl2+ZOnzBI47J1Ss=,tag:DjBN2/V5SUYq0zeYkdJylQ==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.1
version: 3.13.3
+26
View File
@@ -0,0 +1,26 @@
{
"data": "ENC[AES256_GCM,data://9IEHIVCfHjyMNao5sCu2zNlZ/CaW+JyxVpGpD/vab2qvURunCUY7eMfSOyvOx/2WPXnWWlkoVJPCR1ec/yUg09EaSMfxrvqlu4UJI3Sxvu9xNuDszMwMMD/sCulJDiMWFNLp8qaYt7UGzexp4+GlGiqWDxk3ZEu/iLmSApzBrpciTF0lfehT4qblDovo9QXG2KDWFhCt2SwEKmHJ61Yl3pVAQnPLyTWaNhwWD/mYL76mIiDVKq7DJlvi9MBxzi3aYh/ttuHgRCvnCL0C9oUvOyT2cljwra0LXuOrum7FhPIXXboA7WTEbTHJm1LB5yLfxDrnWCrGxQzFQAwNixVHIcjuvjjvA/LifTvbj5FYM8x7an+DwXPfPhruFrDew1paAWsEGNNYXSmAlju2QLI/OwLHFFLuDyLnBKQ6RLtZbCEAOUKf2h4iZj7nH86eb/aGU=,iv:rj76+MJBCpiYyx2Ogut5UxJj3Gn+bygvu2mtuY5hFLA=,tag:yQmTvWVu8ZLug/7fo6RnwA==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqcURBS2tIMGl1aWxLMHFQ\nS3ZGVTE0cnYzTmpnQ0daL1F3QnlpemRCMmtjClJiZzIrc2syN29sRXVoaXlPRUNF\nakw4RndmbEduTFg5ZVRKUTVwRWpGRjgKLS0tIHZ5ZFlyODlGWTJoNGpXR3RhY0ZH\nOGpPdDZQVmt6UWZXbHkxQTBoeW1JencKbsfH1V1lUj8mmHyLNj36VaRDgaBojcDU\ndoQWmSEXxjticJqdadbVKb3UABpvzAZxASCy81sa3wH0gT+7zLaNyQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrbFdaMmRjcHgyV3Y0eWZL\neG1uMEtpQVUxQmNTU0Z3aC95Zi9uQlgvMEhRCmJiVlRTR2NocGlnbVU1UmlLVVA2\neERiMXpiQlVPNVhRU09XVGxrY0Ixa2sKLS0tIFRuOHdoelJxOXNRRHBJNnlhSE9j\nUUZHZmQzOVFwRmhFV3pvdnpRMTMraGcKRHBuSUpbHaEzH2tuSBE5MsLJDCuH3vUx\nO0jnDldCWkCw7Wvr/tQAkaDI8axZcYVDUkCEk+xqAdxDozLCPhEO6g==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZMW5taXh2QkJkT0JjaVpq\nQlBjVFpFYUhXTHhOT3prbFcvUFB0YnRGUzMwCjhHT01lY3dPWDE2dDZWZnJza1hN\nS1AvZWIvdjZoYmRjWlliK1hiOEdrT00KLS0tIDljWWY0NlQveS9VR2hOSUNyTUtH\nK2gvbEVibmZSdktPemdEQ2p5U0Y4d2MKKitoTi2vbxJ41IoWMlj4vO91Ahpj0hHP\nrKCPyx7ws/IUMmKGyvDLpZ3pYqHD9jl5pLfB05Hh4Emhv4lA1qtwwQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3K2dVV1BHbDk2Z3FhNmdV\nZEN2Um94K0pma1k4b1V5ZUJxQ1hWU1grZjJNCi9iRVVqOVpEMmtBUi80NThlYldY\nTUtZOVErT2VSWk43NndpVzBlL1lQaTgKLS0tIEpCM3Q4NjM3N1lpUE56N04rTXN0\nZkZ6TkV1TU9OV3dpc25RNWRpVnprM00KItUzKBdShakOfX8Sr+k906nsvYPl8QLb\nge//1GA+ukGsaS9rcChOY89vFdm61JDmj1jXSJ0CN4wLMW9/eblZRw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c"
}
],
"lastmodified": "2026-07-28T01:44:28Z",
"mac": "ENC[AES256_GCM,data:efFXIqbauOURR7lrVpK7kqRIPgYjgWfenBYoX7mUrQ/thFc+ApcAx58Fi1zg4biwIs7NarJAgDqAxZi+yKb2Sll8H/wlsEjWDU4iQlLJdIQw7wey9eDW8pgBLd+6E7FXrgn1Vh49dS5GXlC6clAYk1lCiB4NvfzPDy5Ktpl+Chs=,iv:+zCqj5I1MLJfRRiIrqgocYB49PZnlV45PUTH4gYlfrQ=,tag:WxY1sF4kFOTEzbSFcfJFbQ==,type:str]",
"version": "3.13.2"
}
}
+35 -17
View File
@@ -3,31 +3,49 @@ sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrWFpRSURBR0gvRzVIMElk
ZlJGZlJvc3lFOVJIZ29YbFp3akFCNG42TWkwCjhNM0J6cjIwRXBzc3VWNVpZSGdL
Q1M1bm90OE1tTG9GbUxvS1dvRkZ0UGsKLS0tIE43L1dnbWhQOUhhYjg5bEIvZkVD
Zkh1NDhvZjlDc2c0cUZUYVRLdEozS2sKU/r6JnEnUs2WPj/J724B+lgiV84iteZa
uMlhgnwYJFLkH7ZyydQqjYcHL7xEInr0taYJN+M0nZIsZTvzAEi7iw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAzTFFlZHdGUzk0b1Zva1U5
V1UwREEwS05icWxYNEdKTE8rQ3lJMjM5YXdzCkw4OE4xWVUzVWZMaXg5OFo1UG8z
WkNwK20yb09rV2VVSENwNWUvTmhJNk0KLS0tIGVPWUhFS2RDcTNjY2JLaWxvcTZt
Z2RscURQdDVCMUdiVng2YWRMSlEvVVEKmyd3re6AaKn4gBjoT0x3e/zJznvJFYKn
ugKu3EsUX+gbailPmY1ss9+MVtpJFGZa2FiM0x1wSMKm6UJH0aPhVA==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXZmYrVTZxWW1QKzhLWThX
UFAwSEhId24yMitmVk1kT215OGdqbGp4akFjCjRzN2dkTURXS0wzbzhYL0YxRW41
TWJ3UExXaHNhSUxaYkVsMFNCekZHZDgKLS0tIGQrSWNzalhCbGJZdkxvT1N5ZWlo
a0l0Nk9DY1BKYU9ReWovaHJ0Z2NyWGcKFoIYS1M4EbR6H6QG3Wjv2ZdX3r2W8zKp
S9f578O5ZLh2OWaawcSb0oecZJykT9pgudVcuKRunzZN7NQvFxBRnQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtK2tud3pWSmd2aC95ZkV4
RkVvZXNrK09ZVmZsYWhqZ3JlaXBCd1NRWGhzCk5lelhSN2N6N2VhWEVXUjhLTjZH
V2VlMm5XdUtuK2d0MjIyRi9xeTh2ZXcKLS0tIDh5Rk1UT0dWblBkUXYzU3YzVGkw
OFk0bkJ5RXZpWE4rK05QUHlLQktYSmsK/HsVEIhBEIo3qqVWdUJEWnHZiKB3uHVH
R+nJGuXa2B/oUoxEwMP2YBHwwjLLiJCTYy+aQtiPdTrVq0YJ0HmC7w==
-----END AGE ENCRYPTED FILE-----
recipient: age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05
recipient: age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlWnpPUk10YW82TVRzbk80
WHJqbXVVSUFrSTVjMEdqUTdSQWwvdG1TcW1nClQ5T0E0akFtSlFGdC9WaU1LbGkw
allOSjJ2SjQ0MkVMcVNtdzN3cW1hVzgKLS0tIERlQ2l3NitPQndERTYrNVA3ajF0
bCtnYzhHcnZkalRNK1BHbjdIS2JLNmcK5RwFzeaK1KafO4cAtdFh5Tnz1lpZbj1Q
aeMmTGMkxJYVGzXKS6SRspVT3MvY4Fvay8B0MezjG5Y5HKNIv22Vog==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpVVVxWE1nUDBLTVFUaGJJ
Vm9EUldwRGcvbXhuT3JPd3N0WXo2S0gwRnh3CktabkdaSndaZUNSeUJGRzFKcjlH
b0x0SFdEQ0VqNWdQcEkxb0drVVJNcVUKLS0tICtyVGRqUmFtRHV1SlpXSVd3N2VN
elQrVFdhWTJ1R2pJbjdYa2w4NmRmc2sKJHqLYdNQJcna71KNhGF80iS1hIYG1U1w
I2kihepJsrmYr76ld9k+u1ZfnuIuJ1ozYsZothE+dr4pV0k8s6wkpg==
-----END AGE ENCRYPTED FILE-----
recipient: age1ehkswwz2pqaz4svzh7ela5tdnssl8kn6d4vwwxd6zwg8exfpd43syyrrjp
recipient: age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSWXJxUEk5Mkp5eTZXamFR
SXZ4L1JTSGNaQjFiVXFGaEdvWkV2ZzBlVzJJCmd4NmUzZEdCbi9vdWdkVGZRL2Qv
cnVXa05xd2gzaXh1SnlMZmtueGpZMHMKLS0tIFowdnhFVGFheURQU1V2M0ZuM1Y0
NTJFWXBEYUxmOU9ROTkxWGhYUmlqMHMK3pexvc16BLKjh2meqtNm3M1zyLQ3eEsz
7C5WkdcSpCkW1lPDGtW7pEdAIL15StD4x7ut4MkSk0BjG1S+RpDbzA==
-----END AGE ENCRYPTED FILE-----
recipient: age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkR0o0SFh1L2xjZ1RjSC9K
ZnFyb1lJbnlMbENYN0VjSVQyU1F6RDBrL2hjCnRwTUp6Y1hJWldVeFdQZSttZ1Vw
L0dCS0ZROFArb0ppVzB5WmV4bWI5alEKLS0tIDh4VDV2TFhhaUp4L09jYm52UCsr
NGh5a3VMY2ZMZVBQbmRHeWsrQnZVWDQKR1UeSZ/EzZEXMqyjB1I2SHELv8Ha/tmI
kJKs2WT1RtDhAiTrbty3f4oVrXWSYKZr40kNiP/RLbUcH1s65ys/tQ==
-----END AGE ENCRYPTED FILE-----
recipient: age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn
lastmodified: "2026-07-22T01:15:21Z"
mac: ENC[AES256_GCM,data:dC/oIqMUHkOh3AocOwP7Gc6XGH3L+nTqJfhFNts1DNbRXsopNIxVBtIz2pEhwnWSQrqPisDLmPHFBwRpGVn01u8w8IU1FKbAKC0J2nJXF8ozpInbjzDOmehqPWZG7yaKoq8cwAnp5XOk+IVO4l6tPxLxkExU5fT2ALuMq+sgOko=,iv:jaVyArpf6zMCFa6J9X1aQMGrmFq+W2CPZdWO6vVW68c=,tag:S+qF8/FkgHc4uW0e4ICmSQ==,type:str]
unencrypted_suffix: _unencrypted
+22
View File
@@ -0,0 +1,22 @@
{
"data": "ENC[AES256_GCM,data:JwjmYg0qzoer+8/jv7KOfK/BxmpObHOn04GNsUFQd6hSFBWbQlRbODfluL54hnWQ4gsS9MHgwWh8nrLciBhye05kRQCqADB7Crm0CZYubNb65WzM/devh14jmerc3MYIp1M3VmDisML3x5IqhJkFMSqKM/VaNAC3f0otcVR8TTgZw/fJyvPlfqpPTjikGmo5xg++Yk7Cy8A1Dj70kYk10+EQjQ78jf4k/agLoaS+YvMmKXFk4EljtHg8Fe5H6Rn9nfMSSKGTZeAH3Riix2e+fIH+nWEZiPHC0UlPjJrB3SeBqqGRSUUJshVKHFIdKI1Tq2Ea+GDWrAyIRI0qO8e7WQ==,iv:mr39NxbZuYUDHDw0e+fJDshbt9R1hLgVqkUxUVX/+l0=,tag:u31xZRPfvH5j3Sw5U8/nqg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtVW5ENXlkUW83elc3ZnhE\nOEZ4MWIvQWZ4aXhuUmZGYXZNQnhJemsvOWlVCjIwNmFjS3pJOWEwRnhBOGNRamlM\ncXZid1RNYVBXOFZpMlpnU3ZnVE1scnMKLS0tIGFhWFZFd2JPTFNrV1VHSDVKVHRK\nbmhoeFh6YlUySU1ZRVhHbjZnM1IxMFkKo7aHkz2pEeV64m+OEkBZ2V1e+PUzoChu\nUc/Mnh37dNXSSJtg2KnocHdyzDGi1yQbA72xKTxx6QjYJWrAC/jurg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBReWRLb09reXUxS0YyMUVV\nQlR2bEdYY08rK0RoR2pIelpLVEtYL0pqNUVNCklRaDNoSjBWT20xbk1UdmNqWHNt\nR3JoSlVwODVMUVBzMERUNktHMVViZm8KLS0tIHc0SXRIS1IreDF1VCthb0FBR0Na\ncGFRWTROUHF3clFtTmY1azEwNUpPNzQKsgCND/BZcMTgBTAcnHunQcT6LG1jNrtO\n+W7Yx7bFtFBajWnRYiNpUZPibQJlv5SE9os47WDH1gs86xftgV+uyA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGbllNTlhxNVVkd01GSUNi\nR0lxQWZoRXI5NWxoTExRUnBSb2FPR1BONnpzCkxCN3pNOXVrTW5GR2J0WFhZUE9r\ncm1QTThINDVVVlZEalJiS1RXcHJRS28KLS0tIHV6aGh5QjB0M1VPWDVMT1k3cTBh\nNTNGY25NV0tWSUl3UHlJSExFQVdpVDQK3ARj8xhFRYU6oqYxNQ5+Ryza86ALNUwX\n7yK/8ATnquYC8/ZIYUbgTPxzsocmXX9lVT0+ktIALqjtG8PgNTlOow==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r"
}
],
"lastmodified": "2026-07-28T16:04:51Z",
"mac": "ENC[AES256_GCM,data:9X0dkAEGJiug16LDy/8//QOkHITTzt4zwTKiIU13tRgSRXRpt/7b27+3eDauscXDMamg5wbYo1YF0+VwLl21Ip4icnnwrZBHd6HWleP30HTgef8rmo21SstshnclZz6RH8SEGVDO/vjrMDChaZ3A2WD+nbcomw47DpK3cG3kBIk=,iv:G7CScSD5Z1Z9WMEMydyeGK/RD4W43xA0PlcpmTCxLOc=,tag:DZmby/xkxpinoJztZZRWwA==,type:str]",
"version": "3.13.3"
}
}
+19
View File
@@ -0,0 +1,19 @@
{
"data": "ENC[AES256_GCM,data:VpxnzrdX5FY2gfDcZpqg1f2wqL6mAawcrdE5A9A3fQOKr4BWXmCT/hjQEtt8qG+yVOTApdG+vF8SKDj1AKZ25KsZ253WS7W5C4j8oiT/xHo0CtvVani4fZqN60lB/9381h7aUZ+lcBMPaQcNn35zoQqYthuCGrC9PH1qQgD2VVHr0v8J6kceXKp0KTJldtDsQxky6ROC9Zyc44wwkrFPgPss4/yGaRWWqiojDvK5fK8238hmR8HyfD6Tf5KhgbqCsSGS2g6yt1muUhmegaZwtpi/KpoqBmf1XFaObuHGRPKvblq4Fa0x/sSjpPogBZyNuvRKIwRhla0FkKOH1bX4gC9bzOlgiUUfVcRJT6Nr9jocYuhIv9wvIT4TujGbgHyorYaDSQLFPJoJEUa3fgpGq+AC6p1DJJE3b/wqUSngNbdOtAPE/imaEuprOfnbS7cRx+Ti/Yu+ZYZuAlDtLkSVdXi1hQ4fLQ==,iv:rq/BzePXa/w/Gqewz8JfM/NdU+x4ShiH1OuQT7wRyCQ=,tag:IcO3ddoj4M57CkndHEmNkQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRdWFEUlRJMlZId2Yvc29F\nNnkwREkwQkllY25vZnRjV3Nob3BvNE1Zd21NCm56UVg5YUFGbEc2ak5MakxDbCs2\nTFVsRFhjVlRwMXdUbGpMKzQ1N2hBUUEKLS0tIEhmdm9hZEJqUEhRN0dsSk80bDY2\nY1JpajNOVFpEejhibjdKZlNHVjE1bGcKycyDfEBZ1WYc4EfAK2y5x/nKRqq7mnb4\nDlpR1Som4bSt7+B+OCZa48mC0Zv05HbP8PnoZdCe83swGNljvRONcQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0SGFVbXYycTE3amR3ZVhi\nVFJHVVhWNWVQQlJKZUJvZ2p4ditqdnYrWnd3ClZaaUVLUXFKaWlDdDQ4QW5SQUZY\nbG05c0RGNnZUenI1QTBlemVjT0szRncKLS0tIHBWTmNpd2FjWHc3MmR3cWt4SCtH\nWS8xZnVON1lGZVc4YzZFRmRBamdSVlUKdlQTlwiExTUVsiY05MXFG2/IQt1bZwKU\nlFDnuy6YCQzawBLQZuSUAp7WSGedupexlZKQKNUzPf+dis26XEYZoA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6"
}
],
"lastmodified": "2026-07-28T11:53:06Z",
"mac": "ENC[AES256_GCM,data:PMsPSffQkxRmqT6HwFezIoT4WKJ9WDlAcflWynsFLtzbqVTREN0YGOJgMwLd0yBR2sbGBWogG5xAuKjO16aiWUVpCFPRvP97um351679ZoXIlrOL+GlFn4NKwoLH9/eCp8BteB9mv5huhEEN631lqCr+jtYO6yS0ULk6QS4+7/c=,iv:oYWV4J1JUuGryvqntVXkj/HOzRDRr9E2C1RKNvr2bT4=,tag:e4p+I40y1LrgdlK9tLRicA==,type:str]",
"unencrypted_suffix": "_unencrypted",
"version": "3.13.3"
}
}
+19
View File
@@ -0,0 +1,19 @@
{
"data": "ENC[AES256_GCM,data:Ea5AGwloFMyRgmQhJaDNTW19mpDvx0+dSc3ibENEXEniEIBV4Wv135mAwE6jQPbN8wF6f5ki3B5mDXHOMuhDQEqUXg6jcs4uEm2nwz7wAKYXEX4T8p0gUj7FPISiSQrw8O3jwzt4qF7AVcPsNhMtZLgrx9ssMtVmGct5do+E61Y7dFe0XbpJLynfzeiMKYvaBngTbHwQjv1mn9AMpX98apsp3tyzg8avLDR+WXRcMusds2mdov0jCoKuWPLG/srhMwjKOp1Z1Tu+aZCr9lHTT76mytpGIWGSbrP5l3UTp3oPLmy87Kwi/FJStZYdYikMnvmSo5qovePiYsLKVD74dS+d3oFGARyW4TILd9OQdXEjzdDRak5HcPv2/HhR638kvukVMdEdH0hpeJcqcV0h+BoLyIHe1e037iF1ZkCjuXE+A8LtlzRR0wqP202uzBIyg21ca4X5bRiHkkAOX4P3riaEcMgKfw==,iv:IXq4sHHsjnK6maVd3RRVZzhnxx1hOnxfsvYX8IKLn+4=,tag:jRZinAVLH8nGVf3gE4aSKA==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSByOTkrVVd0TXlBRzdkTktO\nalU3WlVOb2ltc1k5RG5sRE9EN2M2WUlnQVRnClFGZFYvSHdpVlhtNXExSXZMZnU3\ncnl6aHRzU3k2V3R3dU9RVDA3VWp3VkkKLS0tIHcwREM1dDREK0h0UFp0bU5JTW9Q\ncWdyanNMVjY0SnZUWUl3Y2R2SXNyc1UKI1FPIE66to19oQK2TwM5B4snGOVMLFfx\nl6JMEilDVSgy2RR/tSiovmo4NwOsv0hmmF1t34547rbZb+dB5KrMEQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnbWp2UWphSnFibnA4N0ZV\nRHlNUnBNYUNzMzNZakFVc3JXT1d0R3Yyakg0CkowMFlOcEhPcHE5dXd4ZUd3cDAx\nbjQ2T1NqejNRMnFvbTdNRGFGdW1pM2sKLS0tIDhGb1JBQ0RFQ3NxbmpkdXg2WVF6\nS25uZW1hMVo4Q1JGdGNQWWRyWE84b0EK0FmFZB6sN7uZynhzYU932x461zSZIUWU\n8oyDEKNoUuPqpWK8RktgjWlKHtB0oLXC+SLAOJEhtnhWF+GSqloGfw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r"
}
],
"lastmodified": "2026-07-28T11:53:32Z",
"mac": "ENC[AES256_GCM,data:EhWEEIamG62xHI6SqO4tzdK6gcEXxUU/UGcF25m+X11kUVXzHd6yLcfiM2DvqcwGLmwmOOf3SpFLABlQUzTka4QRzTMFIAJi7x1rzQV4J0YIGITHk9rgvy0V50TI0loONBb2du+vDt8IlTBNvF9WMbNCki+fHPAjIemyoRBdubo=,iv:wfsx0H4fD3L7WeGzz57SzZtajBr/xTRDljSpz22DD7Y=,tag:naDyGH0WTKkqgd954Ya8jA==,type:str]",
"unencrypted_suffix": "_unencrypted",
"version": "3.13.3"
}
}
+26
View File
@@ -0,0 +1,26 @@
{
"data": "ENC[AES256_GCM,data:z2bazlADvtWkGcY4an7NgZgIDSDq8KmYpSq8tcBleW/33HlElMNhdmje8m8hd7WxgrVvCQDdlAfodBLj2mjbgSwBptlPX2zbwcmJdm0g0Ope+YyQZucIio7NnBaYa+OK8meWqmxO2WEeOk+hnNfNPbt8MkzF/FMMbP8xv9wSTKP9CDdmO7s8rdg8G1JxVfn9dyaSDUTuA2JGNtbh+osd4CtNCV/bc84MjXDZTgTfT0W/uv8VH3iPNT7BhQbU9FIY4dsFRYBf758VtJ+3mTBiQkR38IevJBvhFB7Wiqn118LSBgjdTKJnBwmGo6lMdo0LRYHEgUnb2ejhWMNie19x+LEAErDki0RBmjc2BbnOF908kUlW2fUPo5fz/hlfqlhDLS1uOxESKhfnKt7s5qzWiMX/nohpknVfyMbmu9j/M571z9sgFnJAylpW3NLp53lX8N9mINwRGXsyL/ijbNE=,iv:M/MNquAGN+lIIyVVvFRgKR6DKAhLclc/OY1HV5Xwoog=,tag:eEuURMErSqsoh+l9ZaHY9Q==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqZlUxRG0zR3IzMlpDL3Vl\nNmRWQXUvc1NDek5wbWtKVEFTcTlxL3ZoWldNCmJVMkFtbDkxN1FGQ2VLTlhBUW5Q\nZ3VzL0UydUQ2YnlUMmtqa1Z0d0FSVFUKLS0tIGJFelc4Rm84T3ZHSGZDdnNUbUFM\nRXZHV3N3aVlSOXhUZk04UFRMWU5TbFUKhPU80PVYuDFUCxu1CA8+W8bqkr+Ne2fh\n+nBUPJbGxfN9TyD9tUC77AMbcL1R2L5x+SSAh0bEgSWE24/RjsuVKw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3M1o3S2FibHV6N2tIOFEx\nRytJRDFIUU05azNDbWlpaDdFQXFRWlBUNnlFCnBwNVB0SEVwL3NWNS9aeG4xSUI1\nV3NOcUFxUUhaODYyWXdFMEhWeFpvancKLS0tIHY4NHFQRGJwUURQbWZ2cGlSdm9s\nOFJodGZkekk3UnBRRUwxb2YrQ1ZPcU0K0tCapb1hfVHFSNpmESXexYa5k9OE9Tha\n51QpU4mZHKGrnWPK/kyj7rHiz95TLsmwUdA8Q2hOiYNSxEaVBjYDQw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkTmNrbUFoVFVGcytzQkk1\nMUxDdmJGL0xDcUNZaDFRQnUwNXoyMU90SEVvCi9EaTlVOGlMeUZtQzRJS1J2TTJY\neWdkMGNFMWZzQXVKOU5KcXVvalk1ekkKLS0tIDN2MlRQcG5aQnN3bUMrSDdvc09X\nM29hQ0pJRzk1amRHaS9mRlhrb2FMZG8KEWkSSP+MqGRU75qo7ctOCL7qHhyCM3l4\nL+ga1XOKxRsQkKQPozCi5Bm+k28W9hIDaC34Rw9difxICM9Z1rAbOA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1TUtxVjJ2STFpU3RNdWpJ\nVjcwUHVORVN6L1pKVC9TeWt4cTNoR2pPWVRBCkR4QVJ1bHFUa2Z5aWw1bG0vQWVy\ncjNGRDl5REZzbzdZWTFMRkU1eHFTc1kKLS0tIHNLVDFiRkRKTHhrSEx2S0J1R3cx\nSkdEQWJya1ptTTRqUjJZT3FFaFhkd0kKgecHrnzW9+Eb+b7c0z1yR+Y0Czsr9Kjh\nx1UUOfleHntJUCs8oYcOogKknEnhBJJoJ5oe/gWruRSwle1fs3cBqQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68"
}
],
"lastmodified": "2026-07-27T21:22:58Z",
"mac": "ENC[AES256_GCM,data:VG2ygV4X6yMxGlJgq3sN4GAzgVqiD06noyMxB2yb/FCmiFVYZAH/9LqF//G7MTInfIjFrTDyoHAny+m/ztUrJkHXI1Fzg0U8R/zYyQ6wj/GguL0gyhA70uriQhvHsRHefHPa7Km61Blo9cME6CBJWejtRvh1IMw9itAioVrSIZE=,iv:zjxNG6IYxkcNqLk08NADoDiIbS6at86y136SBzadW1U=,tag:sj4ZRR7ixU8h7aOUJmTCYg==,type:str]",
"version": "3.13.2"
}
}
+29 -21
View File
@@ -1,35 +1,43 @@
beszel-token: ENC[AES256_GCM,data:meuzUP/6wCssJDVTgbC0XwiLZPMGyDl55HEIiON9xOXCD9k6,iv:TDqWcp+8Mxd8wN09r5otQRQXq3XTeQphaTWxvvuLTAs=,tag:cRPZQGlwB/dTguBAheWPQg==,type:str]
cache-priv-key: ENC[AES256_GCM,data:6vQKIf7eS0WNL2Eptoi4VWr18SRMZfN/H/aFUUtXdMYQY5LLyBp2EHRKqZcGFuh1nZhUdAxUztq/CVXx+QFxKW+ElHxCxUSp0QqI1fdSkBkKZb8hlit5SoX9JtLzZGg0HBNM3nJu,iv:0J+xmrPJhInHhFR/c41ACjuTfaIoMkQFSfbL2KkgFa8=,tag:f4s9Szs5oprVVRSyXaX48A==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBPWE1HTUhiSUp5ZEUwWEpI
bGpkZlBIMUo5ZlYrQ09SN3Q1a0ZkQ0ZnOEhVCnJPNEZQenVWWGZiODlzQzNEc1Zq
c3l4OWZJTElJc2Y2UE15OGtEUzhyY1EKLS0tIHNJUStyWnlQWjZBbEZjQ3UwdUpz
ZndoUDR6bisrNGJCUHk3TGI4bTZaMFUK87fFsm9ne9s+PK2pcwtrDjqyGBss2r2E
8lhqoeiKZ2j96z8kP/7ChzovwTCmqdcmAQuyNQD+ZAFijseipSvfbQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwMklZTFJuR3NYbkFvV0l0
eUhMWU4vMHpnN1NKVThuVVdiOFpkZW9rTjBVCjJabWkvOFpOSm1hdEdlZTYxc3BP
WkRURDFEQzMvRFlka1VnaU9zRjhiSEkKLS0tIHZXRG9GaE5iVjg3M3I0SzhtN0JP
UlJVUzRzN3NEZWxXZHJ6RW1oYWwxS2MKonnhq7YDg4v93PZtoaLANDy8mdRCenjo
FjRUzozMLpgWBll4DwRWikejsrRofRBwlcsiIdrBr90f8Lr9pHdQ/Q==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUYi9SRFFGV3Z6cFd2Znk5
b2FLbWtzTllJMDBUaGk0NTViOTNBa2hQclVZClZHKzNhbGVjQUJhWkFWdTFBMG5a
cUFJdUdyVG5HQXJRRnJId3hqRTN2cXMKLS0tIEFMRjh3WE1ON0U2TTNTZ3hxMTR4
ZGRlemlIbDZKeExmVHROc3Eyak5DdzQKaLwIVDi6BN4cxpVxJoqTYvJETPOp4thc
l9uVMvIGuEsEZgDsvShw1dYLljd+uGy/A+dXbcxIUCP/mmPkwmd1Pw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsWDJQYzM5Z1JRT3FlVkJy
TkVaSXdzanYwbnpOM1d2RFh0c2NaSUFmdW1ZCit4SVFxSS9HNXhVM2gveERGVS81
NVFQaUV2RGR5Q3kyUTQ5eEpDVXJVMWcKLS0tIHdHSy9WM0o3Q0o1THhXZW11K2Vp
NTNtSGM2UDFuWDBEdS9tbTY1ZWt6UlUK8z5qoi0kGn0ES3m9khummuU51rkR0Sb9
TWT92+BWvPdNrAsDFjv0fgpUKyTMzN72EzHZKAJCIM3crUG9I0tX2g==
-----END AGE ENCRYPTED FILE-----
recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
recipient: age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArOWovSW9DeFpxL0VDUDQ3
SHUwTzJVZUtPV01ZRkdCUXZGL2lTRCtCNFNnCjBSNExqRW5mTEN5SFVucHJHSzZt
cDlNc3BjY3M1c1k1Z2tkVEg4R1pacGsKLS0tIEFWbHNKZW0vbVh1Y2VhQW93OUwx
MWV0eW9sOXdQd0l2ZjlWOEVVc1dwcTgK2s4p9xoNkawH2OkGsl80bNIo3ad5vn4W
Z2w+jwppSoUmbQnD3WFbLmSSxmuobmU8HILwElv6SZu+KE3aspF6XA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpY3o4SEpwZEdFQmVnOTVV
d2NqL0VudHM4VjdDK1N4dWdlS0lGR1V6a0c0CmJUTHlsbEMwaEU0Kzdaa21lRFRm
RTRnTUlGUG9GQVB4U2pzTHdRY09udkEKLS0tIGlFQW9Wd0N5WFg2WUpCQzhWUm5v
aG1VVWV5ajBmc2o4ckgyQWpWaFVxVmcKisAw40bGQBRH+u6uNygfYpfb7iEgfEHj
E+g9n2WeVH8kUzD2O1o6VAu4m/SVuI4+IQ77j9GEWmlI/wgp8wKXgQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
lastmodified: "2026-07-19T23:30:21Z"
mac: ENC[AES256_GCM,data:kLGE2xawQT7mx+sfw68hmGk5nCEGiEjZrqTEl9B1dtQmTrMwmoVr/1RISi4LfJrwxy31mDgff4lcIL4wIJuM373uk3X8j4RNyYQNTfKEkORT6r8NHeepNs267O77pKGd7OmcM4MT/BqOnB8ELS7Wlf2ect7CAlvUUVyc8icxgZE=,iv:EYLDsHYHZ1XOQXafOTqHHWpk/OBNq/R6IJnOBYV33E4=,tag:thxrCPC5oGvDjhK7Dz87YA==,type:str]
recipient: age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1elpNZXVJbmJWSU85aUV6
NG9YTWVBeWxiRHUveStTQnl4NC8rT2VNdHpNCldxZVJYNVhUR2V3Vk41VnJxenVT
WS9rRWlKcGpVdVJPakkwUTY2a2xQTlkKLS0tIFhFVTRucFNuS0pMK0FNRk1ndnFO
SlVvakczaktUa2VLY3RLYUdVRzFyamcKIhctg0mbYL7OE08dRwj5wMu2x+O8/BMu
IqA477+noQ/Rjrszb2hEvxID7keogcDUWMWQzQvdMc22+3mvAr9qIg==
-----END AGE ENCRYPTED FILE-----
recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
lastmodified: "2026-07-29T01:59:11Z"
mac: ENC[AES256_GCM,data:/nbcfause6G6F8IvMoyPZtkWS1XRLAivhwTFu6y5P0Mm0eCcO6M7/rgioN9dngKzPXrCUl3Dx/EvhrrWKe2/Saq9WEOFgvS2V05pTRbQgYjuugVzW2paPq1fgmoDYNjHz2yFYWAovbIFtjVxMR9tmcASMTe6r/FocvMXbFCJh7Y=,iv:n3KoaPPtDhWK8mxJNJRk7WPVUUNR59nZSA8JyysWNDc=,tag:tYhkNXsj2ohVjHkvxife2w==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.1
version: 3.13.3
+30
View File
@@ -0,0 +1,30 @@
{
"data": "ENC[AES256_GCM,data:apiijrtrqd77CTizITg0R35BfCi8PBnufpxIyC+hLYqwoBzP//3z/yjFyHPLG98m/c/qywoi3Kn+zsaTT7MjP++9OMhhX94YKlSHV1/cHB76OkwsNc+ClqWxl6vpaFX29Qvh3gFX9c/NR3xvYQutYwrIrQ9NR+t/M52IMC8hvtR1LQy0ak3VIuXJlSnG2r4kF2Ym1iP7phjuq39Gd245Axzw8OB7yGvOjNxSdTPxW/qL0fMlzNcMrjr9hw15WlqnZfWPOsB1+gZjHXpGfPD5BCbAAMoTRJd75vhKKXP/ERhIffewuuH2x/QHfSFvXVB3QyhBQMxd2b8QEEE5cjvcExOST3tkj6QARkzoUpRT7AE3jhl3XZ0uA2qu9SwyrSvbr0tBRKxCdK0g2E2/hqwcK/Tck5GB1eKb4aN+UkqxOblNDH+B1RfDoyNAuN+KEg==,iv:0p+ScrKpP4kQvO52gBAlwAis6oAzZ0EHFnU74hYPrn4=,tag:ON7qOjztF52xsJWAou7ogg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlYXB3cVAzb2xEZ2pGa1RJ\nbS9ZVTc4Ums5eUZJUjEvd1g1aGVyNUNramowCnptZXFOZVB3MFRFcUtzSXBEZk1B\neEtKcDdLS0h0b1h3VjRjRXRvV3V5V3MKLS0tIHBDemkyUnV6ZXhTeE5VOVVOMlky\nWWMzVGVzZlAxMjZYUGpQUCs5QmxiYkkKcuBshCgWX4TwfVlQ5lHikzvwWdLEXWD1\n/uSiy0J6yMSiu8u6cg2SxeFrlKJ3j47dDlT6WHCxS0PfeEA0bJb3LA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGa2VTc2NWdkFRNUJxelVR\nWFk4RWxoelYzNHo1UFVhU2ZkLzEySlRWN2xNCmNmcmJod2crL3NMRlVsSmpmVkU2\nMjlXMktjc3piUVNhUXlTdnVGTWJkUTQKLS0tIGQrMUxrNDlNTkRCSUtFWkxRdXgw\nRlV4ZmtYSGhPQU84eWtiQXVqTmxUK3cKk5fn72UZPH68t5ZappfAhZJwzpLkfKmT\ny9TbUPIr4Pbrexau6YiH43QIbDQFdwYPfkBjGkd57zCg8AVo1+MBRw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArNWkwQXFWT1RpaDNvbzYy\nQWc1aHhmNHFEUXVsQjZqb0EzM0wrV0dwN1hnCjFsUFJiT3REK05uSGRWTEw2SFE4\nY1FleE1XVjhBbndiMmZxTWNTYmhYeVEKLS0tIEJiZzJvS3BsYzB3cHIxa2k5N1Ro\nenFFZDVaODNnVGdBZTBOYWJwRjQzc1kKlXJgee8wTSN4Beq4P0t9cYbk0BWHCseQ\nyaWpiPT9aZBEGLFmuEd3zKABc8lrilX/ySTmOG49vRg6CPmr7cT0Wg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEa0JYenFMNmNzWnVmcXdz\ndG90ZUZ0WWlIU0FCZG9OaWpBM3ZDWnFhZFhNCjhuV1FTOTJ2WVJGa2RuNVV2MjR0\naVNXa3diaWxWUlJtdkNOQXZ2R2NsQkUKLS0tIDcyYXh3N3B2QmNiK3dzemFFMGV1\nNTZpTk5yNGV5YVo3cGswK0NLWFQxQlEKIe0N5OxooWXzt1cUViBmjihmGEe3G6/f\nkz2/IscnG78ZvNgYKjdoG1jlsyje/3zI4C8aWXLq2DnIyxUyAhPgsQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2b0NVVm9QNm9waUUzcXBi\nWnZWU0JETjZRZmprZVRUL1h6ZHB4cXkrdm5rCm9GZ0VnTXB3S1BYSmlGWFJVcDhJ\naUl3RjR0ak9BRmQvVk1GRnQxNmtYM00KLS0tIFlTU1p2OHhWUGlOWngwbE56NEhF\nRW5QSkVVUWZpdDZXWEIxZ1BkbzVwclEK2P25nBgf8255vaKW/+T97aNTecRgNjLu\nedIUiPdXbFATCe3v/YRo6sqzFwIsvM6Bl9yHh/SXo6Ftc7eWZZd8zQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs"
}
],
"lastmodified": "2026-07-28T01:44:24Z",
"mac": "ENC[AES256_GCM,data:J0D8bEs5mHLraLS6TvYuCgfiNU1xKM2Yfb5Y0f/q/4wM4LzXufNzv3+SWDHumTe328U8UnNXLqjNHEKL0bZi0coxpU5hVM+BvPcmqD72vscETzbQ2hnU05sfW+XjfZhcN8/ke0bpLt7nP0crD5hsZv3esV1E2UWvzjEiYtWzFHY=,iv:lcmXYG2H469UKBYDndWKMO+GP0mSGLztenm+kBaUdYI=,tag:ujiXbLM9CUsuoFwQQWI84Q==,type:str]",
"version": "3.13.2"
}
}
+22
View File
@@ -0,0 +1,22 @@
{
"data": "ENC[AES256_GCM,data:Q++XWxg9tvY7ugT8+8FWCC5jgOfQ1+LYLsnqN0/WGxTf5XT2VmoXvszcE/ow2BUOBG3qBTXS3OHYFPwmj1GgaBHB8rpdXX6+LveSBE2gmx1VR+NUDTxy+0/DBq411MK9n/J9eIYcybdFIE11biFSAob9EgfxBF5roCDXIPDPyVCSe6LyhNvqYPnGQsfHCbejSewLTcRQEiguP9BX96CpMPIpmaB9fHN25t5RWCgMI7MtacrRxRsyKg44+2FZstXdZrp2Wv9u86BxqdAFqtZE8qpPeGdrdzluZx9jhnw0wZPzHdKg5wS7/UrLCb0UxIQiDxDMDuuBuLGkKXfAhb6SGZU41wWAWYk548iGRUaG+79BO4HhRZNObOFvfsjpMXEfcH/Vbv/wHVY4OpJUPe/ZWK4wpL9YrY4nT0t62HH7MirOy1uhwLE50h2+6dHKj6ur5G7thkSqgzVWXQ==,iv:zaRBwS+gfXLhH30havn6Q2+oPWuLV3qBfbOj00kewlQ=,tag:Vr5iEQ2u+9YNahryhgzwSw==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArTGJnaVhleUtsWnlITE1s\nNmRCaW1QWTFNSG5LRFdmb1lzRlV5NWl3YTBNCmF6eUN6RkJBZzc0MmJJa0dKTW01\ncVh4K1VLR2lURUpKQXpxNGpQNnpSUlUKLS0tIFJONnJFWkNCR3pqalRsUW9POVBj\nQ0pFQ3ltKzBETTVXTW5sV1ppWTFJc1kKzxUboNZO+Nwn2eTWy11VP9w1pRswCHaJ\nE2dYU0oUOClVzc0oSuIJxraG6TPj1N4WGC24gS+UmpkmSuCiOeZBsw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBDSmxNR1I3dUFKT0xUVG9h\nWkRIakNVeWRQOEN3blNRVjZlWHF4K2NRa0hBClRiOXlmTTJ4T2JTUEw2c1l0R2N2\nMnRwdDA5bEZlQWJRTm9vUmNKclBSU1EKLS0tIFcyeDFjbTZyVEVDUjN1VzU1VHly\nNWNDMW9rTXY2bHNWYVR0SmtMckovUzQKhTWr6yFVW9am3okCiIswwqR5+/p9OLmB\nWCgPtwoFaBt1RjUXPK4/eS4LlucR2K6V/mNMn4xVsnkIl193U9632g==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2ME9CbmVGcENRWksyL0pW\nOEJyUGhpbGMwWlBhVXBSeXQ2MW1EWnFuR0E4CnMwU0pjdk1YMzF5ZEhTVFlBaHZq\nam94UWVGbjhZSEx6VHBmem9JRWgwYzAKLS0tIHFJZWRyRjRHNzhXdDJSYWN3bDlR\nYVp3eGJWWkh3Y09ZWElyclZQN1ZSVFkKjR32//EcFAdMjVlNgky5zvVkwXwEN68D\nrkTuHKjiO5aV7yAQGPkdNw0UM0oRGF0u4YF3oOUcZfSvnKgDeoi2Zw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz"
}
],
"lastmodified": "2026-07-28T00:27:46Z",
"mac": "ENC[AES256_GCM,data:AbJIHYcFpeanQsJ3x7RPL9Yjlg5BJgkepKax0fL9L/PpA03Antab93iUNG95Mp6k/duovp8Jm445lbuppDZq1dh9ij/deBa8GbzJ50wwEe9zMc3EwRKScpqZEhRPF7KlJsIjsHJyd8NkcI5ji49XkHb4Ae1//8zG5HpVgy+3b04=,iv:uxQCbMwMIfP5S1dbsvIx3F79YWEguxwox8T0YZvUBdc=,tag:3utjmBGDmPc8q4JjaXvCkA==,type:str]",
"version": "3.13.2"
}
}
+26
View File
@@ -0,0 +1,26 @@
{
"data": "ENC[AES256_GCM,data:aqlMnoVkGtH9z3fJRweeC0OYf7LGqJU2sWA9Q25dKK6NuNyJd4BvjPtpfeg/WhVtJsaOtcbwVm4WAhVK9FARE8g8j+vmq0f6BAU4s6mx0ZIhl+mP+/hIpt//LOdd+9YezelJxdpzUyZbdAngU99rsTluLRe2XmZ7Fquxd8yH/OHenSDY6dizp9+5jfEi8EU+EmuXvuWMPY59xnlnqYNPfSFxs43/pS402LzJoJ5H+cBPprddkUBVzy4cBQvMnrRFUSjnqp74ovZkfIWFqDWQ5YgSU2PjatBg18oulZ7wNRhQ6OLqj6gsu+xrMjNFwnp7rMlA3X//hIidxTkVcYITycXd8KzuMIaofUpnwoyT34fy6+H35/39iiEyG4LRTrOOKRDzXkY2rhJUxFSZ8GlhNhMd0RlkmLngVYrtjsswJ9meIwoAFLPYt7BC61PJf0TXdtk=,iv:mCA819J9LpAOj8QxFAkrHI9wFJIy8qVxv31D6IwWFnk=,tag:qqMtM0eyHbQEyl6ND/wf7g==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOZVpRYmNMQ1E4ZlFXMDVq\nbW1FcnYzWTBHQmdhSjlsZUtZK3dzNUZjY1FvCkV4VXU4MTcwNVZoZUpVVS9JeHE5\nbGk3UzE3YmpZQ1JISHV0RHJqSG9ZNXcKLS0tIHhBdnIvMjJkeWlVRE9Fb0FhWjNX\nUjBVWDltK2w0akkwOTJaTGNSWWNrRXMKqZRNnHiXvn1QBoSGdABp7vOqNlsEN6Xr\nDp3NByXow6PuRuWvQXHzd+WC+ADkwNaaiT6TUrbZcd/Pl8Ges9kcZg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2VFBxc2ZMUFp0YkJhVFhW\nc0ZNbnJUcGtkT1VLR2YrVGhRRFVTZFlyZXlJCmVnblNBNGxIdHh4Q0IweWR1b0dq\nRWFyOURuWkdkRE1RTnJRMEpXdk9HaUUKLS0tIFhKV3FoWW1zRk1pMlhuWWlhV2E5\nQzJTRHAyc0JtSjd2NHlJODZVbndaVDgKFA4565X/4FqNq/fZDZTg81/55hZi4c7b\nTti2AnyE3OcY/kurXJFHRinVMqURQf1fx9MxqUYRitiCz4qe5zFF+A==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwRWlKTFNocnBRN01pZ0hm\na3pyRUkxMWM3dE1iTkZZcGRYUFcyb1RDb2h3Ckt4blpGZHBjcnZ4SVE5UnF4cmY4\nWW0rZG5wUkZINVk5a2lmSzN2L3JJY1kKLS0tIExpWGlBRFZJVGpGbFhxRE5ZYjBo\nQVliTlFIZ1U5dE9xbDhHMUtxenpBSm8KY6sIFEfK8p+70IXsC4Jwb9Lm/pd9+V6K\n4JAzGrpA6mAuIwwSNnbdcA5j8FmBhCpK6nLBWmFhGm9Y+MRTaM7Jrw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoV0pIUXlyOFFTUHVTdllu\neG10VENOYWl5Wk0ycnpLUytTbnlrQ25pMkFRCnhyRU5xWW9vQXJJOHdFMGZqMkR1\nSktqT3lOdVVPRUN4YTlveGp0NXpqd0kKLS0tIGRaTktra0ZtSVpzSHBrY3VSSUph\nRVRZOSsxTTNmMmltMlJnVy9oT2VEWU0Krxf49B1BsrWn05fqg+cZ0k0PtfJJNfn0\nUL44RUWXWbK2igQHaIct9DfYe7DEonBJeROuxDYm8g7yNOv15S+P4Q==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp"
}
],
"lastmodified": "2026-07-28T01:44:26Z",
"mac": "ENC[AES256_GCM,data:eyOSn03dzHSgkshPzLVwc95382eEFaDQarHs9l83dtcsb1Ui9CjkKipl2DVSUb6bdMUH1qKYmXqJhwFnAZbFZFjT4VTKtutNtM+OkhVXfT+fJs+1+u7k+ZUYFL9HuxKA6AWpwX3eJ8vmJDJZaAayJbm4PRzOyJywvKeneQqdUS0=,iv:QRKltR4qzVofo/Elt2Us/lrHlD7BenX605X31x+Ng78=,tag:SZ97CcfgMJeu3yqNk8Y/cA==,type:str]",
"version": "3.13.2"
}
}
-43
View File
@@ -1,43 +0,0 @@
beszel-token: ENC[AES256_GCM,data:cbQOXhLzNk4g9d6hvm2DH7Q5ApTPCTzsW2txflDT2dD/UPIE,iv:V19MI1GEo5/0205Hrt7JImfkjduFiZ7f9aIkDVaI8mU=,tag:WCArgdrnIOudVe/Tw+oxRw==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaYU9HR0lETDhkYXR5NWlj
L0diSE5JQU9KWjRoU3hrVjdqZ2tPNUtOdFZvCjdUVGNFbDVYa3pVSC9ZWVNORER4
QWozUlhoSEtjTk9IRlM3VkZoYlc1RTAKLS0tIDlHS01WVWlOMWFEQU5GTVRLZVhV
VTcwekhrRHB3SlVYT2MzOW5GbE52dkkKKCWehPhpdGapdyzpll20NJUcZwvW/7X8
KQ1EqAgI2fewnbwuIDYCleN0b0SLJNUeSV/tFKDDoTMnHWCdeD4ECg==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4YndybFBTQ2p4SGZ4SDJs
a2p0eFRQOVVWcGd2a25ESW9ESGx1RWw0Zno4CnhpVVh3cGI5UjY0YmFINFFPMTh4
b1B5SjJ3NTNvUE1QUmJjVFozY1dYS2MKLS0tIC9HZGNpOFZhZGNFZGt5blJuZXVV
SXpkRzV4d2ppV3ZQZSt1dmxYNGVFMUEKmSe9dkrmkND81Hw2/ATAmFvcmhk1tUC1
LxxBw54IVHUqwYKgRYUYRNu+pykDT5OnFDPiskd49Xso99LY87PyiA==
-----END AGE ENCRYPTED FILE-----
recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLd1FSSHVTSGdHWm8wMVQz
d0dlOEsxeGwxdHU0eUlFSUxka0ZmcVpnOTM0CndYNUVjVy92QVhNY2orQmpSQjYy
VS9KM0NUTXhuM0lCSDBZMWtISEdtWTgKLS0tIExTL2wvS3FEdVViUmRYZEFsR3R0
YTJFM200RjF6MjNxOFA4eHRpWVhtRkUKOqBIT445HnPXrrH/qV6FIuAhAuJmSL6V
+PQopM/m3PAnK5m5Mu3cfjYfDiB8+GWTABhljfT+GbcoK7CqWLehrw==
-----END AGE ENCRYPTED FILE-----
recipient: age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvVXF4a2NmaW15R2VXS3FP
SDVPUGpUWTlIWjl2N2N2SXR1UlRkdllWNVRJClZDbVo4dUhrZytqZkMrYWtpOEZx
c0dSZGViN04zQ1B1WEZEWm1QM1lsejQKLS0tIE0wa0k5Rm1xZmw3OHFESkVXc25j
NlFpYTJSckQ4MVlZQ01reDlkaWY3TkEKsmQlreRhRAjVZ/q5x52FNATDF2sLhbHo
djOZigZx2rs6shqQ6It/XRJ8CiPaXZBPOt529Gwmplu+hWlAU1+l2Q==
-----END AGE ENCRYPTED FILE-----
recipient: age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
lastmodified: "2026-07-19T02:30:40Z"
mac: ENC[AES256_GCM,data:rKHZjU/MH08ASTlu32HZO9uWmsBYuMCEC6M8gwVhzuWvmablnP05tS2z13XfaWaCEUXk6kmGJKuU0zu5+IKVZgamCF6DAMtxQb6bVCaLsoAm/GSqWQ5VI9eHqgnSSdN/o3ul/33Rf8iBQo4aw8FFAmDVuNz8bfAn0QefFTj0ByI=,iv:JD2gtqRinOY77etg6PUmZNovkYl1Q3F6ZvRi4x7RznQ=,tag:/5IMpWKRVt+l1luCTQE0BA==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.1
+26
View File
@@ -0,0 +1,26 @@
{
"data": "ENC[AES256_GCM,data:+/1AAVha+86abQIX8tebqPJ6BIcCCiTJgPe0OfFOwA2Mcx0NJdiPtQgyeo3G8fAolIQOFkk5reL+GUhlOz4iEbPOCWwFCckdH0tXiCiVcD35qQNMRurY8HmpM5FWFnyzBkKuyNZ8okPuo2+fA3NQh3gs94rpm2kBsfS18xvrlv9b8u1JvAxlH6GRMN1uUgFGmOXlpGAVjDUFiKUHN8tRSZpsxG0aKKPBZ82JdKgYfCiCQifS1366gIFrsjGriUC+P8wkadRnD1JE2ZAGSL7wJLaRmzA4LAMGXFGbitOsFxKxX3q8VWEXaaL+9h4rTe0WCrvmgDM9NUeVHpfRiJrLvQgrFZMUxuZF66vQVsLybjIWpYYJcjC6nfR0U3lMa9gjzOTMPxm8HdGKC53FhLM1HiPoe1a2Nno283fV0uaByMPmo257O6l1CXqb6KoZOGFccm7fKR/VnFAS1AInF+szI7/r+UcaTW6LwYeV3OwLE4a8b4OiqTLgCu6v,iv:kUlVXF4Yl5HGvoLwu9loiuuDVtz0kARRiUU3K+BPL2A=,tag:pVaGzQAX9Etoqc5XMGmNeg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB6OUVUdkoxM1M5THlYZ1ky\nYWJ5RjhtRGoxTldwRkI2T1p6dHR4Y3gwR1ZrCjRDS1hTM01lZ0FSalBYQ1B0bENs\nUytXNmlrUkV4TmJkMUM1QW1ZaExIcGMKLS0tIHdHKzdEcUxvM0ZYUEp0a2d6SHp5\nYXQydy9uNG15V2FhUUd4NEFTMTNNMEUKkIzKEoYzoVs+nhnpkHFgDkQqrWykatND\ntsNxcr1SXSKeEW1m/QpXZnn/aW3zSQR09PqOHf7PYU47/AdkwrUaAg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSByZENrck42dWJjR2hYUVBl\ncERtRjR6dTdING5nWXlBc3o3eXFhaSs5NkFRCk43QXFUeTBJR0U5dG9YTDRmQ0Yx\naEo2blUvUXNZY3dpbEZRTUx1elNzdkkKLS0tIHJwNjRNM3V6WktWZ1BlUUtLRTNI\nMTVEbCtYbllIbTdxTGozWUluS3pIRXMKZCruDIkD/JofdAHWgPuaaKTDsz408ZkY\n77mhO8J+kd03qwt6qhFC5KF1lyjhwEnqrOE195+R/8Yl7hA/DsL2ZQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxaC95RjYybzVlU2tYdHdU\naGJ4M2IvNkJwVlNIU0VCZEFUZk5VaksrVnpFCmJXS3luTG9WejR0Rk9DbHA3dngy\nRElFWjU1N2xuc3JaKzQ0L0U2Yktib1kKLS0tIFlrK3F0ZFh5Ync3ejVWMzRKTUx2\nZDhxdEMwa3B4TFZUcWdTdktDeGt1QUEKfgYnK2lW3cZuJGaw+bAKDipLuC4S5vK2\nxK2eJB5TP/xXrp0F3lx9sc2b1FOY9Vt9IQ7zVlBqJFkzJrcw8zYJJw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwOGFkSXNxTUIzLzR1Skk5\nMGZveHFDN3YwVmpwT3VTYkppWEU1aGVpdkhRCjZtRC92bVphd0ZFbzFxVzRUcGk4\nMUd1aWdEbFRaM01FT25JSVRoMTRsNUUKLS0tIDAybmcrYjFVWkFYaGY5TFZjcjFO\nYlZoQjJjN2lFNzd4RlJuSHFlRlNCOFEKgUIPnL2/OJgz9oMYt86/llHa7adTkhs6\n8yGYGV1wtU9aYtUMIR907SfYyZ6M4z8jH2wwzpQLbwQQMLgkejl1jg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m"
}
],
"lastmodified": "2026-07-27T23:14:51Z",
"mac": "ENC[AES256_GCM,data:CYhzR0Y29GXvWUBbb13s16v9hDQ4k4Xmd4FUpcyNtHe0L/IrDGkx1WHzAohNGZjGMNB9W3BgiEH85OoOo8r1F82El4/8s2prEJi8AARvQU3+2cmRjjhNCCYQWmJkoF/cZNpw6nVyWzZdfUpPvHjbuf6utlI3rtR0qiCpSo/32S0=,iv:XLKTU2Ute4jMkfRAbXiVGxqP9+fjSs6HwRv1JfTTe7w=,tag:oIqS/1HWQZ7mLWaO7GwLFA==,type:str]",
"version": "3.13.2"
}
}
+18
View File
@@ -0,0 +1,18 @@
{
"data": "ENC[AES256_GCM,data:QvViw/s1N7eIN3CoD27llEjriuSrohOou4Cv310nigcW8xMnA2SDN2id3H3AoKii1JlJ+qWpKn+gUmt5HOM0UlbeNe3os2RVwiX38O7eN9xFH9F5kA3TFs6Umqq3EoI586PwIVmB2LyxDnTeEEXVd7v5PFkBcfu7u8YIcNF7lpcj+6rOyHMB8uxPhrGep3yiKawFd9c9wWD0hlSSatV5tMHA1qmdK8VmDbCU/iuGwIoMzN1eZwGAXzG6LkCA63bUfdxU6yGuTboD+kN2Wbo+GZB0EACmiZoofl2wqlXuiw6qwTvlXkyauc9O5EG//PUkIECzDwiXcX+qSOM9DIBlZNth4ebhtic/PskyF09etL/gICz5YvV4ph5lyrWHq41KxljSU7QXOkGhzagruuMrYzhZb35wFH4Tie1ee2DXbGhreJr8V3Zse/zTMaD+iM57V8bvNcarTzOFXyKfp7Y=,iv:XBPhj2wT0k/yRCRHU4d+BQA/k00ZHWSKOucnZ5+PGys=,tag:Nh5MOxZIUXAYVZY5SZh/JA==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0aUtETXlBMmRlTC9vK3NF\nRW4zS1FrSmo1dDN0OGN4SmV5ODd5MTRVejFJCnpPQkM2MVJ5WFlYRW1NVjduV1hv\nUWUzY0hHV05LV1BROVZVUlM2NldsTlEKLS0tIHFsTzI2SVZzYUtJWTM0MmFiUlVQ\neHEvUXgzc1pxSU1OZFo0cXhSZDdGUVEKpLVfzQEnntluUGsblnkHZJ9Jezu8tFte\nxEoV96GVHxUca6TFWpTLMqdR7NtuQGCkx295W3i1tkp58DP1OzRTUA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA4ZW5ybVBvTkxibWJMWGRL\nZ1lnWEkxZ3l6d0FWbmxMY2VsTVlzOHJJM3owClNYd3BlQi9pL1lIdzJmekJUVEVi\ncWhSK3ZkZEMyeTNoZlpTT0NMUXZFWVEKLS0tIHVkVlhmQXdRQktTK3J5dXZ4aTNt\ncUw3WCt5dXJhTkdUbVpmeWoxWkoxNnMKj4XtdwmgFVOiVsIJs2Du7QJ09A9tv/Lf\nkFOq8y4tlZe0nCwRjq43sVz7hdCTdQ0rsaWjBGY90LLkJbOA+f+Wrw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2"
}
],
"lastmodified": "2026-07-28T01:44:30Z",
"mac": "ENC[AES256_GCM,data:zR2WLWX7NaHA15gi4kX0jDvzUIe9jtz5bMCAggbPW+IXOEedPrddAHZ8OfPErVMfx8O1pJKkAKSzoPTAbEle54FisSLMHXp8fI0297MByJrF9pOsMFpVcDy/L4Q+pBzmB7aS9r7+u7KRVVTZT3QwG1rFWZaDs5dFTP80RhtCbWQ=,iv:ZzPoO+h7ebS+jsSH7tWMx6QK8umpa2/HFQmx9dnJN+Y=,tag:vG9+ifxw4HaBE6YsmOwXcg==,type:str]",
"version": "3.13.2"
}
}
+94 -12
View File
@@ -4,8 +4,20 @@
homeDomain = "sweet.home"; # base LAN domain for service subdomains (pve., docker.)
tailnetDomain = "tail13f623.ts.net"; # Tailscale MagicDNS suffix
lanCidr = "192.168.2.0/24"; # LAN subnet
pxeServerIp = "192.168.2.247"; # pxe-boot host's LAN IP
pbsIp = "192.168.2.108"; # Proxmox Backup Server LAN IP
lanGateway = "192.168.2.254"; # LAN default gateway (router)
lanPrefixLength = 24; # LAN subnet prefix length (/24 = 255.255.255.0)
lxcLanInterface = "eth0"; # LAN NIC name in LXC containers (set by Proxmox --net0 name=eth0)
vmLanInterface = "ens18"; # LAN NIC name in Proxmox VMs (virtio, first NIC)
vmStorageInterface = "ens19"; # storage NIC name in HA server VMs (virtio, second NIC on vmbr1)
pxeServerIp = "192.168.2.223"; # pxe-boot LXC container LAN IP
nixCacheIp = "192.168.2.224"; # nix-cache LXC container LAN IP
tailscaleRouterIp = "192.168.2.222"; # tailscale-router LXC container LAN IP
torRelayIp = "192.168.2.221"; # tor-relay LXC container LAN IP
serverIp = "192.168.2.226"; # server (NFS/ZFS) Proxmox VM LAN IP
dockerIp = "192.168.2.225"; # docker Proxmox VM LAN IP
pbsIp = "192.168.2.244"; # Proxmox Backup Server LAN IP (not NixOS-managed)
domainControllerIp = "192.168.2.253"; # FreeIPA domain controller — authoritative DNS for sweet.home (not NixOS-managed)
ipaServer = "domain-controller.sweet.home"; # FreeIPA server hostname (used by security.ipa and Kerberos; must be a resolvable FQDN, not an IP)
# Cross-host references (LAN hostnames/users other hosts reach over the network)
nixCacheHost = "nix-cache"; # substituter/remote-builder hostname
@@ -26,19 +38,27 @@
# fresh client that has never manually ssh'd to nix-cache before. Update
# this if nix-cache's host key is ever rotated or the host is rebuilt
# from scratch.
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache";
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICuHUxGNH6ei3BZD+EfZs3l4X8uJNcjQiOsM/G4yo4O/ lxc-nix-cache";
# Beszel hub's SSH public key — used by every agent to authenticate the
# hub's incoming connection. Update if the docker host is ever rebuilt and
# the hub generates a new keypair.
beszelHubKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
# Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
# host (modules/nix-cache/server.nix) — one per client host that's allowed
# to use it as a distributed builder.
remoteBuilderAuthorizedKeys = [
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFDEA1S2ikpObREgbP5uVBWMxIOGbY8B+Wx7VTZK1m6t root@server"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPAYIT9ormlmxZ0SziyDQaUntnKI8HK9/s3Qac1ZKjP2 root@docker"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKKKzoEPl/ZW9KBRHBcp6/ThOngGpwMv5EhkTlgC4aDf root@nixos"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGtOWOCS+ImHc7NehguoyD7PbonGosKMZqc9+QR3v/h root@nixos"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHxXTQxFnArK5HXG7czeoybZebCGfxpUdusJkPn+BCSp root@server"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+ioWPhHixlgCB9KIQ0QTHTz6A+Oo2F3uKiINLip5rO root@docker"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEj26SL/emsVjW2YhRucJVp2kTz8WgcEQgjBEBLRikk root@claude"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII/rLceRhnDobVXQYiPceuhDHHvVjFQ1pc9A6un/eUlA root@server"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHj11bLPpRzH2oslnwFzEvY9cSgfEFtSZbLQaDm4nZMK root@pxe-boot"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH4/Sesm8NYpj73R0cbGhI0Ubvz73vIVWAnbEDTlBTdh root@tor-relay"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIBCFUUtePndW7pqtlawft1QCdHmBVs3O/c8EJO+RcXV root@tailscale-router"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBZ9WKKAlP9Z7GQdgaZ1Xgw9C+vja2lqEZO5rJFpVqYN root@ha-server-1"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEGNKlaaMckd8nLWNGz4B2QokXjnnIvM+rEUv+R6h0sp root@ha-server-2"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+XeMco7OxUpjrjZm54HogMs9QB5xlcKmElASRvrmlW root@nixos"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMseQwPpmaa6cgV5U8KhUsiVSYARG85zGa9rho0LJWks wayne@pve1"
];
# Admin SSH public key, authorized on the primary user of every host and
@@ -68,6 +88,40 @@
# one-line change.
primaryUser = "nixos";
# Primary IPA/domain user. Home Manager is configured for this user on every
# IPA-enrolled host (see modules/ipa/client.nix) to manage the environment
# that IPA itself doesn't cover: dotfiles, user packages, session variables.
ipaUser = "wayne";
# GID of the IPA "docker-access" group (GID 50010 on the IPA server).
# The local "docker" group is pinned to this GID on every host that runs
# Docker so that IPA group membership alone grants docker socket access -
# no per-host users.groups.docker.members entry for the IPA user needed.
dockerAccessGid = 50010;
# HA file server cluster
# LAN IPs (vmbr0 / ens18) — client-facing: iSCSI initiators, NFS, management.
# Storage IPs (vmbr1 / ens19) — isolated internal bridge, used for DRBD
# replication and Corosync heartbeat only; never leaves pve1.
# haServerVip: floating virtual IP managed by Pacemaker's IPaddr2 resource;
# NFS and iSCSI clients connect here regardless of which node is Active.
haServer1Host = "ha-server-1";
haServer2Host = "ha-server-2";
haServer1Ip = "192.168.2.228"; # LAN IP, node 1
haServer2Ip = "192.168.2.227"; # LAN IP, node 2
haServerVip = "192.168.2.229"; # floating VIP (Pacemaker IPaddr2)
haServer1StorageIp = "192.168.4.228"; # storage-net IP, node 1 (vmbr1 / ens19)
haServer2StorageIp = "192.168.4.227"; # storage-net IP, node 2 (vmbr1 / ens19)
haStorageCidr = "192.168.4.0/29"; # storage subnet — internal to pve1 only
haStoragePrefixLength = 29; # storage subnet prefix length (/29)
haStorageRoot = "/srv/ha-data"; # XFS-over-DRBD mount point on the Active node
haIscsiIqn = "iqn.2026-01.home.sweet:ha-storage";
# DRBD backing disk — identified by SCSI controller path so it resolves to the
# correct block device regardless of OS-level naming (sda vs sdb can differ
# between Proxmox VMs depending on disk-add order). drive-scsi1 is always the
# dedicated data disk on all HA nodes; drive-scsi0 is the OS disk.
haServerDrbdDisk = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1";
# Storage
storageRoot = "/tank"; # ZFS pool root on `server`
@@ -83,6 +137,7 @@
# dataset or moving where it's mounted only needs changing it here — the
# export and every client reference follow automatically.
nfsShares = {
options = "(rw,sync,no_subtree_check,no_root_squash)";
dockerConfig = {
subpath = "docker/config";
mountpoint = "/mnt/docker/config";
@@ -103,6 +158,18 @@
subpath = "raspi/volumes";
mountpoint = "/mnt/raspi-backup";
};
proxmoxIsos = {
subpath = "proxmox/iso";
mountpoint = "/mnt/iso";
};
proxmoxLxcImages = {
subpath = "proxmox/lxc";
mountpoint = "/mnt/lxc";
};
pxebootImages = {
subpath = "pxe-boot/images";
mountpoint = "/mnt/pxe-images";
};
};
# The Raspberry Pi's own NFS export — not under storageRoot/nfsServerHost,
@@ -129,10 +196,25 @@
# (modules/build-types/pxe-boot.nix).
pxeBootTftp = 69;
# `server`'s NFS exports need both the portmapper (rpcbind) and the
# NFS data port itself opened (modules/build-types/server.nix).
# `server`'s NFS exports: portmapper (rpcbind), NFS data, and the
# mountd RPC service (used by showmount/NFSv3 mount protocol).
# Mountd listens on a fixed port so the firewall can whitelist it
# explicitly rather than opening all of rpcbind's dynamic range.
# All three need both TCP and UDP (modules/build-types/server.nix and
# modules/build-types/ha-server.nix).
nfsRpcbind = 111;
nfsd = 2049;
nfsMountd = 20048;
# HA cluster ports opened on ha-server-1 and ha-server-2
# (modules/build-types/ha-server.nix / modules/ha/cluster-config.nix).
haServerDrbd = 7789; # DRBD replication (TCP)
haServerIscsi = 3260; # iSCSI target (TCP)
haServerCorosync1 = 5404; # Corosync totem ring (UDP)
haServerCorosync2 = 5405; # Corosync totem ring (UDP)
haServerCorosyncCrypto = 5407; # Corosync crypto sync (UDP)
haServerPacemakerRemoted = 3121; # pacemaker-remoted (TCP)
haServerPcsd = 2224; # pcsd cluster daemon (TCP)
# Opened on the docker host's firewall for the Traefik-fronted
# container stack (docker-compose config lives in the separate
@@ -166,7 +248,7 @@
# build (modules/disko/proxmox.nix, config.system.build.diskoImagesScript
# — see docs/proxmox-images.md). Root fills whatever's left after the ESP
# and swap partitions within this total.
proxmoxImageSize = "20G";
proxmoxImageSize = "50G";
# nix-cache's Nix store garbage collection retention
# (modules/nix-cache/server.nix).
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPwUaVHP4MHZXicMAYGOe2ME1tp+CJShr8WocevlGWMh baremetal-gui
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:iVu2256Uzr2ThOGmFvCe48vWxCTJFflKUZ5vFSlFn+NomD69+y7RXf0ZICYOIH9t8pDAT3ZQDoMDw/kbPHxIj2R2U4AAt2H5QmKddTK3YCR8GgPde6Jp//v8NxLv+ZkAfj4W532rdaJ4CYrga9m8rOZi/6FubldUjBAVLVBiHxX86nCMbP5z2YTdHi2/IfEVK4Agy63oPZ0yD0T7QcXEX5F5eMcd6ceIDjJoROsQw5yVwUnjREo/Sgv5mQy9wBaf4ZtLw+eY5mJECIXjxhSkz417N31GB0D57A9N+WrEm8BEeuIbd+x7zdTSnZRsCZXLI3HRmkFH8bK75G9kPpXrpTQ5TT58LwDNuAGUK9K6dS/h8WAD8JRhtLhnEdCZObpxwoxOT2cCredwYZSdoVVirhuQI5EsGiTU1SGP78guAqpXvwvfbgIo2Qvic/e9OPQCHWLO4YrWonc6VUHhh/HmPOmaSrMcCFHE027fGXP8uTUEzOyE5hzmcM0d0WOX26xJ7PttbdKcLF3dZDnJQphn,iv:vpaX2prOsN/4Ke2HvoNHHp86DQpcPC6EtfgWvlgRGlQ=,tag:2Fv0oy99E88owAnB1fkvDw==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsU2pBT3M1TGJUYVlWUjNp\nN3JudVdVamN0TnF6cjlXN3BJRW5HbDFIZTBZCmhyRlQyNkl4bjBjOEhMZVhmQWxK\nVEJvaEpnTDREOXlKV3VxYzdkcW5tUEkKLS0tIHROY2RZeXFKQXBUeTZLODEzTGxZ\nSHl4WFhmOWVVR2NEcUxQSCtObU8xWkUKa/qmGyWYuEjf+BCoag5H9cA4ovW+ro8V\nhGsi3GqPEFfrI+qx/e6JqxazYpwfwIEaMZljhfgoFzg0I1U7OsnpSQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:21Z",
"mac": "ENC[AES256_GCM,data:7xbxAir+/3FbbJU4L4qRpZIYr/gqyoHjGxJvYYVLZ5yZLs8k/UiTG/qlu2zVkN2yfQaUzAu4/bQco5vOin6pwFodRH8BK4fbVIOheExzN1fk2uayejt/wWRiWj8w+qH+HaWLejlvrmbNced9HTiGViAHa+EZv3OUphlZKAdk/3Q=,iv:ZBuC8htjmKKc84kCBRHj7TdRdiPaEGeVMl+8ydOfIMc=,tag:uO4evh0xShWmG8kY/AkOWQ==,type:str]",
"version": "3.13.2"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIO+Ub4qK5Asqgi3A17zEy/+VOQ4ozL85ZBvlJnyhLZHD linode-docker
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:KpK5pPE33YtraIHSylq1MmlDiRvpDdwz9QTHsZCVrGuB3M19y5EXCBnARXreEHQyRgXWgeL4XZUKhR10xzeq/TqqZdKROtGn+LNwDM2f0OxLLxwPoqAqGQ2yEH2WbQdui28zEPcG1qKxzSVH2P2LIdLLghVTVeR1MM+sWgcJC5aoQGjjHt1tgGdQDv/jfGz9gDB9KRlJ6BVCdOdbkR4Ui5QFJX3edwvHRXsZUvlDNoVNrnQnGXUtw9FI2Ma+4mZ06x9Jv53LgAGsVf+Bu5USAn2VUKZDZu6Tfcdg8gb9iDKQrb+1jDq/dIb4/rh8fJwynMyG+pdrLtAoHs1D2K/3itglifs4sX8XDxSotMJWBB/KZUqdI2TIjv3fnBgkENbMFiv2rboWULRKA8z+viP2phS7kwm+mp7bGqRn36Bs90O5iVgifu/PYVykdnKggDbCyTbuKqdoq/C/s2oO7WbQISmjj+HjRrQ4wQd74iUvYjV2364AhwDQ+1L/OCR7HYr/l/NRpBcrjI5Kxaa5Usef,iv:GfqfGPjKxpOhdLJvJWTduYe2FbIKrMlKevErvnxfOa0=,tag:khM375z1e04y/JebLz/VgA==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSbS96Rmc4eDJkNGNUV2c3\nS3FCYnY1TCtOWXh1MzBPbnRnL09VbzNxblZBCkp5NkpxUXlJbUFjay9KbjVZeGc3\nTzgxRk0yKzU0SFBGelBDZnVHVzNaNlUKLS0tIFZRWXFUVW56a2owWk9TYXZPTE9r\nbmZHcG5ZN1dla3pXa3ZhOUdjR2dyTVkKaqAlfNkc2wzjB2//7DzW7JWg2BZd0Vqe\nO9YttWf4ikU6vfM+M/yiWGpWJ4U8p/3PBmT3ZLQBRGRtcyPby5DxZQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:23Z",
"mac": "ENC[AES256_GCM,data:+arWx20i4iAlFhbzH8/R9jggzJEP2PlEzyBnbAiDYZhp+2vfbsn+28WBzmjhi4AXz8CbBrhhFS6IFmYQZSFyYFuT943qoo82d/8E6p1DoyEEBL856SmkuVKpCq6IpWEJmkj8j7z3yyrpXva0aagReP+8+agWI+wWdNmG9M2tBRQ=,iv:Edqregq0OXRM4AWJ9EOtR/dZCObRepHnP4s8SYYkKHo=,tag:bBSiY7EVtli41w9s6JbQVw==,type:str]",
"version": "3.13.2"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOFHRs2HkjhOL/Ilii8PiwxGkQNsstmGn/Bs2Vx/OWsI linode-gui
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:SRq6nUjsMjpM5CXHpAMN2j20/2LbdvHfwqOOoWMTDkW+oC8tLhQfKHpXzKFcKVfqiqjZqaS8J1pmb59buDuYvShXm21i+nRTfhzOHAvdQJNYghBKVLt10RuIDdgPuhpi7r9y5hPZrF+jFSyk4wLcoE+F9FpktTPZwrGnjd5kT1btehuwbjQFIgl23L9E7BtJMBOecUNc3xTLymaEi26kWga8zFXQMLiyLLP4ZPSrjhSeZpXx6zGj2iIHKnPSsaYTFhfoWbiOIzCZrSpV6GsDIsH4MDEwNjwk8fPL7VuGFoMJev0vM0IGeoKB1Jn7HIe6an1t8DoX1TZHk9NA0L2IJR7nglP5I9CHjDJzQZ4hOZvmy7PZd0M8P4be5sQ72Yl2ZgsX9y4ZhANJ+IvYMI7VM1GamcQqbuYrzYCc+zqVeq75FJYUW/fkcKpQihWjQeSO/PAzevdMH4H1k810MEQJSbQxeBB6xc7bArwip2vMvrmDAvBeJCkkJlDvLW5acnbX5MMiP+FiDOsTrggOSL6B,iv:g87PWSi9TyN4GwZdYAxUbRpJvtoYSon6tIoz4SC5bWg=,tag:hDem9tgzD0/lg44g3Us0vg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSByQWUxUHpyZkk5aFNFMTVY\nc09vcHdDUFIwUHA1TVhwblVPMWRiS0dTaHlBCmtHWk9YTVRTcGJndjQ3bmhIWEJL\nMkFVdzNmU3RRUGN6clhZNGRtVnRTQjgKLS0tIFoxWnBsa2Y2MTM2SjBjNjhrUDNx\nU1dyZkhpZHZkb2toeUFpS2tLN2Fic1EKUxQ1J57TJytHqIuLowiSyoS/nJPxSoZI\ni35CHWWE0+Y344m/DmdJPspvYZSI3rY1JbgaqjIKZrshDggZVTka2A==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:25Z",
"mac": "ENC[AES256_GCM,data:ARpnwhj3Y88r6FsqPFBUK/Nth/R6Kfel5cVtlKqjkAdK1FB5BMmdxNbbOn2AH7+zlzdH7qRKvkH49CS6P+bfLaIJBkdLanzF0lSzxq8+88fgqFhJi+mroAZGH6OJaVcHVdTytaE2b8xAMeoz07e1Smp07F6Sp5mwHemAB9EHi38=,iv:JrYUaIGeJYLcgKj845irmKPH05tD9z6Rabv1B5tVmZw=,tag:gK5X/dBJvrWF8LTpdhytAg==,type:str]",
"version": "3.13.2"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPp8ZFWz97BFJJxHE+3w0/RpIXpRV64XE0eNBYl3jpSz linode-minimal
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:b9t5ALXBMo3Y1HJZkIeCsrevIe8ZCmthgvKOU21MKJqV6UZF6JxUwh/ZVZG2aTBDU7yiocGHHoxGOcTqC21RgpEPQTRkklQbO33fvb3bh35pQTxPH8J5ZVgjLgThXDDE7pYcSWClJwxznfJQXiefLyJwxqowBIL2FAdLiWc6Em0lThbcbdlwLhxsmUR375tjRyp41gsYYUZOAdAwlNOVXi3EuMK3l6lEVY14cUJj4JdDGBErPy6l46R/jOkwTeDrsxRWIsOhR+Dq0MVo5TllYoTKpTvQEQQV4pG8v0XrTsjhrPPvbQ0y3QKI6KkU1f50d0eHPKeCrJOe4puQ9K2VAxPnzwWO1UTw3wY4SsE8DXE0wXM2auAdQbDXRpPidOXwyYTR+Rr7DQ7avHUqRDkV9ZW27EadnNQWSCiomdwkbTg0i89+cyDRf6Wh4viYtBH59WFt1qkGX4nFnvSyx84SzH1/uSVjJ2CE7NJQ7/8NWZHVjqDQcKDrONYvP3resCFGKTu3miAArTin1+NQ8HvzfH41FV41BRsMQDXP,iv:OUUUl4lyZPCTQcw+RELE7W6fWoCHnJmKzN2ELJwXp9Q=,tag:jS7TtEsoUipHiMgT/rQUBQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArSGJ0VFAyazlrNGRENGhT\nWDhnd1BreHJ1Ukg0a3B1d1hDcWdRWUdUVDNrCkkwQUtNaEZjdnJXNmx3SjhWVUpV\nVGNlQnh6U1dwY0JCY3NMdXppT1oyc00KLS0tIHlMeitMT21JQkE3UFg4V3FIL2pk\ndVVFV3dpU1RMWGEydHl6TjYyRlJQMGsKrGJc51sxBuSFcVj07otUJoZnaPZhwVPp\nSOat4/BsH/NgWN2W1b+x+qMZQhlwwUy/f4oIC0y2XSvDfif5hUNwJw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:28Z",
"mac": "ENC[AES256_GCM,data:3C/etclYVQ/T3AhF+BfvYxTSCemmhPAHPUUsfLLB/B/VVMYQUMcy8RGpe1jWdtlRfszuJ7UyRsFOgWrpbUQiRTFSDIkTRb6hlPBuK85MVDuEbGCQb8e2ztnYgGBm0fHqTf4c5+QEKoVFZE2QMnAdjGSn1tJmaOW6zcDRFQOHLKE=,iv:0sE1bbWSQabtcMa+mi9ym7/MXEaEKhTFlkDNWruDQ+k=,tag:dRzgBJ//7O1x51M1deCn2Q==,type:str]",
"version": "3.13.2"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOBJ/vcsXr8yc/dGQNkEJcBwtnApunrrK8BdNJZPk/D3 linode-nix-cache
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:+VpjDTycBuzaUzZuWfHPrXEPXAOip17PxYNbNTufr52EQxfME/VbvMlhvHeh7eysPFduH4IUHVdjulpymlSvPt5+BpE+o29WmwvdRThUEWKArXS+dYBdVtcZppHG0DcBvg6ogWWpKpnn/nyC1nqSlOPXbLvH8A/9iF4JrB8L9wvi8CnvhetEbzbUy9T7PnJuG7nv0GUZdqcIr+srxG+imdGPonpce+HZVAWbAUNo2duGMm51SnJKZUvN17wVqgjbFcK/D7GETyU/XOavHCoDrkum7EUcO0i7/9AvWLMdcEqTXekMSNE9v6ZCoFjO6lbWDORwxgaqiNgu9rFh9PtZ7Ji1wyvk0FQ2MXrRCMjRnxrrm3RQNXbYnZpKjEcXT+Y4jCCEL00SwQAwqcR4GvOAkyA2fog+kvT00Kpu2lr3ZV0flxDxmbDOCHc9FWcf34FxWQxKTpTKYAbh3t+9EHF/fZzrpaB/Lwvt4+9grwkq0HtdV409asKMhPObz/b41U1fw9u5OJ/drsKbDBD89vnvp4+Lj7cSbJVkObVk,iv:h5xMpGCXUd/Jo2OfolPGDO9Rcd4Q9pV4bGnfFMhPhwk=,tag:ABE3c9TNH83Bu2ogNj1Zxw==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA5eUQyV1Y5dUdpR0RVUm1F\ndEZseWFRbEhQTFJxQ2hvVlJZVjBBQVRsaUZjCnJBNVFibG5tVmwvNTFYV0IyQ3BR\ncTlZN0U5R2U4RmpVOFU4MTNKOHZqSDAKLS0tIDZNeXF0NDRQM2pNckYwUlFzRVFm\nMkM5MHJJcThqeTh3OWxleWZNblBabFkK1RISRRs4CZn07ounWKO6tZo3OGLY5K52\nW+PAxOpvREtGsQiPq8dQFTxjfg5YyfHkK6o976eKD3Y2VM7pDo8kew==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:30Z",
"mac": "ENC[AES256_GCM,data:ZWtdVL9q67Zo10eiCNCe+bZ8pUqPjUcKDD8M7N6+N4l1jjngkOajJ4wpft4UQOmAmzB4RSWyajpXxwOg324H5KylYAFxfc6QH22bTPY2gaa5hD/NPnpM9kxZlcI3WsElI2xZBGo6n0nD9q5JV254QMBPLlg5EkAs7KXPY1hXFTc=,iv:LDXaZAmxIWQzSBK8KdNJ2nrFOwr3+vuiZDRW54usGj8=,tag:dwXewWjVBtc5qDaLyFz4qg==,type:str]",
"version": "3.13.2"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPW/X9Mdrqs0wLR7XbEDTihk7TEkNZ3LcCeXoa2ITSDA linode-server
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:hzI2cJ81+Psv35MMZO06UHpC8F9FLVkfUzn0ReuR+WoUds8D5zhDFbQHeR6ByeHTuK8Hlj712okhoXh6Vm7l3WmhLJZkr8IGLsV9W+P2PYityjuOtBPrphrKUeSqxJfjQ31/EhZLOqw/508XUmQDNcs1/n2g0TtQ2UQSTAuO8r1OlRfRSPizvVdj7lu+Vqg3dDBRETSOJAYIh8XQXoQWl3M4dS7jOYUvgc4EsYOWrvPPPS+8xXzctBPeToasY0IyWtLzoeNajCs5EpH5u7S5K9S40/vFPa45Ic2IxEkb1HHClyN6lXWbnc5QlJ5HPrq+2YBU2y9dJHd9DQiwEbcEBIvR3/Lo9puVDppXx9kez7i4XR1UHU7WVu39nQZuP50BQQazj+eM0/HJapXfjoqtaaq0qAGN4onnmRd0vEmrZZl4WMXmx8m9HhxdNJoZ4C2Wb2skdHZok88tPzjd4r7xKzVhAbB+wqS4LML6FmCNk+m0USQXa25USXs0JDV7lgiTFQJPE/67JGa820WFicGs,iv:YmOeK2Ha3yBXumVO9strgLgqNmPOcnqwUDJv5QNR1WM=,tag:Z7kZvQLDOKx3uX4dRXJqeQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOSWl5ajRSVndlMHJJQTFp\nYzRRU3RQMEVzazB0Sm5FdWZmMzBRaUl5RUFvCjgwRUZWdUpvUjkzWHFVdThIZ1ZV\nM2RMNy9pUWF4VVBCVHBGWHgwakhKQ2cKLS0tIERIOTVVZm55QWkvL25SRGZkZmtI\nT2o3ZEJNQ2hmTDNoRUx2Z3UxcmlyekUKLOajmvRfLdCJL74PKSgBtIXDuAVd8NwM\nh4BtDs0hONOz82JaBqFw8Uz28hVFG/gcS80br1o2klqPd2gN62PCVw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:33Z",
"mac": "ENC[AES256_GCM,data:JJC+12gTCsDVvMWRtL5cj50kf1n6Xn2j1hNBnwvZXUA9Pdd96SXt61U/Q80h8GZ7Ycs/slsV7h5f3g6+8tV0PcdBM/vy0vPM5qX1zMySyMv2p+dkJb9MwQpPg2xAQ9jjYM9237p5n9nysgu74h4V7ccBqmBzp764bL9wx6hEzS4=,iv:U6u1GFvoxaqxmHv1zCht24nW0ZMJR2b4pZqCG3bGNzs=,tag:MocwywOBg/1rUuoSADuJVA==,type:str]",
"version": "3.13.2"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEPWLRc9oCcn9tqY+a+yMJPGaqDx67vNpzC6mpyXqFp linode-tailscale-subnet-router
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:OeRskwkEPIKfgzK1409bt/5Raw4HNgUvkfB1nBUlFZghjTn9nuZqWoIjclgaV+qcZWqi86YHL2QYU20fAOOINM8pppknh4iH2WLgkde8LCxFdijWq6hn1TWaLOBVOn5UxNNj/rSzyXqjImaKOXbxrOHJ4/snDaVJccIhqo7BfQfJlXf1eHyuqdAl33y/2NRYxd/6+cqntlALT61yNFEpX1iyoO7NyCsniYYvfOU3hzlW4Y7OLQT2wiVuns5Tt/ty1IRk2LU3AnpyklMnSalwFSBs4CWjzow36AOmmDRL/fUiQ2I4JJjsYmXlatvl1zYoG5Cr0utnclr6juUkpb7W4Pa4ZtA4xQIpbIBpo2b6RHedM9i48RA+JhrxaFnyM8Qhq0HJAd7usVRPtXLWEP7uHs89e8emSoam84gW27Ayyv6zWEZGungAzghNLcCEvc0JlQXYfkVla/gxCIKTTUNtch+aGAn/hSNNPXQH4E2Jcs8/zdXvyHl67Y6okrmcytn9ZdV7nSJ2HbjzOyLZOomiix0uHSeU5/t9BlhfBKPeHpapt0xv+ZAujKAuXalkSapP,iv:3doK6Rx3p8dLHBmqa6x/gi2qdQul0m5Er9fEbYaHa54=,tag:k+bzF7uBu/gt+Qt43vPQkw==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlUmxGdGhpQmZXQmtURXBp\nVHdhMlpHWjh3R0lvSzNJcFZ2RFhoMFBYZTNrCitpQXFpRlU5RlpOSnJkQ3BVb0I4\nWWt5aWtPbDVHRUlPOEpLRGZTQzI1U3MKLS0tIG5semJrN3RIMjcwdW42NGgxcWh3\nQjlKL0N6Ly9ZWEUyazVINC9YL0dLeXcKh46uyhNdyOYketjOHBP71Ad5Yz3TRb4u\nVZrcMuXT3+fbTumtvySu7sCMh2/UsKJxVXGnW20Rac7MrJEk4xX38g==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:35Z",
"mac": "ENC[AES256_GCM,data:UeBDJaEaiA/6FvmTDnyKqXqNH079RG/W2iRY/92i5DRw94+DkXckcQbv0Cw30maRAJnu04JUhPSwdCsrDM8m/ajqZYTK/VuWKVt7CMFxeA3quHYD4IlPpvh9lLuBLZ5/7sfvRMxRUN83Acr5JqFUiX4wUGcgMTUdnwU+t96hDyw=,iv:LoErePhZTtvxvxNwyllTafup/Q79xgpH306jx0xwUjI=,tag:qvTLlr33yi2eVwsQqAX8Ag==,type:str]",
"version": "3.13.2"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILWVWRVrhvVLI8e65/nQXUXoM4PctQDC2HQpA3VHMFKE lxc-docker

Some files were not shown because too many files have changed in this diff Show More