Compare commits

...
Author SHA1 Message Date
beatzaplentyandClaude Sonnet 4.6 e18b605706 feat(ha): add resize-data-disk.sh for online data disk expansion
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m27s
Three-phase online resize: Proxmox qm resize → guest block device
rescan → drbdadm resize + xfs_growfs. No downtime required.
Detects active node via crm_mon, pre-checks DRBD UpToDate/UpToDate,
auto-discovers VMIDs from qm list, and resolves the block device name
from the stable scsi-0QEMU... by-id symlink with a slot-index fallback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 09:59:39 +10:00
beatzaplenty ee2451d87c remove stray pihole backup
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
2026-07-29 09:50:50 +10:00
beatzaplentyandClaude Sonnet 4.6 71a6c4738c fix(ha): fix crm_mon active-node detection in all three scripts
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
crm_mon 2.x formats the Promoted line as "    * Promoted: [ node ]" — the
asterisk bullet means the previous grep -E '^\s*(Promoted|Masters):' never
matched, so active-node detection silently returned empty in health.sh,
failover.sh, and acceptance-tests.sh.

Fix: pipe through grep -v Unpromoted first, then grep -E '(Promoted|Masters):'
without anchoring to start-of-line.

Also: remove the SSH_OPTS=-i ~/.ssh/... variable pattern in health.sh and
failover.sh; tilde is not expanded inside double-quoted strings, so $SSH_OPTS
was passing a literal ~ to SSH.  Inline the key path in each function
definition instead (same as acceptance-tests.sh already did).

Also: drop the 2>/dev/null embedded in the crm_mon argument string — the
outer 2>/dev/null on the n1/n2 call already suppresses SSH stderr; the
embedded one was harmless but noisy to reason about.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 02:36:22 +10:00
beatzaplentyandClaude Sonnet 4.6 793a2b5924 feat(ha): add failover.sh and health.sh operator scripts
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
failover.sh: graceful active→standby resource migration with pre-checks
(DRBD UpToDate/UpToDate, quorum), spinner wait for XFS mount on target,
rollback on timeout, --to/--force/--dry-run/--timeout flags.

health.sh: read-only status panel — node reachability, quorum, DRBD
role/dstate/cs per node, Pacemaker resources, failure history, XFS mount
usage, and service port checks (NFS 2049, iSCSI 3260) via VIP.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 02:23:47 +10:00
beatzaplenty 98b5429fb9 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 02:04:51 +10:00
beatzaplentyandClaude Sonnet 4.6 f658bdbabc fix(ha/cluster-init): enable Pacemaker maintenance-mode during DRBD sync
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Even with both nodes in standby, Pacemaker's monitor operations keep running.
When the monitor sees DRBD is Primary on a standby node (that it didn't start),
it triggers a stop action — killing the initial sync after ~10 s.

Enable maintenance-mode after standby stops the existing resources but before
DRBD is manually brought up for the sync.  Maintenance-mode suspends all
start/stop/monitor actions so Pacemaker is completely hands-off during the
sync.  Disable it alongside crm_standby -v off once UpToDate/UpToDate is
confirmed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:49:15 +10:00
beatzaplentyandClaude Sonnet 4.6 d7e63cd80e fix(ha/cluster-init): fix DRBD metadata skip condition and StandAlone grace period
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
The metadata creation check used grep -q "UpToDate" which matches when the
resource is DOWN — drbdadm dstate reads metadata directly and returns just
"UpToDate" (no slash) for a down-but-previously-synced resource.  This caused
metadata creation to be silently skipped, leaving mismatched UUIDs from an
interrupted sync which cause DRBD to go WFConnection→StandAlone immediately.

Fix: require exact "UpToDate/UpToDate" (with slash, only possible when the
resource is UP and both disks are confirmed synced) before skipping.

Also give the StandAlone detection a 5-iteration (15 s) grace period before
dying, to avoid false-positive on the first few ticks while the peer is still
coming up.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:45:03 +10:00
beatzaplentyandClaude Sonnet 4.6 b88ea49880 fix(ha/cluster-init): keep Pacemaker in standby until DRBD sync completes
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Clearing crm_standby before the initial sync finished caused Pacemaker's OCF
DRBD agent to race with the manual drbdadm up/primary calls.  The agent saw
DRBD in WFConnection or SyncSource and tore it down, driving the resource back
to StandAlone and killing the sync in seconds.

Move the crm_standby -v off calls to immediately after the sync-complete
break, so Pacemaker only resumes once DRBD is UpToDate/UpToDate and safe
to hand back.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:41:15 +10:00
beatzaplentyandClaude Sonnet 4.6 c7268ade8e fix(ha/cluster-init): detect StandAlone DRBD state in sync wait loop
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
If the DRBD peer connection drops during the initial sync wait (cs:StandAlone),
the loop would spin forever printing "waiting for sync progress" with no
indication of what's wrong. Now parses cs: from /proc/drbd each tick and
dies with a clear error if StandAlone is detected. Also shows the cs: field
in the non-syncing message so the actual connection state is always visible.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:29:04 +10:00
beatzaplentyandClaude Sonnet 4.6 367158548e fix(ha): wrap crm-fence-peer.sh to set PATH for kernel UMH callout
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
When the DRBD kernel module invokes fence-peer via the User Mode Helper
mechanism it provides a minimal PATH that omits /run/current-system/sw/bin.
crm-fence-peer.sh calls cibadmin, crm_mon etc.; without those in PATH a
pipeline breaks with SIGPIPE.  A signal-killed process has WEXITSTATUS()==0,
so the kernel sees exit code 0, logs "fence-peer helper broken, returned 0",
and retries forever — blocking Pacemaker failover completely.

Fix: use pkgs.writeShellScript to create thin PATH-fixing wrappers in the
Nix store.  The wrappers prepend /run/current-system/sw/bin before exec-ing
the real crm-fence-peer.sh / crm-unfence-peer.sh, giving them a working
Pacemaker toolchain regardless of what PATH the kernel provides.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:22:46 +10:00
beatzaplenty 585126beee secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 01:11:23 +10:00
beatzaplentyandClaude Sonnet 4.6 ceb491b18d fix(ha/cluster-init): drop TTY detection, always use \r for sync progress
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
The TTY check ([[ -t 1 ]]) evaluated false under sudo, causing each update
to print on a new line via log(). Since \r worked correctly in the original
code (user confirmed), drop the branching and always use printf \r.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:55:31 +10:00
beatzaplentyandClaude Sonnet 4.6 6d16eaff89 fix(ha/cluster-init): fix /proc/drbd sync progress parsing and TTY display
Check NixOS configurations / eval-hosts (push) Successful in 10m29s
The sync progress grep was silently failing because /proc/drbd formats the
percentage line as "sync'ed:  5.2%" (two spaces after colon), but the
pattern matched only one.  Use [[:space:]]+ throughout to tolerate any
whitespace.

Also make the display TTY-aware: interactive sessions get the spinning
\r in-place update; piped/logged sessions get a plain log line printed
once per unique percentage point (avoiding scroll spam).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:51:35 +10:00
beatzaplentyandClaude Sonnet 4.6 e085d4707c fix(ha): add DRBD sync progress indicator and pre-flight sync gate
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
cluster-init.sh: replace the fixed 300-iteration sync wait with an
indefinite loop that prints an in-place progress line (%done, ETA, speed)
from /proc/drbd every 3 s. Clears the line with printf \r before logging
completion, so the output stays clean alongside the [cluster-init] log
lines.

acceptance-tests.sh: add a pre-flight check that hard-exits if
drbdadm dstate is not UpToDate/UpToDate, with a hint to the watch
command for monitoring progress. Tests cannot give accurate results
while the initial full-sync is in progress.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:46:57 +10:00
beatzaplenty c8d4440787 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:40:11 +10:00
beatzaplentyandClaude Sonnet 4.6 e7531b276e fix(ha): use crm_mon for Active node detection and add cleanup before VIP wait
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
acceptance-tests.sh: replace drbdadm-role-based Active node detection with
crm_mon-based detection. DRBD role briefly reads Secondary during Pacemaker
transitions, causing T2/T3/T4 to target the wrong node. crm_mon Promoted:/
Masters: lines are authoritative; wait up to 90 s for Pacemaker to settle
before falling back to NODE1.

cluster-init.sh: add crm_resource --cleanup before the VIP wait loop so
stale migration-threshold failure counters from previous cluster-init runs
are cleared before Pacemaker decides resource placement.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:24:47 +10:00
beatzaplenty 49a10d7cc5 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:16:51 +10:00
beatzaplentyandClaude Sonnet 4.6 ead4f55805 fix(ha): use stable by-id disk path for DRBD instead of /dev/sd*
Check NixOS configurations / eval-hosts (push) Failing after 9m44s
/dev/sda and /dev/sdb are assigned by the OS based on Proxmox disk-add
order, which is not consistent across VMs. Use the SCSI controller path
instead — drive-scsi1 is always the dedicated data disk on all HA nodes
regardless of which sda/sdb it gets assigned to.

variables.nix: replace per-node haServer{1,2}DrbdDisk with a single
haServerDrbdDisk using /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1.

cluster-config.nix: revert to shared volume block (per-node block was
needed for /dev/sd* but the by-id path is identical on both nodes).

cluster-init.sh:
- single DRBD_DISK variable (matching haServerDrbdDisk)
- robust /etc/drbd.conf patch: NixOS manages this file as a symlink to
  a read-only Nix store path; cp --remove-destination breaks the symlink
  before sed -i so the edit actually takes effect
- scp helper script to NODE2 rather than bash -c over SSH to avoid
  quoting complexity

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:11:13 +10:00
beatzaplenty cac5ec45cc secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:02:48 +10:00
beatzaplentyandClaude Sonnet 4.6 ec90753a09 fix(ha): per-node DRBD disk assignment (NODE2 data disk is /dev/sda not /dev/sdb)
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Proxmox VM disk ordering differs between the two HA nodes:
  ha-server-1: sda=OS (50G), sdb=DRBD data (32G)
  ha-server-2: sda=DRBD data (32G), sdb=OS (50G)

The DRBD resource config was using a shared disk=/dev/sdb which targeted
the OS disk on ha-server-2, causing drbdmeta and drbdadm up to operate
on the mounted root filesystem (hence "Device or resource busy").

Changes:
- variables.nix: add haServer1DrbdDisk/haServer2DrbdDisk
- cluster-config.nix: move volume block inside per-host on{} sections
  so each node uses the correct backing disk
- cluster-init.sh: use NODE1_DRBD_DISK/NODE2_DRBD_DISK variables;
  add runtime check that patches /etc/drbd.d/*.res on the running nodes
  if the deployed config points to the wrong disk (workaround for VMs
  built before this fix; redeploy with --force-rebuild to make permanent)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:57:07 +10:00
beatzaplenty df1ddee735 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:48:23 +10:00
beatzaplentyandClaude Sonnet 4.6 8e8261be31 fix(ha/cluster-init): drop bash -c wrapper for NODE2 drbdmeta calls
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
drbdmeta is in /run/current-system/sw/bin which is already on sudo's
PATH on the HA nodes; the bash -c wrapper with PATH manipulation was
adding complexity without benefit and introduced quoting hazards that
likely caused write-dev-uuid to fail or run in the wrong context.

Use direct n2_ssh calls instead — the same pattern that works when
tested manually on NODE2.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:42:36 +10:00
beatzaplenty 5e9541741f secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:30:12 +10:00
beatzaplentyandClaude Sonnet 4.6 07543d7d56 fix(ha/cluster-init): add drbdmeta to PATH and remove openssl dependency
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
drbdmeta lives in the same Nix store dir as drbdadm but sudo doesn't
inherit the full PATH, so drbdmeta was not found (exit 127) even though
drbdadm was.  Resolve drbdmeta's directory from drbdadm's location and
prepend it to PATH.

Replace openssl rand for UUID generation with /proc/sys/kernel/random/uuid
— openssl is not guaranteed to be on PATH in a minimal NixOS root
environment, but /proc/sys/kernel/random/uuid is always present.

Apply the same PATH fix on NODE2 inline in the bash -c invocations that
call drbdmeta over SSH.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:24:01 +10:00
beatzaplentyandClaude Sonnet 4.6 a7c4a24fc3 fix(ha): use drbdmeta --force directly; fix T4 grep-c arithmetic bug
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
cluster-init.sh: replace 'drbdadm create-md --force' with direct drbdmeta
calls using --force on both steps.  drbdadm create-md --force passes --force
to the create-md sub-call but NOT to write-dev-uuid, which then fails when
/dev/sdb is busy and stdin is not a TTY ("stdin not a TTY, not waiting for
confirmation" → exit 20).  write-dev-uuid failing means DRBD has no UUID,
so the peer can't identify the device → connection fails → no sync → wrong
Active node detected by acceptance tests.

acceptance-tests.sh T4: grep -c returns exit code 1 when the count is 0
(no matches), triggering '|| echo "0"' and producing "0\n0" which breaks
[[ "$IQN_COUNT" -ge 1 ]] with "arithmetic syntax error".  Fixed by running
the pipe inside bash -c with '|| true' on the grep to suppress the
non-zero exit code.  Same fix applied to T5's IQN_ON_STANDBY check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:09:06 +10:00
beatzaplenty a98955a8da secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:08:32 +10:00
beatzaplenty 7d60741e73 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:00:18 +10:00
beatzaplenty 427be2b287 secrets(ha): refresh sops host-key registrations for new VM instances 2026-07-28 22:55:02 +10:00
beatzaplentyandClaude Sonnet 4.6 684351b89b fix(ha): pre-seed SSH host key in disko image; fix DRBD init race
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
Root cause of recurring sops failures on new VM boots: disko builds raw
disk images, and create-proxmox-resource.sh only syncs the clan-var SSH
host key to the Proxmox node (for proxmox.nix to bake into the image) when
it actually builds — reusing a cached image skips sync_remote_host_keys, so
destroy+recreate reuses a stale image with the wrong or randomly-generated
key baked in.  On first boot the VM gets a different key than what .sops.yaml
was encrypted for, and sops fails permanently.

Fix 1 — deploy.sh Phase 3: always pass --force-rebuild so every VM creation
rebuilds the disko image fresh with the current clan-var key baked in via
NIXOS_HOST_KEYS_DIR (proxmox.nix already reads this under --impure).

Fix 2 — deploy.sh Phase 5.5: after VMs boot, scan their actual ed25519 host
keys and, if they drift from clan vars, update the clan var pub-key files,
rewrite the .sops.yaml age anchors, and re-encrypt all affected sops files.
Defence-in-depth: normally a no-op after Fix 1, but catches any residual
mismatch (e.g. --skip-create-vms reuse of an older image).

Fix 3 — cluster-init.sh: add crm_standby -v on for both nodes before DRBD
metadata init.  Without this, Pacemaker's OCF DRBD agent races: it sees
drbdadm down as a failure and immediately calls drbdadm up again, leaving
/dev/sdb busy when create-md / write-dev-uuid runs (drbdmeta exits 20 with
"stdin not a TTY, not waiting for confirmation").  Standby suppresses
resource scheduling during init; crm_standby -v off restores it after DRBD
is up on both nodes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 22:51:29 +10:00
beatzaplenty 327ff0b44d secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 22:27:29 +10:00
beatzaplenty 43b4cc6aa6 secrets(ha): regenerate IPA keytabs for ha-server-1 and ha-server-2
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
2026-07-28 21:54:00 +10:00
beatzaplentyandClaude Sonnet 4.6 dd984019a1 fix(ha): prevent drbd.service from starting alongside Pacemaker
Check NixOS configurations / eval-hosts (push) Failing after 9m47s
drbd.service runs drbdadm up all at activation time, but Pacemaker's
OCF drbd agent manages the resource lifecycle (up/down/promote/demote).
When both run simultaneously the device is busy, causing drbdmeta
apply-al to abort with exit 20.

Set wantedBy = [] so the kernel module and config are still present
(via services.drbd.enable = true) but systemd doesn't auto-start the
service. Pacemaker's OCF agent calls drbdadm directly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 21:48:43 +10:00
beatzaplenty cb8a51b2fd updated sops keys
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
2026-07-28 21:41:31 +10:00
beatzaplenty 18ab0ff254 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 21:19:22 +10:00
beatzaplenty 5c5f22f84a +x on scripts and remove beszel from ha for now
Check NixOS configurations / eval-hosts (push) Failing after 5m35s
2026-07-28 20:48:51 +10:00
beatzaplenty 094eaa752b secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 20:35:12 +10:00
beatzaplentyandClaude Sonnet 4.6 fe9fc7364b fix(ha/cluster-init): explicitly start pacemaker after corosync restart
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
At first boot, pacemaker fails with 'Dependency failed' because ipa-activation
fails (IPA not enrolled yet) before corosync/authkey are ready, causing a
systemd boot-ordering race.  The service recovers fine when started manually.
Add an explicit 'systemctl start pacemaker' on both nodes immediately after
restarting corosync so cluster-init doesn't time out waiting for it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:47:33 +10:00
beatzaplentyandClaude Sonnet 4.6 1ce4589830 fix(ha/deploy): remove _patch_targetctl — iscsi-target.nix now correct
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
The _patch_targetctl runtime dropin was a workaround for the wrong targetctl
binary path in the original iscsi-target.nix (used pkgs.targetcli-fb, but
targetctl is in rtslib-fb's python3 env).  The module is now fixed to use
\${python3}/bin/targetctl, so new VM images have the correct ExecStart/ExecStop
baked in and the patch finds nothing — failing with exit 1 and killing the deploy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:41:20 +10:00
beatzaplenty 7417cc1b0a Merge pull request 'feat(gui): add wayne's GUI Home Manager config matching the nixos user' (#94) from worktree-functional-mapping-russell into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #94
2026-07-28 09:29:28 +00:00
beatzaplentyandClaude Sonnet 4.6 ec44b7955b feat(gui): add wayne's GUI Home Manager config matching the nixos user
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
Adds nextcloud-client, chromium, git, vim, claude-code, fish, sops,
the Nextcloud tray service, SOPS_AGE_KEY_FILE, and Proxmox/PBS desktop
shortcuts to wayne's HM config on GUI hosts. Uses lib.mkDefault on the
EDITOR setting in ipa/client.nix so the GUI layer can override to vim.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 19:28:22 +10:00
beatzaplentyandClaude Sonnet 4.6 cf3d8ea9a5 fix(sops): resolve merge conflicts in ha-server keytab files
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
Both secrets/ha-server-{1,2}.keytab had unresolved conflict markers from
the stash/merge interaction.  Keep the upstream version (2 age recipients:
admin + the host's own key) so each host can decrypt its own keytab on boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:26:39 +10:00
beatzaplentyandClaude Sonnet 4.6 756e45743c fix(sops): resolve merge conflict in .sops.yaml keytab recipients
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
The HA branch merge left conflict markers in the ha-server-{1,2}.keytab
path_regex entries.  Keep the upstream side (adds *proxmox-ha-server-{1,2}
as recipients) so each host can decrypt its own IPA keytab on boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:23:07 +10:00
beatzaplenty 7e8755d141 added ipa keytabs for ha servers
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
2026-07-28 19:12:23 +10:00
beatzaplenty 8194707478 Merge pull request 'Worktree ha file server test' (#93) from worktree-ha-file-server-test into main
Check NixOS configurations / eval-hosts (push) Successful in 10m37s
Reviewed-on: #93
2026-07-28 09:09:04 +00:00
beatzaplentyandClaude Sonnet 4.6 d740064a35 fix(ha): all 7 acceptance tests pass — targetctl, fencing, failover, data integrity
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s
Deploy/init fixes:
- iscsi-target.nix: targetctl binary is in rtslib-fb (python3 env), not
  targetcli-fb — fixes ExecStart and ExecStop for the targetctl.service
- deploy.sh: _patch_targetctl() applies runtime dropin to both nodes before
  cluster-init so Pacemaker can manage the iSCSI target from first start
- cluster-init.sh: replace crm configure heredoc with cibadmin --replace XML
  (pacemaker-4.0 schema: globally-unique in meta_attributes, promoted-max/
  promoted-node-max, Promoted role in constraints); force_unmount=true on
  xfs-data; DRBD promote timeout 240s
- cluster-config.nix: add crm-fence-peer.sh/crm-unfence-peer.sh handlers;
  update fencing comment to reflect resource-only + Pacemaker-aware handler
  replacing STONITH during testing phase
- ha-server.nix: add openiscsi to systemPackages for T4 iscsiadm availability

Acceptance test fixes:
- acceptance-tests.sh: fix ((PASS++)) set -e bug → PASS=$((PASS+1));
  detect Active/Standby dynamically via drbdadm role (Pacemaker can promote
  either node); T4 bash TCP probe instead of iscsiadm; T5 timeout 120s;
  T6 echo|sudo tee for root-owned XFS write (bash -c redirect runs as nixos
  not sudo — permission denied); use ns cat / ns rm for root-owned reads

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:03:39 +10:00
beatzaplenty 164d14eb87 Merge pull request 'fix(docker): declaratively apply groupmod --non-unique for IPA GID conflict' (#92) from worktree-parsed-mapping-raven into main
Check NixOS configurations / eval-hosts (push) Successful in 10m36s
Reviewed-on: #92
2026-07-28 08:19:53 +00:00
beatzaplentyandClaude Sonnet 4.6 720399b00d fix(docker): declaratively apply groupmod --non-unique for IPA GID conflict
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
NixOS's group activation uses plain groupmod, which silently skips the
GID change when SSSD exposes the same GID (50010) via the IPA
docker-access group through NSS.  Add an activation script that runs
after the normal 'groups' step and applies groupmod --non-unique so the
local docker group can share GID 50010 with the SSSD-provided IPA group.
If the GID actually changes on a live system the script also restarts
docker.socket + docker.service so the socket is recreated with the new
GID.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 18:17:43 +10:00
beatzaplentyandClaude Sonnet 4.6 da4d808c6a fix(ha/cluster-init): use crm configure instead of cibadmin XML for resources
cibadmin raw XML fails schema validation under pacemaker-4.0: globally-unique
is not a valid direct <clone> attribute, and master-max/master-node-max are
renamed.  Switch to crm configure commands which are schema-version-aware:
- promotable clone with promoted-max/promoted-node-max
- order/colocation constraints using Promoted role (Pacemaker 4.0 naming)
- crm configure handles schema differences automatically

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:57:49 +10:00
beatzaplentyandClaude Sonnet 4.6 79cde50e27 fix(ha/cluster-init): use targetcli delete commands for LIO teardown
clearconfig does not reliably clear kernel LIO configfs state — the kernel
still holds backing-file references, blocking umount.  Replace clearconfig
with explicit targetcli delete commands (/iscsi delete, /backstores/fileio
delete) which do release kernel state.  Also make DRBD secondary demote
idempotent (skip if already Secondary).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:55:26 +10:00
beatzaplentyandClaude Sonnet 4.6 d31d9fa584 fix(ha/cluster-init): fix LIO teardown, mount idempotency, VIP portal
- Remove VIP-specific portal binding — the VIP doesn't exist until Pacemaker
  assigns it; the default all-IPs portal (::0:3260) is correct for Pacemaker
  to manage
- Clear existing LIO targets before re-running targetcli (idempotent on
  partial failures)
- Tear down LIO kernel objects after saveconfig so umount succeeds (LIO holds
  the backing file open otherwise)
- Guard mount with mountpoint check so re-runs don't fail when already mounted
- Use --replace for cibadmin constraints (idempotent vs --create)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:50:24 +10:00
beatzaplentyandClaude Sonnet 4.6 c76698efb7 fix(ha): add xfsprogs to system packages; fix cluster-init PATH and DRBD check
- ha-server.nix: add xfsprogs to systemPackages so mkfs.xfs is on PATH for
  root (needed by cluster-init.sh during initial setup)
- cluster-config.nix: create /var/lib/drbd via tmpfiles to silence
  lk_bdev_save warnings from drbd-utils
- cluster-init.sh: dynamically find xfsprogs in /nix/store if not on PATH
  (fallback for running VMs before xfsprogs is in the system profile)
- cluster-init.sh: fix DRBD metadata check on node2 — broken regex now uses
  grep -E for ERE alternation to correctly skip create-md when DRBD is already
  set up (previous regex would have triggered create-md on a live secondary)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:43:27 +10:00
beatzaplenty 601db79689 Merge pull request 'fix(ipa): restore IPA password login via LightDM and su' (#91) from worktree-parsed-mapping-raven into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #91
2026-07-28 07:31:46 +00:00
beatzaplentyandClaude Sonnet 4.6 acebbdbe26 fix(ha/deploy): use nixos+sudo instead of root SSH; temp key for inter-node comms
Root SSH was failing because only the RSA admin key was authorized but the
local dev box only has an ed25519 key. Fix:

- cluster-config.nix: add ed25519 keys to root (same set as nixos user) so
  future deployments work without the temp-key workaround
- deploy.sh/acceptance-tests.sh: SSH as nixos user with sudo instead of root@
- cluster-init.sh: HA_USER/HA_KEY env vars + n2_ssh()/n2_scp() helpers so
  inter-node SSH works regardless of whether root-to-root is available
- deploy.sh Phase 6: generate temp keypair, authorize on node2, place on node1
  for root to use during cluster-init, clean up afterward

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:30:53 +10:00
beatzaplentyandClaude Sonnet 4.6 fc8f7baf3e fix(ipa): restore IPA password login via LightDM and su
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
Two independent fixes:

1. hashedPassword = "!" on the IPA user stub
   Without any shadow entry, pam_unix returns PAM_AUTHINFO_UNAVAIL
   before prompting, so PAM_AUTHTOK is never set. The "!" marker
   (account locked, not a real hash) ensures a shadow entry exists so
   pam_unix prompts and sets PAM_AUTHTOK — which the subsequent
   pam_sss module can then use.

2. pam_sss try_first_pass instead of use_first_pass (login + su)
   use_first_pass silently fails when PAM_AUTHTOK is unset.
   try_first_pass prompts independently in that case, making IPA
   password auth work even if pam_unix returns early for any reason.

Root cause found during incident: wayne (uid 50002) was outside IPA's
auto-SID range (163800000+), so sidgen never assigned him an
ipaNTSecurityIdentifier. Without it, ipadb's handle_authdata step fails
with "Generic error" after SPAKE pre-auth succeeds. Fixed by manually
adding objectClass: ipaNTUserAttrs + ipaNTSecurityIdentifier RID 550002
to wayne's LDAP entry on domain-controller. Any future IPA user with a
manually-assigned uid outside the auto-range needs the same treatment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 17:25:37 +10:00
beatzaplentyandClaude Sonnet 4.6 3f9b968a41 fix(ha/deploy): use non-sudo SSH for /nix writability check
pve_check() always uses sudo, so "sudo test -w /nix" passes as root
regardless of whether the SSH user can actually write there.  Use a
direct non-sudo SSH command for the writability probe so the check
reflects wayne's own access, not root's.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:15:20 +10:00
beatzaplentyandClaude Sonnet 4.6 1263c7540c fix(ha/deploy): ensure remote clone is on correct branch before building
When create-proxmox-resource.sh clones the repo to pve1, it stays on
whatever branch was checked out. Add a pre-build phase that detects
branch mismatch and switches the remote clone to the current local
branch before building, so the Proxmox node always builds from the
same commits we're deploying.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:12:39 +10:00
beatzaplentyandClaude Sonnet 4.6 a8d8b1465c fix(ha/deploy): fix /nix ownership on pve1 before codex-setup
pve1 has a pre-existing /nix store owned by a different UID; wayne's IPA
UID (50002) can't write to it.  Add a pre-phase-3 check that uses wayne's
passwordless sudo to chown -R the store before create-proxmox-resource.sh
runs codex-setup.sh on the node.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:11:05 +10:00
beatzaplentyandClaude Sonnet 4.6 b76d54e702 fix(ha): fix deploy script bugs and correct cluster configuration
- Add root SSH key + passwordless sudo to ha-server nodes (needed for
  deploy script to run cluster-init.sh via SSH as root on node1)
- Fix cluster-init.sh: correct default IPs (228/227/229 per variables.nix),
  use \${VAR:-default} for all config so deploy.sh can override via env
- Fix acceptance-tests.sh: same IP corrections, add -i flag to SSH calls,
  use \${VAR:-default} pattern
- Fix deploy.sh dry-run bugs: pve_check() always runs SSH for read-only
  probes so bridge existence check is accurate; wait_for_ssh skips in
  dry-run instead of timing out
- Fix cluster-init invocation: upload script via scp and run via SSH as
  root on node1 (was incorrectly trying to run the script locally)
- Fix acceptance-tests invocation: pass IP env vars from deploy.sh

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:04:13 +10:00
beatzaplentyandClaude Sonnet 4.6 59854a0229 feat(ha): implement clan vars and fully encrypt all HA secrets
- Run sync-host-keys.sh for proxmox-ha-server-{1,2}: generates SSH host
  key pairs in vars/per-machine/, registers age anchors in .sops.yaml,
  adds both hosts as recipients for common.yaml, ha-corosync-authkey,
  and per-host secrets/keytab files
- Re-encrypt secrets/common.yaml with both new host keys
- Convert all stub secrets to real sops-encrypted files:
    secrets/ha-server-{1,2}.yaml    (YAML, beszel-token = PLACEHOLDER)
    secrets/ha-server-{1,2}.keytab  (binary, stub text encrypted)
    secrets/ha-corosync-authkey      (binary, stub text encrypted)
- Add scripts/ha/deploy.sh: full lifecycle script (bridge setup, VM
  creation, DRBD disk + storage NIC attachment, boot wait, cluster-init,
  acceptance tests, --destroy)

Bootstrap order (operator runs these before first deploy):
  1. bash scripts/ha/deploy.sh            # deploys, tests
  # Post-deploy secret replacement:
  2. sops secrets/ha-server-{1,2}.yaml   (set real beszel-token)
  3. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
  4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
  5. Set services.beszel.agent.environment.KEY in host.nix after hub pairing
  6. nixos-rebuild switch on both nodes to pick up real secrets

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 16:48:04 +10:00
beatzaplentyandClaude Sonnet 4.6 dc83333526 feat(ha): pre-stage IPA enrollment for ha-server-1 and ha-server-2
- Add stub keytab files (secrets/ha-server-{1,2}.keytab) so modules/ipa/client.nix
  activates for both nodes — IPA security.ipa.enable = true confirmed by eval
- Add .sops.yaml creation rules for both keytabs (admin-only until
  sync-host-keys.sh registers the host age keys)

Eval: both proxmox-ha-server-1 and proxmox-ha-server-2 build cleanly
with IPA enabled, beszel agent enabled, corosync authkey + beszel-token
+ ipa-host-keytab all wired via sops-nix.

Bootstrap order before first deploy:
  1. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
  2. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
  3. sops updatekeys secrets/common.yaml
  4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
  5. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
  6. sops secrets/ha-server-1.yaml  (set beszel-token)
  7. sops secrets/ha-server-2.yaml  (set beszel-token)
  8. On node1 after first boot: corosync-keygen, then
     sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey
  9. Set services.beszel.agent.environment.KEY in each host.nix after hub pairing
 10. git add/commit the real secrets, then nixos-rebuild switch on both nodes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 16:30:29 +10:00
beatzaplenty a960c662f0 Merge pull request 'feat(ipa/docker): grant docker access via IPA group membership' (#90) from worktree-docker-ipa-group into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #90
2026-07-28 05:48:45 +00:00
beatzaplenty 6f602b2245 Merge pull request 'chore: full sweep — docs sync, SSH key module extraction, NFS dedup, dead code removal' (#89) from worktree-full-sweep into main
Check NixOS configurations / eval-hosts (push) Successful in 10m39s
Reviewed-on: #89
2026-07-28 05:26:57 +00:00
beatzaplentyandClaude Sonnet 4.6 e62e9c9a6a chore: full sweep — docs sync, SSH key module extraction, NFS dedup, dead code removal
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m40s
Documentation fixes:
- README/AGENTS: rename tailscale-exit-node → tailscale-router, add ha-server
  build type and proxmox-ha-server-{1,2} host table rows, add baremetal to
  platform list, remove references to non-existent flake-target-refactor-spec.md
  and remove-sensetive-info-refactor.md
- docs/auto-installer.md: fix lxc-tailscale-exit-node → lxc-tailscale-router,
  add pxe-minimal to the flake outputs list
- variables.nix: fix domainControllerIp comment — IPA is the authoritative DNS
  at .253 (Pi-hole is gone), not a forwarding intermediary

Code deduplication:
- Extract duplicate SSH host-key preservation activation scripts from
  modules/platforms/lxc.nix and modules/platforms/proxmox.nix into a shared
  modules/common/preserve-ssh-host-key.nix; both platforms now import it
- Replace 8-line hand-enumerated NFS export lists in server.nix and ha-server.nix
  with a mkNfsExports helper that generates exports from vars.nfsShares — adding
  a share to variables.nix now propagates to both exporters automatically

Dead code removal:
- modules/common/configuration.nix: remove leftover NixOS skeleton comments
  (hardware-configuration import, grub lines) that were never used
- modules/docker/enable-service.nix: remove commented-out listenOptions and
  daemon.settings blocks
- hosts/server/host.nix, hosts/nix-cache/host.nix: remove #DOCKER_HOST comments

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 15:18:06 +10:00
beatzaplentyandClaude Sonnet 4.6 5beed2d75c feat(ipa/docker): grant docker access via IPA group membership
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m39s
Create an IPA group "docker-access" (GID 50010) and pin the local
"docker" group to that GID on all Docker hosts.  Any IPA user in the
docker-access group automatically gains docker socket access through
SSSD supplementary-group resolution — no per-host docker.members
entry needed.

Specific changes:
- variables.nix: add dockerAccessGid = 50010
- modules/docker/enable-service.nix: lib.mkForce docker GID to
  dockerAccessGid, removing the need to name individual IPA users
- modules/build-types/docker.nix: remove direct wayne docker.members
  entry (access now comes from IPA group)
- modules/ipa/client.nix: refactor repeated security.* / systemd.*
  top-level keys into merged attribute sets (fixes statix W20); add
  security.pam.services.lightdm.makeHomeDir so the GUI login path
  also creates the home dir on first login

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 15:16:05 +10:00
beatzaplentyandClaude Sonnet 4.6 fe7fc55c04 feat(docker): add IPA wayne user to docker group
Check NixOS configurations / eval-hosts (push) Failing after 12m13s
Grants the domain wayne account docker socket access on docker.sweet.home
without needing sudo, alongside the existing nixos local user membership.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 14:07:34 +10:00
beatzaplentyandClaude Sonnet 4.6 6cdae391f4 fix(docker): pin IPA dyndns to eth0 to prevent Docker bridge IP registration
SSSD's dyndns_iface defaults to "*" which registers every interface's IP in
IPA DNS, including all Docker bridge networks (172.x.x.x). This caused
docker.sweet.home to resolve to Docker bridge IPs instead of 192.168.2.225.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 14:07:34 +10:00
beatzaplenty 95eb494370 Merge pull request 'Worktree ipa lxc kerberos fix' (#88) from worktree-ipa-lxc-kerberos-fix into main
Check NixOS configurations / eval-hosts (push) Failing after 9m47s
Reviewed-on: #88
2026-07-28 03:39:23 +00:00
beatzaplentyandClaude Sonnet 4.6 9294d2fd25 fix(ipa): skip home-manager-wayne service when home dir absent
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m42s
On first enrollment /home/wayne doesn't exist until the IPA user's first
login (pam_mkhomedir creates it). home-manager-<user>.service cd's into
the home dir immediately and fails with ENOENT, causing the whole rebuild
activation to return exit code 4.

Add ConditionPathExists so systemd skips the service (condition not met,
no failure) instead. After first login the dir exists and subsequent
rebuilds activate Home Manager normally.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 13:38:49 +10:00
beatzaplentyandClaude Sonnet 4.6 0fe7ddf6e8 fix(ipa): reject FQDN input in create-nixos-ipa-host-account.sh
Passing a FQDN like "nixos.sweet.home" instead of the short hostname
"nixos" caused the script to create a double-FQDN IPA host account
(nixos.sweet.home.sweet.home). Add an early check that rejects any
TARGET containing a dot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 13:23:48 +10:00
beatzaplenty 5c2d61d35a Merge pull request 'fix(ipa): use NOPASSWD sudo for IPA user to bypass broken PAM path' (#87) from worktree-ipa-lxc-kerberos-fix into main
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Reviewed-on: #87
2026-07-28 03:16:03 +00:00
beatzaplentyandClaude Sonnet 4.6 186e9187ce fix(ipa): use NOPASSWD sudo for IPA user to bypass broken PAM path
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m44s
HM's useUserPackages creates a users.users stub for every configured HM
user, which lands wayne in /etc/passwd. NixOS adds pam_sss.so with the
"localusers" flag to the sudo PAM stack when SSSD is enabled; that flag
causes pam_sss to skip SSSD for any user found in local /etc/passwd,
falling through to pam_unix which has no shadow password for the stub.
Result: sudo auth always fails for the IPA user despite being in wheel.

Use NOPASSWD for the IPA user in sudoers instead. The IPA user already
authenticated to reach a shell (SSH key from IPA or Kerberos), so
re-prompting via a broken PAM path is security theater on a homelab.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 13:12:42 +10:00
beatzaplenty e6f15404f5 Merge pull request 'fix(ipa): add wheel group to IPA user stub for sudo access' (#86) from worktree-ipa-lxc-kerberos-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 10m54s
Reviewed-on: #86
2026-07-28 03:00:33 +00:00
beatzaplentyandClaude Sonnet 4.6 44a0acc18f fix(ipa): add wheel group to IPA user stub for sudo access
Check NixOS configurations / eval-hosts (pull_request) Failing after 28m20s
initgroups() uses NSS (groups: files sss) to build the supplemental
group list at login. Adding wheel to the local users.users stub means
the IPA user gets wheel membership from /etc/group, satisfying sudo's
group check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 12:59:46 +10:00
beatzaplenty 1fc6e63178 Merge pull request 'Worktree ipa lxc kerberos fix' (#85) from worktree-ipa-lxc-kerberos-fix into main
Check NixOS configurations / eval-hosts (push) Failing after 33m33s
Reviewed-on: #85
2026-07-28 02:55:20 +00:00
beatzaplentyandClaude Sonnet 4.6 3589fc31d7 feat(ipa): add Home Manager config for IPA primary user
Check NixOS configurations / eval-hosts (pull_request) Failing after 40m50s
Any enrolled host now automatically gets a Home Manager profile for the
IPA primary user (vars.ipaUser = "wayne"), covering what IPA doesn't:
dotfiles, user-scoped packages (tmux, sshfs), and EDITOR variable.

The home directory is pre-created by systemd-tmpfiles so HM activation
succeeds on steady-state systems before first login; pam_mkhomedir remains
as a fallback for fresh deploys where SSSD hasn't cached the user yet.

A minimal users.users stub satisfies NixOS's assertion requirements
(isNormalUser + group) that arise because home-manager.useUserPackages
creates a users.users entry to install packages to /etc/profiles/per-user/.
The stub is shadowed by SSSD at runtime (security.ipa sets passwd: sss files).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 12:53:10 +10:00
beatzaplentyandClaude Sonnet 4.6 89746718a9 feat(shell): make Switch-nix/Test-nix/buildImage system-wide
IPA users (e.g. wayne@) don't get Home Manager so the aliases defined in
aliases.nix were invisible to them. Move Switch-nix, Test-nix, and
buildImage into programs.bash in configuration.nix so every user on every
host gets them via /etc/bashrc. Stub out aliases.nix for future per-user
HM-only additions.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 12:32:33 +10:00
beatzaplenty 232d0e7c40 Merge pull request 'fix(ipa): suppress Kerberos NFS services in LXC containers' (#84) from worktree-ipa-lxc-kerberos-fix into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #84
2026-07-28 02:13:47 +00:00
beatzaplentyandClaude Sonnet 4.6 0b60d3ff2d fix(ipa): suppress Kerberos NFS services in LXC containers
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m30s
The NixOS IPA module adds ConditionPathExists=/etc/krb5.keytab drop-ins
for auth-rpcgss-module.service and rpc-gssd.service via systemd.units.
In LXC containers with keytabs those conditions pass, the services start,
and then fail because auth_rpcgss can't be loaded and rpc_pipefs doesn't
exist in the container namespace.

Use lib.mkForce on our systemd.units text to win the conflict with NixOS's
existing definitions, and include ConditionVirtualization=!container
alongside the ConditionPathExists conditions so the services are skipped
(inactive, not failed) in containers that have a keytab.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 12:07:58 +10:00
beatzaplenty ff82e8885d added remaining ipa keytabs
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
2026-07-28 11:45:06 +10:00
beatzaplenty 9a7411d1dd Merge pull request 'Worktree ipa common config' (#83) from worktree-ipa-common-config into main
Check NixOS configurations / eval-hosts (push) Failing after 9m46s
Reviewed-on: #83
2026-07-28 01:43:53 +00:00
beatzaplentyandClaude Sonnet 4.6 2f829ba3e7 refactor(ipa): move FreeIPA client to common config with auto-enrollment
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m48s
modules/ipa/client.nix is now a self-contained NixOS module (no longer a
parameterized function): it checks builtins.pathExists for
secrets/<hostname>.keytab at eval time and enables itself automatically if
found, making it a no-op for hosts without a keytab.

modules/common/configuration.nix imports it so every host in the flake is
a candidate for IPA enrollment — no per-host wiring needed. Adding a
keytab (via scripts/ipa/create-nixos-ipa-host-account.sh) is now the
only step required to enroll a host.

The module also sets networking.domain and networking.nameservers via
mkDefault when active, so new hosts don't need those set explicitly.

Also:
- Remove explicit IPA imports from hosts/nix-cache and hosts/tailscale-router
- Add secrets/pxe-boot.keytab + creation rule; remove incorrect
  secrets/nixos.sweet.home.keytab and its creation rule
- Add .sops.yaml creation rules for all remaining host keytabs
  (server, docker, tor-relay, nix-minimal, nixos) so the creation script
  can target them without manual .sops.yaml edits
- Fix duplicate tailscale-router.keytab rule and corrupted gui.yaml comment
  block in .sops.yaml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 11:15:59 +10:00
beatzaplenty dedd69dc42 secrets: add IPA keytab for nixos.sweet.home 2026-07-28 10:43:43 +10:00
beatzaplenty f7f670ca4c Merge remote-tracking branch 'origin/worktree-docker-gui-fix'
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
2026-07-28 10:33:15 +10:00
beatzaplenty 55ba283c82 Merge branch 'main' into worktree-docker-gui-fix
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m24s
2026-07-28 00:32:06 +00:00
beatzaplenty 2d46d25a67 Merge pull request 'fix(ipa): harden script and update module docs' (#82) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Reviewed-on: #82
2026-07-28 00:25:13 +00:00
beatzaplentyandClaude Sonnet 4.6 f5d29be041 fix(ipa): harden script and update module docs
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m53s
Script fixes:
- Rename HOSTNAME variable to TARGET (shadowed the bash builtin)
- Fix ipa-getkeytab -s to always use IPA_SERVER, not DC_HOST (diverge if
  --dc is overridden to a jump host)
- Remove dead REALM variable
- Add EXIT trap to delete the plaintext keytab if the script aborts before
  sops encryption completes; cleared after successful encrypt
- Distinguish real ipa host-add failures from "already exists" instead of
  swallowing all errors with || true
- Warn explicitly when no platform age keys exist for the target (keytab
  would be admin-only and the host couldn't decrypt it at boot)
- Fix sops fallback from pinned nixos-25.11 channel to nixpkgs (uses the
  repo's own flake.lock)
- Expand "next steps" output to include networking.domain and nameservers
  lines that host.nix requires for IPA membership

Module docs:
- Point to the script as the primary setup path; move manual steps to a
  fallback section
- Note that certs/ipa-ca.crt is already committed (no need to re-fetch)
- Document the networking.domain and nameservers requirements in the header
- Add sync-host-keys.sh as explicit step 0

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 10:24:23 +10:00
beatzaplenty e10a1572c3 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
2026-07-28 10:23:34 +10:00
beatzaplenty 578ef70aa9 updated flake.lock 2026-07-28 10:23:23 +10:00
beatzaplenty e9fcbbbbcb Merge pull request 'Worktree ipa client module' (#80) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Failing after 9m47s
Reviewed-on: #80
2026-07-28 00:13:08 +00:00
beatzaplentyandClaude Sonnet 4.6 f46ae18672 fix(ipa): work around OpenSSH 10 AuthorizedKeysCommand path check
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m43s
OpenSSH 10.0 tightened AuthorizedKeysCommand security by checking every
path component of the command binary for group/world-write permission.
/nix/store is 1775 (group-writable by nixbld), so sshd silently skips
the command for any binary in the Nix store — causing IPA pubkey auth to
silently fail with no diagnostic.

Fix: copy sss_ssh_authorizedkeys to /usr/local/bin via systemd tmpfiles
(C+ copies the file rather than symlinking, so the path at runtime is
root-owned/755 throughout), and point AuthorizedKeysCommand at the copy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 10:08:57 +10:00
beatzaplenty fc277294f3 add vars to module
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
2026-07-28 10:02:58 +10:00
beatzaplenty f3e5ea67a0 add docker group to user in enable docker service module
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
2026-07-28 09:59:48 +10:00
beatzaplenty 7a8aebf679 Merge branch 'worktree-docker-gui-fix' 2026-07-28 09:59:40 +10:00
beatzaplenty a8d95aad02 enable docker
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
2026-07-28 09:49:14 +10:00
beatzaplentyandClaude Sonnet 4.6 dac5fbd574 feat(gui): enable docker service on workstation
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m45s
Imports docker/enable-service.nix and pins virtualisation.docker to
docker_29 via a nixpkgs overlay. Fixes indentation on `imports` block
and `programs.direnv.enable` line so nixpkgs-fmt passes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:48:59 +10:00
beatzaplenty da0c651c60 Merge branch 'worktree-ipa-client-module' 2026-07-28 09:48:55 +10:00
beatzaplentyandClaude Sonnet 4.6 97019205da fix(ipa): create home dir on first login + AuthorizedKeysCommand
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m45s
- security.pam.services.sshd.makeHomeDir: IPA users have no pre-created
  home directory on the host; without this, sshd opens a session to a
  missing directory and resets the connection immediately after auth
- AuthorizedKeysCommand was already added in previous commit

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:47:11 +10:00
beatzaplentyandClaude Sonnet 4.6 5487490b8e feat(ipa): add AuthorizedKeysCommand + enroll tailscale-router
- modules/ipa/client.nix: add AuthorizedKeysCommand so sshd fetches SSH
  public keys from IPA via sss_ssh_authorizedkeys, enabling pubkey login
  without per-host authorized_keys files

- hosts/tailscale-router/host.nix: add IPA client module + networking.domain
  so SSSD runs and wayne can authenticate on this host

- secrets/tailscale-router.keytab: sops-encrypted keytab for
  tailscale-router.sweet.home (generated by create-nixos-ipa-host-account.sh)

- .sops.yaml: creation rule for secrets/tailscale-router.keytab

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:42:38 +10:00
beatzaplenty 543ea432f0 Merge pull request 'fix(tailscale-router): stop dnsmasq from intercepting host DNS queries' (#79) from worktree-peaceful-stirring-noodle into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #79
2026-07-27 23:27:34 +00:00
beatzaplentyandClaude Sonnet 4.6 c7bbf88dce fix(tailscale-router): stop dnsmasq from intercepting host DNS queries
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m30s
NixOS's dnsmasq module defaults resolveLocalQueries to true, which adds
127.0.0.1 to networking.nameservers and binds dnsmasq to listen-address=127.0.0.1.
This made the host route all its own DNS through dnsmasq, which had
no-resolv=true and no upstream for anything outside the tailnet domain —
so every non-tailscale DNS query from the host itself (including SSSD
resolving the IPA server FQDN after the IPA client module was added) failed.

Setting resolveLocalQueries=false limits dnsmasq to its intended role: a
forwarding proxy reachable on the LAN interface for IPA's conditional
forwarder. The host uses domainControllerIp directly (already set in
networking.nameservers in host.nix).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:26:50 +10:00
beatzaplenty d57145b31e add tailscale-router to domain
Check NixOS configurations / eval-hosts (push) Failing after 9m44s
2026-07-28 09:17:29 +10:00
beatzaplenty 1cbe80c0ca secrets: add IPA keytab for tailscale-router 2026-07-28 09:15:21 +10:00
beatzaplenty 01ee261cf8 Merge pull request 'fix(ipa): stream keytab via sudo cat instead of scp' (#78) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Reviewed-on: #78
2026-07-27 23:14:41 +00:00
beatzaplentyandClaude Sonnet 4.6 f6f30c675f fix(ipa): stream keytab via sudo cat instead of scp
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m25s
ipa-getkeytab runs as root via sudo so the temp file is root-owned;
scp as wayne gets Permission denied. Pipe through `sudo cat` over SSH
instead, which reads as root but writes locally as the invoking user.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:08:19 +10:00
beatzaplenty 60b80cbd96 Merge pull request 'fix(ipa): SSH as wayne with sudo instead of root on domain controller' (#77) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
Reviewed-on: #77
2026-07-27 23:06:49 +00:00
beatzaplentyandClaude Sonnet 4.6 27a8c7fad9 fix(ipa): SSH as wayne with sudo instead of root on domain controller
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m23s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 09:02:32 +10:00
beatzaplenty d9cee0a674 Merge pull request 'feat(ipa): add create-nixos-ipa-host-account script' (#76) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Reviewed-on: #76
2026-07-27 22:58:46 +00:00
beatzaplentyandClaude Sonnet 4.6 6c1891812e feat(ipa): add create-nixos-ipa-host-account script
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m24s
Single command to enroll a NixOS host in FreeIPA and produce a
sops-encrypted keytab at secrets/<hostname>.keytab:
  - Adds the .sops.yaml creation rule automatically (with all registered
    platform-variant age keys as recipients)
  - SSHes to the domain controller to run ipa host-add + ipa-getkeytab
  - Refreshes the admin Kerberos ticket via `ssh -t ... kinit admin` if
    missing or expired, so no manual kinit step is needed
  - SCPs the keytab and encrypts it in-place with sops (file must be at
    secrets/<hostname>.keytab before encryption so the path-based creation
    rule matches — the common failure point when doing this manually)

Also adds HOME_DOMAIN and IPA_SERVER to scripts/env.sh, matching
variables.nix's homeDomain/ipaServer (same manual-sync pattern as
NIX_CACHE_HOST/LAN_DOMAIN).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 08:57:55 +10:00
beatzaplenty 59316c982e Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
2026-07-28 07:26:40 +10:00
beatzaplenty 9eca3bd719 Merge pull request 'Worktree ipa client module' (#75) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Reviewed-on: #75
2026-07-27 21:25:57 +00:00
beatzaplenty f2f0fcf756 certs: add FreeIPA CA certificate
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m45s
2026-07-28 07:25:28 +10:00
beatzaplenty b4fb9c25f2 secrets(nix-cache): add sops-encrypted IPA host keytab 2026-07-28 07:25:25 +10:00
beatzaplentyandClaude Sonnet 4.6 8955840f0a fix(ipa): use in-place sops encryption in module docs
sops matches creation rules against the input file path, so encrypting
/tmp/<host>.keytab directly with stdout redirect fails to find the rule.
Copy to secrets/ first, then use -i to encrypt in-place.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 07:19:53 +10:00
beatzaplentyandClaude Sonnet 4.6 a4b49c9909 fix(ipa): correct ipaServer to domain-controller.sweet.home
ipa-getkeytab confirmed ipa.sweet.home doesn't respond to LDAP;
domain-controller.sweet.home is the actual IPA server FQDN.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 07:16:53 +10:00
beatzaplenty c0936a10e7 Merge pull request 'feat(ipa): add reusable declarative FreeIPA client module' (#74) from worktree-ipa-client-module into main
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Reviewed-on: #74
2026-07-27 21:06:28 +00:00
beatzaplentyandClaude Sonnet 4.6 f4bbd6331d feat(ipa): add reusable declarative FreeIPA client module
Check NixOS configurations / eval-hosts (pull_request) Failing after 10m0s
Adds modules/ipa/client.nix — a parameterized module that joins a NixOS host
to the sweet.home FreeIPA domain without ipa-client-install. It configures
security.ipa (SSSD, Kerberos, PAM, NSSwitch) and places a pre-provisioned host
keytab via sops-nix binary secret so enrollment is fully reproducible from the
flake.

- variables.nix: adds ipaServer (FQDN of the FreeIPA KDC; security.ipa.server
  requires a hostname, not an IP, for Kerberos/TLS)
- certs/ipa-ca.crt: placeholder for the IPA CA public certificate (operator
  replaces with: curl http://<ipa-server>/ipa/config/ca.crt)
- secrets/nix-cache.keytab: placeholder binary sops file (operator replaces
  with the encrypted keytab after ipa host-add + ipa-getkeytab)
- .sops.yaml: adds creation rule for secrets/nix-cache.keytab (same recipients
  as secrets/nix-cache.yaml)
- hosts/nix-cache/host.nix: imports the IPA client module; adds
  networking.domain so the host's FQDN resolves correctly

Module header documents the three operator steps needed per host before deploy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 23:31:21 +10:00
beatzaplenty b99ba87cf6 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos 2026-07-27 23:05:35 +10:00
beatzaplenty 2128353f9f Merge pull request 'Fix/static ips' (#73) from fix/static-ips into main
Check NixOS configurations / eval-hosts (push) Failing after 42m27s
Reviewed-on: #73
2026-07-27 12:06:33 +00:00
beatzaplentyandClaude Sonnet 4.6 7b4ce0ab3d fix(server): prevent zfs-init-tank from wiping pool on udev race
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m46s
zpool create -f was called if `zpool import -d /dev/disk/by-id` failed,
which could happen due to a race with systemd-udev-settle. The disk
would then be visible by the time zpool create ran, silently destroying
all data on an otherwise-intact pool.

Fix: locate the data disk first, retry the import directly against it
as a fallback, then check zdb -l for existing ZFS label metadata before
concluding the disk is blank. Remove -f so zpool create refuses rather
than overwrites if a pool is present.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 21:52:39 +10:00
beatzaplentyandClaude Sonnet 4.6 3123565011 fix(tailscale-router): scope ts.net forwarder to tailnet subdomain
IPA refuses to create a forward zone for ts.net because it's a real
public TLD with DNSimple nameservers. The forward zone must use the
tailnet-specific subdomain (vars.tailnetDomain, e.g. tail13f623.ts.net)
instead. Update dnsmasq server selector and comments to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 19:23:46 +10:00
beatzaplentyandClaude Sonnet 4.6 36f5ebdf86 feat(tailscale-router): serve ts.net DNS forward zone for LAN hosts
FreeIPA (the new authoritative DNS) cannot reach 100.100.100.100
(Tailscale's internal MagicDNS resolver) directly because the DC is not
a Tailscale node. The tailscale-router IS a Tailscale node and can
reach 100.100.100.100 via tailscale0, so it now runs a dnsmasq
instance on its LAN interface that forwards all ts.net queries to
Tailscale's resolver.

After deploying this host, configure FreeIPA with:
  kinit admin
  ipa dnsforwardzone-add ts.net \
    --forwarder=192.168.2.222 \
    --forward-policy=only

This replaces Pi-hole's conditional forwarder for ts.net and restores
resolution of Tailscale MagicDNS names (e.g. raspberrypi.tail13f623.ts.net)
for all LAN hosts using FreeIPA as their DNS server.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 19:20:12 +10:00
beatzaplenty bba054db85 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
# Conflicts:
#	variables.nix
2026-07-27 18:29:07 +10:00
beatzaplenty b63a529a1d Merge branch 'fix/static-ips' 2026-07-27 18:25:37 +10:00
beatzaplentyandClaude Sonnet 4.6 ee96713a50 feat(networking): declare static IPs for all NixOS-managed hosts in flake
Adds static IP configuration to every NixOS host in the flake that
has a fixed LAN address, and centralises all network primitives
(IPs, gateway, prefix length, interface names) in variables.nix so
there is one place to update if any of them change.

variables.nix additions:
- lanGateway / lanPrefixLength — LAN gateway and /24 prefix, replacing
  every hardcoded 192.168.2.254 / 24 across host files
- lxcLanInterface / vmLanInterface / vmStorageInterface — NIC names for
  LXC containers (eth0), Proxmox VMs (ens18), and the HA storage NIC
  (ens19), used as attribute keys so changing the name is a one-line edit
- haStoragePrefixLength — /29 for the storage subnet, mirrors haStorageCidr
- Per-host IP variables: nixCacheIp (.224), tailscaleRouterIp (.222),
  torRelayIp (.221), serverIp (.226), dockerIp (.225)

host.nix changes:
- tailscale-router, tor-relay, nix-cache, pxe-boot: useDHCP = false,
  static address on eth0 (lxcLanInterface), struct-form defaultGateway
  (required when using systemd-networkd which LXC containers use)
- server, docker: useDHCP = false, static address on ens18 (vmLanInterface),
  struct-form defaultGateway (works for both scripted networking and networkd)
- ha-server-1, ha-server-2: replace hardcoded 192.168.2.254 / 24 / 29
  with the new variables; no functional change for these hosts

modules/build-types/pxe-boot.nix:
- Domain-controller kickstart template: replace hardcoded 192.168.2.138
  and 192.168.2.254 with vars.domainControllerIp / vars.lanGateway /
  vars.lanPrefixLength / vars.homeDomain so the template stays correct
  if the DC IP or domain is ever changed again

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 18:14:31 +10:00
beatzaplenty 1ece0c75d9 Merge pull request 'chore(pxe-boot): remove log-dhcp debug flag now that PXE boot is confirmed working' (#86) from fix/pxe-dnsmasq-port-conflict into main
Reviewed-on: #86
2026-07-27 05:52:18 +00:00
beatzaplentyandClaude Sonnet 4.6 548c6f5041 chore(pxe-boot): remove log-dhcp debug flag now that PXE boot is confirmed working
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 15:50:19 +10:00
beatzaplenty bae4c8171f Merge pull request 'fix(pxe-boot): use current pxe-boot IP (.247) in dnsmasq until Stage 5 renumber' (#85) from fix/pxe-dnsmasq-port-conflict into main
Reviewed-on: #85
2026-07-27 05:41:38 +00:00
beatzaplentyandClaude Sonnet 4.6 3748c86049 fix(pxe-boot): use current pxe-boot IP (.247) in dnsmasq until Stage 5 renumber
vars.pxeServerIp was already set to the post-renumber target (.223) but the
pxe-boot CT is still at .247, so dnsmasq was advertising .223 as the TFTP server
and the VM couldn't reach it.  Update to .247 so PXE boot works now; the comment
reminds us to flip it back to .223 when Stage 5 step 6 renumbers the CT.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 15:40:54 +10:00
beatzaplenty 9dd969cf4c Merge pull request 'fix(pxe-boot): open UDP 4011 for PXE boot service discovery' (#84) from fix/pxe-dnsmasq-port-conflict into main
Reviewed-on: #84
2026-07-27 05:22:46 +00:00
beatzaplentyandClaude Sonnet 4.6 7e4b2d33fb fix(pxe-boot): open UDP 4011 for PXE boot service discovery
After the proxy DHCP handshake, the UEFI client queries the PXE boot
server on UDP port 4011 to get the actual boot filename. This port was
not open in the NixOS firewall, so all four PXE discovery attempts from
the test VM silently timed out. dnsmasq was bound to 0.0.0.0:4011 but
never received the packets.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 15:12:12 +10:00
beatzaplenty 288d50fd33 Merge pull request 'Fix/pxe dnsmasq port conflict' (#83) from fix/pxe-dnsmasq-port-conflict into main
Reviewed-on: #83
2026-07-27 05:05:42 +00:00
beatzaplentyandClaude Sonnet 4.6 f0e76f8aff fix(pxe-boot): use pxe-service for proxy DHCP initial boot offers
dnsmasq 2.93 does not send proxy DHCP offers from dhcp-boot alone in proxy
mode — pxe-service is required to trigger proxy offers for the initial boot.

Replace the dhcp-boot-based arch detection with pxe-service entries for
arch 0 (BIOS), 7 (EFI BC / OVMF), and 9 (EFI x86-64), then use dhcp-boot
only for the iPXE second-boot HTTP chain (tag:ipxe clients). Add log-dhcp
for post-deploy verification.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 15:03:01 +10:00
beatzaplenty e80d195284 Merge branch 'fix/pxe-dnsmasq-port-conflict' 2026-07-27 14:32:00 +10:00
beatzaplentyandClaude Sonnet 4.6 c770feebc9 fix(pxe-boot): disable dnsmasq DNS listener to avoid port 53 conflict
dnsmasq binds port 53 by default even when only proxy DHCP is needed.
systemd-resolved already owns port 53 on the pxe-boot container, so
dnsmasq crashes at startup. Setting port=0 disables the DNS listener
entirely — pxe-boot only needs the proxy DHCP path for iPXE chainloading.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 14:29:30 +10:00
beatzaplenty 7fd6d558d5 updated dnsmasq service 2026-07-27 14:07:28 +10:00
beatzaplentyandClaude Sonnet 4.6 58c40292e2 fix(pxe-boot): move dnsmasq out of systemd.services into services.dnsmasq
dnsmasq config was placed inside systemd.services{} block, making Nix
interpret it as systemd.services.dnsmasq.settings (which doesn't exist).
Move to top-level services.dnsmasq and add UDP 67 to firewall for DHCP
proxy mode.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 14:02:41 +10:00
beatzaplenty 94842875d0 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos into worktree-network-cutover-plan 2026-07-27 14:01:24 +10:00
beatzaplenty cfa36b97fc added dnsmasq to pxe-boot 2026-07-27 13:44:29 +10:00
beatzaplentyandClaude Sonnet 4.6 4444398cac fix(network): wire correct IPs throughout and add cutover plan
variables.nix:
- HA server LAN IPs: .200/.201/.202 → .228/.227/.229 (from ip-addressing.md)
- Add haServer1StorageIp (.228), haServer2StorageIp (.227) for 192.168.4.0/29
- Add haStorageCidr for firewall rules

ha-server host.nix (both nodes):
- Add ens19 interface on storage subnet (/29)
- Fix defaultGateway: 192.168.2.1 → 192.168.2.254
- Fix nameservers: 192.168.2.1/8.8.8.8 → domainControllerIp (.253)

cluster-config.nix:
- DRBD replication addresses: LAN IPs → storage IPs (keep replication off LAN)
- Corosync ring_addrs: LAN IPs → storage IPs
- Firewall: add haStorageCidr to allowed sources

docs/network-cutover.md: step-by-step cutover plan with rollback at every stage

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 12:32:54 +10:00
beatzaplenty f80378f92f Merge pull request 'docs(network): establish IP addressing scheme and update infra IPs' (#82) from worktree-network-ip-scheme into main
Reviewed-on: #82
2026-07-27 02:23:01 +00:00
beatzaplentyandClaude Sonnet 4.6 cd4997f429 docs(network): establish IP addressing scheme and update infra IPs
Defines the new structured 192.168.2.0/24 layout:
- .10–.59   client DHCP (router-assigned, DNS → .253)
- .220–.229 virtual nodes (VMs / LXC containers)
- .230–.239 expansion buffer
- .240–.249 physical nodes (pve1 at .245, PBS at .244)
- .250–.253 network services (router .254, FreeIPA/DC .253)

Storage network 192.168.4.0/29 defined for HA DRBD replication
(internal vmbr1 bridge, no uplink). Host octet matches LAN throughout.

Updates variables.nix: pxeServerIp .247→.223, pbsIp .108→.244,
adds domainControllerIp .253. Updates pxe-boot.md IP references.
Full migration before/after table in docs/ip-addressing.md.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 12:19:45 +10:00
beatzaplenty 6ce4784376 Merge pull request 'Worktree ha file server test' (#81) from worktree-ha-file-server-test into main
Reviewed-on: #81
2026-07-27 02:18:06 +00:00
beatzaplentyandClaude Sonnet 4.6 5856d45575 feat(ha): wire sops secrets and disable NetworkManager for HA servers
- cluster-config.nix: add corosync_authkey sops binary secret
  (/etc/corosync/authkey, mode 0400) and force-disable NetworkManager
  (common config enables it; HA nodes need stable static IP networking)
- hosts/ha-server-{1,2}/host.nix: add host-token.nix import for
  sops-managed beszel-token; add KEY placeholder for beszel hub pairing
- .sops.yaml: add creation rules for secrets/ha-server-{1,2}.yaml and
  secrets/ha-corosync-authkey (admin-only until sync-host-keys.sh runs)
- secrets/ha-server-{1,2}.yaml, secrets/ha-corosync-authkey: stub files
  so eval passes before real secrets are provisioned

Bootstrap order (post-merge):
  1. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
  2. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
  3. sops updatekeys secrets/common.yaml  (grants HA nodes common secrets)
  4. sops secrets/ha-server-{1,2}.yaml   (set beszel-token values)
  5. On node1: corosync-keygen; sops -e --input-type binary
     /etc/corosync/authkey > secrets/ha-corosync-authkey; git add/commit

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 11:45:36 +10:00
beatzaplentyandClaude Sonnet 4.6 e3498b1087 feat(ha): promote HA file server to production flake targets
Adds proxmox-ha-server-1 and proxmox-ha-server-2 as real mkTarget entries
alongside the existing proxmox-server, backed by a new ha-server build type.

New modules
  modules/ha/cluster-config.nix — DRBD resource + corosync nodelist sourced
    from vars (haServer1Host/Ip, haServer2Host/Ip); resource-only fencing for
    production STONITH; HA port firewall rules for DRBD, iSCSI, Corosync, pcsd
  modules/build-types/ha-server.nix — imports pacemaker-stack + iscsi-target
    + cluster-config + beszel; NFS exports from vars.haStorageRoot (XFS-over-DRBD
    mount); nfs-server.service.wantedBy force-cleared so Pacemaker controls
    start/stop on the Active node only

New hosts
  hosts/ha-server-{1,2}/host.nix — static IP from vars, unique hostId; sops
    secrets (beszel, corosync authkey) are TODOs pending sync-host-keys.sh

variables.nix
  haServer1/2Host, haServer1/2Ip, haServerVip, haStorageRoot, haIscsiIqn
  ports.haServerDrbd/Iscsi/Corosync{1,2,Crypto}/PacemakerRemoted/Pcsd

scripts/ha/ (migrated + updated from test-lab/ha/)
  cluster-init.sh — generates corosync authkey, initialises DRBD/XFS/iSCSI,
    creates NFS dataset dirs, configures Pacemaker with DRBD + XFS + iSCSI
    + nfs-server + VIP; STONITH disabled initially (enable separately)
  cluster-enable-stonith.sh — enables fence_pve_ssh STONITH after key deploy
  fence-pve-ssh.py — Proxmox SSH fence agent (node names updated to ha-server-1/2)
  acceptance-tests.sh — T1–T7 production acceptance tests

test-lab/ha/ removed — all Nix config moved to modules/ha/ and
  modules/build-types/; scripts moved to scripts/ha/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 11:26:37 +10:00
beatzaplentyandClaude Sonnet 4.6 724d9a45af feat(ha): add NixOS modules for DRBD+XFS+LIO+Corosync+Pacemaker HA stack
All 7 acceptance tests pass on live NixOS 25.11 VMs (VMIDs 200/201 on
pve1).  Failover completes in ~5 s with data integrity verified.

modules/ha/pacemaker-stack.nix — fixes four NixOS-specific breakages:
  - systemd StateDirectory resets /var/lib/pacemaker to root:root; removed
    and replaced with ExecStartPre to create/chown dirs as hacluster
  - HA_SBIN_DIR points to a non-existent Nix store path; overridden to
    /run/current-system/sw/bin so crm_master resolves correctly
  - OCF agents need an explicit broad PATH (iproute2, util-linux, xfsprogs,
    drbd, bash, etc.) — NixOS services have no implicit PATH
  - FUSER=true bypasses the psmisc fuser check_binary call in the
    Filesystem OCF agent (psmisc not installed on minimal hosts)

modules/ha/iscsi-target.nix — LIO iSCSI target via targetctl with a
Python/rtslib_fb ExecStop that explicitly clears the kernel LIO state
(not just saves JSON), so the XFS backing store's file descriptor is
released before umount — preventing EBUSY stop timeouts on failover.
Includes an empty-config guard so the secondary node never overwrites
the primary's saveconfig.json with an empty one.

test-lab/ha/common.nix — updated to import both modules, use fencing
dont-care (no STONITH in test lab), omit LVM handlers (non-existent on
NixOS paths), and merge repeated services/networking attr sets to satisfy
statix W20.  test-lab/ha/acceptance-tests.sh — final v4 with crm_standby
fix (pacemaker 3.x API).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 09:13:55 +10:00
beatzaplenty 109429c7da Merge pull request 'feat(pxe-boot): add FreeIPA Server (Rocky Linux 9) iPXE menu entry' (#80) from worktree-freeipa-pxe into main
Reviewed-on: #80
2026-07-26 21:36:58 +00:00
beatzaplentyandClaude Sonnet 4.6 40856b2e5e feat(pxe-boot): add FreeIPA Server (Rocky Linux 9) iPXE menu entry
Adds an unattended install option to the PXE boot menu that installs
Rocky Linux 9 and configures FreeIPA on the domain-controller.sweet.home
host without any operator interaction after selecting the menu entry.

How it works:
- fetch-rocky-pxeboot.service downloads the Rocky 9 Anaconda pxeboot
  kernel and initrd from the Rocky mirror on first pxe-boot deploy
  (idempotent, same pattern as fetch-debian-netboot)
- rocky-freeipa.ipxe boots Anaconda with inst.ks pointing at the
  hosted Kickstart and net.ifnames=0 biosdevname=0 for stable eth0
- rocky-freeipa.ks (generated, includes vars.adminSshKey) performs:
  - Minimal Rocky 9 install with ipa-server + ipa-server-dns
  - Static IP 192.168.2.138 via NM connection file written in %post
  - /etc/hosts fixed for FreeIPA FQDN requirement
  - Random DM + admin passwords generated and saved to
    /root/ipa-credentials.txt (chmod 600, never hardcoded)
  - freeipa-first-boot.service oneshot enabled to run
    ipa-server-install on the first real boot (~20 min)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Jbvxx4xbHVcx1NkK3vtmK
2026-07-27 07:35:13 +10:00
beatzaplentyandClaude Sonnet 4.6 23634134f0 fix(ha-test): remove Debian LVM handlers and fix corosync authkey length
- Remove before/after-resync-target LVM handlers (snapshot-resync-target-lvm.sh
  doesn't exist in NixOS; exit code 127 caused DRBD to drop connections on sync)
- Remove split-brain handler pointing to /usr/lib/drbd/ (Debian path)
- Fix testAuthKey from 126 to 128 bytes (corosync minimum is 1024 bits)
- Fix cluster-init.sh quorum check: corosync-quorumtool has no -q flag;
  use -s | grep 'Quorate: Yes' instead

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 06:40:57 +10:00
beatzaplentyandClaude Sonnet 4.6 34c55f27ca test-lab/ha: nixpkgs-fmt formatting
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 06:18:42 +10:00
beatzaplentyandClaude Sonnet 4.6 dc36a47ac9 test-lab/ha: add session key + enable qemu-guest-agent
Add nixos@nixos session key so the Claude Code session can SSH into
test VMs directly.  Also enable services.qemuGuest.enable so
qm guest exec works as a fallback for key injection.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 06:06:53 +10:00
beatzaplentyandClaude Sonnet 4.6 750121e9dd test-lab: add two-node HA file-server test cluster config
Disposable test VMs (ha-test-node1 / ha-test-node2, VMIDs 200/201 on pve1)
to evaluate whether the DRBD + XFS + LIO + Corosync + Pacemaker stack
runs correctly on NixOS before deciding NixOS vs Debian for production.

Includes:
- test-lab/ha/disko.nix: 20G boot disk layout (smaller than production)
- test-lab/ha/common.nix: shared HA stack (drbd, corosync, pacemaker,
  targetcli-fb, xfsprogs), OCF PATH workaround for nixpkgs#207891
- test-lab/ha/node1.nix / node2.nix: per-node hostname + static IP
- test-lab/ha/fence-pve-ssh.py: Proxmox SSH fence agent for STONITH
- test-lab/ha/cluster-init.sh: one-shot cluster bootstrap script
- test-lab/ha/cluster-enable-stonith.sh: enables STONITH post-key-deploy
- flake.nix: adds ha-test-node1 / ha-test-node2 nixosConfigurations
  (bypasses mkTarget / clan-core / sops-nix — test-only)

These VMs must be destroyed once acceptance testing is complete.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:46:04 +10:00
beatzaplenty 479444d26a Merge pull request 'fix(sops): hard-fail on missing admin key and expand literal ~ in key path' (#79) from worktree-structured-nibbling-nova into main
Reviewed-on: #79
2026-07-26 19:43:57 +00:00
beatzaplentyandClaude Sonnet 4.6 8c19ee9d72 fix(sops): hard-fail on missing admin key and expand literal ~ in key path
SOPS_AGE_KEY_FILE was set in hosts/nixos/home.nix sessionVariables with a
literal ~ that Home Manager injects as-is into the environment.  In bash,
tilde expansion does not happen inside double-quoted variable references, so
DEFAULT_SOPS_AGE_KEY_FILE resolved to ~/... literally and the -s file-existence
check in ensure_admin_decrypt_key silently failed.  The script then generated
a brand-new age key (to ~/... relative to the repo root) while the real admin
key at ~/.config/sops/age/keys.txt went untouched -- making it appear the key
was lost when it was actually still intact.

Fix the home.nix root cause by using config.home.homeDirectory so the path
is fully resolved.  Add tilde expansion in ensure_admin_decrypt_key as a
belt-and-suspenders guard for any caller whose environment has the same issue.

Also replace the auto-generate-a-new-key fallback with a hard failure: auto-
generating a new admin key is never useful (it cannot decrypt existing secrets)
and created serious confusion about whether the original key was lost.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:42:32 +10:00
beatzaplenty 2514d3bc89 Merge pull request 'fix(lxc-pxe-boot): make privileged so NFS mounts work' (#78) from worktree-debian-pxe into main
Reviewed-on: #78
2026-07-26 19:29:25 +00:00
beatzaplentyandClaude Sonnet 4.6 48d6d6f7a2 fix(lxc): auto-derive privileged from NFS fileSystems, not hostname list
Replace the hardcoded hostname check (docker, pxe-boot) with a check
on config.fileSystems: any lxc-* host whose NixOS config declares an
NFS fileSystem entry is automatically made privileged. The script
already reads proxmoxLXC.privileged dynamically via
flake_target_lxc_privileged, so no logic change is needed there —
only the comment is updated to describe the new derivation.

Result: lxc-docker and lxc-pxe-boot (the two with NFS mounts) evaluate
as privileged=true; lxc-nix-cache, lxc-minimal, lxc-server,
lxc-tailscale-router, lxc-tor-relay evaluate as privileged=false.
Any future lxc-* host that declares an NFS mount gets the correct
privilege level for free without a separate manual edit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:27:56 +10:00
beatzaplentyandClaude Sonnet 4.6 cf0d62696f fix(lxc-pxe-boot): make privileged so NFS mounts work
The kernel's NFS client (FS_USERNS_MOUNT not set) rejects NFS mounts
from inside any unprivileged container's user namespace with EPERM —
AppArmor's mount=nfs feature only whitelists the AppArmor layer; the
VFS-level rejection happens before AppArmor is consulted.

lxc-pxe-boot mounts server.sweet.home:/tank/pxe-boot/images at
/mnt/pxe-images so nginx can serve large ISOs without filling the
container's root disk. Same pattern as lxc-docker (already privileged
for the same reason since it also mounts several NFS shares).

Operator action required: VMID 103 must be recreated as a privileged
container (the UID mapping on disk differs between privileged and
unprivileged; changing it in-place with pct set is unsafe). Rebuild
the tarball and use create-proxmox-resource.sh to replace it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:23:22 +10:00
beatzaplenty b2a3d5fbdd Merge pull request 'docs(pxe-boot): fix NFS path, note nesting=1 requirement for LXC' (#77) from worktree-debian-pxe into main
Reviewed-on: #77
2026-07-26 19:18:48 +00:00
beatzaplentyandClaude Sonnet 4.6 86e55ff954 docs(pxe-boot): fix NFS path, clarify NFSv3/NFSv4 split, document nesting=1 requirement
- Correct the NFS path from /tank/proxmox/pxe-images to /tank/pxe-boot/images
  (matches variables.nix's proxmoxPxeImages.subpath)
- Clarify that LXC uses NFSv3+nolock while VM uses NFSv4.2+automount
- Add explicit note that lxc-pxe-boot needs features: nesting=1,mount=nfs and
  why: nesting=1 is required by systemd 260+ for userns/credential isolation
  (AppArmor denies userns_create without it), mount=nfs for NFSv3 access.
  pct set replaces the whole features string — include both or the container
  will fail to boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:13:21 +10:00
beatzaplenty d3d6382360 Merge pull request 'fix: wire vars.nfsShares.options and correct stale share key names' (#76) from worktree-tidy-giggling-pizza into main
Reviewed-on: #76
2026-07-26 18:45:53 +00:00
beatzaplentyandClaude Sonnet 4.6 b8d21d78d9 fix: wire vars.nfsShares.options and correct stale share key names
- Filter non-attrset values from nfsShares in server.nix so the
  poolDatasets loop skips the new `options` string entry
- Fix typo proxomoxLxcImages → proxmoxLxcImages in server.nix exports
- Rename proxmoxPxeImages → pxebootImages in mount-pxe-images.nix and
  pxe-boot.nix to match the actual key in variables.nix

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 04:45:16 +10:00
beatzaplenty dcce023b14 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos 2026-07-27 04:34:53 +10:00
beatzaplenty 4c5ade5605 updated NFS shares and added vars.nfsShares.options 2026-07-27 04:34:32 +10:00
beatzaplenty b232daf5e1 Merge pull request 'fix(pxe-boot): fix NFS mount in LXC — NFSv3+nolock and skip rpc_pipefs' (#75) from worktree-debian-pxe into main
Reviewed-on: #75
2026-07-26 18:15:58 +00:00
beatzaplentyandClaude Sonnet 4.6 b65736c0dc fix(pxe-boot): fix NFS mount in LXC — NFSv3+nolock and skip rpc_pipefs
Proxmox LXC containers block the sunrpc filesystem (rpc_pipefs) via AppArmor
unless the container has `features: mount=nfs` set. NFSv4 requires rpc_pipefs
for client state management, so the mount fails outright in a default LXC.

Two fixes for the LXC case (config.boot.isContainer):
- Switch from nfsvers=4.2 to nfsvers=3,proto=tcp,nolock,nofail: NFSv3 doesn't
  need rpc_pipefs at the protocol level, and nofail keeps boot clean if the
  NFS server is unreachable.
- Add ConditionVirtualization=!container to var-lib-nfs-rpc_pipefs.mount via
  systemd drop-in: NixOS pulls this unit into nfs-client.target for any NFS
  fileSystems entry. With the condition, systemd skips (not fails) the unit in
  containers, keeping nfs-client.target green and activation reporting clean.

Proxmox VM hosts (not isContainer) continue to use nfsvers=4.2 with
x-systemd.automount unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 04:14:18 +10:00
beatzaplenty 4f54a1f0cd Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos 2026-07-27 04:05:15 +10:00
beatzaplenty 2d85ecec8f update pxe-boot images share path 2026-07-27 04:04:59 +10:00
beatzaplenty 34bb14d9f6 Merge pull request 'fix(server): open mountd port 20048 so NFS clients can scan and mount' (#74) from worktree-fix-nfs-mountd-firewall into main
Reviewed-on: #74
2026-07-26 17:57:40 +00:00
beatzaplenty 515da66db9 Merge pull request 'Worktree debian pxe' (#73) from worktree-debian-pxe into main
Reviewed-on: #73
2026-07-26 17:57:31 +00:00
beatzaplentyandClaude Sonnet 4.6 2e9d3da301 fix(server): open mountd port 20048 so NFS clients can scan and mount
showmount and the NFSv3 mount protocol need mountd reachable after querying
portmapper on 111; the server was only opening TCP 111 and 2049, causing
clients (e.g. Proxmox GUI NFS storage scan) to time out connecting to
mountd on 20048. Also adds UDP for all three ports — portmapper, nfsd, and
mountd all use both protocols.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 03:56:39 +10:00
beatzaplentyandClaude Sonnet 4.6 321048626e feat(pxe-boot): mount pxe-images NFS share and symlink to /srv/pxe/http/images
Adds modules/pxe-boot/mount-pxe-images.nix, which mounts
server.sweet.home:/tank/proxmox/pxe-images at /mnt/pxe-images via NFSv4.2
(x-systemd.automount on Proxmox VMs, nofail on LXC containers — same pattern
as docker/mount-data.nix). The pxe-boot build-type now imports this module and
replaces the previous local /srv/pxe/http/images directory rule with an L+
symlink pointing to /mnt/pxe-images, so large images (ISOs, disk images) live
on the NFS share rather than the host's own root disk.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 03:54:37 +10:00
beatzaplenty 16d6baea5f Merge remote-tracking branch 'origin/main' into worktree-debian-pxe 2026-07-27 03:50:56 +10:00
beatzaplenty d082c6a084 updated nfs shares on server build type 2026-07-27 03:40:19 +10:00
beatzaplenty ee93322ae2 add proxmox nfs shares 2026-07-27 03:28:46 +10:00
beatzaplenty d1ce8d3e71 add proxmox iso share 2026-07-27 03:26:31 +10:00
beatzaplentyandClaude Sonnet 4.6 f7c32aff12 feat(pxe-boot): add Debian bookworm minimal netboot menu entry
Adds a `fetch-debian-netboot.service` oneshot that downloads the Debian
bookworm netboot kernel and initrd from deb.debian.org on first boot,
stages them under /srv/pxe/http/debian/, and serves them via a generated
debian.ipxe chain script. The service is idempotent — it skips the
download if both files are already present.

Also merges the previously split systemd.tmpfiles.rules and
systemd.services blocks into a single systemd = { ... } attrset to
satisfy statix's repeated-keys lint.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 03:23:14 +10:00
beatzaplenty bf88a6ebb0 added tailscale-router beszel secret
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
2026-07-26 11:14:05 +10:00
beatzaplenty de508141a8 removed tailscale from docker updated secrets
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
2026-07-26 11:12:46 +10:00
beatzaplenty 18cd9c2342 Merge pull request 'feat(beszel): add beszel agent to tailscale-router' (#72) from worktree-beszel-tailscale-router into main
Check NixOS configurations / eval-hosts (push) Failing after 11m27s
Reviewed-on: #72
2026-07-26 01:08:29 +00:00
beatzaplentyandClaude Sonnet 4.6 997918e2f7 feat(beszel): add beszel agent to tailscale-router
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m33s
Wires beszel-agent into all tailscale-router variants (lxc/linode/proxmox)
by importing enable-agent.nix in the build type and host-token.nix in the
host file. Adds the sops creation rule for secrets/tailscale-router.yaml
(all three platform variants as recipients). The secrets file must be
created manually before deploying — see instructions in PR.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 11:07:51 +10:00
beatzaplenty 9872b8ff1d Merge pull request 'fix(tailscale-router): add MASQUERADE to POSTROUTING, not nixos-nat-post' (#71) from worktree-lovely-spinning-bubble into main
Check NixOS configurations / eval-hosts (push) Successful in 10m43s
Reviewed-on: #71
2026-07-26 00:01:48 +00:00
beatzaplentyandClaude Sonnet 4.6 e9b225d2d6 fix(tailscale-router): add MASQUERADE to POSTROUTING, not nixos-nat-post
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
extraCommands runs after nixos-nat-post is deleted but before it is
re-created, so -A nixos-nat-post silently fails every time.  POSTROUTING
is a built-in chain that always exists; target it directly instead.

The -C idempotency check prevents duplicate rules on firewall reloads.
Drop networking.nat.enable -- it was only needed for the sub-chain that
turned out to be the wrong target.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 10:00:47 +10:00
beatzaplenty 2860f750b4 Merge pull request 'fix(tailscale-router): actually insert MASQUERADE rule via extraCommands' (#70) from worktree-lovely-spinning-bubble into main
Check NixOS configurations / eval-hosts (push) Successful in 10m34s
Reviewed-on: #70
2026-07-25 23:56:11 +00:00
beatzaplentyandClaude Sonnet 4.6 781b1d324e fix(tailscale-router): actually insert MASQUERADE rule via extraCommands
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
networking.nat.externalInterface without internalInterfaces creates the
nixos-nat-post chain but inserts no MASQUERADE rule into it — confirmed
by inspecting the live firewall-start script on the deployed host.

Add the rule explicitly via firewall.extraCommands targeting nixos-nat-post,
scoped to LAN source traffic (vars.lanCidr) going out tailscale0.
extraStopCommands removes it on firewall stop.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 09:54:33 +10:00
beatzaplenty 3014a45936 Merge pull request 'fix(tailscale-router): masquerade LAN traffic into Tailscale' (#69) from worktree-lovely-spinning-bubble into main
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
Reviewed-on: #69
2026-07-25 23:48:13 +00:00
beatzaplentyandClaude Sonnet 4.6 cda2132d6a fix(tailscale-router): masquerade LAN traffic into Tailscale
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
Without SNAT on tailscale0, Tailscale drops forwarded packets from LAN
source IPs (192.168.2.x) because they are not recognised Tailscale
addresses.  With networking.nat.externalInterface = "tailscale0", all
traffic leaving through the Tailscale tunnel is masqueraded to the
router's own Tailscale IP (100.x.x.x), making it indistinguishable from
locally-originated traffic.  Conntrack handles the return path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 09:45:07 +10:00
beatzaplenty 6c1cc821a0 Merge pull request 'feat(tailscale): enable UDP GRO forwarding on subnet router uplink' (#68) from worktree-polished-wandering-toast into main
Check NixOS configurations / eval-hosts (push) Successful in 10m34s
Reviewed-on: #68
2026-07-25 23:14:33 +00:00
beatzaplentyandClaude Sonnet 4.6 4be064572d feat(tailscale): enable UDP GRO forwarding on subnet router uplink
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m42s
Adds a oneshot systemd service that sets ethtool rx-udp-gro-forwarding on
and rx-gro-list off on the default-route interface at boot, silencing
Tailscale's warning about suboptimal UDP GRO forwarding on subnet routers.
Interface is discovered dynamically via `ip route get` so it works on all
platforms regardless of NIC naming.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 09:13:28 +10:00
beatzaplenty 89d506180d Merge pull request 'fix(create-proxmox-resource): fix VM disk never attaching after import' (#67) from worktree-warm-discovering-moon into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #67
2026-07-25 22:57:21 +00:00
beatzaplentyandClaude Sonnet 4.6 6e1e992652 fix(proxmox): embed SSH host key via NIXOS_HOST_KEYS_DIR so sops can decrypt on first boot
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m30s
--pre-format-files placed the key on the QEMU builder VM's rootfs, not the
target disk. nixos-install chroots into the target and runs sshd-keygen, which
found no key in the chroot and generated a fresh (unregistered) one. sops then
could not decrypt on first boot because the key didn't match .sops.yaml, leaving
both root and nixos with '!' in /etc/shadow even after mutableUsers = false was
set (hashedPasswordFile pointed to paths sops never wrote).

Fix modules/platforms/proxmox.nix to embed the clan SSH host key in
environment.etc via NIXOS_HOST_KEYS_DIR at eval time -- the same pattern
lxc.nix uses. nixos-install's own activation places the key on the target disk,
sshd-keygen finds it already present and skips generation, and sops decrypts
correctly on first boot. Includes the same preserveSshHostKey/restoreSshHostKey
activation scripts as lxc.nix so subsequent nixos-rebuild switch calls (without
NIXOS_HOST_KEYS_DIR) don't remove the key as "obsolete" from environment.etc.

Update create-proxmox-resource.sh: switch VM builds from
  ./result-<target> --pre-format-files ... --build-memory 2048
to
  NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build --impure ... diskoImagesScript
  ./result-<target> --build-memory 2048
matching the LXC build path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
2026-07-26 08:54:39 +10:00
beatzaplentyandClaude Sonnet 4.6 5467c2e140 fix(create-proxmox-resource): fix VM disk never attaching after import
Three bugs combined to leave every VM build with a shell but no boot disk:

1. The remote build script moved the raw image to /var/lib/vz/import/ before
   qm importdisk could use it. If the mv failed (cross-filesystem copy, sudo
   path, or any other reason) the remote script exited non-zero -- but the
   local script's set -e handling of the SSH heredoc was inconsistent, so
   qm create sometimes ran anyway, leaving a diskless VM shell.

   Fix: skip the mv entirely. The diskoImagesScript writes <hostname>.raw into
   its CWD (the remote repo dir, $out = $PWD at invocation). Import directly
   from that path; clean it up after a successful import.

2. The qm importdisk output regex expected "Successfully imported disk as '...'"
   but current Proxmox emits "unusedN: successfully imported disk '...'"
   (lowercase, no "as"). The grep returned no match and exited 1.

3. The disk_id assignment used $(... | grep ...) without || true inside the
   substitution. With set -euo pipefail, a non-zero grep exit aborts the
   script before the fallback could run -- so the VM was always left with an
   unattached unused0 disk.

   Fix: update the primary regex to match the actual PVE format; add || true
   inside the substitution so set -e never fires on a grep miss; add a qm
   config fallback (scan for unusedN: lines) that works regardless of PVE
   output format changes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
2026-07-26 08:54:39 +10:00
beatzaplenty 123cd2b3d7 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Successful in 10m28s
2026-07-26 08:23:44 +10:00
beatzaplenty 006dd8097a updated vm HDD size 2026-07-26 08:23:30 +10:00
beatzaplenty 18cd6e884e Merge pull request 'fix(common): set mutableUsers = false to fix password setup on disk images' (#65) from worktree-warm-discovering-moon into main
Check NixOS configurations / eval-hosts (push) Successful in 10m36s
Reviewed-on: #65
2026-07-25 21:37:34 +00:00
beatzaplenty 3102d66337 Merge pull request 'fix(create-proxmox-resource): case-insensitive importdisk parse + warn on --disk-size for VMs' (#64) from worktree-gentle-cuddling-hippo into main
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Reviewed-on: #64
2026-07-25 21:37:14 +00:00
beatzaplentyandClaude Sonnet 4.6 852ba2240f fix(create-proxmox-resource): case-insensitive importdisk parse + warn on --disk-size for VMs
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m22s
qm importdisk in QEMU 11.x outputs lowercase "successfully imported disk
as '...'" rather than the capitalised form the original grep expected.
The case mismatch made disk_id always empty, which caused the script to
exit 1 after qm create had already run -- leaving the VM with only an
EFI disk, no scsi0, and boot order still set to net0.

Fix by adding -i (case-insensitive) to the grep. Both the old capitalised
format (where the disk id had an "unused0:" prefix inside the quotes) and
the new lowercase format are handled correctly: the sed strip of unused0:
is preserved for backward compatibility, and the regex result is identical
either way.

Also add an early warning when --disk-size is passed for --type vm: the
flag is LXC-only for create mode and was silently ignored, leaving users
expecting a different size than the proxmoxImageSize in variables.nix.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 06:40:54 +10:00
70 changed files with 5453 additions and 423 deletions
+130
View File
@@ -22,6 +22,8 @@ keys:
- &proxmox-pxe-boot age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz - &proxmox-pxe-boot age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
- &proxmox-server age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp - &proxmox-server age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
- &proxmox-tailscale-router age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0 - &proxmox-tailscale-router age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
- &proxmox-ha-server-1 age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
- &proxmox-ha-server-2 age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
creation_rules: creation_rules:
# Shared across every currently-deployed host: root/nixos password hash, # Shared across every currently-deployed host: root/nixos password hash,
@@ -54,6 +56,8 @@ creation_rules:
- *proxmox-pxe-boot - *proxmox-pxe-boot
- *proxmox-server - *proxmox-server
- *proxmox-tailscale-router - *proxmox-tailscale-router
- *proxmox-ha-server-1
- *proxmox-ha-server-2
- path_regex: secrets/nix-cache\.yaml$ - path_regex: secrets/nix-cache\.yaml$
key_groups: key_groups:
@@ -77,6 +81,44 @@ creation_rules:
- *admin - *admin
- *lxc-tor-relay - *lxc-tor-relay
- path_regex: secrets/tailscale-router\.yaml$
key_groups:
- age:
- *admin
- *linode-tailscale-router
- *lxc-tailscale-router
- *proxmox-tailscale-router
# HA file server per-node secrets (beszel-token).
# proxmox-ha-server-1 / proxmox-ha-server-2 keys are added automatically
# by scripts/secrets/sync-host-keys.sh once the hosts are provisioned;
# until then only the admin key can decrypt these files.
- path_regex: secrets/ha-server-1\.yaml$
key_groups:
- age:
- *admin
- *proxmox-ha-server-1
# proxmox-ha-server-1 added by sync-host-keys.sh
- path_regex: secrets/ha-server-2\.yaml$
key_groups:
- age:
- *admin
- *proxmox-ha-server-2
# proxmox-ha-server-2 added by sync-host-keys.sh
# Shared HA cluster corosync authkey (binary sops file).
# Encrypted for both HA nodes so either can decrypt on boot.
# Both host keys added by sync-host-keys.sh; admin key allows initial creation.
- path_regex: secrets/ha-corosync-authkey$
key_groups:
- age:
- *admin
- *proxmox-ha-server-1
- *proxmox-ha-server-2
# proxmox-ha-server-1 added by sync-host-keys.sh
# proxmox-ha-server-2 added by sync-host-keys.sh
# gui-host-specific secrets (currently: wifi-password, see # gui-host-specific secrets (currently: wifi-password, see
# modules/networking/wifi.nix). Only *lxc-gui has a registered key today # modules/networking/wifi.nix). Only *lxc-gui has a registered key today
# -- proxmox-gui/linode-gui/baremetal-gui haven't been provisioned via # -- proxmox-gui/linode-gui/baremetal-gui haven't been provisioned via
@@ -91,3 +133,91 @@ creation_rules:
- *baremetal-gui - *baremetal-gui
- *linode-gui - *linode-gui
- *proxmox-gui - *proxmox-gui
# IPA host keytabs (binary sops files).
# Each keytab is encrypted for all platform variants of that host so any
# deployed variant can decrypt it at boot. Run
# scripts/ipa/create-nixos-ipa-host-account.sh <hostname> to enroll a new
# host and produce the keytab; this section is updated by that script.
- path_regex: secrets/nix-cache\.keytab$
key_groups:
- age:
- *admin
- *linode-nix-cache
- *lxc-nix-cache
- *proxmox-nix-cache
- path_regex: secrets/tailscale-router\.keytab$
key_groups:
- age:
- *admin
- *linode-tailscale-router
- *lxc-tailscale-router
- *proxmox-tailscale-router
- path_regex: secrets/pxe-boot\.keytab$
key_groups:
- age:
- *admin
- *lxc-pxe-boot
- *proxmox-pxe-boot
# nixos = the workstation (hosts/nixos/host.nix). All gui platform variants
# share the hostname "nixos" and must be able to decrypt at boot.
- path_regex: secrets/nixos\.keytab$
key_groups:
- age:
- *admin
- *baremetal-gui
- *lxc-gui
- *proxmox-gui
- *linode-gui
- path_regex: secrets/server\.keytab$
key_groups:
- age:
- *admin
- *linode-server
- *lxc-server
- *proxmox-server
- path_regex: secrets/docker\.keytab$
key_groups:
- age:
- *admin
- *linode-docker
- *lxc-docker
- *proxmox-docker
- path_regex: secrets/tor-relay\.keytab$
key_groups:
- age:
- *admin
- *lxc-tor-relay
- path_regex: secrets/nix-minimal\.keytab$
key_groups:
- age:
- *admin
- *lxc-minimal
- *proxmox-minimal
- *linode-minimal
# Host keytab for ha-server-1 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/ha-server-1\.keytab$
key_groups:
- age:
- *admin
- *proxmox-ha-server-1
# proxmox-ha-server-1 added by sync-host-keys.sh
# Host keytab for ha-server-2 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/ha-server-2\.keytab$
key_groups:
- age:
- *admin
- *proxmox-ha-server-2
# proxmox-ha-server-2 added by sync-host-keys.sh
+8 -6
View File
@@ -6,12 +6,14 @@ This repository contains flake-based NixOS configurations for Wayne's LAN
servers and workstation. servers and workstation.
The flake exposes NixOS configurations named `<platform>-<buildtype>` The flake exposes NixOS configurations named `<platform>-<buildtype>`
(platforms: `linode`, `proxmox`, `lxc`; build types: `minimal`, `nix-cache`, (platforms: `linode`, `proxmox`, `lxc`, `baremetal`; build types: `minimal`,
`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`, `tor-relay`), generated from `modules/platforms/*` `nix-cache`, `server`, `docker`, `gui`, `pxe-boot`, `tailscale-router`,
and `modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not `tor-relay`, `ha-server`), generated from `modules/platforms/*` and
every combination is built — `pxe-boot` has no `linode` variant. See `modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not every
`README.md` for the full current target list; treat `flake.nix` as the combination is built — `pxe-boot` has no `linode` variant, `ha-server` only
source of truth since this list can drift. exists on `proxmox`, and `tor-relay` only exists on `lxc`. See `README.md`
for the full current target list; treat `flake.nix` as the source of truth
since this list can drift.
Do not deploy, switch, reboot, repartition, format disks, or run destructive Do not deploy, switch, reboot, repartition, format disks, or run destructive
install commands from this repository unless explicitly asked. install commands from this repository unless explicitly asked.
+10 -11
View File
@@ -10,13 +10,13 @@ pieces composed in `flake.nix`:
- **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`, `baremetal` - **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`, `baremetal`
- **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`, - **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
`gui`, `pxe-boot`, `tailscale-exit-node`, `tor-relay` `gui`, `pxe-boot`, `tailscale-router`, `tor-relay`, `ha-server`
Not every combination exists — `pxe-boot` has no `linode` variant, since Not every combination exists — `pxe-boot` has no `linode` variant, since
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have, PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have,
`tor-relay` currently only exists as `lxc-tor-relay`, and `baremetal` `tor-relay` and `ha-server` currently only exist on `lxc`/`proxmox`, and
currently only exists as `baremetal-gui` (the real gui-host hardware). The `baremetal` currently only exists as `baremetal-gui` (the real gui-host
full list: hardware). The full list:
| Target | Purpose | | Target | Purpose |
| --- | --- | | --- | --- |
@@ -29,8 +29,9 @@ full list:
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target | | `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target |
| `baremetal-gui` | Same Cinnamon desktop workstation, on the real gui-host hardware — ZFS RAID0 root, systemd-boot | | `baremetal-gui` | Same Cinnamon desktop workstation, on the real gui-host hardware — ZFS RAID0 root, systemd-boot |
| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host — previously the flat `pxe-boot` target | | `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host — previously the flat `pxe-boot` target |
| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node | | `linode-tailscale-router` / `proxmox-tailscale-router` / `lxc-tailscale-router` | Tailscale subnet router + MagicDNS forwarder for the LAN |
| `lxc-tor-relay` | Tor middle relay | | `lxc-tor-relay` | Tor middle relay |
| `proxmox-ha-server-1` / `proxmox-ha-server-2` | HA file-server cluster nodes — DRBD + XFS + iSCSI + NFS, managed by Corosync + Pacemaker |
Which variant of a given buildtype is actually deployed isn't tracked Which variant of a given buildtype is actually deployed isn't tracked
anywhere in this repo — that's live infrastructure state, not something a anywhere in this repo — that's live infrastructure state, not something a
@@ -47,8 +48,7 @@ section for which is which.
Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
(hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay (hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay
fixed regardless of which platform it's built for — see fixed regardless of which platform it's built for. Every deployed host
`flake-target-refactor-spec.md` for the full rationale. Every deployed host
stamps its own active target name into `/etc/flake-target` at build time, so stamps its own active target name into `/etc/flake-target` at build time, so
`nixos-rebuild switch --flake .#$(cat /etc/flake-target)` always picks up the `nixos-rebuild switch --flake .#$(cat /etc/flake-target)` always picks up the
right one even after a platform migration changes the flake attribute name. right one even after a platform migration changes the flake attribute name.
@@ -167,7 +167,6 @@ per-boot host key for sops-nix to derive from on ephemeral media) — see
(`vars/per-machine/<target>/openssh/`, committed and sops-encrypted) for (`vars/per-machine/<target>/openssh/`, committed and sops-encrypted) for
their SSH host keys. their SSH host keys.
This repository's git *history* still contains secrets committed before this This repository's git *history* still contains secrets committed before the
migration (see `remove-sensetive-info-refactor.md`) — those are being sops-nix migration — those are being scrubbed and rotated separately; don't
scrubbed and rotated separately; don't treat the repo as safe to make public treat the repo as safe to make public until that's finished.
until that's finished.
+25
View File
@@ -0,0 +1,25 @@
-----BEGIN CERTIFICATE-----
MIIESDCCArCgAwIBAgIBATANBgkqhkiG9w0BAQsFADA1MRMwEQYDVQQKDApTV0VF
VC5IT01FMR4wHAYDVQQDDBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwHhcNMjYwNzI2
MjExMzQxWhcNNDYwNzI2MjExMzQxWjA1MRMwEQYDVQQKDApTV0VFVC5IT01FMR4w
HAYDVQQDDBVDZXJ0aWZpY2F0ZSBBdXRob3JpdHkwggGiMA0GCSqGSIb3DQEBAQUA
A4IBjwAwggGKAoIBgQCzljYktbHdMGVJ6Wq0XQJuHLN6dkCSOgtoIzQtriPQkkNI
uo28LwobaiQQ8sX4kGRH/BTKnH8QlId/jug4Uc+sDHnABYu++AiOhPbBX8gCpRQ0
hebBjZiktHSBUEJR31siWOVdBoKBDJEoxehx7XUXvcxIJcaRN+LHYjO86nJN55HB
VwFU2JcYDk98c+144dFJxXdr++MjWe4Z/oVVU8JHIOtNtKhVhvij6oOSWxcYoJO/
S80LRj1vx/o6o/3G6bYug7PjY7JjZk/Oj61whijZkcsoO1MXSYI6UywJZGflv+ZB
7HyufdYAsK3WhE8O2FX3/kq64Ol83HNtoR8Dt68rTg1xpW6K45jS6iDPKueYGkb0
oSx7e++90VAW2PDhj6QQ3JJ4O5VQwrrecekJzUrAean0FOEbmgyi4PsEp1Vk6LDQ
SsIn1x0euyxVivQMlzNX2XrZL3urn1BNPqAdntXQMkR0Wl8sbUiJPe0kxG52CGXs
6yfNEXbPmVGcC0TBdGECAwEAAaNjMGEwHQYDVR0OBBYEFLh5QbI1UWMH0WR4z8bG
lhrOX3X5MB8GA1UdIwQYMBaAFLh5QbI1UWMH0WR4z8bGlhrOX3X5MA8GA1UdEwEB
/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgHGMA0GCSqGSIb3DQEBCwUAA4IBgQCVodVN
owwo53OQe02QhtEbIur2PL7zIfvhvCTRD4J8gwpbMIqT7JQK0tV6Mvsg2L8yTb2O
KjrWeLKHGWaZZlhGSPTbkMFdb/Ls8M9FSnkc2bwcdWW3Z1lOiCjBYYqwLCG6JhvB
5SXVwWNJwXeasL2m7oFTSwhsqPpARJ2t25u2N35o+tqIoCjijKwkmEOT66N9EAbu
2VQjtYZWPkBtP4YCe0Ey6u4oy7sy8ThNAjOylZok+J4JW7QEFjK4Q/emhA4aQq5H
gg9qgMuG+5oi6D1g2Wy+fMTRBaukJtLYZbBpQMQhMYWg44uPp/2bbNPTID/nV1KB
GcPyHaskcVxPdYWxAPMwk3AeJXWyOq7atAPTF5sbk0kQQf2m+vyOqcli5CxRMUgV
rcyi9l6+dZW4U+38Q0ET5M3OuxNI4hA7kVY2cfTakXWNqh97+TIHnstblDhAxECK
6ZLMJQYUy7LqJTX84H27CBWLexEMjXwdr5HCV88Fj6mAK0fRufnIw5FeneA=
-----END CERTIFICATE-----
+5 -3
View File
@@ -22,7 +22,7 @@ see "LXC hosts" immediately below for why those are different.**
## LXC hosts ## LXC hosts
`lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`, `lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`,
`lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-exit-node`, `lxc-tor-relay`) are **not** installed via `auto-install.sh` — the `lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-router`, `lxc-tor-relay`) are **not** installed via `auto-install.sh` — the
interactive menu deliberately excludes them. Don't try to select one there; interactive menu deliberately excludes them. Don't try to select one there;
`nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw `nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw
disk to partition) and then refuse to touch the filesystem it's currently disk to partition) and then refuse to touch the filesystem it's currently
@@ -133,13 +133,15 @@ Flake outputs:
```nix ```nix
nixosConfigurations.installer # ISO/netboot installer image nixosConfigurations.installer # ISO/netboot installer image
packages.x86_64-linux.iso # installer ISO/netboot image packages.x86_64-linux.iso # installer ISO/netboot image
packages.x86_64-linux.pxe # netboot-ipxe + netboot-initrd + netboot-kernel, bundled packages.x86_64-linux.pxe # auto-installer netboot bundle (kernel + initrd + ipxe script)
packages.x86_64-linux.pxe-minimal # vanilla NixOS minimal netboot bundle (no installer wiring)
``` ```
```sh ```sh
nix build .#iso nix build .#iso
nix build .#pxe nix build .#pxe
nix build .#pxe-minimal
``` ```
There's no `nixosConfigurations.proxmox-lxc` (installer-boots-as-an-LXC- There's no `nixosConfigurations.proxmox-lxc` (installer-boots-as-an-LXC-
+128
View File
@@ -0,0 +1,128 @@
# IP Addressing Scheme
## Subnets
| Subnet | CIDR | Purpose | Routed? |
|---|---|---|---|
| LAN | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
| Storage | `192.168.4.0/29` | HA file server DRBD replication | No — internal `vmbr1` only, no uplink |
The storage subnet never leaves pve1. `vmbr1` is a Proxmox Linux bridge with no physical port
attached; traffic between the two HA file server VMs stays in-kernel.
The host octet is consistent across subnets for any host that has multiple interfaces — e.g.
ha-node1 is always `.228` (LAN: `192.168.2.228`, storage: `192.168.4.228`).
---
## LAN — 192.168.2.0/24
### Address map
| Range | Purpose |
|---|---|
| .1.9 | Reserved, never assign |
| .10.59 | Client DHCP pool (router-assigned) |
| .60.219 | Unallocated buffer |
| .220.229 | Virtual nodes (VMs / LXC containers) |
| .230.239 | Expansion buffer (reserved, unallocated) |
| .240.249 | Physical nodes (bare-metal hosts) |
| .250.253 | Network services |
| .254 | Router / gateway |
### Network services (.250.253)
| IP | Hostname | Role |
|---|---|---|
| `192.168.2.254` | router | Gateway (TP-Link) |
| `192.168.2.253` | domain-controller | FreeIPA — authoritative DNS for `sweet.home`, Kerberos, LDAP |
| `192.168.2.250``.252` | — | Reserved for future network services |
### Physical nodes (.240.249)
| IP | Hostname | Role |
|---|---|---|
| `192.168.2.245` | pve1 | Proxmox VE hypervisor |
| `192.168.2.244` | pbs | Proxmox Backup Server |
| `192.168.2.243` | nixos | Bare-metal workstation (`baremetal-gui`) |
| `192.168.2.246``.249` | — | Reserved — second Proxmox node and associated services |
| `192.168.2.240``.242` | — | Reserved |
pve1 sits mid-range deliberately so a second Proxmox node can slot in on either side.
### Virtual nodes (.220.229)
All VMs and LXC containers run on pve1.
| IP | Hostname | Role | Status |
|---|---|---|---|
| `192.168.2.229` | ha-vip | HA file server iSCSI floating VIP (Pacemaker) | Future |
| `192.168.2.228` | ha-node1 | HA file server node 1 (DRBD + XFS + iSCSI) | Future |
| `192.168.2.227` | ha-node2 | HA file server node 2 (DRBD + XFS + iSCSI) | Future |
| `192.168.2.226` | server | Current NFS/ZFS file server — retires when HA is live | Retiring |
| `192.168.2.225` | docker | Docker / Traefik stack | Active |
| `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
| `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active |
| `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active |
| `192.168.2.221` | tor-relay | Tor relay | Active |
| `192.168.2.220` | pdm | Proxmox Deploy Manager | Active |
### Client DHCP pool (.10.59)
Assigned by the router. DNS option points to `192.168.2.253` (domain-controller).
Devices in this range: phones, laptops, IoT, Canon printer, any non-infrastructure host.
No static reservations for infrastructure hosts — all infra uses static IP configuration
on the guest itself (not DHCP reservations), so IPs survive VM recreation regardless of
MAC address churn.
---
## Storage network — 192.168.4.0/29
Internal to pve1 only. Proxmox bridge `vmbr1`, no physical NIC attached.
| IP | Hostname | Interface role |
|---|---|---|
| `192.168.4.228` | ha-node1 | DRBD replication NIC |
| `192.168.4.227` | ha-node2 | DRBD replication NIC |
| — | no gateway | Isolated — not routed to LAN or internet |
---
## Migration reference
Current → target IP for every host being renumbered.
| Host | Current IP | New IP | Config location |
|---|---|---|---|
| router | `192.168.2.254` | `192.168.2.254` | unchanged |
| domain-controller | `192.168.2.138` | `192.168.2.253` | `/etc/sysconfig/network-scripts/ifcfg-eth0` on guest |
| pve1 | `192.168.2.250` | `192.168.2.245` | `/etc/network/interfaces` on Proxmox host |
| pbs | `192.168.2.108` | `192.168.2.244` | static config on PBS host |
| nixos workstation | `192.168.2.119` | `192.168.2.243` | `networking.interfaces` / NetworkManager on guest |
| ha-node1 | — | `192.168.2.228` | future |
| ha-node2 | — | `192.168.2.227` | future |
| ha-vip | — | `192.168.2.229` | future (Pacemaker resource) |
| server | `192.168.2.252` | `192.168.2.226` | static config on guest |
| docker | `192.168.2.249` | `192.168.2.225` | static config on guest |
| nix-cache | `192.168.2.120` | `192.168.2.224` | static config on guest |
| pxe-boot | `192.168.2.247` | `192.168.2.223` | static config on guest; update `vars.pxeServerIp` in `variables.nix` ✓ |
| tailscale-router | `192.168.2.121` | `192.168.2.222` | static config on guest |
| tor-relay | `192.168.2.107` | `192.168.2.221` | static config on guest |
| pdm | `192.168.2.248` | `192.168.2.220` | static config on guest |
### Cutover notes
- **Do domain-controller first** — it becomes the DNS server; everything else depends on it
having its new IP and FreeIPA DNS configured before Pi-hole is retired.
- **pve1 last among physical hosts** — changing the Proxmox management IP drops the web UI
briefly; all guests keep running.
- **Update Pi-hole custom.list / FreeIPA DNS A records** to new IPs before flipping any host,
so name resolution stays valid throughout the migration.
- **variables.nix already updated** for `pxeServerIp` (.247→.223), `pbsIp` (.108→.244), and
new `domainControllerIp` (.253). Rebuild affected hosts after renumbering.
- **Router DHCP**: once domain-controller is at .253 and FreeIPA DNS is serving `sweet.home`,
switch router DHCP on with pool .10.59 and DNS option pointing to .253; retire Pi-hole CT.
- **Pi-hole's iPXE dnsmasq config** (`99-ipxe-chainload.conf`) moves to the pxe-boot CT as a
dnsmasq proxy-mode config before Pi-hole is decommissioned.
+366
View File
@@ -0,0 +1,366 @@
# Network Cutover Plan
Moves the LAN from the current flat/Pi-hole-managed state to the new IP scheme
defined in `docs/ip-addressing.md`. Works in five independent stages — each
stage is safe to pause after and resume later. Rollback steps are given at
every point where something can break.
**Before starting anything:** confirm you have
- SSH access to `192.168.2.138` (domain-controller, current IP)
- SSH access to `192.168.2.250` (pve1)
- Browser access to Pi-hole admin at `http://192.168.2.253`
- Browser access to router admin at `http://192.168.2.254`
- The FreeIPA `admin` password to hand
---
## Stage 1 — Prepare FreeIPA DNS (zero downtime)
Everything here is additive. Pi-hole keeps running. Nothing breaks if you stop
mid-stage.
### 1a. Add NextDNS forwarders
```bash
ssh wayne@192.168.2.138
kinit admin # enter FreeIPA admin password when prompted
ipa dnsconfig-mod \
--forwarder=45.90.28.142 \
--forwarder=45.90.30.142 \
--forward-policy=only
```
**Verify external resolution works through FreeIPA before continuing:**
```bash
dig @127.0.0.1 google.com +short # must return an IP, not SERVFAIL
```
### 1b. Add A records for every host at their CURRENT IPs
These represent the live state now. You'll update each record to the new IP
when you renumber that host in Stage 5.
```bash
ipa dnsrecord-add sweet.home pve1 --a-rec 192.168.2.250
ipa dnsrecord-add sweet.home pbs --a-rec 192.168.2.108
ipa dnsrecord-add sweet.home nixos --a-rec 192.168.2.119
ipa dnsrecord-add sweet.home server --a-rec 192.168.2.252
ipa dnsrecord-add sweet.home docker --a-rec 192.168.2.249
ipa dnsrecord-add sweet.home nix-cache --a-rec 192.168.2.120
ipa dnsrecord-add sweet.home pxe-boot --a-rec 192.168.2.247
ipa dnsrecord-add sweet.home tailscale-router --a-rec 192.168.2.121
ipa dnsrecord-add sweet.home tor-relay --a-rec 192.168.2.107
ipa dnsrecord-add sweet.home pdm --a-rec 192.168.2.248
ipa dnsrecord-add sweet.home router --a-rec 192.168.2.254
```
### 1c. Clean up stale reverse-zone PTR records
FreeIPA already has PTR records from an earlier import but some are wrong.
Fix them now so reverse DNS is accurate from day one.
```bash
# Remove stale "win11" entry at .250 (should be pve1)
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec win11.
ipa dnsrecord-add 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
# Fix unqualified PTR records (missing .sweet.home. suffix)
ipa dnsrecord-mod 2.168.192.in-addr.arpa 108 --ptr-rec pbs.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 248 --ptr-rec pdm.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 249 --ptr-rec docker.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 252 --ptr-rec server.sweet.home.
# Add any missing PTR records
ipa dnsrecord-add 2.168.192.in-addr.arpa 119 --ptr-rec nixos.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 120 --ptr-rec nix-cache.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 121 --ptr-rec tailscale-router.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 247 --ptr-rec pxe-boot.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 254 --ptr-rec router.sweet.home.
```
### 1d. Point domain-controller's own DNS at itself
```bash
sudo nmcli connection modify "System eth0" ipv4.dns "127.0.0.1"
sudo nmcli connection up "System eth0"
```
**Verify:**
```bash
dig pve1.sweet.home +short # must return 192.168.2.250
dig google.com +short # must return an IP (NextDNS forwarding)
```
**Rollback 1d:** `sudo nmcli connection modify "System eth0" ipv4.dns "192.168.2.253" && sudo nmcli connection up "System eth0"`
---
## Stage 2 — Move pxe-boot DHCP options off Pi-hole (zero downtime)
Pi-hole's dnsmasq currently serves the iPXE boot options via
`99-ipxe-chainload.conf`. Before Pi-hole is retired, that config must move to
the pxe-boot CT running dnsmasq in proxy mode so PXE boot keeps working.
### 2a. Add dnsmasq proxy config to the pxe-boot NixOS module
In `modules/build-types/pxe-boot.nix`, add:
```nix
services.dnsmasq = {
enable = true;
settings = {
# Proxy mode: respond only to PXE DHCP requests, leave normal leases to router
dhcp-range = [ "192.168.2.0,proxy" ];
# iPXE client detection
dhcp-match = [
"set:ipxe,175"
"set:efi64,option:client-arch,7"
"set:efi64,option:client-arch,9"
];
dhcp-userclass = "set:ipxe,iPXE";
# Boot file selection
dhcp-boot = [
"tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
"tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
"tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
"undionly.kpxe,,${vars.pxeServerIp}"
];
};
};
```
### 2b. Rebuild and deploy the pxe-boot CT
```bash
# On pve1 — build the new tarball
nix build .#lxc-pxe-boot.config.system.build.tarball
# Verify dnsmasq starts correctly in the CT after deploy
ssh nixos@192.168.2.247 systemctl status dnsmasq
```
### 2c. Remove the iPXE config from Pi-hole
In the Pi-hole CT, remove `/etc/dnsmasq.d/99-ipxe-chainload.conf` and
restart the FTL service:
```bash
ssh wayne@pve1.sweet.home \
"sudo pct exec 100 -- bash -c 'rm /etc/dnsmasq.d/99-ipxe-chainload.conf && systemctl restart pihole-FTL'"
```
**Verify:** PXE boot a test machine — it should still get an iPXE response and
reach the boot menu.
**Rollback 2c:** restore the file from the Pi-hole config backup at
`/etc/pihole/config_backups/` and restart pihole-FTL.
---
## Stage 3 — DHCP migration: Pi-hole → router (brief maintenance window)
**Do this in the evening.** Existing DHCP leases stay valid during the
switchover so connected devices don't drop — only new lease requests fail
during the gap, which is under 60 seconds if you follow the steps in order.
The key: configure the router's DHCP DNS option to point at `.253` (Pi-hole's
current IP). This way, all new leases issued by the router still get the same
DNS server address — clients never need to change their DNS config. When Pi-hole
is retired and the DC takes `.253` in Stage 4, `.253` just starts answering
differently. No client reconfiguration.
### 3a. Pre-configure router DHCP (do not enable yet)
Log into `http://192.168.2.254`, find the DHCP settings and fill in — but
leave DHCP **disabled** until step 3b:
| Setting | Value |
|---|---|
| Start IP | 192.168.2.10 |
| End IP | 192.168.2.59 |
| Subnet mask | 255.255.255.0 |
| Gateway | 192.168.2.254 |
| Primary DNS | 192.168.2.253 |
| Secondary DNS | *(leave blank)* |
| Lease time | 24h |
Save without enabling.
### 3b. Switchover (do steps in quick succession)
1. **Disable Pi-hole DHCP:** Pi-hole admin UI → Settings → DHCP → uncheck
"DHCP server enabled" → Save
2. **Enable router DHCP** immediately after step 1
### 3c. Verify router DHCP is working
On a phone or laptop, disconnect from WiFi and reconnect (or run
`sudo dhclient -r && sudo dhclient` on a Linux host):
```bash
ip addr show # IP should be in 192.168.2.1059 range
dig google.com # should resolve (Pi-hole DNS still running at .253)
dig pve1.sweet.home # should resolve via FreeIPA at .138 (relayed via Pi-hole)
```
Wait 1015 minutes for the most active devices to renew their leases. There's
no need to wait for all leases to expire before proceeding.
**Rollback 3b:** Re-enable Pi-hole DHCP. Disable router DHCP. Done — existing
leases remain valid so most devices are unaffected.
---
## Stage 4 — Move domain-controller from .138 to .253
Pi-hole lives at `.253`. The DC must take `.253` the moment Pi-hole stops so
clients that still have `.253` as their DNS server don't notice the change.
Script these commands in advance and run them in rapid succession.
**Pre-stage: have this SSH command ready before running step 4a:**
```bash
ssh wayne@192.168.2.138 "
sudo nmcli connection modify 'System eth0' \
ipv4.addresses '192.168.2.253/24' \
ipv4.gateway '192.168.2.254' \
ipv4.dns '127.0.0.1' \
ipv4.method manual && \
sudo nmcli connection up 'System eth0'
"
```
**Also update the Proxmox VM config to match (run from pve1):**
```bash
sudo qm set 108 \
--ipconfig0 ip=192.168.2.253/24,gw=192.168.2.254 \
--nameserver 192.168.2.253
```
### 4a. Stop Pi-hole
```bash
ssh wayne@pve1.sweet.home "sudo pct stop 100"
```
### 4b. Immediately: change DC's IP to .253
Run the pre-staged SSH command from above. You have ~30 seconds before any
client notices Pi-hole is gone. If SSH to `.138` refuses (the IP is already
changing), open a Proxmox console to VM 108 and run the `nmcli` commands
there.
### 4c. Update Proxmox VM config
Run the pre-staged `qm set 108` command from above.
### 4d. Verify
```bash
ssh wayne@192.168.2.253 # must connect (new DC IP)
dig @192.168.2.253 pve1.sweet.home +short # must return 192.168.2.250
dig @192.168.2.253 google.com +short # must return an IP
```
From a client device that renewed its DHCP lease in Stage 3:
```bash
cat /etc/resolv.conf # should show 192.168.2.253
dig pve1.sweet.home # should resolve
```
**Rollback 4:** `ssh wayne@pve1.sweet.home "sudo pct start 100"`. Change DC IP
back to .138 via Proxmox console. This restores full Pi-hole DNS/DHCP service.
Leave Pi-hole CT stopped-but-intact for 48 hours before deleting it.
---
## Stage 5 — Host renumbering (one at a time, any order)
For each host:
1. Update FreeIPA DNS A record and PTR record to the new IP
2. Change the static IP on the host itself
3. Verify SSH to new IP
4. Update `variables.nix` if that host has an IP variable (pxe-boot, pbs — already done in this PR)
**FreeIPA record update template** (run as admin on domain-controller):
```bash
ipa dnsrecord-mod sweet.home <hostname> --a-rec <new-ip>
ipa dnsrecord-del 2.168.192.in-addr.arpa <old-last-octet> --ptr-rec <hostname>.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa <new-last-octet> --ptr-rec <hostname>.sweet.home.
```
### Renumbering order
| # | Host | Old IP | New IP | How to change IP |
|---|---|---|---|---|
| 1 | nixos workstation | .119 | .243 | NetworkManager on guest; or `nmcli connection modify` |
| 2 | nix-cache | .120 | .224 | `pct set 102 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.224/24,gw=192.168.2.254` then `pct reboot 102` |
| 3 | tailscale-router | .121 | .222 | Static config on guest; check Tailscale ACLs if IP is referenced there |
| 4 | tor-relay | .107 | .221 | `pct set 104 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.221/24,gw=192.168.2.254` then `pct reboot 104` |
| 5 | pdm | .248 | .220 | `pct set 106 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.220/24,gw=192.168.2.254` then `pct reboot 106` |
| 6 | pxe-boot | .247 | .223 | `pct set 103 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.223/24,gw=192.168.2.254` then rebuild NixOS (already updated in variables.nix) |
| 7 | server | .252 | .226 | Static config on guest; NFS clients (docker) lose mounts briefly — they remount automatically |
| 8 | docker | .249 | .225 | Static config on guest; do this after server is at .226 |
| 9 | pbs | .108 | .244 | Static config on PBS host itself; update in `pbsIp` already done in variables.nix |
| 10 | pve1 | .250 | .245 | Edit `/etc/network/interfaces` on the Proxmox host — see below |
### pve1 renumber (step 10 — do last)
All guests keep running; only the Proxmox web UI is briefly unreachable.
```bash
ssh wayne@pve1.sweet.home
# Edit /etc/network/interfaces: change address from .250 to .245
sudo nano /etc/network/interfaces
# Change: address 192.168.2.250/24
# To: address 192.168.2.245/24
sudo systemctl restart networking
# SSH will drop here — reconnect to new IP
```
```bash
ssh wayne@192.168.2.245 # verify
```
Update FreeIPA DNS:
```bash
ipa dnsrecord-mod sweet.home pve1 --a-rec 192.168.2.245
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 245 --ptr-rec pve1.sweet.home.
```
**Rollback any step 5 host:** change the IP back on the guest and update the
FreeIPA record back to the old IP. The old IP is unoccupied so you can
temporarily use either.
---
## Stage 6 — Final cleanup
Once all hosts are at their new IPs and verified:
```bash
# Delete the Pi-hole CT (already stopped since Stage 4)
ssh wayne@pve1.sweet.home "sudo pct destroy 100"
# Remove stale FreeIPA records for retired addresses
ipa dnsrecord-del sweet.home pihole --del-all
ipa dnsrecord-del 2.168.192.in-addr.arpa 253 --ptr-rec pihole.sweet.home.
# Rebuild any NixOS hosts that reference pbsIp or pxeServerIp to pick up
# the updated variables.nix values (pxe-boot mandatory; others as convenient)
```
---
## Rollback summary
| What broke | How to roll back |
|---|---|
| FreeIPA DNS not resolving | Check `systemctl status named` on DC; restart if failed |
| FreeIPA DNS unreachable | `pct start 100` on pve1 (restores Pi-hole) |
| Router DHCP not handing out leases | Re-enable Pi-hole DHCP; disable router DHCP |
| DC unreachable after IP change | Proxmox console on VM 108 → `nmcli connection up "System eth0"` with old IP |
| Host unreachable after renumber | Proxmox console → revert IP; or `pct set <id> --net0 ...` old IP and reboot CT |
| pve1 web UI gone after renumber | SSH to .245 and check `/etc/network/interfaces`; if wrong, fix and restart networking |
+97 -7
View File
@@ -15,6 +15,7 @@ rescue/inspection use.
- TFTP root for first-stage bootloaders: `/srv/pxe/tftp` - TFTP root for first-stage bootloaders: `/srv/pxe/tftp`
- iPXE entry script: `/srv/pxe/http/boot.ipxe` - iPXE entry script: `/srv/pxe/http/boot.ipxe`
- Generated iPXE menu: `/srv/pxe/http/menu.ipxe` - Generated iPXE menu: `/srv/pxe/http/menu.ipxe`
- Debian Minimal iPXE script: `/srv/pxe/http/debian.ipxe`
- SystemRescue iPXE script: `/srv/pxe/http/systemrescue.ipxe` - SystemRescue iPXE script: `/srv/pxe/http/systemrescue.ipxe`
- TFTP fallback script: `/srv/pxe/tftp/autoexec.ipxe` - TFTP fallback script: `/srv/pxe/tftp/autoexec.ipxe`
- Boot binaries copied from the Nix `ipxe` package: - Boot binaries copied from the Nix `ipxe` package:
@@ -28,32 +29,51 @@ The host creates these directories with systemd tmpfiles:
```text ```text
/srv/pxe /srv/pxe
/srv/pxe/http /srv/pxe/http
/srv/pxe/http/images /srv/pxe/http/images -> /mnt/pxe-images (symlink to NFS share)
/srv/pxe/http/auto-installer /srv/pxe/http/auto-installer
/srv/pxe/http/nixos-minimal /srv/pxe/http/nixos-minimal
/srv/pxe/http/debian
/srv/pxe/http/systemrescue /srv/pxe/http/systemrescue
/srv/pxe/http/ubuntu /srv/pxe/http/ubuntu
/srv/pxe/http/rescue /srv/pxe/http/rescue
/srv/pxe/tftp /srv/pxe/tftp
``` ```
Mount shared image storage under `/srv/pxe/http`, preferably `/srv/pxe/http/images` is a symlink to `/mnt/pxe-images`, which is an NFS
`/srv/pxe/http/images` unless a menu entry expects files in a specific mount of `server.sweet.home:/tank/pxe-boot/images`
directory such as `/srv/pxe/http/auto-installer`. (`modules/pxe-boot/mount-pxe-images.nix`). Place large images there (ISOs,
disk images) rather than on the pxe-boot host's own root disk. For an LXC
pxe-boot container the mount uses NFSv3+nolock with `nofail` (eager,
non-blocking on server unavailability); for a Proxmox VM it uses NFSv4.2
with `x-systemd.automount` (lazy, triggered on first access).
When running as `lxc-pxe-boot`, the Proxmox container must have
`features: nesting=1,mount=nfs` (at minimum) in its Proxmox config. `nesting=1`
is required by systemd 260+ for credential isolation (user namespace creation
and internal move-mounts); without it, AppArmor denies both, and every
systemd service that uses `PrivateUsers`, `PrivateDevices`, or credential
passing fails on boot. `mount=nfs` allows the NFSv3 mount. Both are set
automatically by `scripts/proxmox/create-proxmox-resource.sh` (via
`PROXMOX_DEFAULT_LXC_FEATURES` in `scripts/env.sh` which defaults to
`nesting=1,keyctl=1,mount=nfs;nfs4`). If you ever change these features
manually via `pct set`, be sure to include both — `pct set` replaces the
entire features string, it does not append to it.
The HTTP iPXE chain is: The HTTP iPXE chain is:
```text ```text
undionly.kpxe or ipxe.efi undionly.kpxe or ipxe.efi
-> autoexec.ipxe from the TFTP root, when iPXE requests it -> autoexec.ipxe from the TFTP root, when iPXE requests it
-> http://192.168.2.247/boot.ipxe -> http://192.168.2.223/boot.ipxe
-> http://192.168.2.247/menu.ipxe -> http://192.168.2.223/menu.ipxe
``` ```
The generated menu currently exposes entries for: The generated menu currently exposes entries for:
- NixOS Auto-Installer - NixOS Auto-Installer
- NixOS Minimal - NixOS Minimal
- Debian Minimal
- FreeIPA Server (Rocky Linux 9)
- SystemRescue environment - SystemRescue environment
- iPXE shell - iPXE shell
- Reboot - Reboot
@@ -84,19 +104,82 @@ directory name (`auto-installer` / `nixos-minimal`), so each one's
generated system name (`nixos-system-<name>-*`) is self-describing rather generated system name (`nixos-system-<name>-*`) is self-describing rather
than the nixpkgs default of `nixos-system-nixos-*` for both. than the nixpkgs default of `nixos-system-nixos-*` for both.
The Debian Minimal entry chains `http://<pxeServerIp>/debian.ipxe`, which loads
the Debian bookworm netboot kernel and initrd from `/srv/pxe/http/debian/`. The
`fetch-debian-netboot.service` oneshot downloads these files from
`deb.debian.org` on first boot (idempotent — skips if files are already
present):
```text
/srv/pxe/http/debian/linux (Debian bookworm netboot kernel)
/srv/pxe/http/debian/initrd.gz (Debian bookworm netboot initrd)
```
The service requires outbound internet access on the pxe-boot host. To
re-download (e.g. after a Debian point release), delete the files and restart
the service:
```bash
rm /srv/pxe/http/debian/linux /srv/pxe/http/debian/initrd.gz
systemctl restart fetch-debian-netboot.service
```
To update to a different Debian release, change `debianRelease` in
`modules/build-types/pxe-boot.nix` and redeploy.
The **FreeIPA Server (Rocky Linux 9)** entry chains
`http://<pxeServerIp>/rocky-freeipa.ipxe`, which boots the Rocky Linux 9
Anaconda installer with a Kickstart file (`rocky-freeipa.ks`) hosted on the
same server. The `fetch-rocky-pxeboot.service` oneshot downloads the pxeboot
kernel and initrd from the Rocky Linux mirror on first boot (idempotent):
```text
/srv/pxe/http/rocky/vmlinuz (Rocky Linux 9 Anaconda pxeboot kernel)
/srv/pxe/http/rocky/initrd.img (Rocky Linux 9 Anaconda pxeboot initrd)
```
The Kickstart file is generated from the NixOS module and staged at
`/srv/pxe/http/rocky-freeipa.ks`. It performs a fully unattended install:
1. Installs Rocky Linux 9 with `ipa-server` + `ipa-server-dns` packages
2. Configures static IP `192.168.2.138`, hostname `domain-controller.sweet.home`
3. Creates user `wayne` with the `adminSshKey` from `variables.nix`
4. Generates random IPA passwords and writes them to `/root/ipa-credentials.txt`
5. Creates a `freeipa-first-boot.service` oneshot that runs `ipa-server-install`
on first reboot (~20 minutes)
After the install completes:
- SSH in as `wayne@domain-controller` using the admin key
- Monitor FreeIPA install progress: `sudo tail -f /root/freeipa-install.log`
- Retrieve credentials: `sudo cat /root/ipa-credentials.txt` (save to password manager)
- Configure Pi-hole: `server=/sweet.home/192.168.2.138` in dnsmasq
To refresh the pxeboot files (e.g. after a Rocky point release):
```bash
rm /srv/pxe/http/rocky/vmlinuz /srv/pxe/http/rocky/initrd.img
systemctl restart fetch-rocky-pxeboot.service
```
To update to a different Rocky release, change `rockyRelease` in
`modules/build-types/pxe-boot.nix` and redeploy.
The SystemRescue entry expects the source ISO at: The SystemRescue entry expects the source ISO at:
```text ```text
/srv/pxe/http/images/systemrescue.iso /srv/pxe/http/images/systemrescue.iso
``` ```
Since `/srv/pxe/http/images` is the NFS-backed symlink, place the ISO on the
NFS share at `server.sweet.home:/tank/pxe-boot/images/systemrescue.iso`.
The `stage-systemrescue.service` oneshot extracts that ISO into: The `stage-systemrescue.service` oneshot extracts that ISO into:
```text ```text
/srv/pxe/http/systemrescue /srv/pxe/http/systemrescue
``` ```
The rescue menu entry then chains `http://192.168.2.247/systemrescue.ipxe`, The rescue menu entry then chains `http://192.168.2.223/systemrescue.ipxe`,
which loads the SystemRescue kernel and initramfs from the extracted tree and which loads the SystemRescue kernel and initramfs from the extracted tree and
uses `archiso_http_srv` to fetch the squashfs payload over HTTP. uses `archiso_http_srv` to fetch the squashfs payload over HTTP.
@@ -113,6 +196,13 @@ After deployment by an operator, basic service checks are:
```bash ```bash
curl http://pxe-boot/boot.ipxe curl http://pxe-boot/boot.ipxe
curl http://pxe-boot/menu.ipxe curl http://pxe-boot/menu.ipxe
curl http://pxe-boot/debian.ipxe
curl -I http://pxe-boot/debian/linux
curl -I http://pxe-boot/debian/initrd.gz
curl http://pxe-boot/rocky-freeipa.ipxe
curl http://pxe-boot/rocky-freeipa.ks
curl -I http://pxe-boot/rocky/vmlinuz
curl -I http://pxe-boot/rocky/initrd.img
curl http://pxe-boot/systemrescue.ipxe curl http://pxe-boot/systemrescue.ipxe
curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/vmlinuz curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/vmlinuz
curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/sysresccd.img curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/sysresccd.img
Generated
+6 -6
View File
@@ -173,11 +173,11 @@
] ]
}, },
"locked": { "locked": {
"lastModified": 1784350909, "lastModified": 1785119570,
"narHash": "sha256-ZWyzLbS1yKUTeFJLmdVuWNnHttL333/ldJbEE+KzCrM=", "narHash": "sha256-Rgs2xKnGLFWQscxUaXX07oyZeuMDOHEbqDOsgliLFGM=",
"owner": "nix-community", "owner": "nix-community",
"repo": "home-manager", "repo": "home-manager",
"rev": "4ce190229c73d44536caa7072f6308fb2d8feeb3", "rev": "d4fd24667c8cbef124bb70a20380cab75ec8474d",
"type": "github" "type": "github"
}, },
"original": { "original": {
@@ -259,11 +259,11 @@
}, },
"nixpkgs_2": { "nixpkgs_2": {
"locked": { "locked": {
"lastModified": 1784432872, "lastModified": 1785104993,
"narHash": "sha256-n3gKTBIV4ZA5VQpUakffBe3KGu4+mhPoA34rrqS0GkA=", "narHash": "sha256-eKbrvPoAOFutbYMdbB3r5EQVmFxKv24iKqHPPUXA0gM=",
"owner": "NixOS", "owner": "NixOS",
"repo": "nixpkgs", "repo": "nixpkgs",
"rev": "fd1462031fdee08f65fd0b4c6b64e22239a77870", "rev": "8623c4c20aa4ca2f5fb81510d2944066c3fb0d96",
"type": "github" "type": "github"
}, },
"original": { "original": {
+3
View File
@@ -130,6 +130,9 @@
lxc-tailscale-router = mkTarget { platform = "lxc"; buildType = "tailscale-router"; hostPath = ./hosts/tailscale-router/host.nix; }; lxc-tailscale-router = mkTarget { platform = "lxc"; buildType = "tailscale-router"; hostPath = ./hosts/tailscale-router/host.nix; };
lxc-tor-relay = mkTarget { platform = "lxc"; buildType = "tor-relay"; hostPath = ./hosts/tor-relay/host.nix; }; lxc-tor-relay = mkTarget { platform = "lxc"; buildType = "tor-relay"; hostPath = ./hosts/tor-relay/host.nix; };
proxmox-ha-server-1 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-1/host.nix; };
proxmox-ha-server-2 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-2/host.nix; };
}; };
# Auto-install environments (migrated from the former nix-auto-installer # Auto-install environments (migrated from the former nix-auto-installer
+17 -3
View File
@@ -1,10 +1,24 @@
_: { vars, ... }:
{ {
networking.hostName = "docker"; networking = {
networking.hostId = "007f0200"; hostName = "docker";
hostId = "007f0200";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.dockerIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
boot.zfs.forceImportRoot = false; boot.zfs.forceImportRoot = false;
# Only advertise the LAN interface to IPA DNS. Without this, SSSD registers
# every Docker bridge (172.x.x.x) as an A record for docker.sweet.home —
# the default dyndns.interface = "*" catches them all.
security.ipa.dyndns.interface = vars.lxcLanInterface; # eth0
# Preserved from the pre-refactor `docker` target — stateVersion must never # Preserved from the pre-refactor `docker` target — stateVersion must never
# be bumped on an already-installed machine. # be bumped on an already-installed machine.
system.stateVersion = "25.05"; system.stateVersion = "25.05";
+30
View File
@@ -0,0 +1,30 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "ha-server-1";
sopsFile = ../../secrets/ha-server-1.yaml;
})
];
networking = {
hostName = vars.haServer1Host;
hostId = "3a4b5c6d";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.haServer1Ip;
prefixLength = vars.lanPrefixLength;
}];
interfaces.${vars.vmStorageInterface}.ipv4.addresses = [{
address = vars.haServer1StorageIp;
prefixLength = vars.haStoragePrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# Set KEY after pairing this host with the beszel hub; the token is sops-managed.
# services.beszel.agent.environment.KEY = "";
system.stateVersion = "26.05";
}
+30
View File
@@ -0,0 +1,30 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "ha-server-2";
sopsFile = ../../secrets/ha-server-2.yaml;
})
];
networking = {
hostName = vars.haServer2Host;
hostId = "7e8f9a0b";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.haServer2Ip;
prefixLength = vars.lanPrefixLength;
}];
interfaces.${vars.vmStorageInterface}.ipv4.addresses = [{
address = vars.haServer2StorageIp;
prefixLength = vars.haStoragePrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# Set KEY after pairing this host with the beszel hub; the token is sops-managed.
# services.beszel.agent.environment.KEY = "";
system.stateVersion = "26.05";
}
+10 -2
View File
@@ -8,10 +8,18 @@
}) })
]; ];
networking.hostName = vars.nixCacheHost; networking = {
hostName = vars.nixCacheHost;
useDHCP = false;
interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
address = vars.nixCacheIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
services.beszel.agent.environment = { services.beszel.agent.environment = {
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG"; KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
}; };
+1 -1
View File
@@ -27,7 +27,7 @@
# Optional: set environment vars # Optional: set environment vars
sessionVariables = { sessionVariables = {
EDITOR = "vim"; EDITOR = "vim";
SOPS_AGE_KEY_FILE = "~/.config/sops/age/keys.txt"; SOPS_AGE_KEY_FILE = "${config.home.homeDirectory}/.config/sops/age/keys.txt";
}; };
file = { file = {
+11 -2
View File
@@ -1,7 +1,16 @@
_: { vars, ... }:
{ {
networking.hostName = "pxe-boot"; networking = {
hostName = "pxe-boot";
useDHCP = false;
interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
address = vars.pxeServerIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# Preserved from the pre-refactor `pxe-boot` target — stateVersion must # Preserved from the pre-refactor `pxe-boot` target — stateVersion must
# never be bumped on an already-installed machine. # never be bumped on an already-installed machine.
+11 -3
View File
@@ -8,11 +8,19 @@
}) })
]; ];
networking.hostName = vars.nfsServerHost; networking = {
networking.hostId = "6689f93e"; hostName = vars.nfsServerHost;
hostId = "6689f93e";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.serverIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
services.beszel.agent.environment = { services.beszel.agent.environment = {
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG"; KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}"; EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
LOG_LEVEL = "debug"; LOG_LEVEL = "debug";
+22 -2
View File
@@ -1,7 +1,27 @@
_: { vars, ... }:
{ {
networking.hostName = "tailscale-router"; imports = [
(import ../../modules/beszel/host-token.nix {
name = "tailscale-router";
sopsFile = ../../secrets/tailscale-router.yaml;
})
];
networking = {
hostName = "tailscale-router";
useDHCP = false;
interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
address = vars.tailscaleRouterIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
services.beszel.agent.environment = {
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
};
# No networking.hostId: only ZFS-touching hosts (server, docker) need one # No networking.hostId: only ZFS-touching hosts (server, docker) need one
# for pool-import safety, and this host does neither. # for pool-import safety, and this host does neither.
+13 -4
View File
@@ -1,4 +1,4 @@
{ ... }: { vars, ... }:
{ {
imports = [ imports = [
@@ -8,10 +8,19 @@
}) })
]; ];
networking.hostName = "tor-relay"; networking = {
hostName = "tor-relay";
useDHCP = false;
interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
address = vars.torRelayIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# No networking.hostId: only ZFS-touching hosts (server, docker) need one # No networking.hostId: only ZFS-touching hosts need one for pool-import
# for pool-import safety, and this host does neither. # safety, and this host does neither.
services.beszel.agent.environment = { services.beszel.agent.environment = {
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG"; KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
-1
View File
@@ -15,7 +15,6 @@
../docker/enable-service.nix ../docker/enable-service.nix
../docker/nextcloud-cron-job.nix ../docker/nextcloud-cron-job.nix
../docker/docker-health-to-gotify.nix ../docker/docker-health-to-gotify.nix
../tailscale/enable-service.nix
../traefik/rotate-logs.nix ../traefik/rotate-logs.nix
../raspi/mount-data.nix ../raspi/mount-data.nix
../services/enable-rpcbind.nix ../services/enable-rpcbind.nix
+78 -1
View File
@@ -1,6 +1,17 @@
{ config, pkgs, lib, inputs, vars, ... }: { config, pkgs, lib, inputs, vars, ... }:
{ {
imports = [
../docker/enable-service.nix
];
nixpkgs.overlays = [
(final: prev: {
docker = prev.docker_29;
docker_cli = prev.docker_29;
})
];
environment.systemPackages = with pkgs; [ environment.systemPackages = with pkgs; [
inputs.nixos-conf-editor.packages.${pkgs.stdenv.hostPlatform.system}.nixos-conf-editor inputs.nixos-conf-editor.packages.${pkgs.stdenv.hostPlatform.system}.nixos-conf-editor
nodejs nodejs
@@ -18,7 +29,7 @@
]; ];
boot.loader.grub.useOSProber = true; boot.loader.grub.useOSProber = true;
programs.direnv.enable = true; programs.direnv.enable = true;
services = { services = {
xserver = { xserver = {
enable = true; enable = true;
@@ -70,4 +81,70 @@ programs.direnv.enable = true;
programs.firefox.enable = true; programs.firefox.enable = true;
nixpkgs.config.allowUnfree = true; nixpkgs.config.allowUnfree = true;
# GUI-specific Home Manager additions for the IPA primary user, extending
# the baseline in modules/ipa/client.nix with desktop apps and services
# that only make sense on a graphical workstation.
home-manager.users.${vars.ipaUser} = { pkgs, ... }: {
home = {
packages = with pkgs; [
git
vim
nextcloud-client
chromium
claude-code
fish
sops
];
sessionVariables = {
EDITOR = "vim";
SOPS_AGE_KEY_FILE = "/home/${vars.ipaUser}/.config/sops/age/keys.txt";
};
file = {
".local/share/applications/proxmox-chromium-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox (Chromium)
Exec=chromium --app=https://pve.${vars.homeDomain}:${toString vars.ports.pveWeb} --window-size=1920,1080 --window-position=0,0
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=Hypervisor;
StartupWMClass=PVE
'';
".local/share/applications/pbs-chromium-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox Backup Server (Chromium)
Exec=chromium --app=https://${vars.pbsIp}:${toString vars.ports.pbsWeb} --window-size=1920,1080 --window-position=0,0
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=backup;
'';
".local/share/applications/proxmox-firefox-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox (Firefox)
Exec=firefox --new-instance https://pve.${vars.homeDomain}:${toString vars.ports.pveWeb} --profile ProxmoxWebApp --window-size=1920,1080 --class ProxmoxWebApp
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=Hypervisor;
StartupWMClass=PVE
'';
".local/share/applications/pbs-firefox-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox Backup Server (Firefox)
Exec=firefox --new-window https://${vars.pbsIp}:${toString vars.ports.pbsWeb} --profile PbsWebApp --window-size=1920,1080 --class PbsWebApp
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=backup;
StartupWMClass=PBS
'';
};
};
services.nextcloud-client = {
enable = true;
startInBackground = true;
};
};
} }
+48
View File
@@ -0,0 +1,48 @@
# HA file server build type: DRBD + XFS + LIO iSCSI + NFS, managed by
# Corosync + Pacemaker. Both ha-server-1 and ha-server-2 use this type.
#
# NFS start/stop:
# services.nfs.server.enable = true configures /etc/exports, wires up
# rpcbind, and loads kernel modules — but nfs-server.service.wantedBy is
# force-cleared so systemd does NOT auto-start it at boot. Pacemaker's
# ha-group resource group (configured by scripts/ha/cluster-init.sh)
# starts and stops nfs-server as part of the failover sequence after the
# XFS mount and iSCSI target are brought up on the new Active node.
#
# Beszel agent:
# Enabled here via enable-agent.nix. The agent KEY (used to pair with
# the Beszel hub) is not set yet — add it to hosts/ha-server-{1,2}/host.nix
# under services.beszel.agent.environment.KEY once the hub accepts the
# new agents, following the pattern in hosts/server/host.nix.
{ lib, pkgs, vars, ... }:
let
# Generates /etc/exports lines for all nfsShares data entries. Shared
# pattern with modules/build-types/server.nix — both export the same
# set of shares, differing only in the storage root they serve from.
mkNfsExports = storageRoot:
lib.concatMapStrings
(share: " ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n")
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares));
in
{
imports = [
../ha/pacemaker-stack.nix
../ha/iscsi-target.nix
../ha/cluster-config.nix
# ../beszel/enable-agent.nix
];
# xfsprogs: mkfs.xfs/xfs_info needed by cluster-init.sh.
# openiscsi: iscsiadm needed by acceptance-tests.sh T4 (iSCSI discovery check).
environment.systemPackages = [ pkgs.xfsprogs pkgs.openiscsi ];
services.nfs.server = {
enable = true;
exports = mkNfsExports vars.haStorageRoot;
};
# Pacemaker controls nfs-server — prevent systemd from starting it at boot
# on both nodes (only the Active node should be serving NFS).
systemd.services.nfs-server.wantedBy = lib.mkForce [ ];
}
+316 -33
View File
@@ -21,6 +21,216 @@ let
chain ${pxeBaseUrl}/boot.ipxe chain ${pxeBaseUrl}/boot.ipxe
''; '';
debianRelease = "bookworm";
debianMirror = "https://deb.debian.org/debian";
debianNetbootBase = "${debianMirror}/dists/${debianRelease}/main/installer-amd64/current/images/netboot/debian-installer/amd64";
rockyRelease = "9";
rockyArch = "x86_64";
rockyMirror = "https://dl.rockylinux.org/pub/rocky/${rockyRelease}";
rockyPxebootBase = "${rockyMirror}/BaseOS/${rockyArch}/os/images/pxeboot";
debianIpxe = pkgs.writeText "debian.ipxe" ''
#!ipxe
set base ${pxeBaseUrl}
kernel ''${base}/debian/linux
initrd ''${base}/debian/initrd.gz
boot
'';
fetchDebianNetboot = pkgs.writeShellScript "fetch-debian-netboot" ''
set -eu
dir="${httpRoot}/debian"
mirror="${debianNetbootBase}"
if [ -f "$dir/linux" ] && [ -f "$dir/initrd.gz" ]; then
echo "Debian ${debianRelease} netboot files already present; skipping download."
exit 0
fi
echo "Downloading Debian ${debianRelease} netboot kernel and initrd from $mirror ..."
${pkgs.curl}/bin/curl -fsSL -o "$dir/linux.tmp" "$mirror/linux"
${pkgs.curl}/bin/curl -fsSL -o "$dir/initrd.gz.tmp" "$mirror/initrd.gz"
mv "$dir/linux.tmp" "$dir/linux"
mv "$dir/initrd.gz.tmp" "$dir/initrd.gz"
echo "Debian ${debianRelease} netboot files staged."
'';
# Rocky Linux 9 iPXE script — boots vmlinuz+initrd.img from the staged
# /rocky/ directory and hands Anaconda the hosted Kickstart URL.
# net.ifnames=0 biosdevname=0 ensures the NIC is eth0 in both the
# installer and the installed system (matches the Kickstart NM config).
rockyFreeIpaIpxe = pkgs.writeText "rocky-freeipa.ipxe" ''
#!ipxe
set base ${pxeBaseUrl}
kernel ''${base}/rocky/vmlinuz inst.ks=''${base}/rocky-freeipa.ks inst.repo=${rockyMirror}/BaseOS/${rockyArch}/os/ net.ifnames=0 biosdevname=0 ip=dhcp quiet
initrd ''${base}/rocky/initrd.img
boot
'';
# Kickstart file for domain-controller.sweet.home.
# Installs Rocky Linux 9, sets a static IP, creates wayne with the
# admin SSH key, then on first reboot runs ipa-server-install via a
# systemd oneshot service. Passwords are generated at %post time,
# written to /root/ipa-credentials.txt (chmod 600), and read back by
# the first-boot script — never hardcoded here or in the repo.
rockyFreeIpaKs = pkgs.writeText "rocky-freeipa.ks" ''
#version=RHEL9
# Unattended Rocky Linux 9 + FreeIPA install
# Target: domain-controller.${vars.homeDomain} ${vars.domainControllerIp}
url --url=${rockyMirror}/BaseOS/${rockyArch}/os/
repo --name=appstream --baseurl=${rockyMirror}/AppStream/${rockyArch}/os/
lang en_US.UTF-8
keyboard us
timezone UTC --utc
# DHCP during install; static IP configured in %post via NM config file
network --bootproto=dhcp --device=link --activate
network --hostname=domain-controller.sweet.home
selinux --enforcing
firewall --enabled --service=ssh
rootpw --lock
user --name=wayne --groups=wheel --shell=/bin/bash
sshkey --username=wayne "${vars.adminSshKey}"
zerombr
clearpart --all --initlabel --drives=sda
# Keep net.ifnames=0 biosdevname=0 in the installed GRUB so the NIC
# stays eth0 after reboot (matches the NM connection file below).
bootloader --location=mbr --boot-drive=sda --append="net.ifnames=0 biosdevname=0"
part /boot --fstype=xfs --size=1024 --ondisk=sda
part swap --fstype=swap --size=2048 --ondisk=sda
part / --fstype=xfs --grow --size=1 --ondisk=sda --asprimary
%packages
@^minimal-environment
ipa-server
ipa-server-dns
%end
reboot
%post --log=/root/ks-post.log
set -euo pipefail
# -- Static IP: write NM connection file directly (NM not running in chroot) --
mkdir -p /etc/NetworkManager/system-connections
cat > /etc/NetworkManager/system-connections/eth0.nmconnection << 'NMCONN'
[connection]
id=eth0
type=ethernet
interface-name=eth0
autoconnect=true
[ethernet]
[ipv4]
method=manual
addresses=${vars.domainControllerIp}/${toString vars.lanPrefixLength}
gateway=${vars.lanGateway}
dns=${vars.domainControllerIp};
dns-search=${vars.homeDomain};
[ipv6]
method=auto
NMCONN
chmod 600 /etc/NetworkManager/system-connections/eth0.nmconnection
# -- /etc/hosts: FQDN must resolve to the real IP (not loopback) for IPA --
sed -i '/domain-controller/d' /etc/hosts
echo '${vars.domainControllerIp} domain-controller.${vars.homeDomain} domain-controller' >> /etc/hosts
# -- Generate IPA passwords and store securely --
DM_PASS=$(openssl rand -base64 24 | tr -dc 'A-Za-z0-9' | head -c 24)
ADMIN_PASS=$(openssl rand -base64 24 | tr -dc 'A-Za-z0-9' | head -c 24)
printf 'Directory Manager: %s\nIPA Admin: %s\n' "$DM_PASS" "$ADMIN_PASS" \
> /root/ipa-credentials.txt
chmod 600 /root/ipa-credentials.txt
# -- First-boot script: reads passwords back, runs ipa-server-install --
cat > /usr/local/sbin/freeipa-first-boot.sh << 'FIRSTBOOT'
#!/bin/bash
set -euo pipefail
exec >> /root/freeipa-install.log 2>&1
echo "=== FreeIPA first-boot install started at $(date) ==="
DM_PASS=$(grep '^Directory Manager:' /root/ipa-credentials.txt | awk '{print $NF}')
ADMIN_PASS=$(grep '^IPA Admin:' /root/ipa-credentials.txt | awk '{print $NF}')
ipa-server-install \
--realm=SWEET.HOME \
--domain=sweet.home \
--hostname=domain-controller.sweet.home \
--ds-password="$DM_PASS" \
--admin-password="$ADMIN_PASS" \
--setup-dns \
--forwarder=192.168.2.253 \
--no-dnssec-validation \
--no-ntp \
--unattended
echo "=== FreeIPA install complete at $(date) ==="
echo "Credentials: /root/ipa-credentials.txt (save to password manager)"
echo "CA backup: /root/cacert.p12 (encrypted with Directory Manager password)"
systemctl disable freeipa-first-boot.service
FIRSTBOOT
chmod 700 /usr/local/sbin/freeipa-first-boot.sh
# -- Systemd oneshot service: runs freeipa-first-boot.sh on first real boot --
cat > /etc/systemd/system/freeipa-first-boot.service << 'UNIT'
[Unit]
Description=FreeIPA first-boot installation
After=network-online.target
Wants=network-online.target
ConditionPathExists=/root/ipa-credentials.txt
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/freeipa-first-boot.sh
TimeoutStartSec=1800
RemainAfterExit=yes
[Install]
WantedBy=multi-user.target
UNIT
mkdir -p /etc/systemd/system/multi-user.target.wants
ln -sf /etc/systemd/system/freeipa-first-boot.service \
/etc/systemd/system/multi-user.target.wants/freeipa-first-boot.service
echo "Kickstart %post complete. FreeIPA installs on first reboot (~20 min)."
%end
'';
fetchRockyPxeboot = pkgs.writeShellScript "fetch-rocky-pxeboot" ''
set -eu
dir="${httpRoot}/rocky"
base="${rockyPxebootBase}"
if [ -f "$dir/vmlinuz" ] && [ -f "$dir/initrd.img" ]; then
echo "Rocky Linux ${rockyRelease} pxeboot files already present; skipping download."
exit 0
fi
echo "Downloading Rocky Linux ${rockyRelease} pxeboot kernel and initrd from $base ..."
${pkgs.curl}/bin/curl -fsSL -o "$dir/vmlinuz.tmp" "$base/vmlinuz"
${pkgs.curl}/bin/curl -fsSL -o "$dir/initrd.img.tmp" "$base/initrd.img"
mv "$dir/vmlinuz.tmp" "$dir/vmlinuz"
mv "$dir/initrd.img.tmp" "$dir/initrd.img"
echo "Rocky Linux ${rockyRelease} pxeboot files staged."
'';
systemRescueIpxe = pkgs.writeText "systemrescue.ipxe" '' systemRescueIpxe = pkgs.writeText "systemrescue.ipxe" ''
#!ipxe #!ipxe
@@ -68,11 +278,13 @@ let
set base ${pxeBaseUrl} set base ${pxeBaseUrl}
menu PXE Boot Menu menu PXE Boot Menu
item auto-installer NixOS Auto-Installer item auto-installer NixOS Auto-Installer
item nixos-minimal NixOS Minimal item nixos-minimal NixOS Minimal
item rescue Rescue Environment item debian Debian Minimal
item shell iPXE Shell item rocky-freeipa FreeIPA Server (Rocky Linux 9)
item reboot Reboot item rescue Rescue Environment
item shell iPXE Shell
item reboot Reboot
choose target && goto ''${target} choose target && goto ''${target}
@@ -82,6 +294,12 @@ let
:nixos-minimal :nixos-minimal
chain ''${base}/nixos-minimal/netboot.ipxe chain ''${base}/nixos-minimal/netboot.ipxe
:debian
chain ''${base}/debian.ipxe
:rocky-freeipa
chain ''${base}/rocky-freeipa.ipxe
:rescue :rescue
chain ''${base}/systemrescue.ipxe chain ''${base}/systemrescue.ipxe
@@ -95,6 +313,7 @@ in
{ {
imports = [ imports = [
../pxe-boot/stage-installer-artifacts.nix ../pxe-boot/stage-installer-artifacts.nix
../pxe-boot/mount-pxe-images.nix
]; ];
environment.systemPackages = with pkgs; [ environment.systemPackages = with pkgs; [
@@ -129,37 +348,101 @@ in
openssh.settings.PermitRootLogin = "yes"; openssh.settings.PermitRootLogin = "yes";
}; };
systemd.tmpfiles.rules = [ systemd = {
"d ${pxeRoot} 0755 root root -" tmpfiles.rules = [
"d ${httpRoot} 0755 root root -" "d ${pxeRoot} 0755 root root -"
"d ${httpRoot}/images 0755 root root -" "d ${httpRoot} 0755 root root -"
"d ${httpRoot}/auto-installer 0755 root root -" "L+ ${httpRoot}/images - - - - ${vars.nfsShares.pxebootImages.mountpoint}"
"d ${httpRoot}/nixos-minimal 0755 root root -" "d ${httpRoot}/auto-installer 0755 root root -"
"d ${httpRoot}/systemrescue 0755 root root -" "d ${httpRoot}/nixos-minimal 0755 root root -"
"d ${httpRoot}/ubuntu 0755 root root -" "d ${httpRoot}/systemrescue 0755 root root -"
"d ${httpRoot}/rescue 0755 root root -" "d ${httpRoot}/debian 0755 root root -"
"d ${tftpRoot} 0755 root root -" "d ${httpRoot}/ubuntu 0755 root root -"
"C+ ${httpRoot}/boot.ipxe 0644 root root - ${bootIpxe}" "d ${httpRoot}/rescue 0755 root root -"
"C+ ${httpRoot}/menu.ipxe 0644 root root - ${menuIpxe}" "d ${httpRoot}/rocky 0755 root root -"
"C+ ${httpRoot}/systemrescue.ipxe 0644 root root - ${systemRescueIpxe}" "d ${tftpRoot} 0755 root root -"
"C+ ${tftpRoot}/autoexec.ipxe 0644 root root - ${autoexecIpxe}" "C+ ${httpRoot}/boot.ipxe 0644 root root - ${bootIpxe}"
"C+ ${tftpRoot}/ipxe.efi 0644 root root - ${pkgs.ipxe}/ipxe.efi" "C+ ${httpRoot}/menu.ipxe 0644 root root - ${menuIpxe}"
"C+ ${tftpRoot}/undionly.kpxe 0644 root root - ${pkgs.ipxe}/undionly.kpxe" "C+ ${httpRoot}/debian.ipxe 0644 root root - ${debianIpxe}"
]; "C+ ${httpRoot}/rocky-freeipa.ipxe 0644 root root - ${rockyFreeIpaIpxe}"
"C+ ${httpRoot}/rocky-freeipa.ks 0644 root root - ${rockyFreeIpaKs}"
systemd.services.stage-systemrescue = { "C+ ${httpRoot}/systemrescue.ipxe 0644 root root - ${systemRescueIpxe}"
description = "Stage SystemRescue ISO contents for HTTP PXE boot"; "C+ ${tftpRoot}/autoexec.ipxe 0644 root root - ${autoexecIpxe}"
after = [ "C+ ${tftpRoot}/ipxe.efi 0644 root root - ${pkgs.ipxe}/ipxe.efi"
"local-fs.target" "C+ ${tftpRoot}/undionly.kpxe 0644 root root - ${pkgs.ipxe}/undionly.kpxe"
"systemd-tmpfiles-setup.service"
]; ];
wantedBy = [ "multi-user.target" ];
serviceConfig = { services = {
Type = "oneshot"; fetch-debian-netboot = {
ExecStart = stageSystemRescue; description = "Download Debian ${debianRelease} netboot kernel and initrd for HTTP PXE boot";
after = [
"local-fs.target"
"systemd-tmpfiles-setup.service"
"network-online.target"
];
wants = [ "network-online.target" ];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
ExecStart = fetchDebianNetboot;
RemainAfterExit = true;
};
};
fetch-rocky-pxeboot = {
description = "Download Rocky Linux ${rockyRelease} pxeboot kernel and initrd for HTTP PXE boot";
after = [
"local-fs.target"
"systemd-tmpfiles-setup.service"
"network-online.target"
];
wants = [ "network-online.target" ];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
ExecStart = fetchRockyPxeboot;
RemainAfterExit = true;
};
};
stage-systemrescue = {
description = "Stage SystemRescue ISO contents for HTTP PXE boot";
after = [
"local-fs.target"
"systemd-tmpfiles-setup.service"
];
wantedBy = [ "multi-user.target" ];
serviceConfig = {
Type = "oneshot";
ExecStart = stageSystemRescue;
};
};
};
};
services.dnsmasq = {
enable = true;
settings = {
# Disable DNS listener — only proxy DHCP is needed here.
# Without this dnsmasq tries to bind port 53 which systemd-resolved
# already owns, causing startup failure.
port = 0;
dhcp-range = [ "192.168.2.0,proxy" ];
dhcp-match = [
"set:ipxe,175"
"set:efi64,option:client-arch,7"
"set:efi64,option:client-arch,9"
];
dhcp-userclass = "set:ipxe,iPXE";
dhcp-boot = [
"tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
"tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
"tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
"undionly.kpxe,,${vars.pxeServerIp}"
];
}; };
}; };
networking.firewall.allowedTCPPorts = [ vars.ports.pxeBootHttp ]; networking.firewall.allowedTCPPorts = [ vars.ports.pxeBootHttp ];
networking.firewall.allowedUDPPorts = [ vars.ports.pxeBootTftp ]; networking.firewall.allowedUDPPorts = [ vars.ports.pxeBootTftp 67 ];
} }
+43 -18
View File
@@ -10,8 +10,18 @@ let
in lib.imap1 (i: _: lib.concatStringsSep "/" (lib.take i parts)) parts; in lib.imap1 (i: _: lib.concatStringsSep "/" (lib.take i parts)) parts;
poolDatasets = lib.unique ( poolDatasets = lib.unique (
lib.concatMap (share: ancestors share.subpath) (lib.attrValues vars.nfsShares) lib.concatMap (share: ancestors share.subpath)
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares))
); );
# Generates /etc/exports lines for all nfsShares data entries (every
# attrset value — excludes the bare `options` string). Both server and
# ha-server export the same share set from different storage roots, so
# this helper is the single source of truth for the export line format.
mkNfsExports = storageRoot:
lib.concatMapStrings
(share: " ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n")
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares));
in in
{ {
imports = [ imports = [
@@ -43,14 +53,8 @@ in
exit 0 exit 0
fi fi
# Pool exists on a device but not yet imported let the standard # Locate the data disk first used for both the fallback import
# zfs-import-${poolName}.service handle it normally. # attempt and, only if the disk is genuinely blank, pool creation.
if zpool import -d /dev/disk/by-id -N "${poolName}" 2>/dev/null; then
exit 0
fi
# No pool found at all. Create it on the Proxmox data disk (scsi1),
# which appears as /dev/disk/by-id/scsi-*drive-scsi1 inside the VM.
DATA_DISK="" DATA_DISK=""
for candidate in /dev/disk/by-id/scsi-*drive-scsi1; do for candidate in /dev/disk/by-id/scsi-*drive-scsi1; do
[[ "$candidate" == *-part* ]] && continue [[ "$candidate" == *-part* ]] && continue
@@ -62,8 +66,31 @@ in
exit 1 exit 1
fi fi
# Try importing via the by-id symlink directory first (normal path),
# then fall back to scanning the disk directly. The two-step exists
# because of a udev race: systemd-udev-settle.service can clear before
# /dev/disk/by-id/ entries are fully populated, causing the first
# import to fail even when the pool is intact on the disk.
if zpool import -d /dev/disk/by-id -N "${poolName}" 2>/dev/null; then
exit 0
fi
if zpool import -d "$DATA_DISK" -N "${poolName}" 2>/dev/null; then
exit 0
fi
# Both import attempts failed. Before creating a new pool, verify the
# disk is genuinely blank if ZFS label metadata is present the import
# failed for some other reason and we must not clobber existing data.
if zdb -l "$DATA_DISK" 2>/dev/null | grep -q "name: '${poolName}'"; then
echo "zfs-init-${poolName}: $DATA_DISK has ZFS pool '${poolName}' metadata but import failed refusing to overwrite existing data. Run 'zpool import -d $DATA_DISK ${poolName}' manually to investigate." >&2
exit 1
fi
# Disk is genuinely blank: create the pool. -f is intentionally
# omitted so that if we somehow reach this point with an existing pool
# on the disk, zpool refuses rather than silently destroying data.
echo "zfs-init-${poolName}: creating pool on $DATA_DISK" echo "zfs-init-${poolName}: creating pool on $DATA_DISK"
zpool create -f "${poolName}" "$DATA_DISK" zpool create "${poolName}" "$DATA_DISK"
${lib.concatMapStrings (ds: '' ${lib.concatMapStrings (ds: ''
zfs create "${poolName}/${ds}" zfs create "${poolName}/${ds}"
'') poolDatasets} '') poolDatasets}
@@ -77,14 +104,12 @@ in
services.nfs.server = { services.nfs.server = {
enable = true; enable = true;
exports = '' exports = mkNfsExports vars.storageRoot;
${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath} ${vars.lanCidr}(rw,sync,no_subtree_check,no_root_squash)
${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath} ${vars.lanCidr}(rw,sync,no_subtree_check,no_root_squash)
${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath} ${vars.lanCidr}(rw,sync,no_subtree_check,no_root_squash)
${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath} ${vars.lanCidr}(rw,sync,no_subtree_check,no_root_squash)
${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath} ${vars.lanCidr}(rw,sync,no_subtree_check,no_root_squash)
'';
}; };
networking.firewall.allowedTCPPorts = [ vars.ports.nfsRpcbind vars.ports.nfsd ]; # mountd (20048) is needed for showmount/NFSv3 mount protocol — without it
# clients can reach portmapper (111) and get the mountd port back, then
# time out trying to connect to it. All three ports need TCP and UDP.
networking.firewall.allowedTCPPorts = [ vars.ports.nfsRpcbind vars.ports.nfsd vars.ports.nfsMountd ];
networking.firewall.allowedUDPPorts = [ vars.ports.nfsRpcbind vars.ports.nfsd vars.ports.nfsMountd ];
} }
+30 -5
View File
@@ -1,8 +1,10 @@
{ ... }: { vars, ... }:
{ {
imports = [ imports = [
../tailscale/subnet-router.nix ../tailscale/subnet-router.nix
../tailscale/ts-dns-forwarder.nix
../beszel/enable-agent.nix
]; ];
# "server", not "both": this build type advertises LAN subnet routes but # "server", not "both": this build type advertises LAN subnet routes but
@@ -12,8 +14,31 @@
# own setting) so the intent is clear at the build-type level. # own setting) so the intent is clear at the build-type level.
services.tailscale.useRoutingFeatures = "server"; services.tailscale.useRoutingFeatures = "server";
# Forwarded subnet-router traffic arrives on tailscale0 already # Advertise the LAN subnet so Tailscale peers can route back to LAN machines.
# tailscale-authenticated -- the firewall's normal per-port allow-list # Must also be approved in the Tailscale admin console (Machines → Edit route settings).
# would otherwise drop it. Standard NixOS/Tailscale subnet-router guidance. services.tailscale.extraUpFlags = [ "--advertise-routes=${vars.lanCidr}" ];
networking.firewall.trustedInterfaces = [ "tailscale0" ];
networking.firewall = {
# Forwarded subnet-router traffic arrives on tailscale0 already
# tailscale-authenticated -- the firewall's normal per-port allow-list
# would otherwise drop it. Standard NixOS/Tailscale subnet-router guidance.
trustedInterfaces = [ "tailscale0" ];
# SNAT LAN traffic going into Tailscale so the remote peer sees it as
# coming from this router's Tailscale IP rather than a raw LAN IP.
# Without this, Tailscale drops forwarded packets whose source is not a
# recognised Tailscale address.
#
# We target POSTROUTING directly (always-existing built-in chain) rather
# than nixos-nat-post: extraCommands runs after the old nixos-nat-post is
# deleted but before the new one is created, so -A nixos-nat-post silently
# fails. The -C check makes the rule idempotent across firewall reloads.
extraCommands = ''
iptables -t nat -C POSTROUTING -s ${vars.lanCidr} -o tailscale0 -j MASQUERADE 2>/dev/null || \
iptables -t nat -A POSTROUTING -s ${vars.lanCidr} -o tailscale0 -j MASQUERADE
'';
extraStopCommands = ''
iptables -t nat -D POSTROUTING -s ${vars.lanCidr} -o tailscale0 -j MASQUERADE 2>/dev/null || true
'';
};
} }
+4 -47
View File
@@ -1,50 +1,7 @@
{ config, pkgs, lib, vars, ... }: _:
let
# Flake attribute names are now <platform>-<buildtype> (e.g. proxmox-docker)
# and no longer match networking.hostName, since a host's hostname stays
# fixed while the platform backing it can change. Each nixosConfiguration
# stamps its own active target name into /etc/flake-target at build time.
mySwitchCmd = ''
sudo nixos-rebuild switch \
--no-write-lock-file \
--refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
'';
myTestCmd = ''
sudo nixos-rebuild test \
--no-write-lock-file \
--refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
'';
# lxc-* hosts pre-seed their SSH host key at build time (see
# modules/platforms/lxc.nix) so sops-nix's .sops.yaml recipient matches on
# first boot -- without it, secrets permanently fail to decrypt (see that
# file's comment for the confirmed failure). That requires --impure plus
# NIXOS_HOST_KEYS_DIR pointing at the repo's host-keys/ dir, same pattern
# docs/auto-installer.md uses for the installer ISO. A function, not a
# shellAlias, since the target name has to interpolate into the middle of
# the flake attribute path, not just append after it. Must be run from the
# repo root, same as every other host-keys/ command in this repo.
buildImageFn = ''
buildImage() {
if [ -z "$1" ]; then
echo "usage: buildImage <flake-target> (e.g. lxc-docker)" >&2
return 1
fi
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
".#nixosConfigurations.$1.config.system.build.tarball"
}
'';
in
{ {
programs.bash = { # Switch-nix, Test-nix, and buildImage are defined system-wide in
enable = true; # modules/common/configuration.nix so all users (including IPA accounts)
shellAliases = { # get them. Add any Home-Manager-only per-user shell config here.
"Switch-nix" = mySwitchCmd;
"Test-nix" = myTestCmd;
};
initExtra = buildImageFn;
};
} }
+38 -10
View File
@@ -1,17 +1,45 @@
{ config, lib, pkgs, vars, ... }: { config, lib, pkgs, vars, ... }:
let
switchCmd = ''
sudo nixos-rebuild switch \
--no-write-lock-file \
--refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
'';
testCmd = ''
sudo nixos-rebuild test \
--no-write-lock-file \
--refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
'';
buildImageFn = ''
buildImage() {
if [ -z "$1" ]; then
echo "usage: buildImage <flake-target> (e.g. lxc-docker)" >&2
return 1
fi
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
".#nixosConfigurations.$1.config.system.build.tarball"
}
'';
in
{ {
imports = imports = [
[ ./set-locale.nix
# Include the results of the hardware scan. ../ipa/client.nix
# ./hardware-configuration.nix ];
./set-locale.nix
];
# Use the GRUB 2 boot loader.
# boot.loader.grub.enable = true;
#boot.loader.grub.device = "/dev/sda"; # or "nodev" for efi only
networking.networkmanager.enable = true; # Easiest to use and most distros use this by default. # System-wide shell config so all users (including IPA accounts) get the
# same management aliases as the local nixos user's Home Manager provides.
programs.bash = {
shellAliases = {
"Switch-nix" = switchCmd;
"Test-nix" = testCmd;
};
interactiveShellInit = buildImageFn;
};
networking.networkmanager.enable = true;
# Recommended over the true default (bypasses ZFS's own import safeguards) # Recommended over the true default (bypasses ZFS's own import safeguards)
# per the option's own docs; matches hosts/docker/host.nix and # per the option's own docs; matches hosts/docker/host.nix and
+35
View File
@@ -0,0 +1,35 @@
# Shared activation-script logic to preserve the SSH host key across
# nixos-rebuild on platforms that embed the key via environment.etc (lxc and
# proxmox). When NIXOS_HOST_KEYS_DIR is not set the key is absent from
# environment.etc, and NixOS's etc activation removes any /etc file not in
# the new generation — which would destroy the live key and break sops-nix
# decryption permanently. These scripts save the key to /run before etc
# removes it, then restore it afterward.
#
# Explicit deps enforce the correct ordering: without them the topological
# sort places preserveSshHostKey after etc (confirmed live on lxc-tor-relay:
# position 7 vs etc's position 5), so the key is gone before it can be saved.
_: {
system.activationScripts = {
preserveSshHostKey = ''
if [ -f /etc/ssh/ssh_host_ed25519_key ]; then
cp /etc/ssh/ssh_host_ed25519_key /run/sshd-host-key-preserve.tmp
cp /etc/ssh/ssh_host_ed25519_key.pub /run/sshd-host-key-preserve.pub.tmp
fi
'';
restoreSshHostKey = {
deps = [ "etc" ];
text = ''
if [ ! -f /etc/ssh/ssh_host_ed25519_key ] && [ -f /run/sshd-host-key-preserve.tmp ]; then
install -m 0600 /run/sshd-host-key-preserve.tmp /etc/ssh/ssh_host_ed25519_key
install -m 0644 /run/sshd-host-key-preserve.pub.tmp /etc/ssh/ssh_host_ed25519_key.pub
fi
rm -f /run/sshd-host-key-preserve.tmp /run/sshd-host-key-preserve.pub.tmp
'';
};
etc = { deps = [ "preserveSshHostKey" ]; };
setupSecrets = { deps = [ "restoreSshHostKey" ]; };
};
}
+34 -12
View File
@@ -1,23 +1,45 @@
{ pkgs, ... }: { lib, pkgs, vars, ... }:
let
gid = toString vars.dockerAccessGid;
in
{ {
# virtualisation.docker.enable = true;
virtualisation.docker = { virtualisation.docker = {
enable = true; enable = true;
package = pkgs.docker; package = pkgs.docker;
# listenOptions = [
# "unix:///var/run/docker.sock"
# "tcp://0.0.0.0:2375"
#];
# daemon.settings = {
# metrics-addr = "0.0.0.0:9323";
# experimental = true;
# };
}; };
# Pin the docker group GID to match the IPA "docker-access" group so that
# IPA group membership alone grants access to the Docker socket. Any user
# whose supplementary groups (resolved by SSSD from IPA) include GID
# vars.dockerAccessGid will pass the socket group-permission check without
# any per-host users.groups.docker.members entry.
users.groups.docker.gid = lib.mkForce vars.dockerAccessGid;
users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
environment.systemPackages = with pkgs; [ environment.systemPackages = with pkgs; [
docker-compose docker-compose
docker-buildx docker-buildx
]; ];
# NixOS's group activation uses plain `groupmod` without --non-unique.
# When SSSD is active it exposes the IPA "docker-access" group at
# vars.dockerAccessGid via NSS, so groupmod sees that GID as already in
# use and silently skips the change (warning: "not applying GID change").
# This script runs after the normal "groups" step and applies the change
# with --non-unique (which lets the local docker group share the GID with
# the SSSD-provided IPA group). If the GID actually changed it also
# restarts docker.socket so the socket is recreated with the new GID.
system.activationScripts.docker-group-gid = {
deps = [ "groups" ];
text = ''
current=$(grep "^docker:" /etc/group | cut -d: -f3)
if [ "$current" != "${gid}" ]; then
${pkgs.shadow}/bin/groupmod --non-unique -g ${gid} docker
if ${pkgs.systemd}/bin/systemctl is-active --quiet docker.socket; then
${pkgs.systemd}/bin/systemctl stop docker.service docker.socket
rm -f /var/run/docker.sock
${pkgs.systemd}/bin/systemctl start docker.socket docker.service
fi
fi
'';
};
} }
+154
View File
@@ -0,0 +1,154 @@
# Cluster-wide HA config shared by both ha-server nodes.
#
# Covers everything that is identical on both nodes and references cluster
# topology (node IPs, hostnames, DRBD resource). Per-node identity
# (hostname, static IP, stateVersion) lives in hosts/ha-server-{1,2}/host.nix.
#
# Corosync authkey:
# /etc/corosync/authkey (mode 0400) is managed by sops-nix below.
# Bootstrap: run scripts/ha/cluster-init.sh on node1 to generate the key,
# then encrypt it with: sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey
# Both host keys must be registered via sync-host-keys.sh first so both nodes can decrypt it.
#
# DRBD fencing:
# resource-only with crm-fence-peer.sh: DRBD calls the Pacemaker-aware
# crm-fence-peer.sh handler before promoting. The handler checks the CIB
# to confirm the peer's DRBD resource is stopped and returns 7 (successfully
# fenced), allowing safe promotion without requiring power-fencing (STONITH).
# The unfence handler crm-unfence-peer.sh clears the outdate flag when the
# peer reconnects. This is the correct setting for Pacemaker+DRBD clusters
# with STONITH disabled; crm-fence-peer.sh replaces the need for a separate
# STONITH device during the testing phase. Switch to resource-and-stonith
# once the fence_pve_ssh STONITH resource is active (see
# scripts/ha/cluster-enable-stonith.sh).
#
# PATH wrapper: when the DRBD kernel module invokes the fence-peer handler
# via the UMH (User Mode Helper) mechanism it provides a minimal PATH that
# omits /run/current-system/sw/bin. crm-fence-peer.sh calls cibadmin,
# crm_mon etc.; if those aren't found a pipeline in the script breaks with
# SIGPIPE. A process killed by signal has WEXITSTATUS() == 0, so the kernel
# sees exit code 0 and logs "fence-peer helper broken, returned 0", looping
# forever. The writeShellScript wrappers below prepend the NixOS sw path
# before exec-ing the real handler, giving it a working Pacemaker toolchain.
{ lib, pkgs, vars, ... }:
let
fencePeerWrapper = pkgs.writeShellScript "drbd-fence-peer" ''
export PATH="/run/current-system/sw/bin:/run/current-system/sw/sbin:$PATH"
exec /run/current-system/sw/lib/drbd/crm-fence-peer.sh "$@"
'';
unfencePeerWrapper = pkgs.writeShellScript "drbd-unfence-peer" ''
export PATH="/run/current-system/sw/bin:/run/current-system/sw/sbin:$PATH"
exec /run/current-system/sw/lib/drbd/crm-unfence-peer.sh "$@"
'';
in
{
# Root SSH access — same key set as nixos user so all admin keys can reach root.
users.users.root.openssh.authorizedKeys.keys = [
vars.adminSshKey
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGygkCljN6uKpdJbHTOQtn8ZnH+wKXDLAwrDFbLrE/65 nixos@nixos"
];
# Passwordless sudo for wheel — operator SSHes as nixos and uses sudo for
# cluster management commands (drbdadm, crm*, pcs, etc.)
security.sudo.wheelNeedsPassword = lib.mkForce false;
# DRBD lock-file directory (drbd-utils checks for it; missing → harmless but noisy warnings).
systemd.tmpfiles.rules = [ "d /var/lib/drbd 0750 root root -" ];
# Prevent drbd.service from auto-starting at boot / nixos-rebuild switch.
# Pacemaker's OCF drbd agent calls drbdadm up/down directly when managing
# the resource. If drbd.service also runs drbdadm up all while DRBD is
# already Primary under Pacemaker, apply-al fails with "device busy" (exit 20).
systemd.services.drbd.wantedBy = lib.mkForce [];
services.drbd = {
enable = true;
config = ''
global {
usage-count yes;
}
common {
net {
protocol C;
ping-int 1;
verify-alg sha256;
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
}
disk {
fencing resource-only;
}
handlers {
fence-peer "${fencePeerWrapper}";
unfence-peer "${unfencePeerWrapper}";
}
}
resource ha-data {
volume 0 {
device /dev/drbd0;
disk ${vars.haServerDrbdDisk};
meta-disk internal;
}
on ${vars.haServer1Host} {
address ${vars.haServer1StorageIp}:${toString vars.ports.haServerDrbd};
}
on ${vars.haServer2Host} {
address ${vars.haServer2StorageIp}:${toString vars.ports.haServerDrbd};
}
}
'';
};
# /etc/corosync/authkey — sops binary secret, identical on both nodes.
# Decryptable by both ha-server host keys (added by sync-host-keys.sh).
sops.secrets.corosync_authkey = {
sopsFile = ../../secrets/ha-corosync-authkey;
format = "binary";
path = "/etc/corosync/authkey";
mode = "0400";
restartUnits = [ "corosync.service" ];
};
# NixOS common config enables NetworkManager by default; HA cluster nodes
# need stable static IPs with predictable interface names — NM is not suitable.
networking.networkmanager.enable = lib.mkForce false;
# services.corosync.enable is set by modules/ha/pacemaker-stack.nix.
services.corosync = {
clusterName = "ha-cluster";
nodelist = [
{ nodeid = 1; name = vars.haServer1Host; ring_addrs = [ vars.haServer1StorageIp ]; }
{ nodeid = 2; name = vars.haServer2Host; ring_addrs = [ vars.haServer2StorageIp ]; }
];
};
networking.firewall = {
allowedTCPPorts = [
vars.ports.haServerIscsi
vars.ports.haServerPacemakerRemoted
vars.ports.haServerPcsd
vars.ports.haServerDrbd
vars.ports.nfsRpcbind
vars.ports.nfsd
vars.ports.nfsMountd
];
allowedUDPPorts = [
vars.ports.haServerCorosync1
vars.ports.haServerCorosync2
vars.ports.haServerCorosyncCrypto
vars.ports.nfsRpcbind
vars.ports.nfsd
vars.ports.nfsMountd
];
extraCommands = ''
iptables -A INPUT -s ${vars.haServer1Ip}/32 -j ACCEPT
iptables -A INPUT -s ${vars.haServer2Ip}/32 -j ACCEPT
iptables -A INPUT -s ${vars.haStorageCidr} -j ACCEPT
'';
};
}
+99
View File
@@ -0,0 +1,99 @@
# LIO iSCSI target service (targetctl) for NixOS HA clusters.
#
# Provides the targetctl.service that saves/restores LIO configuration from
# /etc/target/saveconfig.json. Pacemaker manages this service via its
# systemd resource agent (class="systemd" type="targetctl").
#
# Why ExecStop is not simply "targetctl save":
# targetctl save writes the LIO config to JSON but does NOT remove the LIO
# target from the kernel's configfs. As a result, any fileio backing store
# that LIO has open (e.g. iscsi-lun.img on an XFS-over-DRBD filesystem)
# stays referenced in the kernel. The subsequent XFS umount from the
# Filesystem OCF resource then returns EBUSY and either hangs for the full
# op-stop timeout or fails outright, blocking the entire failover.
#
# The ExecStop script here additionally tears down the kernel LIO state
# via rtslib_fb after saving, so the backing-store file descriptor is
# released and umount succeeds immediately.
#
# Empty-config guard:
# The save step is skipped when no iSCSI targets are currently active.
# This prevents the secondary node (where LIO was never started) from
# overwriting a valid saveconfig.json with an empty one when Pacemaker
# stops the iscsi-target resource as part of a failover or cleanup.
{ pkgs, ... }:
let
python3 = pkgs.python3.withPackages (ps: [ ps.rtslib-fb ]);
targetctl = "${python3}/bin/targetctl";
targetctlStop = pkgs.writeScript "targetctl-stop" ''
#!${python3}/bin/python3
import subprocess, sys
import rtslib_fb
root = rtslib_fb.RTSRoot()
targets = list(root.targets)
if targets:
subprocess.run(
["${targetctl}", "save", "/etc/target/saveconfig.json"],
capture_output=True,
)
print(f"saved {len(targets)} iSCSI target(s)")
else:
print("no active LIO targets saveconfig.json unchanged")
for target in targets:
try:
for tpg in list(target.tpgs):
tpg.enable = False
target.delete()
except Exception as e:
print(f"warn (target): {e}", file=sys.stderr)
for so in list(root.storage_objects):
try:
so.delete()
except Exception as e:
print(f"warn (backstore): {e}", file=sys.stderr)
print("LIO kernel target cleared")
'';
in
{
boot.kernelModules = [
"target_core_mod"
"iscsi_target_mod"
"target_core_file"
"target_core_pscsi"
"target_core_user"
"configfs"
];
systemd = {
mounts = [{
where = "/sys/kernel/config";
what = "configfs";
type = "configfs";
wantedBy = [ "multi-user.target" ];
before = [ "targetctl.service" ];
}];
services.targetctl = {
description = "LIO iSCSI target config save/restore";
wantedBy = [ "multi-user.target" ];
after = [ "sys-kernel-config.mount" "network.target" ];
requires = [ "sys-kernel-config.mount" ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
ExecStart = "${targetctl} restore /etc/target/saveconfig.json";
ExecStop = "${targetctlStop}";
};
unitConfig.ConditionFileNotEmpty = "/etc/target/saveconfig.json";
};
tmpfiles.rules = [
"d /etc/target 0750 root root -"
"f /etc/target/saveconfig.json 0640 root root -"
];
};
environment.systemPackages = [ pkgs.targetcli-fb ];
}
+94
View File
@@ -0,0 +1,94 @@
# Pacemaker + Corosync HA stack for NixOS with known-good workarounds.
#
# Issues fixed here (confirmed through live testing on NixOS 25.11):
#
# 1. StateDirectory ownership reset: systemd's StateDirectory=pacemaker
# creates /var/lib/pacemaker owned root:root. pacemaker-based (the CIB
# daemon) runs as the hacluster user and calls pcmk__daemon_can_write,
# which requires the CIB directory to be owned by hacluster or be
# group-writable by haclient. Workaround: remove StateDirectory and let
# ExecStartPre create every required subdirectory with correct ownership.
#
# 2. HA_SBIN_DIR wrong path: ocf-shellfuncs sets HA_SBIN_DIR to the Nix
# store path of the resource-agents derivation's /sbin, which doesn't
# exist. The DRBD OCF agent uses ${HA_SBIN_DIR}/crm_master, so it exits
# 127 without this override. Fix: export HA_SBIN_DIR=/run/current-system/sw/bin.
#
# 3. Broad PATH for OCF agents: the resource executor (pacemaker-execd) runs
# OCF agent scripts as children. NixOS provides no implicit PATH for
# system services; without an explicit PATH the agents can't find ip, ss,
# mount, umount, drbdadm, etc.
#
# 4. FUSER=true: the Filesystem OCF agent calls check_binary $FUSER (default:
# fuser from psmisc), which is not installed. Setting FUSER=true makes
# check_binary succeed (true is always in PATH) and the subsequent
# "$FUSER -km $mountpoint" becomes a no-op. Pair with force_unmount=false
# on each Filesystem resource unless you want lazy unmount behaviour.
{ lib, pkgs, ... }:
let
ocfBinPath = lib.concatStringsSep ":" [
"${pkgs.iproute2}/bin"
"${pkgs.iproute2}/sbin"
"${pkgs.iputils}/bin"
"${pkgs.util-linux}/bin"
"${pkgs.util-linux}/sbin"
"${pkgs.gawk}/bin"
"${pkgs.gnugrep}/bin"
"${pkgs.gnused}/bin"
"${pkgs.coreutils}/bin"
"${pkgs.bash}/bin"
"${pkgs.procps}/bin"
"${pkgs.xfsprogs}/bin"
"${pkgs.drbd}/bin"
"${pkgs.python3}/bin"
"/run/current-system/sw/bin"
"/run/current-system/sw/sbin"
"/usr/local/sbin"
"/usr/local/bin"
"/usr/sbin"
"/usr/bin"
"/sbin"
"/bin"
];
# Single pre-start script: schemas symlink + directory ownership.
# Runs before pacemakerd so pacemaker-based finds hacluster-owned dirs.
preStartCmd = "${pkgs.bash}/bin/bash -c '"
+ "ln -sfn ${pkgs.pacemaker}/share/pacemaker /var/lib/pacemaker/schemas; "
+ "for d in /var/lib/pacemaker /var/lib/pacemaker/cib /var/lib/pacemaker/cores "
+ "/var/lib/pacemaker/pengine /var/lib/pacemaker/blackbox "
+ "/var/lib/pacemaker/hostcache; do "
+ "mkdir -p \"\\$d\" && chown hacluster:pacemaker \"\\$d\" && chmod 2770 \"\\$d\"; "
+ "done'";
ocfEnv = {
PATH = lib.mkForce ocfBinPath;
OCF_ROOT = "${pkgs.ocf-resource-agents}/usr/lib/ocf";
HA_SBIN_DIR = "/run/current-system/sw/bin";
FUSER = "true";
};
in
{
users.groups.haclient = { };
services.corosync.enable = true;
services.pacemaker.enable = true;
systemd.services = {
pacemaker = {
serviceConfig = {
StateDirectory = lib.mkForce "";
ExecStartPre = lib.mkBefore [ preStartCmd ];
};
environment = ocfEnv;
};
pacemaker-execd.environment = ocfEnv;
};
environment.systemPackages = with pkgs; [
corosync
pacemaker
ocf-resource-agents
];
}
+210
View File
@@ -0,0 +1,210 @@
# Fully declarative FreeIPA domain membership.
#
# Imported by modules/common/configuration.nix — no per-host wiring needed.
# Enables itself automatically on any host that has a sops-encrypted keytab
# at secrets/<hostname>.keytab; is a no-op for all other hosts.
#
# To enroll a new host:
# 0. scripts/secrets/sync-host-keys.sh <flake-target>
# 1. scripts/ipa/create-nixos-ipa-host-account.sh [--ip <addr>] <hostname>
# (adds .sops.yaml rule, runs ipa host-add, encrypts keytab in one step)
# 2. git add secrets/<hostname>.keytab .sops.yaml && git commit
# 3. Deploy — no further steps required.
#
# Manual fallback (if the script isn't usable):
# a. On the FreeIPA server: ipa host-add <fqdn> [--ip-address=<ip>] --force
# b. On the FreeIPA server: ipa-getkeytab -s <ipa-server> -p host/<fqdn> -k /tmp/<host>.keytab
# c. From the repo root (path must match for sops creation rule to apply):
# cp /tmp/<host>.keytab secrets/<host>.keytab
# sops -e --input-type binary -i secrets/<host>.keytab
# d. Commit secrets/<host>.keytab and the updated .sops.yaml, then deploy.
#
# vars dependencies: homeDomain, ipaServer, domainControllerIp, ipaUser
{ config, lib, pkgs, vars, ... }:
let
keytabPath = ../../secrets + "/${config.networking.hostName}.keytab";
enabled = builtins.pathExists keytabPath;
realm = lib.strings.toUpper vars.homeDomain;
fqdn = "${config.networking.hostName}.${vars.homeDomain}";
# "sweet.home" -> "dc=sweet,dc=home"
basedn = lib.strings.concatMapStringsSep "," (c: "dc=${c}") (lib.strings.splitString "." vars.homeDomain);
# security.ipa.certificate expects a derivation (package), not a raw path.
caCertPkg = pkgs.writeText "ipa-ca.crt" (builtins.readFile ../../certs/ipa-ca.crt);
in
lib.mkIf enabled {
networking.domain = lib.mkDefault vars.homeDomain;
networking.nameservers = lib.mkDefault [ vars.domainControllerIp ];
security = {
ipa = {
enable = true;
domain = vars.homeDomain;
inherit realm;
server = vars.ipaServer;
certificate = caCertPkg;
inherit basedn;
ipaHostname = fqdn;
offlinePasswords = true;
cacheCredentials = true;
};
# Create the home directory on first login if it doesn't exist yet.
# IPA users have no pre-created home on the host; without this sshd
# opens a session to a non-existent directory and resets the connection.
# lightdm also needs this so the GUI login path can create the home dir
# if it was not pre-seeded by the tmpfiles rule above (e.g. on first boot
# before SSSD has resolved the user).
pam.services = {
sshd.makeHomeDir = true;
lightdm.makeHomeDir = true;
# pam_unix returns PAM_AUTHINFO_UNAVAIL without prompting when the local
# stub has "!" in shadow (account locked), so PAM_AUTHTOK is never set
# and pam_sss's use_first_pass fails with "No authentication token".
# Changing to try_first_pass makes pam_sss prompt independently when no
# prior module has set the token, restoring IPA password login via
# LightDM and su.
login.rules.auth.sss.settings = lib.mkForce { try_first_pass = true; };
su.rules.auth.sss.settings = lib.mkForce { try_first_pass = true; };
};
# HM with useUserPackages = true (flake.nix) sets users.users.${ipaUser}.packages,
# which forces the stub into /etc/passwd. pam_sss.so with the "localusers" flag
# (added by NixOS when SSSD is enabled) then skips SSSD for any user it finds in
# local /etc/passwd — including this stub — falling through to pam_unix, which has
# no password for the stub → sudo auth always fails.
#
# Fix: NOPASSWD for the IPA user. The IPA user already authenticated to reach a
# shell (SSH public key from IPA or Kerberos), so re-prompting via a broken PAM
# path is security theater on a single-admin homelab.
sudo.extraRules = [{
users = [ vars.ipaUser ];
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
}];
};
systemd = {
# Fetch SSH public keys from IPA so users can log in with the key stored
# in their IPA profile rather than needing ~/.ssh/authorized_keys on every
# host. sss_ssh_authorizedkeys queries SSSD (which queries IPA LDAP).
#
# /nix/store is 1775 (group-writable by nixbld). OpenSSH 10.0+ rejects
# AuthorizedKeysCommand binaries whose path contains any group-writable
# component, silently skipping the command. Copy to /usr/local/bin (all
# components root-owned, 755) so the path passes sshd's safety check.
tmpfiles.rules = [
"d /usr/local 0755 root root - -"
"d /usr/local/bin 0755 root root - -"
"C+ /usr/local/bin/sss_ssh_authorizedkeys 0555 root root - ${pkgs.sssd}/bin/sss_ssh_authorizedkeys"
# Pre-create the IPA user's home dir so Home Manager activation succeeds
# even before their first login. On a fresh system SSSD may not have
# resolved the user yet — tmpfiles warns and skips in that case (non-fatal),
# and pam_mkhomedir covers the first-login path as a fallback.
"d /home/${vars.ipaUser} 0700 ${vars.ipaUser} ${vars.ipaUser} - -"
];
# security.ipa enables Kerberos (security.krb5) which causes systemd to
# start auth-rpcgss-module.service and rpc-gssd.service for Kerberos NFS
# authentication. LXC containers can't load the auth_rpcgss kernel module
# and don't have /var/lib/nfs/rpc_pipefs, so both services fail.
#
# The NixOS IPA module already adds a drop-in for auth-rpcgss-module.service
# with ConditionPathExists=/etc/krb5.keytab. We use lib.mkForce to win the
# text conflict and add ConditionVirtualization=!container alongside it so
# the service is skipped (not failed) in containers that do have a keytab.
# Same fix for rpc-gssd.service which also fails in containers.
units = lib.mkIf config.boot.isContainer {
"auth-rpcgss-module.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
# rpc-gssd also has ConditionPathExists from the NixOS IPA module (and an
# X-Restart-Triggers store path from systemd.nix). Use mkForce to win;
# omit X-Restart-Triggers since this service is skipped in containers anyway.
"rpc-gssd.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
};
# home-manager-<user>.service fails on first enrollment because /home/wayne
# doesn't exist until the user's first login (pam_mkhomedir creates it then).
# ConditionPathExists makes systemd skip the service (exit 0, condition not
# met) instead of failing. After first login the dir exists and subsequent
# rebuilds activate HM normally.
services."home-manager-${vars.ipaUser}".unitConfig.ConditionPathExists =
"/home/${vars.ipaUser}";
};
services.openssh.extraConfig = ''
AuthorizedKeysCommand /usr/local/bin/sss_ssh_authorizedkeys %u
AuthorizedKeysCommandUser nobody
'';
# Host keytab: pre-provisioned on the IPA server, sops-encrypted binary.
# Placed at /etc/krb5.keytab before SSSD starts so the host authenticates
# to IPA without running ipa-client-install.
sops.secrets."ipa-host-keytab" = {
sopsFile = keytabPath;
format = "binary";
path = "/etc/krb5.keytab";
owner = "root";
group = "root";
mode = "0600";
restartUnits = [ "sssd.service" ];
};
# NixOS requires isNormalUser/isSystemUser + group on any entry in
# users.users. HM with useUserPackages = true (set in flake.nix) adds a stub
# entry for each HM user so it can install packages to
# /etc/profiles/per-user/<name>/. This definition satisfies those assertions.
# With security.ipa setting "passwd: sss files" in nsswitch, SSSD's IPA entry
# takes priority for NSS lookups — this local stub is only a fallback when
# SSSD is unreachable (at which point auth fails anyway).
users.users.${vars.ipaUser} = {
isNormalUser = true;
group = "users";
extraGroups = [ "wheel" ];
createHome = false;
# "!" is not a password hash — it is the standard "account locked" marker.
# It cannot authenticate anyone locally. It exists solely so NixOS generates
# a shadow entry for this stub user; without one pam_unix returns
# PAM_AUTHINFO_UNAVAIL before prompting, which means PAM_AUTHTOK is never
# set and the subsequent pam_sss use_first_pass call has nothing to work
# with — blocking LightDM and su logins even when IPA/SSSD auth succeeds.
hashedPassword = "!";
};
# Home Manager config for the IPA primary user, applied on every enrolled
# host. Manages what IPA doesn't: dotfiles, user-scoped packages, session
# variables. Switch-nix/Test-nix/buildImage are system-wide (configuration.nix)
# so they don't need to be repeated here.
#
# homeDirectory uses mkForce because HM's NixOS integration module sets it to
# "/var/empty" for users not found in config.users.users at eval time (SSSD
# users aren't visible there).
home-manager.users.${vars.ipaUser} = { pkgs, ... }: {
home = {
username = vars.ipaUser;
homeDirectory = lib.mkForce "/home/${vars.ipaUser}";
stateVersion = "26.05";
packages = with pkgs; [ tmux sshfs ];
sessionVariables.EDITOR = lib.mkDefault "nano";
};
programs.home-manager.enable = true;
programs.bash.enable = true;
};
}
+17 -60
View File
@@ -52,6 +52,7 @@ in
# LXC container does). # LXC container does).
imports = [ imports = [
(modulesPath + "/virtualisation/proxmox-lxc.nix") (modulesPath + "/virtualisation/proxmox-lxc.nix")
../common/preserve-ssh-host-key.nix
]; ];
proxmoxLXC = { proxmoxLXC = {
@@ -63,23 +64,22 @@ in
# back to decide `pct create`'s --unprivileged flag, so the two stay # back to decide `pct create`'s --unprivileged flag, so the two stay
# in sync). # in sync).
# #
# lxc-docker is the one exception: the kernel's NFS client doesn't set # Any lxc-* host with an NFS fileSystem must be privileged: the kernel's
# FS_USERNS_MOUNT, so mounting NFS from inside *any* non-init user # NFS client doesn't set FS_USERNS_MOUNT, so mounting NFS from inside
# namespace -- which is exactly what an unprivileged container's # *any* non-init user namespace -- which is exactly what an unprivileged
# UID-mapped root runs in -- is rejected at the VFS layer with EPERM, # container's UID-mapped root runs in -- is rejected at the VFS layer
# no matter what Proxmox's own `mount=nfs;nfs4` container feature # with EPERM, no matter what Proxmox's own `mount=nfs;nfs4` container
# allows at the AppArmor layer (confirmed live: TCP to the NFS server # feature allows at the AppArmor layer (confirmed live: TCP to the NFS
# succeeds, the server's export table matches the container's IP, and # server succeeds, the server's export table matches the container's IP,
# `mount.nfs: Operation not permitted` still fires immediately with no # and `mount.nfs: Operation not permitted` still fires immediately with
# corresponding denial anywhere in the server's logs -- a kernel-level # no corresponding denial anywhere in the server's logs -- a kernel-level
# rejection, not a network or export-permission one). Keying off # rejection, not a network or export-permission one). Deriving this from
# hostName rather than something docker-build-type-specific because # fileSystems rather than a per-host override keeps it self-consistent:
# modules/build-types/docker.nix is also composed for linode-docker/ # any new lxc-* host that declares an NFS mount automatically gets the
# proxmox-docker, which don't import proxmox-lxc.nix at all --setting # privilege level it needs without a separate manual flag.
# this option there would break their eval with "option does not privileged = builtins.any
# exist" regardless of any mkIf guard, since mkIf only makes a value (fs: fs.fsType == "nfs" || fs.fsType == "nfs4")
# conditional, not whether the option needs to exist somewhere. (builtins.attrValues config.fileSystems);
privileged = config.networking.hostName == "docker";
}; };
boot.loader = { boot.loader = {
@@ -106,49 +106,6 @@ in
}; };
}; };
# NixOS's etc activation removes any /etc file that was in the previous
# generation's environment.etc but is absent from the current one — even
# real (non-symlink) copies. On every routine nixos-rebuild switch/test that
# lacks NIXOS_HOST_KEYS_DIR the key is absent from environment.etc, so it
# gets removed as "obsolete". sops-nix derives its age decryption key from
# /etc/ssh/ssh_host_ed25519_key; deletion cascades into every sops secret
# failing with "Error getting data key: 0 successful groups required, got 0".
#
# Fix: activation scripts that bracket the etc step, with explicit deps
# to enforce the correct ordering. Without deps the topological sort places
# preserveSshHostKey AFTER etc (confirmed live on a deployed lxc-tor-relay:
# position 7 vs etc's position 5) -- the key is already gone by the time it
# tries to save it. The etc/setupSecrets entries ADD to existing deps
# (types.listOf concatenates across module definitions).
system.activationScripts = {
# Saves the live key to /run before etc can delete it.
preserveSshHostKey = ''
if [ -f /etc/ssh/ssh_host_ed25519_key ]; then
cp /etc/ssh/ssh_host_ed25519_key /run/sshd-host-key-preserve.tmp
cp /etc/ssh/ssh_host_ed25519_key.pub /run/sshd-host-key-preserve.pub.tmp
fi
'';
# Reinstalls the key after etc runs if it was removed as "obsolete".
# The resulting file is not registered in environment.etc for either
# generation, so subsequent rebuilds leave it alone permanently.
restoreSshHostKey = {
deps = [ "etc" ];
text = ''
if [ ! -f /etc/ssh/ssh_host_ed25519_key ] && [ -f /run/sshd-host-key-preserve.tmp ]; then
install -m 0600 /run/sshd-host-key-preserve.tmp /etc/ssh/ssh_host_ed25519_key
install -m 0644 /run/sshd-host-key-preserve.pub.tmp /etc/ssh/ssh_host_ed25519_key.pub
fi
rm -f /run/sshd-host-key-preserve.tmp /run/sshd-host-key-preserve.pub.tmp
'';
};
# Force etc to wait until the key is saved, and sops to wait until the
# key is restored. Without these the topological sort breaks the chain.
etc = { deps = [ "preserveSshHostKey" ]; };
setupSecrets = { deps = [ "restoreSshHostKey" ]; };
};
# virtualisation/proxmox-lxc.nix (imported above) registers the Nix # virtualisation/proxmox-lxc.nix (imported above) registers the Nix
# store DB via a systemd service (register-nix-paths) -- it never runs # store DB via a systemd service (register-nix-paths) -- it never runs
# an activation script at all. Confirmed live this means neither # an activation script at all. Confirmed live this means neither
+42 -1
View File
@@ -1,9 +1,50 @@
{ ... }: { lib, flakeTarget, ... }:
let
# Bakes this exact flake target's pre-generated SSH host key straight
# into /etc/ssh/ -- mirrors lxc.nix's builtins.getEnv pattern (impure
# and empty under normal `nix build`/`nix eval`, so this is a no-op
# unless explicitly opted into with NIXOS_HOST_KEYS_DIR=... --impure).
#
# Unlike --pre-format-files (which places files on the QEMU builder VM's
# rootfs, not the target disk), embedding via environment.etc here means
# nixos-install's own activation step installs the key onto the target
# disk. sshd-keygen then finds it already present and skips generation,
# so the disk image boots with the clan-registered key and sops can
# decrypt on first boot.
#
# Without this, nixos-install's sshd-keygen activation generates a fresh
# key (unregistered in .sops.yaml), sops decryption fails permanently,
# and password hashes are never applied -- confirmed live: passwords
# stayed '!' even with mutableUsers = false because hashedPasswordFile
# pointed to a path that sops never wrote.
hostKeysDirStr = builtins.getEnv "NIXOS_HOST_KEYS_DIR";
hasHostKeysDir = hostKeysDirStr != "" && builtins.pathExists hostKeysDirStr;
hostKeysDir = /. + hostKeysDirStr;
privKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key";
pubKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key.pub";
hasKeyForThisTarget =
hasHostKeysDir
&& builtins.pathExists privKeyFile
&& builtins.pathExists pubKeyFile;
in
{ {
imports = [ imports = [
../hardware-configuration/vm/proxmox.nix ../hardware-configuration/vm/proxmox.nix
../boot/efi.nix ../boot/efi.nix
../disko/proxmox.nix ../disko/proxmox.nix
../common/preserve-ssh-host-key.nix
]; ];
environment.etc = lib.mkIf hasKeyForThisTarget {
"ssh/ssh_host_ed25519_key" = {
source = privKeyFile;
mode = "0600";
};
"ssh/ssh_host_ed25519_key.pub" = {
source = pubKeyFile;
mode = "0644";
};
};
} }
+42
View File
@@ -0,0 +1,42 @@
{ config, lib, vars, ... }:
let
# Use the same FQDN approach as docker/mount-data.nix — a bare hostname is
# unreliable: systemd-resolved only tries LLMNR for single-label names, and
# a global search domain causes it to skip the interface-scoped LAN DNS.
nfsServer = "${vars.nfsServerHost}.${vars.homeDomain}";
in
{
fileSystems.${vars.nfsShares.pxebootImages.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.pxebootImages.subpath}";
fsType = "nfs";
options = [
"_netdev"
"noatime"
] ++ (if config.boot.isContainer
# NFSv4 requires rpc_pipefs (sunrpc filesystem), which Proxmox LXC
# containers block unless `features: mount=nfs` is set. Use NFSv3+nolock
# instead: no rpc_pipefs dependency at the protocol level, and rpcbind
# on the server handles port resolution without needing client-side
# sunrpc infrastructure. nofail keeps boot clean if server is unreachable.
then [ "nfsvers=3" "proto=tcp" "nolock" "nofail" ]
else [ "nfsvers=4.2" "x-systemd.automount" ]);
};
# NixOS pulls var-lib-nfs-rpc_pipefs.mount (the sunrpc filesystem) into
# nfs-client.target for any nfs fileSystems entry. In LXC containers the
# sunrpc mount is blocked by Proxmox's AppArmor profile, causing it to fail
# and the activation to report an error even though our mount uses nofail.
# Add ConditionVirtualization=!container via drop-in so systemd skips the
# unit entirely in containers (skip = inactive, not failed), which keeps
# nfs-client.target green and activation clean.
systemd.units = lib.mkIf config.boot.isContainer {
"var-lib-nfs-rpc_pipefs.mount" = {
overrideStrategy = "asDropin";
text = ''
[Unit]
ConditionVirtualization=!container
'';
};
};
}
+21 -1
View File
@@ -1,4 +1,4 @@
_: { pkgs, ... }:
{ {
imports = [ ./enable-service.nix ]; imports = [ ./enable-service.nix ];
@@ -12,4 +12,24 @@ _:
# instead of relaying through DERP. # instead of relaying through DERP.
openFirewall = true; openFirewall = true;
}; };
# Tailscale recommends these ethtool flags on the uplink interface to get
# full UDP GRO throughput on subnet routers (https://tailscale.com/s/ethtool-config-udp-gro).
# The interface is derived from the default route so it works regardless of
# what the NIC is named on a given host.
systemd.services.tailscale-udp-gro = {
description = "Enable UDP GRO forwarding on uplink for Tailscale subnet router";
after = [ "network-online.target" ];
wants = [ "network-online.target" ];
wantedBy = [ "multi-user.target" ];
path = [ pkgs.ethtool pkgs.iproute2 ];
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
ExecStart = pkgs.writeShellScript "tailscale-udp-gro" ''
NETDEV=$(ip -o route get 8.8.8.8 | cut -f 5 -d " ")
ethtool -K "$NETDEV" rx-udp-gro-forwarding on rx-gro-list off
'';
};
};
} }
+60
View File
@@ -0,0 +1,60 @@
{ vars, ... }:
{
# Run dnsmasq on the LAN interface as a forwarding-only resolver for
# *.ts.net (Tailscale MagicDNS names). FreeIPA's bind-dyndb-ldap
# cannot reach 100.100.100.100 (Tailscale's internal resolver) directly
# because the DC is not a Tailscale node. This host IS a Tailscale node
# and can reach 100.100.100.100 via its tailscale0 interface, so it
# acts as an intermediary: FreeIPA has a conditional forward zone for
# ts.net pointing here (vars.tailscaleRouterIp), and this dnsmasq
# instance forwards those queries onward to Tailscale's resolver.
#
# Configure FreeIPA once after deploying this host:
# kinit admin
# ipa dnsforwardzone-add ${vars.tailnetDomain} \
# --forwarder=${vars.tailscaleRouterIp} \
# --forward-policy=only
# Note: IPA refuses to shadow ts.net (a real public TLD); use the
# tailnet-specific subdomain (vars.tailnetDomain) instead.
services.dnsmasq = {
enable = true;
# NixOS's dnsmasq module defaults resolveLocalQueries to true, which adds
# 127.0.0.1 to networking.nameservers and makes dnsmasq bind to
# listen-address=127.0.0.1. This instance is not the host's local
# resolver — it only serves IPA's conditional forwarder for tailnet names.
# The host uses domainControllerIp directly (networking.nameservers in
# host.nix). Without this, all host DNS goes through dnsmasq, which has
# no upstream for general queries (no-resolv=true), breaking resolution.
resolveLocalQueries = false;
settings = {
# Listen only on the LAN interface — not tailscale0 or loopback.
# bind-interfaces prevents dnsmasq from binding to 0.0.0.0 and
# then filtering by interface later; combined with `interface` this
# ensures it genuinely listens only on eth0.
bind-interfaces = true;
interface = [ vars.lxcLanInterface ];
# Forward-only: no local /etc/hosts or /etc/resolv.conf reading,
# no negative caching of NXDOMAIN for names this instance doesn't
# serve. All ts.net queries come from FreeIPA's conditional forwarder
# and must be answered by Tailscale's resolver.
no-hosts = true;
no-resolv = true;
# Tailscale's internal "Quad100" resolver — reachable from any
# Tailscale node via the tailscale0 interface. Scoped to the
# specific tailnet subdomain (vars.tailnetDomain) rather than
# all of ts.net: FreeIPA refuses to shadow ts.net (a real public
# TLD with DNSimple nameservers) so the conditional forward zone
# in FreeIPA must use the tailnet-specific subdomain instead:
# ipa dnsforwardzone-add ${vars.tailnetDomain} \
# --forwarder=${vars.tailscaleRouterIp} \
# --forward-policy=only
server = [ "/${vars.tailnetDomain}/100.100.100.100" ];
};
};
networking.firewall.allowedUDPPorts = [ 53 ];
networking.firewall.allowedTCPPorts = [ 53 ];
}
Binary file not shown.
+11
View File
@@ -88,6 +88,17 @@ export NIX_CACHE_HOST
: "${LAN_DOMAIN:=gitea.lan.ddnsgeek.com}" : "${LAN_DOMAIN:=gitea.lan.ddnsgeek.com}"
export LAN_DOMAIN export LAN_DOMAIN
# Matches variables.nix's homeDomain -- the base LAN domain for service
# subdomains, FreeIPA Kerberos realm, and host FQDNs.
: "${HOME_DOMAIN:=sweet.home}"
export HOME_DOMAIN
# Matches variables.nix's ipaServer -- the FreeIPA server hostname.
# scripts/ipa/create-nixos-ipa-host-account.sh SSHes here to run
# ipa host-add and ipa-getkeytab.
: "${IPA_SERVER:=domain-controller.sweet.home}"
export IPA_SERVER
# nix_extra_opts: call as a plain statement (NOT inside $(...)/<(...) -- # nix_extra_opts: call as a plain statement (NOT inside $(...)/<(...) --
# that forks a subshell, and the whole point is exporting a decision back # that forks a subshell, and the whole point is exporting a decision back
# into *this* shell) to populate the global NIX_OPTS array with whatever # into *this* shell) to populate the global NIX_OPTS array with whatever
+231
View File
@@ -0,0 +1,231 @@
#!/usr/bin/env bash
# acceptance-tests.sh — HA cluster acceptance tests (T1T7)
#
# Run from a host with SSH access to both HA nodes (or from node1 itself).
# All 7 tests must pass before considering the cluster production-ready.
# Test values below must match variables.nix haServer* values.
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
# All values override-able via environment variables; defaults match variables.nix.
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.2.229}" # vars.haServerVip
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
ISCSI_IQN="${ISCSI_IQN:-iqn.2026-01.home.sweet:ha-storage}" # vars.haIscsiIqn
# ──────────────────────────────────────────────────────────────────────────
PASS=0
FAIL=0
RESULTS=()
# Use PASS=$((PASS+1)) instead of ((PASS++)) — the latter evaluates to 0 when
# PASS=0, which triggers set -e and kills the script after the very first PASS.
pass() { echo " PASS: $1"; PASS=$((PASS+1)); RESULTS+=("PASS $1"); }
fail() { echo " FAIL: $1"; FAIL=$((FAIL+1)); RESULTS+=("FAIL $1"); }
HA_USER="nixos"
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
echo "════════════════════════════════════════════════════"
echo " HA Cluster Acceptance Tests — $(date '+%Y-%m-%d %H:%M:%S')"
echo "════════════════════════════════════════════════════"
# ── Pre-flight: DRBD sync must be complete ────────────────────────────────
# Tests that check disk state, XFS mount, and iSCSI will fail or give false
# results while the initial full sync is in progress. Block until done.
echo ""
echo "Pre-flight: verifying DRBD sync is complete..."
DRBD_PREFLIGHT=$(n1 "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_PREFLIGHT" | grep -q "^UpToDate/UpToDate$"; then
echo ""
echo " ERROR: DRBD initial sync not complete."
echo " Current dstate on $NODE1: $DRBD_PREFLIGHT"
echo ""
echo " Monitor progress:"
echo " ssh nixos@$NODE1_IP 'sudo watch -n3 cat /proc/drbd'"
echo ""
echo " Re-run this script once dstate shows UpToDate/UpToDate."
exit 1
fi
echo " dstate: $DRBD_PREFLIGHT — ready."
# ── Detect Active/Standby nodes ────────────────────────────────────────────
# Use crm_mon to detect which node holds the Promoted (Primary) DRBD resource.
# Pacemaker is authoritative; DRBD role can briefly read as Secondary while
# Pacemaker is mid-transition, giving a false Active/Standby swap.
# Wait up to 90 s for Pacemaker to settle before giving up.
echo ""
echo "Detecting Active/Standby nodes (waiting for Pacemaker to settle)..."
ACTIVE_NODE=""
for i in $(seq 1 30); do
# crm_mon -1 output contains "Promoted: [ <node> ]" for the DRBD master.
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || n2 "crm_mon -1" 2>/dev/null || true)
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" — the * bullet
# means ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | grep -E '(Promoted|Masters):' | grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
[[ -n "$ACTIVE_NODE" ]] && break
sleep 3
done
if [[ -z "$ACTIVE_NODE" ]]; then
echo " WARNING: could not determine Active node from crm_mon after 90 s — defaulting to $NODE1"
ACTIVE_NODE="$NODE1"
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
STANDBY_NODE="$NODE2"; STANDBY_IP="$NODE2_IP"
na() { n1 "$@"; }
ns() { n2 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
STANDBY_NODE="$NODE1"; STANDBY_IP="$NODE1_IP"
na() { n2 "$@"; }
ns() { n1 "$@"; }
fi
echo " Active: $ACTIVE_NODE ($ACTIVE_IP)"
echo " Standby: $STANDBY_NODE ($STANDBY_IP)"
# ── T1: Corosync quorum established ──────────────────────────────────────
echo ""
echo "[T1] Corosync quorum"
if na "corosync-quorumtool -s" 2>/dev/null | grep -q "Quorate:.*Yes"; then
pass "cluster has quorum"
else
fail "cluster does not have quorum — check corosync on both nodes"
fi
# ── T2: DRBD Primary on Active node, Secondary on Standby ────────────────
echo ""
echo "[T2] DRBD roles"
DRBD_ROLE=$(na "drbdadm role ha-data" 2>/dev/null || echo "unknown")
if [[ "$DRBD_ROLE" == "Primary/Secondary" || "$DRBD_ROLE" == "Primary" ]]; then
pass "DRBD Primary on $ACTIVE_NODE ($DRBD_ROLE)"
else
fail "unexpected DRBD role on $ACTIVE_NODE: $DRBD_ROLE (expected Primary/Secondary)"
fi
DRBD_DSTATE=$(na "drbdadm dstate ha-data" 2>/dev/null || echo "unknown")
if echo "$DRBD_DSTATE" | grep -q "UpToDate"; then
pass "DRBD disk state UpToDate ($DRBD_DSTATE)"
else
fail "DRBD disk not UpToDate: $DRBD_DSTATE"
fi
# ── T3: XFS mounted at haStorageRoot on the Active node ──────────────────
echo ""
echo "[T3] XFS mount"
if na "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
pass "XFS mounted at ${XFS_MOUNT} on $ACTIVE_NODE"
else
fail "XFS not mounted at ${XFS_MOUNT} on $ACTIVE_NODE"
fi
if ns "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
fail "XFS unexpectedly mounted on $STANDBY_NODE (should only be on Active node)"
else
pass "XFS not mounted on $STANDBY_NODE (correct — Standby)"
fi
# ── T4: iSCSI target visible on Active node ───────────────────────────────
echo ""
echo "[T4] iSCSI target"
IQN_COUNT=$(na "bash -c 'ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn || true'" 2>/dev/null || echo "0")
if [[ "$IQN_COUNT" -ge 1 ]]; then
pass "iSCSI IQN active on $ACTIVE_NODE ($IQN_COUNT target(s))"
else
fail "no iSCSI IQN active on $ACTIVE_NODE"
fi
# iSCSI port reachable from Standby node via VIP.
# Use bash TCP probe (no iscsiadm needed — just checks port 3260 is open).
if ns "bash -c 'echo >/dev/tcp/${VIP}/3260' 2>/dev/null"; then
pass "iSCSI port 3260 reachable from $STANDBY_NODE via VIP ${VIP}"
else
fail "iSCSI port 3260 not reachable from $STANDBY_NODE via ${VIP}"
fi
# ── T5: Failover — standby Active node, verify resources move to Standby ──
echo ""
echo "[T5] Failover (standby $ACTIVE_NODE)"
ACTIVE_CRMD_NAME=$(na "crm_node -n" 2>/dev/null || echo "")
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v on" 2>/dev/null || true
echo " Waiting up to 120 s for resources to move to $STANDBY_NODE..."
MOVED=false
for i in $(seq 1 120); do
if ns "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
MOVED=true
echo " Resources moved in ${i}s"
break
fi
sleep 1
done
if $MOVED; then
pass "XFS mounted on $STANDBY_NODE after failover"
IQN_ON_STANDBY=$(ns "bash -c 'ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn || true'" 2>/dev/null || echo "0")
[[ "$IQN_ON_STANDBY" -ge 1 ]] \
&& pass "iSCSI target active on $STANDBY_NODE after failover" \
|| fail "iSCSI target NOT active on $STANDBY_NODE after failover"
else
fail "XFS did not mount on $STANDBY_NODE within 120 s — failover incomplete"
fi
# ── T6: Data integrity — file written post-failover readable ─────────────
echo ""
echo "[T6] Data integrity"
# Write a test file on the new Active (former Standby) and verify it.
# Use `echo | sudo tee` for the write: "echo ... > file" via bash -c has the
# redirect interpreted by the remote nixos shell (not sudo), so the file open
# runs as nixos and fails with EACCES on the root-owned XFS mount. Piping
# through sudo tee lets tee (running as root) open the file instead.
TEST_FILE="${XFS_MOUNT}/.acceptance-test-$$"
TEST_CONTENT="ha-acceptance-test-$(date +%s)"
echo "${TEST_CONTENT}" | ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${STANDBY_IP}" sudo tee "${TEST_FILE}" > /dev/null 2>/dev/null || true
READBACK=$(ns cat "${TEST_FILE}" 2>/dev/null || echo "")
if [[ "$READBACK" == "$TEST_CONTENT" ]]; then
pass "test file written and read back correctly on $STANDBY_NODE"
else
fail "data integrity check failed (wrote: '$TEST_CONTENT', read: '$READBACK')"
fi
ns rm -f "${TEST_FILE}" 2>/dev/null || true
# ── T7: Node rejoin — un-standby original Active, verify cluster is healthy ─
echo ""
echo "[T7] Node rejoin"
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
na "crm_resource --cleanup" 2>/dev/null || true
sleep 5
if na "corosync-quorumtool -s 2>/dev/null | grep -q 'Quorate:.*Yes'"; then
pass "$ACTIVE_NODE rejoined — cluster has quorum"
else
fail "$ACTIVE_NODE did not rejoin with quorum"
fi
DRBD_ROLE_AFTER=$(na "drbdadm role ha-data" 2>/dev/null || echo "unknown")
if echo "$DRBD_ROLE_AFTER" | grep -q "Secondary"; then
pass "$ACTIVE_NODE is DRBD Secondary after rejoin ($DRBD_ROLE_AFTER)"
else
fail "unexpected DRBD role on $ACTIVE_NODE after rejoin: $DRBD_ROLE_AFTER"
fi
# ── Summary ───────────────────────────────────────────────────────────────
echo ""
echo "════════════════════════════════════════════════════"
echo " Results: ${PASS} PASS, ${FAIL} FAIL"
echo "════════════════════════════════════════════════════"
for r in "${RESULTS[@]}"; do echo " $r"; done
echo ""
if [[ "$FAIL" -eq 0 ]]; then
echo "ALL PASS — cluster is production-ready."
exit 0
else
echo "SOME TESTS FAILED — investigate before deploying."
exit 1
fi
+86
View File
@@ -0,0 +1,86 @@
#!/usr/bin/env bash
# cluster-enable-stonith.sh — enable STONITH fence agent after the fence SSH
# key is deployed to both nodes and authorised on the Proxmox host.
#
# Run from ha-server-1 as root AFTER:
# - /etc/pacemaker/fence_pve_ssh exists on both nodes (chmod +x)
# (copy from scripts/ha/fence-pve-ssh.py)
# - /etc/fence-pve-ssh-key (SSH private key) exists on both nodes
# - The corresponding public key is in authorized_keys on PVE_HOST
# - VMID_NODE1 / VMID_NODE2 filled in below
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="ha-server-1"
NODE2="ha-server-2"
VMID_NODE1="" # FILL IN: Proxmox VMID for ha-server-1
VMID_NODE2="" # FILL IN: Proxmox VMID for ha-server-2
PVE_HOST="pve1.sweet.home"
PVE_USER="wayne"
FENCE_KEY="/etc/fence-pve-ssh-key"
FENCE_SCRIPT="/etc/pacemaker/fence_pve_ssh"
# ──────────────────────────────────────────────────────────────────────────
log() { echo "[stonith-setup] $*"; }
die() { echo "[stonith-setup] ERROR: $*" >&2; exit 1; }
[[ $(id -u) -eq 0 ]] || die "must run as root"
[[ -n "$VMID_NODE1" ]] || die "VMID_NODE1 not set — edit this script"
[[ -n "$VMID_NODE2" ]] || die "VMID_NODE2 not set — edit this script"
[[ -f "$FENCE_KEY" ]] || die "fence key not found at $FENCE_KEY"
[[ -f "$FENCE_SCRIPT" ]] || die "fence script not found at $FENCE_SCRIPT"
log "Verifying fence agent can reach ${PVE_HOST}..."
ssh -i "$FENCE_KEY" -o BatchMode=yes -o ConnectTimeout=10 \
-o StrictHostKeyChecking=no "${PVE_USER}@${PVE_HOST}" \
"sudo /usr/sbin/qm list" &>/dev/null \
|| die "Cannot SSH to ${PVE_USER}@${PVE_HOST} — check authorized_keys and sudo"
log "Fence agent SSH connectivity confirmed"
log "Creating Pacemaker STONITH resources..."
cibadmin --create --scope resources --xml-text "
<primitive id=\"stonith-${NODE1}\" class=\"stonith\" type=\"external/fence_pve_ssh\">
<instance_attributes id=\"stonith-${NODE1}-attrs\">
<nvpair id=\"stonith-${NODE1}-plug\" name=\"plug\" value=\"${NODE1}\"/>
<nvpair id=\"stonith-${NODE1}-pve-host\" name=\"pve_host\" value=\"${PVE_HOST}\"/>
<nvpair id=\"stonith-${NODE1}-pve-user\" name=\"pve_user\" value=\"${PVE_USER}\"/>
<nvpair id=\"stonith-${NODE1}-key-file\" name=\"key_file\" value=\"${FENCE_KEY}\"/>
<nvpair id=\"stonith-${NODE1}-vmid1\" name=\"vmid_node1\" value=\"${VMID_NODE1}\"/>
<nvpair id=\"stonith-${NODE1}-vmid2\" name=\"vmid_node2\" value=\"${VMID_NODE2}\"/>
<nvpair id=\"stonith-${NODE1}-host-list\" name=\"pcmk_host_list\" value=\"${NODE1}\"/>
</instance_attributes>
<operations>
<op id=\"stonith-${NODE1}-monitor\" name=\"monitor\" interval=\"30s\" timeout=\"30s\"/>
</operations>
</primitive>
" 2>/dev/null || true
cibadmin --create --scope resources --xml-text "
<primitive id=\"stonith-${NODE2}\" class=\"stonith\" type=\"external/fence_pve_ssh\">
<instance_attributes id=\"stonith-${NODE2}-attrs\">
<nvpair id=\"stonith-${NODE2}-plug\" name=\"plug\" value=\"${NODE2}\"/>
<nvpair id=\"stonith-${NODE2}-pve-host\" name=\"pve_host\" value=\"${PVE_HOST}\"/>
<nvpair id=\"stonith-${NODE2}-pve-user\" name=\"pve_user\" value=\"${PVE_USER}\"/>
<nvpair id=\"stonith-${NODE2}-key-file\" name=\"key_file\" value=\"${FENCE_KEY}\"/>
<nvpair id=\"stonith-${NODE2}-vmid1\" name=\"vmid_node1\" value=\"${VMID_NODE1}\"/>
<nvpair id=\"stonith-${NODE2}-vmid2\" name=\"vmid_node2\" value=\"${VMID_NODE2}\"/>
<nvpair id=\"stonith-${NODE2}-host-list\" name=\"pcmk_host_list\" value=\"${NODE2}\"/>
</instance_attributes>
<operations>
<op id=\"stonith-${NODE2}-monitor\" name=\"monitor\" interval=\"30s\" timeout=\"30s\"/>
</operations>
</primitive>
" 2>/dev/null || true
log "Enabling STONITH and restoring quorum policy..."
crm_attribute -t crm_config -n stonith-enabled -v true
crm_attribute -t crm_config -n no-quorum-policy -v stop
log "DRBD fencing mode must also be updated to resource-only (already the"
log "default in cluster-config.nix; confirm with: cat /etc/drbd.d/ha-data.conf)"
log "Testing fence agent..."
stonith_admin --list-devices && log "Fence devices listed successfully." \
|| warn "stonith_admin --list-devices failed — check config"
log "STONITH enabled. Cluster is now fully HA."
+467
View File
@@ -0,0 +1,467 @@
#!/usr/bin/env bash
# cluster-init.sh — one-time HA cluster initialisation script
#
# Run ONCE from ha-server-1 as root AFTER both VMs are booted and have SSH
# access. It:
# 1. Generates and distributes the corosync authkey
# 2. Waits for corosync quorum and pacemaker
# 3. Initialises DRBD metadata, promotes node1 to primary
# 4. Creates XFS on /dev/drbd0 and mounts it
# 5. Creates the directory tree and iSCSI LUN backing file
# 6. Configures LIO iSCSI target (file-backed LUN)
# 7. Configures Pacemaker resources: DRBD → XFS → iSCSI → NFS → VIP
#
# Prerequisites:
# - Both VMs booted with the ha-server config (nixos-rebuild done)
# - SSH key access from node1 to root@NODE2_IP
# - VMID_NODE1 / VMID_NODE2 filled in below (needed for STONITH setup;
# cluster starts without STONITH, which you enable separately via
# scripts/ha/cluster-enable-stonith.sh)
# - Run as root on ha-server-1
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
# All values override-able via environment variables; defaults match variables.nix.
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.2.229}" # vars.haServerVip
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
ISCSI_IQN="${ISCSI_IQN:-iqn.2026-01.home.sweet:ha-storage}" # vars.haIscsiIqn
ISCSI_LUN_FILE="${XFS_MOUNT}/iscsi-lun.img"
ISCSI_LUN_SIZE="10G"
DRBD_DEVICE="/dev/drbd0"
# DRBD backing disk — by-id path that resolves correctly on both nodes
# regardless of whether the OS-level name is sda or sdb (Proxmox VM disk
# ordering is not guaranteed). Matches haServerDrbdDisk in variables.nix.
# Override DRBD_DISK if your hardware uses a different controller/slot path.
DRBD_DISK="${DRBD_DISK:-/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1}"
VMID_NODE1="${VMID_NODE1:-}" # set by deploy.sh; needed for STONITH
VMID_NODE2="${VMID_NODE2:-}"
PVE_HOST="${PVE_HOST:-pve1.sweet.home}"
PVE_USER="${PVE_USER:-wayne}"
# Inter-node SSH: HA_USER is the user to SSH as on NODE2; HA_KEY is the private
# key to use. Default is root-to-root (no key arg). deploy.sh sets HA_USER=nixos
# and HA_KEY=/tmp/cluster-init-key so the script works even when root-to-root SSH
# is not available.
HA_USER="${HA_USER:-root}"
HA_KEY="${HA_KEY:-}"
# NFS dataset subdirectories to create under XFS_MOUNT.
# Must mirror vars.nfsShares subpath values in variables.nix.
NFS_SUBDIRS=(
"docker/config"
"docker/volumes"
"docker/databases"
"docker/nextcloud-data"
"raspi/volumes"
"proxmox/iso"
"proxmox/lxc"
"pxe-boot/images"
)
# ──────────────────────────────────────────────────────────────────────────
log() { echo "[cluster-init] $*"; }
die() { echo "[cluster-init] ERROR: $*" >&2; exit 1; }
warn() { echo "[cluster-init] WARNING: $*" >&2; }
[[ $(id -u) -eq 0 ]] || die "must run as root"
[[ "$(hostname)" == "$NODE1" ]] || die "must run on $NODE1"
# NixOS may not include xfsprogs in root's PATH even when it's in the store.
# If mkfs.xfs is missing, search the Nix store for it.
if ! command -v mkfs.xfs &>/dev/null; then
_xfs_bin=$(find /nix/store -maxdepth 3 -name mkfs.xfs 2>/dev/null | head -1 | xargs dirname 2>/dev/null || true)
[[ -n "$_xfs_bin" ]] && export PATH="$_xfs_bin:$PATH" \
|| die "mkfs.xfs not found — add xfsprogs to ha-server.nix environment.systemPackages and rebuild"
fi
# drbdmeta lives alongside drbdadm but may not be in PATH when run via sudo.
if ! command -v drbdmeta &>/dev/null; then
_drbd_bin=$(dirname "$(command -v drbdadm)" 2>/dev/null || true)
[[ -n "$_drbd_bin" ]] && export PATH="$_drbd_bin:$PATH" \
|| die "drbdmeta not found — is drbd-utils in ha-server environment.systemPackages?"
fi
# Portable 16-hex-char UUID generator (no openssl required).
_rand_uuid() {
cat /proc/sys/kernel/random/uuid 2>/dev/null | tr -d '-' | cut -c1-16 | tr '[:lower:]' '[:upper:]'
}
# Inter-node SSH/SCP helpers — abstract over root-to-root vs nixos+sudo.
_SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=10"
[[ -n "$HA_KEY" ]] && _SSH_OPTS="-i $HA_KEY $_SSH_OPTS"
if [[ "$HA_USER" == "root" ]]; then
n2_ssh() { ssh $_SSH_OPTS "root@${NODE2_IP}" "$@"; }
n2_scp() { scp $_SSH_OPTS "$1" "root@${NODE2_IP}:$2"; }
else
# Non-root user with passwordless sudo; wrap each command with sudo.
n2_ssh() { ssh $_SSH_OPTS "${HA_USER}@${NODE2_IP}" sudo "$@"; }
n2_scp() {
# SCP to a tmp path, then sudo-move to the real destination as the remote user.
local src="$1" dst="$2"
local tmp="/tmp/_cluster_init_scp_$$"
scp $_SSH_OPTS "$src" "${HA_USER}@${NODE2_IP}:${tmp}"
ssh $_SSH_OPTS "${HA_USER}@${NODE2_IP}" sudo mv "${tmp}" "${dst}"
}
fi
# ── 0. Corosync authkey ───────────────────────────────────────────────────
AUTHKEY="/etc/corosync/authkey"
mkdir -p /etc/corosync
if [[ ! -f "$AUTHKEY" ]]; then
log "Generating corosync authkey..."
corosync-keygen -k "$AUTHKEY"
chmod 0400 "$AUTHKEY"
fi
log "Distributing authkey to $NODE2..."
n2_ssh "mkdir -p /etc/corosync"
n2_scp "$AUTHKEY" "$AUTHKEY"
n2_ssh "chmod 0400 '${AUTHKEY}'"
log "Restarting corosync and pacemaker on both nodes..."
systemctl restart corosync
n2_ssh "systemctl restart corosync"
sleep 3
log "Starting pacemaker on both nodes (may have failed at boot before authkey was placed)..."
systemctl start pacemaker 2>/dev/null || systemctl restart pacemaker 2>/dev/null || true
n2_ssh "systemctl start pacemaker 2>/dev/null || systemctl restart pacemaker 2>/dev/null || true"
sleep 2
# ── 1. Corosync quorum ────────────────────────────────────────────────────
log "Waiting for corosync quorum..."
for i in $(seq 1 30); do
if corosync-quorumtool -s 2>/dev/null | grep -q 'Quorate:.*Yes'; then
log "Quorum established"
break
fi
[[ $i -eq 30 ]] && die "corosync quorum not established after 60 s"
sleep 2
done
log "Waiting for pacemaker..."
for i in $(seq 1 30); do
if crm_mon -1 &>/dev/null; then
log "Pacemaker running"
break
fi
[[ $i -eq 30 ]] && die "pacemaker not running after 60 s"
sleep 2
done
# ── 2. DRBD initialisation ────────────────────────────────────────────────
# Put both nodes in Pacemaker standby first so it stops managed resources
# cleanly, then enable maintenance-mode so Pacemaker's monitor operations are
# suspended. Without maintenance-mode, Pacemaker keeps monitoring: when it
# sees DRBD Primary on a standby node (that it didn't start), it triggers a
# stop action — killing the initial sync after ~10 s. Maintenance-mode
# disables all start/stop/monitor actions for the duration of the sync; it is
# cleared after UpToDate/UpToDate is confirmed.
log "Setting both nodes to Pacemaker standby for DRBD metadata init..."
crm_standby -N "$NODE1" -v on 2>/dev/null || true
crm_standby -N "$NODE2" -v on 2>/dev/null || true
# Wait for Pacemaker to actually stop DRBD (if it was managing it).
log "Waiting for DRBD to stop under Pacemaker control..."
for i in $(seq 1 30); do
n1_role=$(drbdadm role ha-data 2>/dev/null || echo "Unconfigured")
n2_role=$(n2_ssh "drbdadm role ha-data 2>/dev/null" 2>/dev/null || echo "Unconfigured")
if [[ "$n1_role" == "Unconfigured" ]] && [[ "$n2_role" == "Unconfigured" ]]; then
log "DRBD stopped on both nodes"
break
fi
[[ $i -eq 30 ]] && warn "DRBD still active after 60s standby — forcing down anyway"
sleep 2
done
log "Enabling Pacemaker maintenance-mode (suspends monitor/start/stop during sync)..."
crm_attribute -t crm_config -n maintenance-mode -v true 2>/dev/null || true
log "Detaching DRBD on $NODE1 (belt-and-suspenders after standby)..."
drbdadm down ha-data 2>/dev/null || true
log "Detaching DRBD on $NODE2..."
n2_ssh "drbdadm down ha-data 2>/dev/null || true"
sleep 2
# Ensure /etc/drbd.conf on both nodes points to DRBD_DISK (the stable by-id
# path). VMs built before this fix may have /dev/sda or /dev/sdb hardcoded.
# NixOS makes /etc/drbd.conf a symlink into the read-only Nix store, so
# sed -i on the symlink target would fail — we break the symlink first with
# cp --remove-destination, creating a regular writable copy.
# Rebuild+redeploy (--force-rebuild) to make this permanent.
_PATCH_DRBD=$(mktemp)
cat > "$_PATCH_DRBD" << 'PATCHEOF'
#!/bin/bash
WANT="$1"
conf=/etc/drbd.conf
if [[ -L "$conf" ]]; then
cp --remove-destination "$(readlink -f "$conf")" "$conf"
fi
cur=$(drbdadm sh-ll-dev ha-data 2>/dev/null | head -1 || true)
if [[ -n "$cur" && "$cur" != "$WANT" ]]; then
echo "[cluster-init] WARNING: patching $conf: $cur → $WANT (rebuild to make permanent)"
sed -i "s,${cur},${WANT},g" "$conf"
fi
PATCHEOF
chmod +x "$_PATCH_DRBD"
bash "$_PATCH_DRBD" "$DRBD_DISK"
n2_scp "$_PATCH_DRBD" "/tmp/patch-drbd-disk.sh"
n2_ssh "bash /tmp/patch-drbd-disk.sh ${DRBD_DISK}"
n2_ssh "rm -f /tmp/patch-drbd-disk.sh"
rm -f "$_PATCH_DRBD"
log "Initialising DRBD metadata on $NODE1..."
# Use drbdmeta --force directly for BOTH create-md and write-dev-uuid.
# drbdadm create-md --force passes --force to drbdmeta create-md but NOT to
# the write-dev-uuid sub-call it makes internally, so write-dev-uuid fails when
# the backing disk is still busy and stdin is not a TTY:
# "stdin not a TTY, not waiting for confirmation" → exit 20.
# Calling drbdmeta --force directly bypasses the exclusive-open confirmation on
# both steps without needing a TTY, regardless of whether the device is busy.
# Skip metadata creation only if DRBD is UP and fully synced (UpToDate/UpToDate).
# When the resource is down, drbdadm dstate reads metadata and returns just
# "UpToDate" (no slash) — that must not be treated as "already synced".
# Mismatched UUIDs from an interrupted sync cause instant WFConnection→StandAlone,
# so we always recreate metadata unless the sync is genuinely complete.
if [[ "$(drbdadm dstate ha-data 2>/dev/null)" != "UpToDate/UpToDate" ]]; then
UUID1=$(_rand_uuid)
drbdmeta --force 0 v08 "${DRBD_DISK}" internal create-md
drbdmeta --force 0 v08 "${DRBD_DISK}" internal write-dev-uuid "$UUID1"
fi
log "Initialising DRBD metadata on $NODE2..."
if [[ "$(n2_ssh "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null)" != "UpToDate/UpToDate" ]]; then
UUID2=$(n2_ssh "cat /proc/sys/kernel/random/uuid 2>/dev/null | tr -d '-' | cut -c1-16 | tr '[:lower:]' '[:upper:]'")
n2_ssh "drbdmeta --force 0 v08 ${DRBD_DISK} internal create-md"
n2_ssh "drbdmeta --force 0 v08 ${DRBD_DISK} internal write-dev-uuid ${UUID2}"
fi
log "Bringing up DRBD on both nodes..."
drbdadm up ha-data 2>/dev/null || true
n2_ssh "drbdadm up ha-data" 2>/dev/null || true
log "Forcing $NODE1 to DRBD Primary for initial sync..."
drbdadm primary ha-data --force
# NOTE: Pacemaker standby is intentionally kept ON until after the sync
# completes. Clearing it here races with the OCF DRBD agent: Pacemaker
# sees DRBD in WFConnection/SyncSource and may call drbdadm-down thinking
# something went wrong, killing the sync. Standby is cleared below, after
# UpToDate/UpToDate is confirmed.
log "Waiting for DRBD initial sync to complete (32 GB may take 1020 min)..."
log " (monitor with: watch -n3 cat /proc/drbd)"
_sync_chars=('|' '/' '-' $'\\')
_sync_iter=0
while true; do
_dstate=$(drbdadm dstate ha-data 2>/dev/null || echo "unknown")
if echo "$_dstate" | grep -q "UpToDate/UpToDate"; then
printf "\r%-80s\r" ""
log "DRBD initial sync complete (dstate: $_dstate)"
break
fi
# Parse connection state from /proc/drbd (cs:SyncSource, cs:Connected, cs:StandAlone …)
_cs=$(grep -oE 'cs:[A-Za-z]+' /proc/drbd 2>/dev/null | head -1 | sed 's/cs://' || echo "unknown")
# /proc/drbd uses variable whitespace: "sync'ed: 5.2%" (two spaces).
_pct=$(grep -oE "sync'ed:[[:space:]]+[0-9.]+" /proc/drbd 2>/dev/null | grep -oE "[0-9.]+" | head -1 || echo "")
_eta=$(grep -oE "finish:[[:space:]]+[0-9:]+" /proc/drbd 2>/dev/null | grep -oE "[0-9:]+$" | head -1 || echo "")
_spd=$(grep -oE "speed:[[:space:]]+[0-9,]+" /proc/drbd 2>/dev/null | grep -oE "[0-9,]+$" | head -1 || echo "")
_sync_iter=$(( _sync_iter + 1 ))
_sc="${_sync_chars[$_sync_iter % 4]}"
if [[ "$_cs" == "StandAlone" && $_sync_iter -gt 5 ]]; then
printf "\r%-80s\r" ""
die "DRBD is StandAlone after 15 s — peer connection lost (dstate: $_dstate). " \
"Check corosync/network and re-run cluster-init."
elif [[ -n "$_pct" ]]; then
printf "\r [%s] syncing: %s%% done — ETA %s @ %s K/s " \
"$_sc" "$_pct" "${_eta:-??:??:??}" "${_spd:-?}"
else
printf "\r [%s] cs:%s dstate:%s — waiting for sync to start " "$_sc" "$_cs" "$_dstate"
fi
sleep 3
done
log "Disabling Pacemaker maintenance-mode and clearing standby — handing DRBD back to Pacemaker..."
crm_attribute -t crm_config -n maintenance-mode -v false 2>/dev/null || true
crm_standby -N "$NODE1" -v off 2>/dev/null || true
crm_standby -N "$NODE2" -v off 2>/dev/null || true
# ── 3. XFS filesystem ─────────────────────────────────────────────────────
log "Creating XFS on ${DRBD_DEVICE}..."
if ! xfs_info "${DRBD_DEVICE}" &>/dev/null; then
mkfs.xfs -f "${DRBD_DEVICE}"
fi
log "Mounting ${DRBD_DEVICE} at ${XFS_MOUNT}..."
mkdir -p "${XFS_MOUNT}"
mountpoint -q "${XFS_MOUNT}" || mount "${DRBD_DEVICE}" "${XFS_MOUNT}"
# ── 4. NFS dataset directories ────────────────────────────────────────────
log "Creating NFS dataset directories..."
for subdir in "${NFS_SUBDIRS[@]}"; do
mkdir -p "${XFS_MOUNT}/${subdir}"
done
# ── 5. iSCSI LUN backing file ─────────────────────────────────────────────
log "Creating iSCSI LUN backing file ${ISCSI_LUN_FILE} (${ISCSI_LUN_SIZE})..."
if [[ ! -f "${ISCSI_LUN_FILE}" ]]; then
fallocate -l "${ISCSI_LUN_SIZE}" "${ISCSI_LUN_FILE}"
fi
# ── 6. LIO iSCSI target ───────────────────────────────────────────────────
log "Configuring LIO iSCSI target via targetcli..."
# Note: do NOT bind portal to ${VIP} here — the VIP isn't assigned yet (Pacemaker
# creates it). The default portal (all IPs, port 3260) is correct; Pacemaker's
# VIP resource will make the target reachable at the VIP address.
#
# Clear any existing LIO state first (idempotent: re-run after a partial failure).
# Use specific delete commands — clearconfig does not reliably clear kernel state.
if ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -q "${ISCSI_IQN}"; then
log "Clearing existing LIO target ${ISCSI_IQN} before reconfiguration..."
targetcli "/iscsi delete ${ISCSI_IQN}" 2>/dev/null || true
fi
if ls /sys/kernel/config/target/core/ 2>/dev/null | grep -q "fileio"; then
log "Clearing existing LIO backstore ha-lun0 before reconfiguration..."
targetcli "/backstores/fileio delete ha-lun0" 2>/dev/null || true
fi
targetcli <<EOF
/backstores/fileio create name=ha-lun0 file_or_dev=${ISCSI_LUN_FILE} size=0 write_back=false
/iscsi create ${ISCSI_IQN}
/iscsi/${ISCSI_IQN}/tpg1/luns create /backstores/fileio/ha-lun0
/iscsi/${ISCSI_IQN}/tpg1 set attribute authentication=0
/iscsi/${ISCSI_IQN}/tpg1 set attribute demo_mode_write_protect=0
saveconfig /etc/target/saveconfig.json
EOF
log "Tearing down LIO kernel objects — Pacemaker will restore via targetctl on the Active node..."
# LIO holds the backing file open; clear kernel state now so the XFS unmount succeeds.
# Use specific delete commands (clearconfig does not reliably clear kernel configfs state).
targetcli "/iscsi delete ${ISCSI_IQN}" 2>/dev/null || warn "LIO iscsi delete failed — umount may fail"
targetcli "/backstores/fileio delete ha-lun0" 2>/dev/null || warn "LIO backstore delete failed"
log "Distributing iSCSI saveconfig to $NODE2..."
n2_scp /etc/target/saveconfig.json /etc/target/saveconfig.json
log "Unmounting ${XFS_MOUNT} — Pacemaker manages it..."
umount "${XFS_MOUNT}" || { sync; umount -l "${XFS_MOUNT}"; }
log "Demoting DRBD to Secondary — Pacemaker manages primary role..."
drbdadm role ha-data 2>/dev/null | grep -q "^Primary" && drbdadm secondary ha-data || true
# ── 7. Pacemaker resources ────────────────────────────────────────────────
log "Configuring Pacemaker cluster properties..."
crm_attribute -t crm_config -n stonith-enabled -v false
crm_attribute -t crm_config -n no-quorum-policy -v ignore
log "Creating Pacemaker resources via cibadmin..."
# Use cibadmin --replace with pacemaker-4.0-compatible XML.
# Key schema rules for pacemaker-4.0:
# - globally-unique must be in <meta_attributes>, not a direct <clone> attribute
# - promoted-max / promoted-node-max (not master-max / master-node-max)
# - constraint with-rsc-role="Promoted" (not "Master")
cibadmin --replace --scope resources --xml-text '<resources>
<clone id="ms-drbd0">
<meta_attributes id="ms-drbd0-meta">
<nvpair id="ms-drbd0-globally-unique" name="globally-unique" value="false"/>
<nvpair id="ms-drbd0-promotable" name="promotable" value="true"/>
<nvpair id="ms-drbd0-promoted-max" name="promoted-max" value="1"/>
<nvpair id="ms-drbd0-promoted-node-max" name="promoted-node-max" value="1"/>
<nvpair id="ms-drbd0-clone-max" name="clone-max" value="2"/>
<nvpair id="ms-drbd0-clone-node-max" name="clone-node-max" value="1"/>
<nvpair id="ms-drbd0-notify" name="notify" value="true"/>
<nvpair id="ms-drbd0-interleave" name="interleave" value="true"/>
</meta_attributes>
<primitive id="drbd0" class="ocf" type="drbd" provider="linbit">
<instance_attributes id="drbd0-attrs">
<nvpair id="drbd0-resource" name="drbd_resource" value="ha-data"/>
</instance_attributes>
<operations>
<op id="drbd0-start" name="start" interval="0" timeout="240s"/>
<op id="drbd0-stop" name="stop" interval="0" timeout="120s"/>
<op id="drbd0-promote" name="promote" interval="0" timeout="240s"/>
<op id="drbd0-demote" name="demote" interval="0" timeout="90s"/>
<op id="drbd0-monitor-promoted" name="monitor" interval="20s" timeout="20s" role="Promoted"/>
<op id="drbd0-monitor-unpromoted" name="monitor" interval="30s" timeout="20s" role="Unpromoted"/>
</operations>
</primitive>
</clone>
<group id="ha-group">
<primitive id="xfs-data" class="ocf" type="Filesystem" provider="heartbeat">
<instance_attributes id="xfs-data-attrs">
<nvpair id="xfs-data-device" name="device" value="/dev/drbd0"/>
<nvpair id="xfs-data-directory" name="directory" value="/srv/ha-data"/>
<nvpair id="xfs-data-fstype" name="fstype" value="xfs"/>
<nvpair id="xfs-data-options" name="options" value="defaults"/>
<nvpair id="xfs-data-force_unmount" name="force_unmount" value="true"/>
</instance_attributes>
<operations>
<op id="xfs-data-start" name="start" interval="0" timeout="60s"/>
<op id="xfs-data-stop" name="stop" interval="0" timeout="60s"/>
<op id="xfs-data-monitor" name="monitor" interval="20s" timeout="40s"/>
</operations>
</primitive>
<primitive id="iscsi-target" class="systemd" type="targetctl">
<operations>
<op id="iscsi-start" name="start" interval="0" timeout="60s"/>
<op id="iscsi-stop" name="stop" interval="0" timeout="60s"/>
<op id="iscsi-monitor" name="monitor" interval="20s" timeout="40s"/>
</operations>
</primitive>
<primitive id="nfs-server" class="systemd" type="nfs-server">
<operations>
<op id="nfs-start" name="start" interval="0" timeout="60s"/>
<op id="nfs-stop" name="stop" interval="0" timeout="60s"/>
<op id="nfs-monitor" name="monitor" interval="30s" timeout="40s"/>
</operations>
</primitive>
<primitive id="vip" class="ocf" type="IPaddr2" provider="heartbeat">
<instance_attributes id="vip-attrs">
<nvpair id="vip-ip" name="ip" value="192.168.2.229"/>
<nvpair id="vip-cidr" name="cidr_netmask" value="24"/>
</instance_attributes>
<operations>
<op id="vip-start" name="start" interval="0" timeout="20s"/>
<op id="vip-stop" name="stop" interval="0" timeout="20s"/>
<op id="vip-monitor" name="monitor" interval="10s" timeout="20s"/>
</operations>
</primitive>
</group>
</resources>'
log "Adding Pacemaker ordering and colocation constraints..."
cibadmin --replace --scope constraints --xml-text '<constraints>
<rsc_order id="order-drbd-group" first="ms-drbd0" first-action="promote" then="ha-group" then-action="start" kind="Mandatory"/>
<rsc_colocation id="coloc-group-with-drbd" score="INFINITY" rsc="ha-group" with-rsc="ms-drbd0" with-rsc-role="Promoted"/>
</constraints>'
log "Clearing stale Pacemaker failure history..."
crm_resource --cleanup 2>/dev/null || true
log "Waiting for resources to start..."
for i in $(seq 1 60); do
if crm_resource -r vip --locate 2>/dev/null | grep -q "running on"; then
log "VIP is up: $(crm_resource -r vip --locate)"
break
fi
[[ $i -eq 60 ]] && { warn "VIP not up after 120 s — check: crm_mon -1"; break; }
sleep 2
done
log ""
log "═══════════════════════════════════════════════════════════════"
log " HA cluster initialised."
log ""
log " crm_mon -1 — cluster status"
log " iscsiadm -m discovery -t st -p ${VIP} — verify iSCSI target"
log " showmount -e ${VIP} — verify NFS exports"
log ""
log " To enable STONITH (after deploying fence SSH key):"
log " 1. Fill in VMID_NODE1 / VMID_NODE2 in cluster-enable-stonith.sh"
log " 2. Copy scripts/ha/fence-pve-ssh.py to /etc/pacemaker/fence_pve_ssh"
log " on both nodes (chmod +x)"
log " 3. Generate and distribute the fence SSH key"
log " (see docs or cluster-enable-stonith.sh header)"
log " 4. bash scripts/ha/cluster-enable-stonith.sh"
log "═══════════════════════════════════════════════════════════════"
+499
View File
@@ -0,0 +1,499 @@
#!/usr/bin/env bash
# deploy.sh — Full lifecycle management for the HA file-server cluster.
#
# Handles everything from zero (no VMs, no secrets) through a running,
# tested cluster, and optionally tears it back down.
#
# Usage:
# scripts/ha/deploy.sh [options]
# scripts/ha/deploy.sh --destroy [options]
#
# Phases (all run by default; skip any with --skip-*):
# 1. ensure-bridge Create storage bridge (vmbr1) on the Proxmox node if absent.
# 2. sync-keys Generate SSH host keys and register age keys for both nodes.
# 3. create-vms Build disk images and create both VMs via create-proxmox-resource.sh.
# 4. add-hardware Attach storage NIC (vmbr1) and DRBD data disk to each VM.
# 5. boot-wait Start VMs, wait for SSH on both nodes.
# 6. cluster-init Form the cluster: DRBD, corosync, Pacemaker, NFS, VIP.
# Also encrypts the generated corosync authkey into the repo.
# 7. run-tests Run acceptance tests (T1T7).
#
# Options:
# --node <host> Proxmox host to deploy on (default: pve1.sweet.home)
# --vmid1 <n> VMID for ha-server-1 (default: 200)
# --vmid2 <n> VMID for ha-server-2 (default: 201)
# --storage <pool> Proxmox storage pool (default: local-zfs)
# --storage-bridge <br> Bridge for HA storage network (default: vmbr1)
# --drbd-disk-gb <n> DRBD data disk size in GB (default: 32)
# --memory <MB> RAM per node (default: 4096)
# --cores <n> vCPUs per node (default: 4)
# --skip-ensure-bridge Skip storage bridge creation/check
# --skip-sync-keys Skip sync-host-keys.sh (clan vars already exist)
# --skip-create-vms Skip VM creation (VMs already exist)
# --skip-add-hardware Skip net1/scsi1 attachment (already attached)
# --skip-boot-wait Skip boot/SSH wait (VMs already running)
# --skip-refresh-sops-keys Skip scanning running VMs for fresh SSH host keys
# --skip-cluster-init Skip cluster formation (cluster already configured)
# --skip-tests Skip acceptance tests
# --force-rebuild Pass --force-rebuild to create-proxmox-resource.sh
# --destroy Stop and delete both VMs (skip all other phases)
# --dry-run Print what would run without executing
# -h|--help Show this message
#
# Prerequisites:
# - SSH access to the Proxmox node as $PROXMOX_SSH_USER (wayne).
# - sops age key in the standard location (used by sync-host-keys.sh).
# - For --skip-sync-keys: clan vars already in vars/per-machine/proxmox-ha-server-{1,2}/.
# - For full tests: secrets/common.yaml decryptable on both nodes (run
# `sops updatekeys secrets/common.yaml` after sync-keys).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# shellcheck source=../env.sh
source "${REPO_ROOT}/scripts/env.sh"
# ── Defaults ──────────────────────────────────────────────────────────────────
NODE="${PROXMOX_HOST:-$PVE1_HOST}"
VMID1=200
VMID2=201
STORAGE="${PROXMOX_STORAGE:-local-zfs}"
STORAGE_BRIDGE="vmbr1"
DRBD_DISK_GB=32
MEMORY_MB=4096
CORES=4
SKIP_ENSURE_BRIDGE=false
SKIP_SYNC_KEYS=false
SKIP_CREATE_VMS=false
SKIP_ADD_HARDWARE=false
SKIP_BOOT_WAIT=false
SKIP_REFRESH_SOPS_KEYS=false
SKIP_CLUSTER_INIT=false
SKIP_TESTS=false
FORCE_REBUILD=false
DESTROY=false
DRY_RUN=false
# ── Variables from repo ───────────────────────────────────────────────────────
NODE1_HOST="ha-server-1"
NODE2_HOST="ha-server-2"
NODE1_IP="192.168.2.228"
NODE2_IP="192.168.2.227"
STORAGE_IP1="192.168.4.228"
STORAGE_IP2="192.168.4.227"
STORAGE_CIDR="192.168.4.0/29"
SSH_USER="${PROXMOX_SSH_USER:-wayne}"
# ── Argument parsing ──────────────────────────────────────────────────────────
usage() {
sed -n '/^# Usage:/,/^[^#]/{ /^#/{ s/^# \?//; p } }' "$0"
exit "${1:-0}"
}
while [[ $# -gt 0 ]]; do
case "$1" in
--node) NODE="$2"; shift 2 ;;
--vmid1) VMID1="$2"; shift 2 ;;
--vmid2) VMID2="$2"; shift 2 ;;
--storage) STORAGE="$2"; shift 2 ;;
--storage-bridge) STORAGE_BRIDGE="$2"; shift 2 ;;
--drbd-disk-gb) DRBD_DISK_GB="$2"; shift 2 ;;
--memory) MEMORY_MB="$2"; shift 2 ;;
--cores) CORES="$2"; shift 2 ;;
--skip-ensure-bridge) SKIP_ENSURE_BRIDGE=true; shift ;;
--skip-sync-keys) SKIP_SYNC_KEYS=true; shift ;;
--skip-create-vms) SKIP_CREATE_VMS=true; shift ;;
--skip-add-hardware) SKIP_ADD_HARDWARE=true; shift ;;
--skip-boot-wait) SKIP_BOOT_WAIT=true; shift ;;
--skip-refresh-sops-keys) SKIP_REFRESH_SOPS_KEYS=true; shift ;;
--skip-cluster-init) SKIP_CLUSTER_INIT=true; shift ;;
--skip-tests) SKIP_TESTS=true; shift ;;
--force-rebuild) FORCE_REBUILD=true; shift ;;
--destroy) DESTROY=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage 0 ;;
*) echo "Unknown option: $1" >&2; usage 1 ;;
esac
done
# ── Helpers ───────────────────────────────────────────────────────────────────
log() { echo "==> $*"; }
logn() { echo " $*"; }
err() { echo "ERROR: $*" >&2; exit 1; }
run() {
if $DRY_RUN; then
echo "[dry-run] $*"
else
"$@"
fi
}
pve() {
# Run a command on the Proxmox node via SSH.
if $DRY_RUN; then
echo "[dry-run] ssh ${SSH_USER}@${NODE} sudo $*"
else
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
fi
}
pve_check() {
# Run a read-only probe on the Proxmox node — always executes even in dry-run.
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
}
HA_USER="nixos"
n1() {
# Run a command on ha-server-1 via SSH as nixos with sudo.
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null
}
n2() {
# Run a command on ha-server-2 via SSH as nixos with sudo.
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null
}
wait_for_ssh() {
local ip="$1" label="$2"
if $DRY_RUN; then
logn "[dry-run] Skipping SSH wait for ${label} (${ip})"
return 0
fi
local deadline=$(( $(date +%s) + 300 ))
log "Waiting for SSH on ${label} (${ip}) — up to 5 min..."
while [[ $(date +%s) -lt $deadline ]]; do
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=3 \
-o BatchMode=yes "${HA_USER}@${ip}" true 2>/dev/null; then
logn "${label} is up."
return 0
fi
sleep 5
done
err "Timed out waiting for SSH on ${label} (${ip})"
}
# ── Destroy mode ─────────────────────────────────────────────────────────────
if $DESTROY; then
log "Destroying HA cluster VMs (${VMID1}=${NODE1_HOST}, ${VMID2}=${NODE2_HOST}) on ${NODE}"
for vmid in "$VMID1" "$VMID2"; do
STATUS=$(pve "qm status ${vmid} 2>/dev/null" 2>/dev/null || true)
if echo "$STATUS" | grep -q "running"; then
log "Stopping VMID ${vmid}..."
pve "qm stop ${vmid} --skiplock 1"
sleep 5
fi
if $DRY_RUN || pve "qm config ${vmid} >/dev/null 2>&1"; then
log "Deleting VMID ${vmid}..."
run pve "qm destroy ${vmid} --purge 1"
else
logn "VMID ${vmid} not found — already gone."
fi
done
log "Done — cluster VMs destroyed."
exit 0
fi
# ── Phase 1: Storage bridge ───────────────────────────────────────────────────
if ! $SKIP_ENSURE_BRIDGE; then
log "Phase 1: Ensuring storage bridge ${STORAGE_BRIDGE} on ${NODE}"
if pve_check "test -d /sys/class/net/${STORAGE_BRIDGE}" &>/dev/null; then
logn "${STORAGE_BRIDGE} already exists — skipping."
else
logn "Creating isolated internal bridge ${STORAGE_BRIDGE} (no upstream port, ${STORAGE_CIDR})"
BRIDGE_CONF="auto ${STORAGE_BRIDGE}
iface ${STORAGE_BRIDGE} inet manual
bridge-ports none
bridge-stp off
bridge-fd 0"
if $DRY_RUN; then
echo "[dry-run] Would write /etc/network/interfaces.d/${STORAGE_BRIDGE}.conf and ifup it"
else
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"echo '${BRIDGE_CONF}' | sudo tee /etc/network/interfaces.d/${STORAGE_BRIDGE}.conf > /dev/null && sudo ifup ${STORAGE_BRIDGE}"
logn "${STORAGE_BRIDGE} created and brought up."
fi
fi
fi
# ── Phase 2: Sync host keys ───────────────────────────────────────────────────
if ! $SKIP_SYNC_KEYS; then
log "Phase 2: Syncing SSH host keys for both HA targets"
for target in proxmox-ha-server-1 proxmox-ha-server-2; do
CLAN_DIR="${REPO_ROOT}/vars/per-machine/${target}/openssh"
if [[ -d "$CLAN_DIR" ]]; then
logn "Clan vars for ${target} already exist — skipping."
else
logn "Generating host keys for ${target}..."
run bash "${REPO_ROOT}/scripts/secrets/sync-host-keys.sh" "$target"
fi
done
fi
# ── Phase 2.5: Prepare Proxmox node for building ─────────────────────────────
if ! $SKIP_CREATE_VMS && ! $DRY_RUN; then
CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"
# Fix /nix ownership if it exists but belongs to a different UID.
# pve1's IPA-enrolled wayne (UID 50002) can't write to a store created by
# another UID — passwordless sudo corrects it once.
# Use direct SSH (no sudo) for the writability check so we test wayne's own
# access, not root's.
local_ssh() { ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "$*"; }
if local_ssh "test -d /nix" &>/dev/null && ! local_ssh "test -w /nix" &>/dev/null; then
logn "/nix exists but not writable by ${SSH_USER} — fixing ownership with sudo (one-time)..."
local_ssh "sudo chown -R ${SSH_USER} /nix"
logn "Done."
fi
unset -f local_ssh
# Ensure the remote clone is on the correct branch so create-proxmox-resource.sh
# builds from the same commits we're deploying.
REMOTE_REPO="/home/${SSH_USER}/nixos"
if pve_check "test -d ${REMOTE_REPO}/.git" &>/dev/null; then
REMOTE_BRANCH=$(ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"cd ${REMOTE_REPO} && git rev-parse --abbrev-ref HEAD 2>/dev/null")
if [[ "$REMOTE_BRANCH" != "$CURRENT_BRANCH" ]]; then
logn "Remote clone is on '${REMOTE_BRANCH}', switching to '${CURRENT_BRANCH}'..."
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"cd ${REMOTE_REPO} && git fetch origin && git checkout '${CURRENT_BRANCH}' && git pull --ff-only"
logn "Done."
fi
fi
fi
# ── Phase 3: Create VMs ───────────────────────────────────────────────────────
if ! $SKIP_CREATE_VMS; then
log "Phase 3: Building and creating VMs on ${NODE}"
CREATE="${REPO_ROOT}/scripts/proxmox/create-proxmox-resource.sh"
for spec in "${VMID1}:ha-server-1:proxmox-ha-server-1" "${VMID2}:ha-server-2:proxmox-ha-server-2"; do
IFS=: read -r vmid host_name flake_target <<< "$spec"
log "Creating ${flake_target} (VMID ${vmid}) on ${NODE}..."
# Always --force-rebuild: create-proxmox-resource.sh only calls
# sync_remote_host_keys (which populates host-keys/ for proxmox.nix to
# bake the clan-var SSH key into the disko image) when it actually builds.
# Reusing a cached image skips that step, so destroy+recreate would reuse
# an image with a stale/random key baked in → sops fails on first boot.
run bash "$CREATE" \
--type vm \
--host "$host_name" \
--vmid "$vmid" \
--node "$NODE" \
--storage "$STORAGE" \
--memory "$MEMORY_MB" \
--cores "$CORES" \
--force-rebuild
done
fi
# ── Phase 4: Add storage NIC and DRBD disk ────────────────────────────────────
if ! $SKIP_ADD_HARDWARE; then
log "Phase 4: Attaching storage NIC (${STORAGE_BRIDGE}) and DRBD disk (${DRBD_DISK_GB}G) to each VM"
for vmid in "$VMID1" "$VMID2"; do
log " VMID ${vmid}: stopping to add hardware..."
pve "qm stop ${vmid} --skiplock 1 2>/dev/null; sleep 3" || true
logn "Adding net1 (${STORAGE_BRIDGE})..."
pve "qm set ${vmid} --net1 virtio,bridge=${STORAGE_BRIDGE},firewall=0"
logn "Adding scsi1 (${STORAGE}:${DRBD_DISK_GB}G for DRBD)..."
pve "qm set ${vmid} --scsi1 ${STORAGE}:${DRBD_DISK_GB},format=raw"
logn "Starting VMID ${vmid}..."
pve "qm start ${vmid}"
done
fi
# ── Phase 5: Wait for SSH ─────────────────────────────────────────────────────
if ! $SKIP_BOOT_WAIT; then
log "Phase 5: Waiting for both nodes to come up"
wait_for_ssh "$NODE1_IP" "$NODE1_HOST"
wait_for_ssh "$NODE2_IP" "$NODE2_HOST"
logn "Both nodes are SSHable."
# Give systemd a few seconds to settle after activation
sleep 10
fi
# ── Phase 5.5: Refresh sops host-key registrations ───────────────────────────
#
# Disko builds raw disk images; each new VM boots with a freshly-generated SSH
# host key rather than the one pre-seeded in clan vars. This phase scans the
# actual running VMs, and if their ed25519 host keys differ from what clan vars
# record: updates the clan var pub-key files, rewrites the .sops.yaml age-key
# anchors, and re-encrypts all affected sops files so the nodes can decrypt
# secrets on the next nixos-rebuild. Safe no-op when keys haven't changed.
if ! $SKIP_REFRESH_SOPS_KEYS; then
if $DRY_RUN; then
logn "[dry-run] Would scan VM host keys and refresh .sops.yaml / secrets if needed"
else
log "Phase 5.5: Refreshing sops host-key registrations (disko key drift fix)"
SOPS_UPDATED=false
for spec in \
"${NODE1_IP}:proxmox-ha-server-1:${NODE1_HOST}" \
"${NODE2_IP}:proxmox-ha-server-2:${NODE2_HOST}"; do
IFS=: read -r node_ip flake_target host_name <<< "$spec"
CLAN_PUB="${REPO_ROOT}/vars/per-machine/${flake_target}/openssh/ssh_host_ed25519_key.pub/value"
logn "Scanning ed25519 host key from ${host_name} (${node_ip})..."
RAW=$(ssh-keyscan -t ed25519 "${node_ip}" 2>/dev/null | grep -v "^#") || true
if [[ -z "$RAW" ]]; then
logn "WARNING: no ed25519 key returned by ssh-keyscan for ${node_ip} — skipping"
continue
fi
# ssh-keyscan returns: <ip> ssh-ed25519 <b64key>
SCANNED_TYPE=$(awk '{print $2}' <<< "$RAW")
SCANNED_KEY=$(awk '{print $3}' <<< "$RAW")
SCANNED_PUBKEY="${SCANNED_TYPE} ${SCANNED_KEY} ${host_name}"
CURRENT=$(tr -d '\n' < "$CLAN_PUB" 2>/dev/null || true)
if [[ "$SCANNED_PUBKEY" == "$CURRENT" ]]; then
logn "${host_name}: clan var matches running key — no update needed"
continue
fi
logn "${host_name}: key drift detected — updating clan var"
logn " old: ${CURRENT}"
logn " new: ${SCANNED_PUBKEY}"
echo "$SCANNED_PUBKEY" > "$CLAN_PUB"
SOPS_UPDATED=true
# Rewrite the .sops.yaml anchor for this host with the new age key.
ANCHOR="${flake_target}" # e.g. proxmox-ha-server-1
NEW_AGE=$(echo "$SCANNED_PUBKEY" | \
nix run --quiet --no-warn-dirty nixpkgs#ssh-to-age 2>/dev/null)
if [[ -z "$NEW_AGE" ]]; then
err "ssh-to-age produced no output for ${host_name} — check nixpkgs#ssh-to-age"
fi
logn " new age key: ${NEW_AGE}"
sed -i "/&${ANCHOR} /s| age[a-z0-9]*$| ${NEW_AGE}|" "${REPO_ROOT}/.sops.yaml"
done
if $SOPS_UPDATED; then
logn "Running sops updatekeys on affected secrets..."
SOPS="nix run --quiet --no-warn-dirty nixpkgs#sops --"
(cd "${REPO_ROOT}" && \
$SOPS updatekeys -y secrets/common.yaml && \
$SOPS updatekeys -y secrets/ha-server-1.yaml && \
$SOPS updatekeys -y secrets/ha-server-2.yaml && \
$SOPS updatekeys -y secrets/ha-server-1.keytab && \
$SOPS updatekeys -y secrets/ha-server-2.keytab)
# Note: ha-corosync-authkey is re-generated and re-encrypted by cluster-init below.
logn "Committing refreshed host keys and re-encrypted secrets..."
(cd "${REPO_ROOT}" && \
git add \
vars/per-machine/proxmox-ha-server-1/openssh/ssh_host_ed25519_key.pub/value \
vars/per-machine/proxmox-ha-server-2/openssh/ssh_host_ed25519_key.pub/value \
.sops.yaml \
secrets/common.yaml \
secrets/ha-server-1.yaml \
secrets/ha-server-2.yaml \
secrets/ha-server-1.keytab \
secrets/ha-server-2.keytab && \
git commit -m "secrets(ha): refresh sops host-key registrations for new VM instances" || true)
logn "Sops keys refreshed and committed."
fi
fi
fi
# ── Phase 6: Cluster init ─────────────────────────────────────────────────────
if ! $SKIP_CLUSTER_INIT; then
log "Phase 6: Initialising HA cluster"
CLUSTER_INIT="${REPO_ROOT}/scripts/ha/cluster-init.sh"
[[ -x "$CLUSTER_INIT" ]] || chmod +x "$CLUSTER_INIT"
if $DRY_RUN; then
logn "[dry-run] Would generate temp key, authorise on ${NODE2_HOST}, scp cluster-init.sh to ${NODE1_HOST}, and run it as root via sudo"
else
# Generate a temp keypair so cluster-init.sh can SSH node1→node2 as ${HA_USER}.
# Root on node1 has no keys; a temp key bridging node1→node2 nixos solves this.
TEMP_KEY="${REPO_ROOT}/.tmp-cluster-init-key"
TEMP_KEY_PUB="${TEMP_KEY}.pub"
rm -f "$TEMP_KEY" "$TEMP_KEY_PUB"
ssh-keygen -t ed25519 -f "$TEMP_KEY" -N "" -C "cluster-init-temp-$(date +%s)" -q
TEMP_PUBKEY=$(cat "$TEMP_KEY_PUB")
logn "Authorising temp key on ${NODE2_HOST} for ${HA_USER}..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE2_IP}" \
"mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo '${TEMP_PUBKEY}' >> ~/.ssh/authorized_keys"
logn "Placing temp key on ${NODE1_HOST} for root..."
scp -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"$TEMP_KEY" "${HA_USER}@${NODE1_IP}:/tmp/cluster-init-key"
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo mkdir -p /root/.ssh && sudo cp /tmp/cluster-init-key /root/.ssh/cluster-init-key && \
sudo chmod 600 /root/.ssh/cluster-init-key && rm -f /tmp/cluster-init-key"
logn "Uploading cluster-init.sh to ${NODE1_HOST}..."
scp -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"$CLUSTER_INIT" "${HA_USER}@${NODE1_IP}:/tmp/cluster-init.sh"
logn "Running cluster-init.sh on ${NODE1_HOST}..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo env NODE1=${NODE1_HOST} NODE2=${NODE2_HOST} \
NODE1_IP=${NODE1_IP} NODE2_IP=${NODE2_IP} \
VIP=192.168.2.229 XFS_MOUNT=/srv/ha-data \
ISCSI_IQN=iqn.2026-01.home.sweet:ha-storage \
VMID_NODE1=${VMID1} VMID_NODE2=${VMID2} \
HA_USER=${HA_USER} HA_KEY=/root/.ssh/cluster-init-key \
bash /tmp/cluster-init.sh"
logn "Cleaning up temp key from both nodes..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE2_IP}" \
"sed -i '/cluster-init-temp/d' ~/.ssh/authorized_keys" 2>/dev/null || true
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo rm -f /root/.ssh/cluster-init-key" 2>/dev/null || true
rm -f "$TEMP_KEY" "$TEMP_KEY_PUB"
# Encrypt the corosync authkey generated by cluster-init and commit it.
log " Encrypting corosync authkey into secrets/ha-corosync-authkey..."
AUTHKEY_TMP="${REPO_ROOT}/secrets/ha-corosync-authkey.tmp"
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo cat /etc/corosync/authkey" > "$AUTHKEY_TMP"
if [[ ! -s "$AUTHKEY_TMP" ]]; then
err "corosync authkey on node1 is empty — cluster-init may have failed."
fi
mv "$AUTHKEY_TMP" "${REPO_ROOT}/secrets/ha-corosync-authkey"
(cd "${REPO_ROOT}" && nix run nixpkgs#sops -- -e --input-type binary -i secrets/ha-corosync-authkey)
logn "Authkey encrypted. Committing..."
(cd "${REPO_ROOT}" && git add secrets/ha-corosync-authkey && \
git commit -m "secrets(ha): encrypt corosync authkey generated by cluster-init")
logn "Committed."
fi
fi
# ── Phase 7: Acceptance tests ─────────────────────────────────────────────────
if ! $SKIP_TESTS; then
log "Phase 7: Running acceptance tests (T1T7)"
if $DRY_RUN; then
logn "[dry-run] Would run acceptance-tests.sh against ${NODE1_HOST}/${NODE2_HOST}"
else
NODE1="$NODE1_HOST" NODE2="$NODE2_HOST" \
NODE1_IP="$NODE1_IP" NODE2_IP="$NODE2_IP" \
VIP="192.168.2.229" \
bash "${REPO_ROOT}/scripts/ha/acceptance-tests.sh"
fi
fi
log "Deploy complete."
+256
View File
@@ -0,0 +1,256 @@
#!/usr/bin/env bash
# failover.sh — graceful HA cluster failover
#
# Detects which node is active and moves all resources to the other node by
# putting the active node into Pacemaker standby. Waits for the XFS mount to
# appear on the target before returning.
#
# Usage:
# scripts/ha/failover.sh [--to node1|node2] [--force] [--timeout <s>] [--dry-run]
#
# --to node1|node2 target node (default: the node that is NOT currently active)
# --force skip the interactive confirmation prompt
# --timeout <s> seconds to wait for resources to move (default: 120)
# --dry-run show what would be done without changing anything
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}"
NODE2_IP="${NODE2_IP:-192.168.2.227}"
VIP="${VIP:-192.168.2.229}"
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"
HA_USER="${HA_USER:-nixos}"
# ──────────────────────────────────────────────────────────────────────────
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
# ── Argument parsing ───────────────────────────────────────────────────────
TARGET_NODE=""
FORCE=false
DRY_RUN=false
TIMEOUT=120
while [[ $# -gt 0 ]]; do
case "$1" in
--to)
shift
case "${1:-}" in
node1|ha-server-1) TARGET_NODE="$NODE1" ;;
node2|ha-server-2) TARGET_NODE="$NODE2" ;;
*) echo "ERROR: --to must be node1, node2, ha-server-1, or ha-server-2"; exit 1 ;;
esac
;;
--force) FORCE=true ;;
--dry-run) DRY_RUN=true ;;
--timeout) shift; TIMEOUT="${1:?--timeout requires a value}" ;;
*) echo "Unknown argument: $1"; echo "Usage: $0 [--to node1|node2] [--force] [--timeout <s>] [--dry-run]"; exit 1 ;;
esac
shift
done
DRY_PREFIX=""
$DRY_RUN && DRY_PREFIX="[dry-run] "
echo "════════════════════════════════════════════════════"
echo " HA Cluster Failover — $(date '+%Y-%m-%d %H:%M:%S')"
$DRY_RUN && echo " MODE: dry-run — no changes will be made"
echo "════════════════════════════════════════════════════"
# ── Detect active node ─────────────────────────────────────────────────────
echo ""
echo "Detecting active node..."
CRM_OUT=""
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" true 2>/dev/null; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
fi
if [[ -z "$CRM_OUT" ]]; then
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" true 2>/dev/null; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
fi
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" — the * bullet
# means ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -z "$ACTIVE_NODE" ]]; then
echo ""
echo "ERROR: could not determine active node from crm_mon."
echo " Is Pacemaker still settling? Try running scripts/ha/health.sh first."
echo " If Pacemaker is down on both nodes, manual recovery is required."
exit 1
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
STANDBY_NODE="$NODE2"
STANDBY_IP="$NODE2_IP"
na() { n1 "$@"; }
ns() { n2 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
STANDBY_NODE="$NODE1"
STANDBY_IP="$NODE1_IP"
na() { n2 "$@"; }
ns() { n1 "$@"; }
fi
echo " Active: $ACTIVE_NODE ($ACTIVE_IP)"
echo " Standby: $STANDBY_NODE ($STANDBY_IP)"
# ── Validate target ────────────────────────────────────────────────────────
if [[ -n "$TARGET_NODE" ]]; then
if [[ "$TARGET_NODE" == "$ACTIVE_NODE" ]]; then
echo ""
echo "ERROR: $TARGET_NODE is already the active node — nothing to do."
exit 1
fi
echo " Target: $TARGET_NODE (as requested)"
else
echo " Target: $STANDBY_NODE (auto — the other node)"
fi
# ── Pre-checks ─────────────────────────────────────────────────────────────
echo ""
echo "Pre-checks..."
DRBD_DSTATE=$(na "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_DSTATE" | grep -q "UpToDate/UpToDate"; then
echo ""
echo " WARNING: DRBD dstate is '$DRBD_DSTATE' (not UpToDate/UpToDate)."
echo " Failing over with a partially-synced disk risks split-brain."
if ! $FORCE; then
echo " Use --force to proceed anyway (not recommended)."
exit 1
fi
echo " --force specified — proceeding despite non-ideal DRBD state."
else
echo " DRBD dstate: $DRBD_DSTATE — OK"
fi
QUORUM_OK=$(na "corosync-quorumtool -s 2>/dev/null | grep -c 'Quorate:.*Yes'" 2>/dev/null || echo "0")
if [[ "$QUORUM_OK" -lt 1 ]]; then
echo " ERROR: cluster does not have quorum — failover would be unsafe."
exit 1
fi
echo " Quorum: OK"
# ── Confirm ────────────────────────────────────────────────────────────────
if ! $FORCE && ! $DRY_RUN; then
echo ""
echo " This will move all resources from $ACTIVE_NODE$STANDBY_NODE."
echo " VIP and services will be unreachable for ~1030 seconds."
printf " Proceed? [y/N] "
read -r ANSWER
[[ "${ANSWER,,}" == "y" || "${ANSWER,,}" == "yes" ]] || { echo "Aborted."; exit 0; }
fi
# ── Capture active node's crm_node name ───────────────────────────────────
# crm_node -n returns the node name as registered in Pacemaker (may differ
# from hostname if Pacemaker was configured with explicit node names).
ACTIVE_CRMD_NAME=$(na "crm_node -n 2>/dev/null" 2>/dev/null || echo "$ACTIVE_NODE")
# ── Perform failover ───────────────────────────────────────────────────────
echo ""
echo "${DRY_PREFIX}Putting $ACTIVE_NODE into standby (resources will migrate to $STANDBY_NODE)..."
if ! $DRY_RUN; then
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v on" 2>/dev/null || true
fi
# ── Wait for resources to move ─────────────────────────────────────────────
echo "${DRY_PREFIX}Waiting up to ${TIMEOUT}s for XFS to mount on $STANDBY_NODE..."
MOVED=false
SPIN_CHARS=('|' '/' '-' '\')
SPIN_I=0
if $DRY_RUN; then
echo " [dry-run] would wait for mountpoint $XFS_MOUNT on $STANDBY_NODE"
MOVED=true
else
for i in $(seq 1 "$TIMEOUT"); do
if ns "mountpoint -q '${XFS_MOUNT}' 2>/dev/null" 2>/dev/null; then
printf "\r%-80s\r" ""
echo " Resources moved in ${i}s"
MOVED=true
break
fi
SPIN_I=$(( SPIN_I + 1 ))
SC="${SPIN_CHARS[$((SPIN_I % 4))]}"
printf "\r [%s] waiting... (%ds) " "$SC" "$i"
sleep 1
done
fi
if ! $MOVED; then
echo ""
echo "ERROR: XFS did not mount on $STANDBY_NODE within ${TIMEOUT}s."
echo ""
echo " Current resource state:"
na "crm_mon -1 2>/dev/null" 2>/dev/null | grep -E 'Started|Stopped|Promoted|Unpromoted|FAILED' | sed 's/^/ /' || true
echo ""
echo " Clearing standby to restore $ACTIVE_NODE (undo the failover attempt)..."
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
na "crm_resource --cleanup" 2>/dev/null || true
exit 1
fi
# ── Clear failure history ──────────────────────────────────────────────────
echo "${DRY_PREFIX}Clearing Pacemaker failure history..."
if ! $DRY_RUN; then
ns "crm_resource --cleanup 2>/dev/null" 2>/dev/null || true
fi
# ── Re-enable original active node as standby ─────────────────────────────
echo "${DRY_PREFIX}Re-enabling $ACTIVE_NODE (now standby — will not claim resources)..."
if ! $DRY_RUN; then
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
fi
# ── Wait briefly for DRBD resync to begin ─────────────────────────────────
if ! $DRY_RUN; then
sleep 5
fi
# ── Final state ────────────────────────────────────────────────────────────
echo ""
echo "Failover complete. Final state:"
echo ""
CRM_OUT_AFTER=""
if ! $DRY_RUN; then
CRM_OUT_AFTER=$(ns "crm_mon -1" 2>/dev/null || na "crm_mon -1" 2>/dev/null || true)
else
CRM_OUT_AFTER="$CRM_OUT"
fi
NEW_ACTIVE=$(echo "$CRM_OUT_AFTER" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -n "$NEW_ACTIVE" ]]; then
if [[ "$NEW_ACTIVE" == "$ACTIVE_NODE" ]]; then
echo " WARNING: $ACTIVE_NODE is still showing as active in crm_mon."
echo " Pacemaker may still be settling — check again in a few seconds."
else
echo " Active: $NEW_ACTIVE"
echo " Standby: $ACTIVE_NODE"
fi
fi
echo ""
RESOURCES_AFTER=$(echo "$CRM_OUT_AFTER" | awk '/Full List of Resources/,0' | tail -n +2 || true)
[[ -z "$RESOURCES_AFTER" ]] && RESOURCES_AFTER=$(echo "$CRM_OUT_AFTER" | \
grep -E 'Started|Stopped|Promoted|Unpromoted|FAILED|Master|Slave' || true)
[[ -n "$RESOURCES_AFTER" ]] && echo "$RESOURCES_AFTER" | sed 's/^/ /'
echo ""
echo " (DRBD resync of $ACTIVE_NODE may take a moment; monitor with:"
echo " ssh nixos@${ACTIVE_IP} 'sudo watch -n3 cat /proc/drbd')"
echo ""
echo "════════════════════════════════════════════════════"
+179
View File
@@ -0,0 +1,179 @@
#!/usr/bin/env python3
"""
fence_pve_ssh - Proxmox VE SSH fence agent for Pacemaker.
Uses SSH to reach the Proxmox host and run 'qm stop/start <vmid>'.
Deploy to /etc/pacemaker/fence_pve_ssh on both HA nodes (chmod +x).
Configuration (as pacemaker stonith resource attributes):
pve_host Proxmox host to SSH to (default: pve1.sweet.home)
pve_user SSH user (default: wayne)
key_file SSH private key path (default: /etc/fence-pve-ssh-key)
vmid_node1 VMID for ha-server-1
vmid_node2 VMID for ha-server-2
plug Node name to act on (set by pacemaker: ha-server-1 or ha-server-2)
action Action: off|on|reboot|status|list|metadata
"""
import argparse
import subprocess
import sys
import os
METADATA = """<?xml version="1.0" ?>
<resource-agent name="fence_pve_ssh" shortdesc="Proxmox VE SSH fence agent (test lab)">
<longdesc>Fences a VM on a Proxmox VE host by SSHing to the PVE host and
running qm stop/start. For test use only.</longdesc>
<vendor-url>https://proxmox.com</vendor-url>
<parameters>
<parameter name="action" required="1" unique="0">
<getopt mixed="-a, --action=[action]"/>
<content type="string" default="reboot"/>
<shortdesc lang="en">Fencing action: off|on|reboot|status|list</shortdesc>
</parameter>
<parameter name="plug" required="0" unique="0">
<getopt mixed="-n, --plug=[nodename]"/>
<content type="string"/>
<shortdesc lang="en">Cluster node name to fence</shortdesc>
</parameter>
<parameter name="pve_host" required="0" unique="0">
<getopt mixed="--pve-host=[host]"/>
<content type="string" default="pve1.sweet.home"/>
<shortdesc lang="en">Proxmox VE host to SSH to</shortdesc>
</parameter>
<parameter name="pve_user" required="0" unique="0">
<getopt mixed="--pve-user=[user]"/>
<content type="string" default="wayne"/>
<shortdesc lang="en">SSH user on the Proxmox host</shortdesc>
</parameter>
<parameter name="key_file" required="0" unique="0">
<getopt mixed="--key-file=[path]"/>
<content type="string" default="/etc/fence-pve-ssh-key"/>
<shortdesc lang="en">SSH private key file path</shortdesc>
</parameter>
<parameter name="vmid_node1" required="1" unique="0">
<getopt mixed="--vmid-node1=[vmid]"/>
<content type="string"/>
<shortdesc lang="en">VMID for ha-test-node1</shortdesc>
</parameter>
<parameter name="vmid_node2" required="1" unique="0">
<getopt mixed="--vmid-node2=[vmid]"/>
<content type="string"/>
<shortdesc lang="en">VMID for ha-test-node2</shortdesc>
</parameter>
</parameters>
<actions>
<action name="off" timeout="60s"/>
<action name="on" timeout="60s"/>
<action name="reboot" timeout="60s"/>
<action name="status" timeout="30s"/>
<action name="list" timeout="10s"/>
<action name="metadata" timeout="5s"/>
</actions>
</resource-agent>
"""
def parse_args():
p = argparse.ArgumentParser(add_help=False)
p.add_argument("-a", "--action", default="reboot")
p.add_argument("-n", "--plug")
p.add_argument("--pve-host", default="pve1.sweet.home")
p.add_argument("--pve-user", default="wayne")
p.add_argument("--key-file", default="/etc/fence-pve-ssh-key")
p.add_argument("--vmid-node1")
p.add_argument("--vmid-node2")
# Allow remaining unknown args (pacemaker may pass extra ones)
return p.parse_known_args()[0]
def ssh(pve_host, pve_user, key_file, cmd):
result = subprocess.run(
[
"ssh",
"-i", key_file,
"-o", "StrictHostKeyChecking=no",
"-o", "BatchMode=yes",
"-o", "ConnectTimeout=10",
f"{pve_user}@{pve_host}",
cmd,
],
capture_output=True,
text=True,
timeout=30,
)
return result
def get_vmid(args):
node = args.plug
if not node:
print("ERROR: --plug not specified", file=sys.stderr)
sys.exit(1)
mapping = {
"ha-server-1": args.vmid_node1,
"ha-server-2": args.vmid_node2,
}
vmid = mapping.get(node)
if not vmid:
print(f"ERROR: unknown node '{node}'", file=sys.stderr)
sys.exit(1)
return vmid
def main():
args = parse_args()
action = args.action.lower()
if action == "metadata":
print(METADATA)
sys.exit(0)
if action == "list":
if args.vmid_node1:
print("ha-server-1")
if args.vmid_node2:
print("ha-server-2")
sys.exit(0)
vmid = get_vmid(args)
if not os.path.exists(args.key_file):
print(f"ERROR: SSH key not found at {args.key_file}", file=sys.stderr)
sys.exit(1)
if action in ("off", "reboot"):
print(f"Stopping VM {vmid} ({args.plug}) on {args.pve_host}...")
r = ssh(args.pve_host, args.pve_user, args.key_file,
f"sudo /usr/sbin/qm stop {vmid}")
if r.returncode != 0:
print(f"ERROR stopping VM: {r.stderr}", file=sys.stderr)
sys.exit(1)
print(f"VM {vmid} stopped")
if action in ("on", "reboot"):
print(f"Starting VM {vmid} ({args.plug}) on {args.pve_host}...")
r = ssh(args.pve_host, args.pve_user, args.key_file,
f"sudo /usr/sbin/qm start {vmid}")
if r.returncode != 0:
print(f"ERROR starting VM: {r.stderr}", file=sys.stderr)
sys.exit(1)
print(f"VM {vmid} started")
if action == "status":
r = ssh(args.pve_host, args.pve_user, args.key_file,
f"sudo /usr/sbin/qm status {vmid}")
if r.returncode != 0:
print(f"ERROR querying VM status: {r.stderr}", file=sys.stderr)
sys.exit(1)
# qm status returns "status: running" or "status: stopped"
status_line = r.stdout.strip()
print(status_line)
if "stopped" in status_line:
sys.exit(2) # pacemaker interprets exit 2 as "off"
sys.exit(0) # running = exit 0
if __name__ == "__main__":
main()
+180
View File
@@ -0,0 +1,180 @@
#!/usr/bin/env bash
# health.sh — HA cluster health snapshot (read-only, non-destructive)
#
# Prints a compact status panel across both nodes: SSH reachability, quorum,
# DRBD state, Pacemaker resources, and service ports via the VIP.
# Run from any host with SSH access to the HA nodes.
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.2.229}" # vars.haServerVip
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
HA_USER="${HA_USER:-nixos}"
# ──────────────────────────────────────────────────────────────────────────
REACHABLE_1=false
REACHABLE_2=false
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
probe_node() {
local ip=$1
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${ip}" true 2>/dev/null && echo "ONLINE" || echo "OFFLINE"
}
section() { echo ""; echo "── $* ──"; }
echo "════════════════════════════════════════════════════"
echo " HA Cluster Health — $(date '+%Y-%m-%d %H:%M:%S')"
echo "════════════════════════════════════════════════════"
# ── Node reachability ──────────────────────────────────────────────────────
section "Nodes"
N1_STATUS=$(probe_node "$NODE1_IP")
N2_STATUS=$(probe_node "$NODE2_IP")
[[ "$N1_STATUS" == "ONLINE" ]] && REACHABLE_1=true
[[ "$N2_STATUS" == "ONLINE" ]] && REACHABLE_2=true
if ! $REACHABLE_1 && ! $REACHABLE_2; then
echo " ERROR: both nodes unreachable — cannot continue."
exit 1
fi
# ── Detect active node ─────────────────────────────────────────────────────
# crm_mon 2.x formats the Promoted line as " * Promoted: [ node ]" — the
# bullet * means ^\s*(Promoted|Masters): never matches. Filter out Unpromoted
# first, then match anywhere on the line.
ACTIVE_NODE=""
CRM_OUT=""
if $REACHABLE_1; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
elif $REACHABLE_2; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
ACTIVE_NODE=$(echo "${CRM_OUT:-}" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
STANDBY_NODE=""
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
STANDBY_NODE="$NODE2"
elif [[ "$ACTIVE_NODE" == "$NODE2" ]]; then
STANDBY_NODE="$NODE1"
fi
n1_tag=""; n2_tag=""
[[ "$ACTIVE_NODE" == "$NODE1" ]] && n1_tag=" [ACTIVE]" || n1_tag=" [STANDBY]"
[[ "$ACTIVE_NODE" == "$NODE2" ]] && n2_tag=" [ACTIVE]" || n2_tag=" [STANDBY]"
[[ -z "$ACTIVE_NODE" ]] && { n1_tag=""; n2_tag=""; }
printf " %-14s [%s]%s\n" "$NODE1" "$N1_STATUS" "$n1_tag"
printf " %-14s [%s]%s\n" "$NODE2" "$N2_STATUS" "$n2_tag"
if [[ -z "$ACTIVE_NODE" ]]; then
echo ""
echo " WARNING: could not determine active node from crm_mon."
echo " Pacemaker may still be settling, or both nodes may be in standby."
fi
# ── Quorum ─────────────────────────────────────────────────────────────────
section "Quorum"
if $REACHABLE_1; then
QUORUM=$(n1 "corosync-quorumtool -s 2>/dev/null" 2>/dev/null || echo "")
elif $REACHABLE_2; then
QUORUM=$(n2 "corosync-quorumtool -s 2>/dev/null" 2>/dev/null || echo "")
fi
if [[ -z "${QUORUM:-}" ]]; then
echo " corosync-quorumtool: unavailable"
else
QUORATE=$(echo "$QUORUM" | grep "Quorate:" | awk '{print $2}' || echo "?")
VOTES=$(echo "$QUORUM" | grep "Total votes:" | awk '{print $3}' || echo "?")
NEEDED=$(echo "$QUORUM" | grep "Quorum votes:" | awk '{print $3}' || echo "?")
echo " Quorate: $QUORATE Votes: $VOTES / Expected: $NEEDED"
fi
# ── DRBD ───────────────────────────────────────────────────────────────────
section "DRBD (ha-data)"
drbd_info_from() {
local node=$1 run=$2
local role dstate cs pct
role=$($run "drbdadm role ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
dstate=$($run "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
cs=$($run "grep -oE 'cs:[A-Za-z]+' /proc/drbd 2>/dev/null | head -1 | sed 's/cs://'" 2>/dev/null || echo "unknown")
pct=$($run "grep -oE \"sync'ed:[[:space:]]+[0-9.]+\" /proc/drbd 2>/dev/null | grep -oE '[0-9.]+$' | head -1" 2>/dev/null || echo "")
printf " %-14s role: %-22s dstate: %-25s cs: %s" \
"$node" "${role:-unknown}" "${dstate:-unknown}" "${cs:-unknown}"
[[ -n "$pct" ]] && printf " syncing: %s%%" "$pct"
echo ""
}
$REACHABLE_1 && drbd_info_from "$NODE1" n1 || echo " $NODE1 [OFFLINE]"
$REACHABLE_2 && drbd_info_from "$NODE2" n2 || echo " $NODE2 [OFFLINE]"
# ── Pacemaker (full crm_mon output) ───────────────────────────────────────
section "Pacemaker"
if [[ -n "${CRM_OUT:-}" ]]; then
echo "$CRM_OUT" | sed 's/^/ /'
else
echo " crm_mon returned no output — trying again without suppression:"
if $REACHABLE_1; then
n1 "crm_mon -1" || true
elif $REACHABLE_2; then
n2 "crm_mon -1" || true
fi
fi
# ── XFS mount ─────────────────────────────────────────────────────────────
section "XFS Mount ($XFS_MOUNT)"
check_mount() {
local node=$1 run=$2
local status
if $run "mountpoint -q '$XFS_MOUNT' 2>/dev/null" 2>/dev/null; then
local usage
usage=$($run "df -h '$XFS_MOUNT' 2>/dev/null | tail -1 | awk '{print \$3\"/\"\$2\" used (\"\$5\")\";}'" 2>/dev/null || echo "")
status="mounted"
[[ -n "$usage" ]] && status="mounted $usage"
else
status="not mounted"
fi
printf " %-14s %s\n" "$node" "$status"
}
$REACHABLE_1 && check_mount "$NODE1" n1 || echo " $NODE1 [OFFLINE]"
$REACHABLE_2 && check_mount "$NODE2" n2 || echo " $NODE2 [OFFLINE]"
# ── Service ports via VIP ──────────────────────────────────────────────────
section "Services via VIP ($VIP)"
check_port() {
local name=$1 port=$2
if bash -c "echo >/dev/tcp/${VIP}/${port}" 2>/dev/null; then
printf " %-10s port %-5s OK\n" "$name" "$port"
else
printf " %-10s port %-5s UNREACHABLE\n" "$name" "$port"
fi
}
if ping -c1 -W2 "$VIP" >/dev/null 2>&1; then
echo " Ping OK"
else
echo " Ping UNREACHABLE"
fi
check_port "NFS" 2049
check_port "iSCSI" 3260
echo ""
echo "════════════════════════════════════════════════════"
if [[ -n "$ACTIVE_NODE" ]]; then
echo " Active: $ACTIVE_NODE Standby: $STANDBY_NODE"
else
echo " Active: unknown (Pacemaker not settled)"
fi
echo "════════════════════════════════════════════════════"
echo ""
+276
View File
@@ -0,0 +1,276 @@
#!/usr/bin/env bash
# resize-data-disk.sh — online resize of the HA cluster data disk
#
# Three-phase process (all online-safe, no downtime required):
# 1. Proxmox: grow scsi1 on both HA VMs (qm resize)
# 2. Guest: rescan block device on both nodes so the kernel sees the new size
# 3. DRBD + XFS: drbdadm resize, then xfs_growfs — active node only
#
# Usage:
# scripts/ha/resize-data-disk.sh --size +20G [--force] [--dry-run]
#
# --size +NNg amount to grow scsi1 by, e.g. +20G, +50G (required)
# XFS and DRBD cannot shrink; only positive deltas accepted
# --force skip the interactive confirmation prompt
# --dry-run show what would be done without changing anything
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=../env.sh
source "${SCRIPT_DIR}/../env.sh"
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}"
NODE2_IP="${NODE2_IP:-192.168.2.227}"
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"
DRBD_RESOURCE="${DRBD_RESOURCE:-ha-data}"
DATA_DISK_SLOT="${DATA_DISK_SLOT:-scsi1}" # Proxmox disk name (scsi1 = data disk)
HA_USER="${HA_USER:-nixos}"
PVE_HOST="${PVE_HOST:-${PVE1_HOST}}"
PVE_SSH_USER="${PVE_SSH_USER:-${PROXMOX_SSH_USER}}"
# By-id symlink for the data disk; basename resolves to the raw block device.
# matches variables.nix's haServerDrbdDisk.
DATA_DISK_BYID="${DATA_DISK_BYID:-scsi-0QEMU_QEMU_HARDDISK_drive-${DATA_DISK_SLOT}}"
# ──────────────────────────────────────────────────────────────────────────
pve() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
"${PVE_SSH_USER}@${PVE_HOST}" "$@"; }
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
# ── Argument parsing ───────────────────────────────────────────────────────
SIZE=""
FORCE=false
DRY_RUN=false
while [[ $# -gt 0 ]]; do
case "$1" in
--size) shift; SIZE="${1:?--size requires a value (e.g. +20G)}" ;;
--force) FORCE=true ;;
--dry-run) DRY_RUN=true ;;
*) echo "Unknown argument: $1"
echo "Usage: $0 --size +NNg [--force] [--dry-run]"
exit 1 ;;
esac
shift
done
if [[ -z "$SIZE" ]]; then
echo "ERROR: --size is required (e.g. --size +20G)"
echo "Usage: $0 --size +NNg [--force] [--dry-run]"
exit 1
fi
# Only positive deltas — qm resize, DRBD, and XFS all refuse to shrink.
if [[ ! "$SIZE" =~ ^\+[0-9]+(G|M|T)$ ]]; then
echo "ERROR: --size must be a positive delta like +20G, +50G, +500M, +2T"
echo " (qm resize, drbdadm resize, and xfs_growfs can only grow, not shrink)"
exit 1
fi
DRY_PREFIX=""
$DRY_RUN && DRY_PREFIX="[dry-run] "
echo "════════════════════════════════════════════════════"
echo " HA Data Disk Resize — $(date '+%Y-%m-%d %H:%M:%S')"
echo " Size delta: $SIZE • Proxmox: $PVE_HOST"
$DRY_RUN && echo " MODE: dry-run — no changes will be made"
echo "════════════════════════════════════════════════════"
# ── Detect active node ─────────────────────────────────────────────────────
echo ""
echo "Detecting active node..."
CRM_OUT=""
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE1_IP}" true 2>/dev/null; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
fi
if [[ -z "$CRM_OUT" ]]; then
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE2_IP}" true 2>/dev/null; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
fi
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" (bullet *),
# so ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -z "$ACTIVE_NODE" ]]; then
echo "ERROR: could not determine active node from crm_mon."
echo " Is Pacemaker still settling? Try running scripts/ha/health.sh first."
exit 1
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
na() { n1 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
na() { n2 "$@"; }
fi
echo " Active node: $ACTIVE_NODE ($ACTIVE_IP)"
# ── Pre-check DRBD state ───────────────────────────────────────────────────
echo ""
echo "Pre-checks..."
DRBD_DSTATE=$(na "drbdadm dstate ${DRBD_RESOURCE} 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_DSTATE" | grep -q "UpToDate/UpToDate"; then
echo " WARNING: DRBD dstate is '$DRBD_DSTATE' (expected UpToDate/UpToDate)."
echo " Resizing with a partially-synced disk may cause issues."
if ! $FORCE; then
echo " Use --force to proceed anyway."
exit 1
fi
echo " --force specified — proceeding despite non-ideal DRBD state."
else
echo " DRBD dstate: $DRBD_DSTATE — OK"
fi
# ── Find VMIDs on Proxmox ─────────────────────────────────────────────────
echo ""
echo "Looking up VM IDs on ${PVE_HOST}..."
QM_LIST=$(pve "qm list 2>/dev/null" || true)
VMID1=$(echo "$QM_LIST" | awk -v name="$NODE1" '$0 ~ name {print $1}' | head -1)
VMID2=$(echo "$QM_LIST" | awk -v name="$NODE2" '$0 ~ name {print $1}' | head -1)
if [[ -z "$VMID1" ]]; then
echo " ERROR: could not find VMID for $NODE1 on $PVE_HOST"
echo " qm list output:"
echo "$QM_LIST" | sed 's/^/ /'
exit 1
fi
if [[ -z "$VMID2" ]]; then
echo " ERROR: could not find VMID for $NODE2 on $PVE_HOST"
echo " qm list output:"
echo "$QM_LIST" | sed 's/^/ /'
exit 1
fi
echo " $NODE1: VMID $VMID1"
echo " $NODE2: VMID $VMID2"
# ── Confirm ────────────────────────────────────────────────────────────────
if ! $FORCE && ! $DRY_RUN; then
echo ""
echo " Plan:"
echo " Phase 1 — qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE} (on $PVE_HOST)"
echo " qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE} (on $PVE_HOST)"
echo " Phase 2 — block device rescan on $NODE1 and $NODE2"
echo " Phase 3 — drbdadm resize + xfs_growfs on $ACTIVE_NODE"
echo " No downtime required (all operations are online-safe)."
printf " Proceed? [y/N] "
read -r ANSWER
[[ "${ANSWER,,}" == "y" || "${ANSWER,,}" == "yes" ]] || { echo "Aborted."; exit 0; }
fi
# ═══════════════════════════════════════════════════════════════
# Phase 1 — Resize both VM data disks in Proxmox
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 1 — Proxmox disk resize (${DATA_DISK_SLOT} ${SIZE} on both VMs) ──"
echo " ${DRY_PREFIX}qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE} ($NODE1 on ${PVE_HOST})"
if ! $DRY_RUN; then
pve "qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE}"
fi
echo " ${DRY_PREFIX}qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE} ($NODE2 on ${PVE_HOST})"
if ! $DRY_RUN; then
pve "qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE}"
fi
echo " Phase 1 done."
# ═══════════════════════════════════════════════════════════════
# Phase 2 — Rescan block device on both guest nodes
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 2 — Block device rescan (both nodes) ──"
rescan_node() {
local node_name=$1 run_fn=$2
# Resolve block device name from the stable by-id symlink on the guest.
local blk_dev=""
if ! $DRY_RUN; then
blk_dev=$($run_fn "bash -c 'basename \$(readlink -f /dev/disk/by-id/${DATA_DISK_BYID})'" 2>/dev/null || true)
fi
if [[ -z "$blk_dev" ]]; then
# Fall back: scsi1 → index 1 → sdb, scsi2 → sdc, etc.
local slot_idx
slot_idx=$(echo "$DATA_DISK_SLOT" | grep -oE '[0-9]+$' || echo "1")
blk_dev=$(printf "sd%s" "$(echo "abcdefghij" | cut -c$((slot_idx + 1)))")
echo " WARNING: could not resolve /dev/disk/by-id/${DATA_DISK_BYID} on $node_name;" \
"falling back to /dev/${blk_dev}" >&2
fi
echo " ${DRY_PREFIX}Rescanning /dev/${blk_dev} on ${node_name}..."
if ! $DRY_RUN; then
$run_fn "bash -c 'echo 1 > /sys/block/${blk_dev}/device/rescan'" 2>/dev/null
local new_size
new_size=$($run_fn "lsblk -nd -o SIZE /dev/${blk_dev} 2>/dev/null" 2>/dev/null || echo "unknown")
echo " /dev/${blk_dev} on $node_name now reports: $new_size"
fi
}
rescan_node "$NODE1" n1
rescan_node "$NODE2" n2
echo " Phase 2 done."
# ═══════════════════════════════════════════════════════════════
# Phase 3 — Grow DRBD metadata, then XFS (active node only)
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 3 — DRBD resize + XFS grow (on active node: $ACTIVE_NODE) ──"
echo " ${DRY_PREFIX}drbdadm resize ${DRBD_RESOURCE}"
if ! $DRY_RUN; then
na "drbdadm resize ${DRBD_RESOURCE}"
fi
echo " ${DRY_PREFIX}xfs_growfs ${XFS_MOUNT}"
if ! $DRY_RUN; then
na "xfs_growfs ${XFS_MOUNT}"
fi
echo " Phase 3 done."
# ── Verify ────────────────────────────────────────────────────────────────
echo ""
echo "── Verify ──"
if ! $DRY_RUN; then
DF_OUT=$(na "df -h '${XFS_MOUNT}' 2>/dev/null" 2>/dev/null || echo "")
if [[ -n "$DF_OUT" ]]; then
echo " ${XFS_MOUNT}:"
echo "$DF_OUT" | sed 's/^/ /'
fi
DRBD_DSTATE_AFTER=$(na "drbdadm dstate ${DRBD_RESOURCE} 2>/dev/null" 2>/dev/null || echo "unknown")
echo " DRBD dstate: $DRBD_DSTATE_AFTER"
if ! echo "$DRBD_DSTATE_AFTER" | grep -q "UpToDate/UpToDate"; then
echo " NOTE: DRBD is resyncing — normal immediately after resize."
echo " Monitor: ssh nixos@${ACTIVE_IP} 'sudo watch -n3 cat /proc/drbd'"
fi
else
echo " [dry-run] would verify df -h ${XFS_MOUNT} and drbdadm dstate on $ACTIVE_NODE"
fi
echo ""
echo "════════════════════════════════════════════════════"
echo " Resize complete."
echo " Active node: $ACTIVE_NODE"
echo "════════════════════════════════════════════════════"
echo ""
+303
View File
@@ -0,0 +1,303 @@
#!/usr/bin/env bash
# Add a NixOS host to the FreeIPA domain and produce a sops-encrypted keytab
# at secrets/<hostname>.keytab, ready for modules/ipa/client.nix.
#
# One command replaces three error-prone manual steps:
# 1. ipa host-add on the domain controller
# 2. ipa-getkeytab on the domain controller + SCP back
# 3. sops encrypt in-place (must be at secrets/<hostname>.keytab for
# the creation rule to match -- the common mistake that breaks sops)
#
# Usage:
# scripts/ipa/create-nixos-ipa-host-account.sh [options] <hostname>
#
# Arguments:
# <hostname> Short hostname, e.g. "tailscale-router". The FQDN is
# derived as <hostname>.<HOME_DOMAIN>.
#
# Options:
# --ip <addr> Register this IP with the IPA host record (optional).
# --dc <host> SSH to this host to run IPA commands.
# Default: $IPA_SERVER (from env.sh / environment).
# --dc-user <u> SSH user on the domain controller. Default: wayne.
# --dry-run Print what would be done without making any changes.
# -h, --help Show this message.
#
# Prereqs:
# 1. Run from the repo root (so .sops.yaml and secrets/ are found).
# 2. SSH access to the domain controller as --dc-user (default: wayne)
# with passwordless sudo (or sudo cached). IPA commands and kinit run
# as root via sudo so the Kerberos ticket is in root's cache where all
# ipa tools expect it. If there's no valid ticket, the script runs
# `sudo kinit admin` interactively — you'll be prompted for the IPA
# admin password once. The password never touches this script.
# 3. The host's age key(s) must already be in .sops.yaml. Run
# scripts/secrets/sync-host-keys.sh <flake-target> first so the host
# can decrypt its own keytab on boot. This script adds the .sops.yaml
# creation rule for secrets/<hostname>.keytab automatically, but the
# host age key anchor (&lxc-<hostname> etc.) must already exist —
# otherwise only the admin key can decrypt the keytab and the deployed
# host will fail to read it.
# 4. sops in PATH, or Nix available to run it via `nix run`.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# shellcheck source=../env.sh
source "${SCRIPT_DIR}/../env.sh"
# --- Argument parsing ---
DC_HOST="${IPA_SERVER}"
DC_USER="wayne"
IP_ADDR=""
DRY_RUN=false
TARGET=""
usage() {
sed -n '/^# Usage:/,/^[^#]/{ /^#/{ s/^# \?//; p } }' "$0"
exit "${1:-0}"
}
while [[ $# -gt 0 ]]; do
case "$1" in
--ip) IP_ADDR="$2"; shift 2 ;;
--dc) DC_HOST="$2"; shift 2 ;;
--dc-user) DC_USER="$2"; shift 2 ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage 0 ;;
-*) echo "Unknown flag: $1" >&2; usage 1 ;;
*)
if [[ -n "${TARGET}" ]]; then echo "Unexpected argument: $1" >&2; usage 1; fi
TARGET="$1"; shift
;;
esac
done
if [[ -z "${TARGET}" ]]; then
echo "Error: hostname required." >&2
usage 1
fi
# Reject FQDNs passed by mistake — the script appends HOME_DOMAIN itself.
# "nixos.sweet.home" → FQDN would become "nixos.sweet.home.sweet.home".
if [[ "${TARGET}" == *"."* ]]; then
echo "Error: <hostname> must be the short name (e.g. 'nixos'), not a FQDN." >&2
echo " The FQDN is derived automatically as ${TARGET}.${HOME_DOMAIN}." >&2
exit 1
fi
FQDN="${TARGET}.${HOME_DOMAIN}"
KEYTAB_SECRET="${REPO_ROOT}/secrets/${TARGET}.keytab"
# Temp path on the domain controller — use a name that won't collide.
DC_TMP="/tmp/nixos-keytab-${TARGET}-$$.keytab"
# --- Helpers ---
log() { echo "==> $*"; }
logn() { echo " $*"; }
run() {
if $DRY_RUN; then
echo "[dry-run] $*"
else
"$@"
fi
}
dc_run() {
# Run a command string on the domain controller via SSH.
if $DRY_RUN; then
echo "[dry-run] ssh ${DC_USER}@${DC_HOST} $*"
else
ssh "${DC_USER}@${DC_HOST}" "$@"
fi
}
# --- Locate sops ---
if command -v sops &>/dev/null; then
SOPS_CMD=(sops)
else
log "sops not in PATH — will use 'nix run nixpkgs#sops'"
SOPS_CMD=(nix run "nixpkgs#sops" --)
fi
# --- Preflight checks ---
cd "${REPO_ROOT}"
[[ -f .sops.yaml ]] || { echo "Error: .sops.yaml not found — run from repo root." >&2; exit 1; }
[[ -d secrets ]] || { echo "Error: secrets/ not found — run from repo root." >&2; exit 1; }
# --- Step 1: Ensure .sops.yaml has a creation rule for this keytab ---
#
# sops matches creation rules against the PATH of the file being encrypted,
# not the output path. To match secrets/<hostname>.keytab, the file must
# already be at that path when sops -e -i is called. The creation rule must
# also exist at that point or sops will refuse with "no matching creation
# rules found."
log "Checking .sops.yaml for creation rule: secrets/${TARGET}.keytab"
RULE_EXISTS=false
# Match "path_regex: secrets/<hostname>...keytab" — using .*keytab rather
# than \.keytab because the file stores the regex verbatim (\.keytab = two
# chars: backslash + dot), which a BRE \. (= escaped literal dot) won't span.
if grep -q "path_regex: secrets/${TARGET}.*keytab" .sops.yaml 2>/dev/null; then
RULE_EXISTS=true
logn "Rule already exists — skipping addition."
fi
if ! $RULE_EXISTS; then
# Collect which platform-variant age anchors exist in .sops.yaml for this
# hostname. The keytab is platform-agnostic (same FQDN regardless of
# whether lxc/proxmox/linode variant is deployed), so all platform anchors
# that have been registered get added as recipients.
RECIPIENTS=("*admin")
for platform in lxc proxmox linode; do
anchor="${platform}-${TARGET}"
if grep -q "^ - &${anchor} " .sops.yaml; then
RECIPIENTS+=("*${anchor}")
fi
done
if [[ ${#RECIPIENTS[@]} -eq 1 ]]; then
echo "Warning: no platform age keys found for '${TARGET}' in .sops.yaml." >&2
echo " Run scripts/secrets/sync-host-keys.sh <flake-target> first," >&2
echo " otherwise only the admin key can decrypt the keytab and the" >&2
echo " deployed host won't be able to read it at boot." >&2
echo " Continuing with admin-only encryption..." >&2
fi
# Build the indented recipient list for the YAML block.
RECIPIENT_YAML=""
for r in "${RECIPIENTS[@]}"; do
RECIPIENT_YAML+=" - ${r}"$'\n'
done
RECIPIENT_YAML="${RECIPIENT_YAML%$'\n'}" # strip trailing newline
NEW_RULE="
# Host keytab for ${TARGET} FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/${TARGET}\\.keytab\$
key_groups:
- age:
${RECIPIENT_YAML}"
if $DRY_RUN; then
echo "[dry-run] Would append to .sops.yaml:"
echo "${NEW_RULE}"
else
logn "Adding creation rule (recipients: ${RECIPIENTS[*]})"
printf '%s\n' "${NEW_RULE}" >> .sops.yaml
logn "Added."
fi
fi
# --- Step 2: Add IPA host account (idempotent) ---
log "Adding FreeIPA host account: ${FQDN}"
# Ensure there's a valid admin Kerberos ticket on the DC.
# ipa host-add and ipa-getkeytab both need one. All IPA commands run via
# sudo so the ticket must be in root's cache — check and refresh as root.
# ssh -t allocates a PTY so kinit (and sudo if needed) can prompt normally;
# no password ever touches this script or the shell history.
if ! $DRY_RUN; then
if ! ssh "${DC_USER}@${DC_HOST}" "sudo klist -s" &>/dev/null; then
log "No valid Kerberos ticket on ${DC_HOST} — running sudo kinit admin"
ssh -t "${DC_USER}@${DC_HOST}" "sudo kinit admin"
if ! ssh "${DC_USER}@${DC_HOST}" "sudo klist -s" &>/dev/null; then
echo "Error: kinit admin failed or produced no valid ticket." >&2
exit 1
fi
else
logn "Kerberos ticket on ${DC_HOST} is valid."
fi
fi
IP_FLAG=""
[[ -n "${IP_ADDR}" ]] && IP_FLAG="--ip-address=${IP_ADDR}"
# --force: create the host record even if DNS doesn't resolve it yet.
if $DRY_RUN; then
echo "[dry-run] ssh ${DC_USER}@${DC_HOST} sudo ipa host-add '${FQDN}' ${IP_FLAG} --force"
else
HOST_ADD_OUT=$(ssh "${DC_USER}@${DC_HOST}" "sudo ipa host-add '${FQDN}' ${IP_FLAG} --force 2>&1") \
&& HOST_ADD_RC=0 || HOST_ADD_RC=$?
if [[ $HOST_ADD_RC -eq 0 ]]; then
echo "${HOST_ADD_OUT}"
elif echo "${HOST_ADD_OUT}" | grep -q "already exists"; then
logn "(host already registered)"
else
echo "Error: ipa host-add failed (exit ${HOST_ADD_RC}):" >&2
echo "${HOST_ADD_OUT}" >&2
exit 1
fi
fi
# --- Step 3: Fetch the keytab from the domain controller ---
log "Fetching keytab for host/${FQDN}"
# Remove the plaintext keytab if the script aborts before encryption completes.
# The trap is cleared at the end of step 4 once sops has encrypted it in-place.
trap 'rm -f "${KEYTAB_SECRET}"' EXIT
dc_run "sudo ipa-getkeytab -s '${IPA_SERVER}' -p 'host/${FQDN}' -k '${DC_TMP}'"
if $DRY_RUN; then
echo "[dry-run] Would stream ${DC_USER}@${DC_HOST}:${DC_TMP} → secrets/${TARGET}.keytab"
else
logn "Streaming keytab from ${DC_HOST}:${DC_TMP} → secrets/${TARGET}.keytab"
# scp can't read a root-owned temp file as ${DC_USER}; pipe through sudo cat instead.
ssh "${DC_USER}@${DC_HOST}" "sudo cat '${DC_TMP}'" > "${KEYTAB_SECRET}"
logn "Removing temp file on ${DC_HOST}"
dc_run "sudo rm -f '${DC_TMP}'"
fi
# --- Step 4: Encrypt in-place ---
#
# The file must already be at secrets/<hostname>.keytab (done above) so
# sops matches the creation rule by path. Using -i (in-place) rather than
# stdout redirect keeps the path intact through the encrypt call.
log "Encrypting secrets/${TARGET}.keytab in-place with sops"
run "${SOPS_CMD[@]}" -e --input-type binary -i "${KEYTAB_SECRET}"
# Encryption succeeded — the file is now sops-encrypted; cancel the cleanup trap.
trap - EXIT
# --- Done ---
if ! $DRY_RUN; then
echo ""
echo "Done. secrets/${TARGET}.keytab is sops-encrypted and ready."
echo ""
echo "Next steps:"
echo " 1. Verify: grep '\"data\": \"ENC' secrets/${TARGET}.keytab"
echo " 2. Stage and commit:"
echo " git add secrets/${TARGET}.keytab .sops.yaml"
echo " git commit -m 'secrets: add IPA keytab for ${TARGET}'"
echo " 3. Add to hosts/${TARGET}/host.nix (networking block and imports):"
echo ""
echo " networking = {"
echo " hostName = \"${TARGET}\";"
echo " domain = vars.homeDomain; # required for Kerberos FQDN"
echo " nameservers = [ vars.domainControllerIp ]; # IPA DNS"
echo " ..."
echo " };"
echo ""
echo " imports = ["
echo " (import ../../modules/ipa/client.nix {"
echo " keytabSopsFile = ../../secrets/${TARGET}.keytab;"
echo " caCertFile = ../../certs/ipa-ca.crt;"
echo " })"
echo " ];"
echo ""
echo " 4. Deploy: nixos-rebuild switch (or create-proxmox-resource.sh)"
fi
+92 -39
View File
@@ -302,6 +302,12 @@ platform_prefix="lxc"
[[ -z "$cores" ]] && cores="$PROXMOX_DEFAULT_CORES" [[ -z "$cores" ]] && cores="$PROXMOX_DEFAULT_CORES"
[[ -z "$memory" ]] && memory="$PROXMOX_DEFAULT_MEMORY_MB" [[ -z "$memory" ]] && memory="$PROXMOX_DEFAULT_MEMORY_MB"
if [[ "$type" == "vm" && -n "$disk_size" ]]; then
echo "WARNING: --disk-size is LXC-only for create mode and is ignored for VMs." >&2
echo " VM disk size comes from proxmoxImageSize in variables.nix (currently ${disk_size}G was requested)." >&2
echo " To expand after creation, use: --modify --vmid <n> --grow-disk <GB>" >&2
fi
# --- discover / resolve the flake target from --host -------------------- # --- discover / resolve the flake target from --host --------------------
# Emits "<target>\t<hostName>" pairs for every ${platform_prefix}-* flake # Emits "<target>\t<hostName>" pairs for every ${platform_prefix}-* flake
# target -- the one source both --list and the --host lookup below read # target -- the one source both --list and the --host lookup below read
@@ -354,6 +360,14 @@ fi
# feeds straight into the guest's real hostname) disagree with host.nix. # feeds straight into the guest's real hostname) disagree with host.nix.
[[ -z "$name" ]] && name="$host" [[ -z "$name" ]] && name="$host"
# For VM builds: the diskoImagesScript (run via QEMU on the node) writes the
# raw disk image as <hostname>.raw into the CWD it was called from (the remote
# repo dir), not to /var/lib/vz/import/ or anywhere else. Import directly from
# there -- no intermediate mv that can fail crossing filesystem boundaries or
# leave a stale file on error.
vm_built_raw=""
[[ "$type" == "vm" ]] && vm_built_raw="${remote_repo_dir}/${host}.raw"
# --- refuse to duplicate a host that's already live on the node --------- # --- refuse to duplicate a host that's already live on the node ---------
# Queries the node itself (qm/pct's own name/hostname config), not any # Queries the node itself (qm/pct's own name/hostname config), not any
# static list in this repo -- a file can't track whether a resource still # static list in this repo -- a file can't track whether a resource still
@@ -688,10 +702,14 @@ if [[ -n "$image" ]]; then
elif [[ "$force_rebuild" -eq 1 ]]; then elif [[ "$force_rebuild" -eq 1 ]]; then
echo "--force-rebuild: skipping the existing-image check on ${node}." echo "--force-rebuild: skipping the existing-image check on ${node}."
else else
echo "==> Checking whether ${node} already has ${remote_path}..." # VMs: check for the raw image in the remote repo dir (where disko writes it).
# LXC: check for the tarball in iso_storage (where the LXC build stages it).
_check_path="$remote_path"
[[ "$type" == "vm" ]] && _check_path="$vm_built_raw"
echo "==> Checking whether ${node} already has ${_check_path}..."
if [[ "$dry_run" -eq 1 ]]; then if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would check: ssh ${ssh_target} -- test -f ${remote_path}" echo "[dry-run] would check: ssh ${ssh_target} -- test -f ${_check_path}"
elif ssh "$ssh_target" "test -f '${remote_path}'" 2>/dev/null; then elif ssh "$ssh_target" "test -f '${_check_path}'" 2>/dev/null; then
echo "Found it -- reusing, skipping build (use --force-rebuild to override)." echo "Found it -- reusing, skipping build (use --force-rebuild to override)."
image_already_remote=1 image_already_remote=1
else else
@@ -766,23 +784,29 @@ REMOTE_SCRIPT
fi fi
else else
if [[ "$dry_run" -eq 1 ]]; then if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would build on ${node}: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\" echo "[dry-run] would build on ${node}: NIXOS_HOST_KEYS_DIR=\$(pwd)/host-keys nix build --impure --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript" echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
echo "[dry-run] would run: ${sudo_display}./result-${flake_target} \\" echo "[dry-run] would run: ${sudo_display}./result-${flake_target} --build-memory 2048"
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\" echo "[dry-run] image will be at ${vm_built_raw} (imported from there; no mv to /var/lib/vz/import/)"
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\"
echo "[dry-run] --build-memory 2048"
echo "[dry-run] would stage the result at ${remote_path}"
local_image="<built-image>.raw" local_image="<built-image>.raw"
else else
echo "==> Building Disko image for ${flake_target} on ${node}..." echo "==> Building Disko image for ${flake_target} on ${node}..."
# See the LXC branch above for why this is one %q-quoted command # See the LXC branch above for why this is one %q-quoted command
# string rather than separate ssh argv elements. # string rather than separate ssh argv elements.
printf -v remote_cmd 'bash -s -- %q %q %q %q %q %q' \ # $7 = image_name (hostname, the diskoImagesScript's own output filename).
"$remote_repo_dir" "$flake_target" "$remote_dir" "$remote_filename" "$NIX_EXTRA_OPTS" "$sudo_prefix" #
# NIXOS_HOST_KEYS_DIR + --impure: modules/platforms/proxmox.nix reads
# this env var at eval time (like lxc.nix) to embed the clan SSH host
# key in environment.etc. nixos-install's own activation then places the
# key on the target disk, so sshd-keygen finds it already present and
# skips generation. --pre-format-files put the key on the QEMU builder
# VM's rootfs (not the target disk), so sshd-keygen regenerated a fresh
# key -- one not registered in .sops.yaml -- and sops could never decrypt.
printf -v remote_cmd 'bash -s -- %q %q %q %q %q %q %q' \
"$remote_repo_dir" "$flake_target" "$remote_dir" "$remote_filename" "$NIX_EXTRA_OPTS" "$sudo_prefix" "$host"
ssh "$ssh_target" "$remote_cmd" <<'REMOTE_SCRIPT' ssh "$ssh_target" "$remote_cmd" <<'REMOTE_SCRIPT'
set -euo pipefail set -euo pipefail
repo_dir="$1"; target="$2"; dest_dir="$3"; dest_name="$4"; nix_extra_opts_str="$5"; sudo_pfx="$6" repo_dir="$1"; target="$2"; dest_dir="$3"; dest_name="$4"; nix_extra_opts_str="$5"; sudo_pfx="$6"; image_name="$7"
declare -a NIX_OPTS=() declare -a NIX_OPTS=()
[[ -n "$nix_extra_opts_str" ]] && eval "NIX_OPTS=(${nix_extra_opts_str})" [[ -n "$nix_extra_opts_str" ]] && eval "NIX_OPTS=(${nix_extra_opts_str})"
cd "$repo_dir" cd "$repo_dir"
@@ -794,24 +818,26 @@ if [[ ! -f "host-keys/${target}_ssh_host_ed25519_key" ]]; then
echo "and ensure it was synced here before starting the build." >&2 echo "and ensure it was synced here before starting the build." >&2
exit 1 exit 1
fi fi
nix build --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \ # Build diskoImagesScript with NIXOS_HOST_KEYS_DIR so proxmox.nix embeds the
# clan SSH key in environment.etc (same as lxc.nix). This causes nixos-install
# to place the key on the target disk, so sshd-keygen finds it and skips
# generation -- the disk image boots with the registered key, sops decrypts.
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
--no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \
".#nixosConfigurations.${target}.config.system.build.diskoImagesScript" \ ".#nixosConfigurations.${target}.config.system.build.diskoImagesScript" \
--out-link "result-${target}" --out-link "result-${target}"
$sudo_pfx "./result-${target}" \ # Remove any stale .raw from a previous failed build so the post-build check
--pre-format-files "$(pwd)/host-keys/${target}_ssh_host_ed25519_key" /etc/ssh/ssh_host_ed25519_key \ # below is unambiguous (diskoImagesScript writes to CWD as ${image_name}.raw).
--pre-format-files "$(pwd)/host-keys/${target}_ssh_host_ed25519_key.pub" /etc/ssh/ssh_host_ed25519_key.pub \ $sudo_pfx rm -f "${image_name}.raw" 2>/dev/null || true
--build-memory 2048 $sudo_pfx "./result-${target}" --build-memory 2048
built="$(find . -maxdepth 1 -name '*.raw' -newer "result-${target}" | head -1)" if [[ ! -f "${image_name}.raw" ]]; then
if [[ -z "$built" ]]; then echo "ERROR: ${image_name}.raw not found in ${repo_dir} after build -- disko/QEMU may have failed." >&2
echo "ERROR: no .raw image found in ${repo_dir} after build." >&2
exit 1 exit 1
fi fi
$sudo_pfx mkdir -p "$dest_dir" echo "Built image: ${repo_dir}/${image_name}.raw"
$sudo_pfx mv "$built" "${dest_dir}/${dest_name}"
echo "Built and staged: ${dest_dir}/${dest_name}"
REMOTE_SCRIPT REMOTE_SCRIPT
local_image="$remote_path" local_image="$vm_built_raw"
echo "Built on ${node}: ${remote_path}" echo "Built on ${node}: ${vm_built_raw}"
fi fi
fi fi
fi fi
@@ -841,17 +867,17 @@ if [[ "$type" == "lxc" ]]; then
local_swap="${swap:-$memory}" local_swap="${swap:-$memory}"
# --unprivileged: read back from modules/platforms/lxc.nix's own # --unprivileged: read back from modules/platforms/lxc.nix's own
# proxmoxLXC.privileged (via flake_target_lxc_privileged) rather than # proxmoxLXC.privileged (via flake_target_lxc_privileged) rather than
# hardcoded, since that's no longer the same for every lxc-* target -- # hardcoded. lxc.nix derives this automatically: any lxc-* host whose
# lxc-docker sets it true so the container's NFS mounts work at all (the # config.fileSystems has an NFS entry gets privileged=true, because the
# kernel's NFS client can't mount from inside any unprivileged # kernel's NFS client (FS_USERNS_MOUNT not set) rejects NFS mounts from
# container's user namespace, no matter what AppArmor allows -- see that # inside any non-init user namespace -- exactly what an unprivileged
# option's own comment). The NixOS config inside the image bakes in # container's UID-mapped root lives in -- with EPERM at the VFS layer,
# cgroup/capability/mount expectations matching whichever value it was # regardless of AppArmor (see lxc.nix's own comment). The NixOS config
# built with, so this must stay in sync with it -- `pct create`'s own # bakes in cgroup/capability/mount expectations matching whichever value
# CLI default for this flag is privileged (unlike the web UI, which # it was built with, so this must stay in sync -- `pct create`'s CLI
# defaults its checkbox the other way), so leaving it unset would create # default is privileged (unlike the web UI, which defaults the other
# a privileged container running a NixOS config that assumes # way), so leaving it unset would create a privileged container running
# unprivileged for every target except lxc-docker, a real mismatch. # a NixOS config that assumes unprivileged, a real mismatch.
privileged_eval="$(flake_target_lxc_privileged "$repo_root" "$flake_target")" privileged_eval="$(flake_target_lxc_privileged "$repo_root" "$flake_target")"
unprivileged_flag=1 unprivileged_flag=1
[[ "$privileged_eval" == "true" ]] && unprivileged_flag=0 [[ "$privileged_eval" == "true" ]] && unprivileged_flag=0
@@ -889,14 +915,35 @@ else
--net0 virtio,bridge=${bridge} --bios ovmf --machine q35 --scsihw virtio-scsi-pci \ --net0 virtio,bridge=${bridge} --bios ovmf --machine q35 --scsihw virtio-scsi-pci \
--efidisk0 ${storage}:1,efitype=4m,pre-enrolled-keys=0 --agent enabled=1" --efidisk0 ${storage}:1,efitype=4m,pre-enrolled-keys=0 --agent enabled=1"
# VMs built on the node: import from the repo dir (where disko/QEMU wrote it).
# VMs from --image: import from remote_path (where scp uploaded it).
_import_path="${remote_path}"
[[ -z "$image" ]] && _import_path="${vm_built_raw}"
if [[ "$dry_run" -eq 1 ]]; then if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] ssh ${ssh_target} -- ${sudo_display}qm importdisk ${vmid} ${remote_path} ${storage}" echo "[dry-run] ssh ${ssh_target} -- ${sudo_display}qm importdisk ${vmid} ${_import_path} ${storage}"
echo "[dry-run] (would parse the resulting disk identifier from that output)" echo "[dry-run] (would parse the resulting disk identifier from that output)"
echo "[dry-run] ssh ${ssh_target} -- ${sudo_display}qm set ${vmid} --scsi0 ${storage}:<parsed-disk-id>" echo "[dry-run] ssh ${ssh_target} -- ${sudo_display}qm set ${vmid} --scsi0 ${storage}:<parsed-disk-id>"
else else
importdisk_output="$(ssh "$ssh_target" "${sudo_prefix} qm importdisk ${vmid} ${remote_path} ${storage}")" if ! importdisk_output="$(ssh "$ssh_target" "${sudo_prefix} qm importdisk ${vmid} ${_import_path} ${storage}" 2>&1)"; then
echo "ERROR: qm importdisk failed:" >&2
echo "${importdisk_output}" >&2
exit 1
fi
echo "$importdisk_output" echo "$importdisk_output"
disk_id="$(echo "$importdisk_output" | grep -oP "(?<=Successfully imported disk as ')[^']+" | sed 's/^unused[0-9]*://')" # PVE output format: "unusedN: successfully imported disk '<storage>:<vol>'"
# (lowercase "successfully", no "as"; the primary regex targets this form; the
# || true inside the substitution prevents set -e from aborting when grep finds
# no match -- without it the script would silently exit before reaching the
# fallback whenever the PVE format doesn't match).
disk_id="$(echo "$importdisk_output" | grep -oP "successfully imported disk '\\K[^']+" || true)"
if [[ -z "$disk_id" ]]; then
# Fallback for other PVE output variants: read qm config directly.
unused_line="$(ssh "$ssh_target" "${sudo_prefix} qm config ${vmid}" | grep '^unused[0-9]*:' | head -1 || true)"
if [[ -n "$unused_line" ]]; then
disk_id="${unused_line#*: }"
echo "Note: disk ID resolved from qm config: ${disk_id}"
fi
fi
if [[ -z "$disk_id" ]]; then if [[ -z "$disk_id" ]]; then
echo "ERROR: couldn't parse the imported disk identifier from qm importdisk's output above." >&2 echo "ERROR: couldn't parse the imported disk identifier from qm importdisk's output above." >&2
echo "The VM shell (${vmid}) and imported disk both exist -- finish attaching it by hand:" >&2 echo "The VM shell (${vmid}) and imported disk both exist -- finish attaching it by hand:" >&2
@@ -905,6 +952,12 @@ else
exit 1 exit 1
fi fi
remote "${sudo_prefix} qm set ${vmid} --scsi0 ${disk_id}" remote "${sudo_prefix} qm set ${vmid} --scsi0 ${disk_id}"
# The disk data is now in ZFS; remove the source raw file (only for images
# we built on the node -- --image uploads are the operator's to manage).
if [[ -z "$image" ]]; then
ssh "$ssh_target" "${sudo_prefix} rm -f '${_import_path}'" 2>/dev/null || \
echo "Warning: couldn't remove ${_import_path} from ${node} -- you can delete it manually" >&2
fi
fi fi
remote "${sudo_prefix} qm set ${vmid} --boot order=scsi0" remote "${sudo_prefix} qm set ${vmid} --boot order=scsi0"
remote "${sudo_prefix} qm start ${vmid}" remote "${sudo_prefix} qm start ${vmid}"
+16 -25
View File
@@ -85,6 +85,10 @@ ensure_admin_decrypt_key() {
fi fi
local key_file="$DEFAULT_SOPS_AGE_KEY_FILE" local key_file="$DEFAULT_SOPS_AGE_KEY_FILE"
# Expand a leading ~ that survived variable substitution without tilde
# expansion (happens when SOPS_AGE_KEY_FILE or XDG_CONFIG_HOME is set with
# a literal ~ in the caller's environment).
key_file="${key_file/#~\//$HOME/}"
if [[ -s "$key_file" ]]; then if [[ -s "$key_file" ]]; then
echo "Found existing sops age key at ${key_file}." echo "Found existing sops age key at ${key_file}."
@@ -93,36 +97,23 @@ ensure_admin_decrypt_key() {
if [[ "$dry_run" -eq 1 ]]; then if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] No sops age decryption key found (checked \$SOPS_AGE_KEY, \$SOPS_AGE_KEY_FILE, ${key_file})." echo "[dry-run] No sops age decryption key found (checked \$SOPS_AGE_KEY, \$SOPS_AGE_KEY_FILE, ${key_file})."
echo "[dry-run] Would generate a new one here -- continuing the dry run without one; any" echo "[dry-run] Continuing dry run without one -- any 'would re-encrypt' output below"
echo "[dry-run] 'would re-encrypt' output below couldn't actually run for real yet." echo "[dry-run] couldn't actually run for real until a key is present."
return return
fi fi
echo "No sops age decryption key found (checked \$SOPS_AGE_KEY, \$SOPS_AGE_KEY_FILE, ${key_file})." cat >&2 <<EOF
echo "Generating a new one at ${key_file}..." No sops age decryption key found (checked \$SOPS_AGE_KEY, \$SOPS_AGE_KEY_FILE, ${key_file}).
mkdir -p "$(dirname "$key_file")"
nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -o '${key_file}'" 2>&1 | grep -v "^Public key:" || true
local new_pub
new_pub="$(age_pubkey_from_identity_file "$key_file")"
cat <<EOF Place your admin age private key at ${key_file}, or set SOPS_AGE_KEY (inline
key) or SOPS_AGE_KEY_FILE (path to a different key file) and re-run.
A brand-new age key was just generated -- it cannot decrypt anything that If the key is truly missing (not just mislocated), this is a manual recovery
already exists in secrets/*.yaml, since nothing was ever encrypted for it. situation -- generating a brand-new admin key won't help, since it cannot
That trust can't be bootstrapped automatically (nobody can decrypt a file decrypt anything already encrypted for the old one. Each secrets/*.yaml is
for a recipient that didn't exist when it was last encrypted). also encrypted for its respective host key(s), so a running deployed host can
still decrypt what it needs -- but the admin key is required for re-encryption
To actually use this key: (e.g. adding new recipients via sops updatekeys).
1. Have someone who currently CAN decrypt replace the &admin entry in
.sops.yaml with this public key:
${new_pub}
2. They re-encrypt every secrets/*.yaml:
sops updatekeys --yes secrets/common.yaml
sops updatekeys --yes secrets/nix-cache.yaml
sops updatekeys --yes secrets/server.yaml
3. Re-run this script.
Exiting without making any other changes.
EOF EOF
exit 1 exit 1
} }
+133 -115
View File
@@ -5,211 +5,229 @@ sops:
age: age:
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrMGd3ZVNlNXdmOUZMUzdQ YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiU2IxZ056SUs3SVZiUTlQ
TW1acEs0NFA2Q01rM2dkc1h0NHkzQmhiWFZZCjMyK202VWdlaGhsZW04MnVwUVdO NXZEYmg3VzJzeWJPRnZXcUtOMGQ1MElBU0dVCm5xSURZZ0sxTjVxQnhpVVNEU092
alA0Q2FETThsYkhSS0hKdHBaS3VaY28KLS0tIG1Gdk8yalREOUtIZTUyY2p1UHlJ bDBHbFAzZ3hzejc5NmM2U0FQTkRNSTQKLS0tIC9OOURPRlcvQXIxMVZXR1JjUWRX
eG5iQnJsaTJBY3Y1dkw1c0VEaDQwdDQKfV04fLy32Lp2ZQ2VnvQ0h/Vsf+qdaJiv bE5ja3FoWXFIZ2ZLTklqWHhMRHBTRWcK2Z4sEcdyvx1rdBIGrbNw7bYvEPXdobvF
DnLXGZ9hE5yzpKWkQIRgqYGBkF8PkH0YC4OIaVkA53wrtjqS4ZHR9Q== h0aNihn817DvtVr8eeucj22iYCS40etElQNcO5i68t5wab4/d3rUbQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjWFNRY2FiK3VkSm1RdHBn YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrN0lNNGtYbE8rRTB0U2hL
cWl1ZStLcTRFZWY5VVI5N0FhODZvR1ltM0FvCnBHZUtTUm9QeHNlbVBoZEx1V3Fa eUNOZUNFRGk1RzE3R1VjV0Q3ckszcDNHWGxJCkhFN25JbDVSaWJIZ09uNzczbkM1
Nk9iMmJKVnhocEpERi9leE1ySUtNMFkKLS0tIDRRYkxnbU90S2RyMHdJNzRJNXBi ZWZ1Y2wrTHZ6VzlnVWRoT2JOdDZHM0UKLS0tIHQ2ZE9BaFRCeDNqbldRQkpLclI4
QjRmZFhVakVic2tYODZHcWtJRmNQTDQK7G8eSJInt11P0DiL9uzNQ/ZHHLVNIYPe dXZqOXNaR2Q2QWJtRlZUa0hLaTh2YlkKeCh3+yIt0sjh2Ig8m/YQifhMDHnTpuZE
bvlhuGkEuQ/+j5sVSKOfSI2Y7CvM7TpE3APyKBcLG3ajYg6F/Ev3SA== zHBE74SUyWFmqLgtd1b8Rx5gn9vBcS/3SMv3yKWispg4DZ3bCOeNFA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age19m0m7vdfg86yqy8l5mmle5jdd0unrn3f55t232w8h5ey42cqw34sfpt32n recipient: age19m0m7vdfg86yqy8l5mmle5jdd0unrn3f55t232w8h5ey42cqw34sfpt32n
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1S3dOcFdwV2NoMW1oMnY1 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5c2luYmpYcVVxRHROQlpk
VGdxeXJVR1lsbzNVZHlGb0NGOVo0SndiakVjCmg0QlNnZDV5RlFja2hCbVRXV1VF cTM4cm9uNEtVS21pK2pDRVRySmxYS3NwdGg0CnQvS3g4Q2tSdlpVeWZxbzArcTM1
S1ZtbC9KU0U1ZW9zeVoyR3hxNW1XTFUKLS0tIFUzWTJhTzM4QnpWV3h1OFU0N3BK YWRMcDAxcWgySFRqRC9XSElsTVorVEEKLS0tIFd6YjdrY2ZXQVBaZ01wMTIvV2sv
RnF1N2k0S0lIVitoNDJLUmZqdHRzZVkKUfNg24p8zxb3749v/A1BOKCNw75AUKpf ZEk0TG0ybENlTlhpZnovMXh0OGd5V1kKEbjrmKP6Su16KRDQ4PMqi0tHRS6+PfuT
RUmFCw5DDWF2aNM0mZqcjjVmJ/FRKV2HXwwUGsHPKSOTnKfOUlPNKA== 0v5DEyi8EDtIYSMzzdF7Jk2C0h5XuRux45Z6E6JhKW5c788dGQHz/A==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39 recipient: age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzazl5MUpUNEVNMTdLWXBp YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1eW5kd1FrSGJaSDkwTzdY
c1hkaVhjcTZXTUNsWS96QVVWb0RVQlZ4VlNvCnE3TFRySU5jTFk2WjBONUQyQUhl SnJhR2hqRkNTTGtSTWsvYmZpSlVWV0xvYkRZCmtqR1FQOTRQR3dGYU8zNFpMdkUr
U1lNTDFTRmZhMUFyZmpVY2xpaUVxRW8KLS0tIE9MQ1M2U2ZXQmRCVll2UGRWL1RG SWN2KzhXaE1rYWZYNHJsOVVvRDY3R00KLS0tIG1aUldSWHlHeHkrcjF2TENvazR3
K2tnU0NOKzU5dkpiekF0Vk1VM0ZZZDgKK13aFypGAqrKWPOr3UwtXI1EoXf1+UzS YXJLUXV3VVNSWEdseFlZQUJPQUF6OFkKOPKH52muUha+MV2leWDiqpuTaIj0ZKzV
rBqcwnX6WPxSKUwWoins4Aojek4QhbhY4R5ei6rRS0KEQeryGxy8bg== keEdyqkIck6KKfIjphNKTW5ujiuKjGQbRliquBknfCRGMnRlMNfmkA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy recipient: age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZUpVT1p4TGZyNC9qMm5G YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmTDExNFpCZHZmWnpwYis4
UzNHV08rWURjY2lqS3FTWFdoc0FYYTFjb0NBCk1mZ2JzaXk5RmExNE9xWGZ5K0pv bFJGZXV5UksyS01MT0d3cXE2Wno2Zzh6OFFnCm9hT3MyZ0hTbnkzdXk3NmN1TUVF
UEtqMkltV0dIWll5eVBUVVRNOUNDWUUKLS0tIGpKNVJudUM1UGNvaGl1UDBOeFA1 OGtCZnlJRmZJaWJWaE1pMGcrTzJzZm8KLS0tIFE1L3ZxeUl6WVA2UXN2b0lIcTBl
RjUyRlZ6a0Y4SXNsL21zSURVRk9KTFEKU1L6BQ6ZlYQQtqx3uF/uM5CQ1ercmvRT YmtXZnZaMml6ejY1Y3lRUFFtTEVMS0UKZbcuunA9GRqctCfC0Fh2foKt7n3l8KkE
TL3r2/Y07gE7CjRn3pR9z0co8KndGzxV6YR+ubyWptwBS8KQh5stkw== FNjjxUzL7OQjNG9M1aAtRMMO5KXL7YKXRUJWOPcpiLeLUNvDLeLXvA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt recipient: age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsaWxxWS9xQXViY3VnUEx2 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQYlp5U0FvVkdIKzBGZVRl
V09FbTI0WGtNbW0yclhOSGZDbG5NUTNaTkFFCitkcjJ3OE9BSnN4bjFWcE9nYVBk UEtOOG5kVHVLQ0ptVTVpdG9Lb1Y3bzlOODNzCmNYN0Y5ZGNzTGFGYmxJcWFxaExn
ZzMyVHlJQ2wwdU5JOXdCQm9oNkhNd2MKLS0tIE9tRzFYS05vSkUwWFRkaTdtc0k0 NkkrT2RLSlZTdmFiNks3RVZXQlBEYnMKLS0tIDdCVlBxUG8vYU05d3c2MXhjQjhM
blVoMWV0QklBVkluT0Z4NHYyS1F0blUKO+Uc0of/V77ZUZOsxTzeH8/LmmAOQt+J TTVJUGxlWUxWalBRYzFKQUQzd1R6dHcKPz1ciNICph7qnqSH5WYmOjUCNHsIAmTX
x/COHxnLCnZ4eWI6q1a0Qn5Br15OJYTxUI2QTV4goTnXBNUDo9wdpQ== lRnbFCZdD/ZIFRK8OYAyCGfQ42UYSTf16kH2jDdTFUy7Fk2+O7lNIA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs recipient: age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsLzFnZkdVdGQ2dHRwcnRt YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiemVqWW9LdlV0RjRuRHpF
Ui8rc1NoQzNKSHVzTk0vYU1vMlRxcjBKZ2hNCmhLYXVGSis4RU9HNGVxZkpvUkd5 U2MwN3RDeHU5SWhuQ2l4YStRVVE2eU9XSUc4ClNGMkNXQVdkWFkwcys0aFVhZnZv
L0xvalhDYTQ4N21OcHRheTlkaUxvTkEKLS0tIFVkRGxtLzhQT0paV2U3ZnNScVdn Y3J6RmxwakdaZVo5VWZhb0lVTUdTYzQKLS0tIEQ2NW0yb2lGQ0NLZm1PSWNFVkQ1
alZnaVppeGI3OUVscGpONkk3YTRXd3MK61na8x5qX7+dyMHasDz2dj7yeaUlX8me a0Y1NlYwZ0dnbHhsQVB5eS9kZVdqbmsKgFLUpZMqGywO9Ext5WRwco2bpenVNrzL
N4/SIk1JDBhv9G7mdKLbKhSF1UJrSY7TJqJqx8/dqEc0uG3vptA1ew== XrGCqhn/7TW9DrJlPnFe8r4fl4DFP6GWByn7axDl58Qy1OzeBZwnNg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx recipient: age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA4ZEtvOUhMU1FRWWpJQjF4 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLRzlzWEcwTUx6M1JNYWgr
NWphNXp5M0dLZXhkZndhT1Y3L09maytHazJVCm1MeEtMWXg1Zjg3bFVnZEorci9J ZWxnVXhFWkhQVjBrSndGTU9oY2MrOTQxTGhFCjhUdERPc3hzQlkzMjBEa3AvaWVX
bkNZQU9Ta1dDTFFHaGFWQVBpK3pYRDQKLS0tIEhPVGliRDR3ZTF2aEl3ZnJEYWtR T0RJWXFFZHdmUVBObGhpQktCQ2NPbWcKLS0tIDN2YkE4bEVtbjJlcndWVmlrQ3ZT
anh0SEpnVW8xdXNkZEZQSjcxU1BHMFEKVRJUA71fi1QawB2TnuTWMYhzQR18u4M2 dlVvRmt4RklOdUJyZ25ia0ZoYjU4ZHcK5fL1yvRcty6+EMzfSkwwJM3qvrr6h46d
s1V4j4TwYyyKZFoNvt8kOUayjC499c5OBUufYs6G2ciC6gK2A9E0EQ== d7Zs+EElXrsGDQvL0fevB/zDcICAtudYnVv4nP3Jf8GQnL19zWlwbg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e recipient: age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTV2IzSjBEMjUwZjB0dCtj YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhZElRNXNLK0tVUjQ0ektQ
aTdZVlEvYkVRbTNjUC9ZRVdNZVhPQTJEd1ZZClc3NDJiR1BVYkZkdVVxMVZGc0VN Tzc5NDJReldiYVFFZnRLS3l5dWw3UVQ0WWhJCm1DYVloZGpYeVJtNUNRcEJvVUZE
dWdSSXBFR2xxR0xrV2thRmUwSS96S0UKLS0tIHRRTXVlUi9UYnFRRlhsU21HZVY4 Q0xMcmR0WWdVVTBaWko0dGI5V1hqaUkKLS0tIFBBeXdNY1pFai9QT2t3NWFLV2sw
SDFYd0NwVEtVZXNsWUI1a1ZZU2xNRGMKuQUhOq2FRD+PGn5OkdODZItbxCzRKjne cWpRU2RISSs1UXh3Wkl1dWI3VEk1bXMKP6v3Lm0XZkp3Gn3iJkB0K0Y0E5BeilqR
E60UOYtHjanuGjJ1svuR9cYsLZz7lLOwItklecYaQYpMRZEwzzBGCQ== plmPh1rKbT/wkefFwpa8e2Wwmhx80YK+bjn633MH7mLQDf6AiK83Xw==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz recipient: age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrSjJoUVUyd1JqRm1ZQzZx YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMVk8yT1FtMVoxYjF2YVBs
SFJ2cWNUY0E4b3FndVA5Y0hIOEZnZUlVWlhBCm1vY1luOXZBelRUTmF5Y1NMeDBn UTFVbmhzblBCRno2V3YvaERRN0xWcHd2NVE4CmVyK3p1SWtodk5iWjdkQjJRc0dM
cE1BTDErc041UjJCWTBQbnk0Wk80dkEKLS0tIDVSZzd1UktvZGdyanFUMkVORUtl Vm8zZHczYlFFSG91QThmUGNaWHBoN0UKLS0tIDZMMXoxbUM0TUVaNFUzRExLc2VM
eVB5TnJkMlp6dUpXSTlxRlplZ2NxUlEK0AYOxIbswjM0SUASDfmZ7PqcEU844fgI OVFrNGZNL256cjg0Q3ZBQW9xUFcxSUEKv4HqaTcTzNYRJCmNEeUQLFfEoFXyOBEC
ycFWVSEPodwUZ6UFoYXhHlJzHFcgpLvwUd1PMktLHe1qrZ7GOQJIMA== yZrg8D+ROBn98484bxNeoyOjQdcVn7OYiCv1y6utnyeA9sdMUB0ong==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBPakhQcE92ZU0zYmk2QS9D YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1Rkl0UXduS3k3WVU1RzdG
TG0xc1JPZXZCZ0tZOXA3MGNLVnBlZGVtRFRzCmIvMHhQKzFVWCtpMTQrQUhGVGJp N2IybE1TZ1kwVmM4Y2lvUS9DVG0zSkdpZGhRCnFkZnJVS2FBZm9Xa0V4SjRwaGdR
RU1jbldYckw3TXI2SlNpZVBIZHRsWWcKLS0tIGtJTUtJejFxem5jajFQUDFTQWU1 TWpaaTh4VEk4cUFrb2pURERzbWtSR0UKLS0tIEsybHdvUWg3UlUvL0lhdHIrUE0y
VnlxYmVlNG04ay9ETi9FRmVYQXVoRkUK9oFNolI7jRjo9RUs1g4ghrx7aYV4U/ce R0pMdnI5QlpIcWVWdGRYQ3JCaElLWTgK5cTfvehj4BPU22tYiR5piZdouUdeh5Gr
ZTc2tFh57+7aKgrDi+2W3jwhfkjvBsThk//p5mLlqEEgw2lwlnhvPA== x/xSbFWqYVauDB6rpzZvhGMI4KCJh2RmDdt7FrlCb/YCB3QoSgvl8w==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th recipient: age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoVnh6dFAwTkY4cHJpaEs4 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwL0NNbGlGS3lTZjEydGJm
QnZPeXZHK0tXYWZPNytmYXVsdGRWQVI3RlhRClFVb2I5OVZzZFNrRXFaa0JTUkRJ MWZRK0Z5YktuNVNudk84MndLUUdXR0RlRFF3ClB4MWRUTFBWYUdwbityeUU2SlR4
OC9GQ1V5K0JhWlhkUjU1WStCa1lPV1kKLS0tIEhzdnBBZkRnK0NtV1FuTkVsNlgv azZMb012YXQzWHUxZnZzeVp3MmdzcU0KLS0tIFJ2SUNyMTBEY1NWejdVQ3hyUytQ
QzVEcEVkQm5NL0Z5dUU1U0ZFaTJITnMKaWE9vlrOpQstr6FGP5ObdilsCYk4kYAj VnlQWHRDYWZFaU1ON2RQTXBwNFNGYUEKwnRUWCdsiStia1jLHJ6UNP/kgEt4Aw8U
/phboR+Ym7QDTyUF9LZXJCU54YJp6vEWkRnlJFqC75UW/v/lgBhBMQ== pA8PKKu6TZVPI8pw7svtI/uvJuW+8ARuNcyb1FdYmX+sOHnOEqHyjg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k recipient: age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkUHAzWk1KblJxVVZZV3FM YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxM0FyaFlTR2FzMkhGTzZx
WndzRGRtbkw2azRVWnBuZmhKWElqRFk0RENzCk5DMHlMVWpwbXEwVUhkaFZUbkp2 d1d3dWNGOG1ESllETEVxRGJlYWkrTUNIdkYwCmI4andLMUUvNVU3R0N2WCsvWWhV
QXNlZFV4SjBEdmR6UEw0N1JOUnhNKzAKLS0tIHo5RkNDUk1ESWRHQmV6bzkvSTlP UmlkcHdMSFA3bFp0ZG5OckE3YmJrejAKLS0tIDhEVEJYOFc5VWVHcGcvMm5yanBl
dk1GQ0Y3V0dTRlByb2xUOERVOTVwbVEKY4sAHyAhvGSYJzPuufWUIQD2xZcSt/nX eGxaNEFxdEN4NWh3cnlGeElBUTJmUzAKvCSJF+LGf1JqEdkzT/Wk6hEzferseyWO
t2ZFXu891/QdEzyUXCIzdwAV+Y/LjvroIlCp5Hkbrk0s7N+ghqsB1A== vAYR8Z+u6GDwQi7JdFz9iZKD21LAs0IMzd7cZqXKIXVedyXuaIhPtg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7 recipient: age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBYVkwYXpYSjdmM1FpbDl3 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGZ2M0cG5nMU5nYVliRzlx
QWpycXoyL1AyOEpZUmtpbjl3MFAwTkJoOWpVCmVDd0FBUWxaQmZCU2VmNkZGMk9o TVpDcHpwclhXNkxuc2VKanBvcnRxRW04UXh3CmNJeEl3K0FrWWVHQWRwK3Q4Tzk2
TUdLNGtac2N4REg2eVF1eVh0WnNaTE0KLS0tIDJvcFErSjRiWmhPMmpadjROOHdt MEZQbFV1QURLZEhCdEhaMjBRUzllaVUKLS0tIFhSUTJ4SXJwRHhVbkZFc2Rwb3Fh
NXp6Y1JpdHFlSlRoa3JTaEt3emdnalUKjoFfZAiKMPF3noX+K0+vc3+p/XUHnhic QzAxTnpWSHZaWlF3NUlFMlBHUkRUa2sK3Kim0mp2adSvruI6Necd3UpZHow0EFmT
k888KdUwcZYl2/dAIc8UDSggbMnncJAJgoezoCHLkj97GNNAD7E+gQ== JFOIpIK+3Pc06gsiqrDtDAx2Qtc10Ja4ILTutPlp4va6zzmzrPABiw==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt recipient: age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqYnQyYzF5cktZNG5PcThy YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwTUZ1RzJKcVF5bnJGamlU
SEszVEwrUkQ1VVRsM3pSTlRQaHVLN0VuSHhrCmN1Z3pwNlFsbDN2UGI0KzYyallM bWJvcU1sTTJuME94YjhMZWRWRWR0aGJ3UUNvCjh6U1BkaEg0TDlCeTNyUlNTRWV6
SHJ5eklQeEIxSlhiYW5PUlpJcG5KNjQKLS0tIEk0QkpMdlBlRjVYMmJaMzJUbDNm Nnd0NjE3NmplSWFjVWpSRy9qWjBUbnMKLS0tIGdMZUFzbGdMejZoMlFBVUJXSE13
K25pZldwd3JoZi9vdURoa3Myb2RQNG8K6N6bO2YKooPfpKihgsYqilfz/yAYCLZD TW4rUHFjZk5oVkw0UGtpaG5mOHRRbHcKFX4ZyNQ0TUeqwXEaz1qmAXQRIOctDkvz
XJ/THgT4URX2VNvSspvBtN8luOiJUVcchp5WtL2m9jARL5txEcDorA== TIwxJseAjS3qVxCslnxzgaqmCh8DzCfjrRxLUS/HVTnwOaE1YAh0vg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f recipient: age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvY0VpeW1nazVvRWJFaGNU YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTWStVRDhlblZDMDJ3ejIr
RGVzVjRmWWV4dnJzRHdsaW9ENVZYck5lWGtVClNtd3ppelowRjZpRFFSMC9EK09n MkwyeHRwcnpWQVBmOXY0a3NCU1JnTjkvVGtVClVvVUpLTVhUNGNNLzY4U0o2SDRD
b2hqNnkzejdrTnhYNGNKblpteDNLRWcKLS0tIE02bWVjazRWNEVKbURITGlQODlR bGZteUU1Uzlzc0NjTW56OVNCcnVpUUEKLS0tIC8wN1V2Nkc0cVU3OHNYa1NBVlhq
cTJJVnBVdGIrdzBoSXExelNrVk1XcEUKc77o2EX7PCm/HjUo5GsUiQdm488WB2mg YTV5QUpsQWppNHRpT1dMb2JDQnpoR3MKCoBngnNg5qpktg5j4CcuTUKiooOVI8IN
wHd/qDbQhF1W75RrVTuIKgtEtrRjZqpmr8toe+aHJizPofcrToUfzw== 04plE+1lra8q5ZK8q4r1/K8hBM23poyfq+XushO1GZ17dPtWs9wITg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2 recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSMmZnd1pXRWh5NDVSN2xq YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUU2ZSazhZNWFZVXk0UXdx
MVhTN3N1OFpkcUh0a0tjYzJnWUlINGNVd3k0CmZkTnBac0l1dllxazdLY2l0Rzli Mjh3R1NLMThlaHUvWVFMZEQ4Z0NYOW83dWgwCkVnd2dhdzQrZkxaL1NRbDJodERQ
bE9sNTBVSkJNaWF1T3c0WktoOHl0NU0KLS0tIHEyeWZTUjdQeUN6U2t0d3JwNTBX SkM5S0kxcVBtY3laNWp5OXpXZHhqZ0UKLS0tIHVOT2VUZU5DZjJjVW1kcU5qS3FD
ZE1Za0tXb0gwc1FSakVYdU9OTHkyd28KkwmlzSYP8XofB0VGag+S18+S2TyQjLrM YXNhRkpaREMzRDB5aWgrdVNvUk1qcWsKQnPqE+5xSBGeDMmLfKB0tEBt2eylBOxT
qaXtbBtLzJGNDhe9FhAKTPFcjTLWbohlG69vxcImyCyCns+QQ+gvug== oi26mC+EoO5T2pCIpbD+DODAAlWKLattqCYHneUO9wRk9DnhuEXRBg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2 recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2Q2RMWmZVOEwrOC9VcUxp YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZUlFlenlOVDk1NmxQVGUx
VVE4R2NqYlZWZlhKSTBHRks4bDNoaUliMWlJCm5FbWdZS05GY0VLc0sxY2x1U21V UHhVK3ZtMHdzaXdtWmE1TXhOekpsTmUwSlJ3CmdncjMxOUw0alBQZkhZZjA0YzJZ
eExwK29GVVBqYlRPZ0l5RWVXRFhRNlEKLS0tIDl6dVZJQndwVStFVEJnRHRyMW1W bFNVOHUySXRuYW8wVjdHMDN4ZThuQmsKLS0tIGdvUUdHcXlpWlZkL0xBcVpMN1E1
NnFqc1F0SGJqT0xmREpaN21EdnlJK3MKRPE5rfFpVnH5wAOkuB5pNMlMd3omcpku NGZCQURYQVY5ZzdDODNNY0o3SG04eDAKiYVuKsJLaSC7/bQRB51tX282/yqpAjWT
do2hFZwyI7t80jxF4+g3J7EolOx8AGjpc9Ba7Gj6IMDjye728q5N+g== XNnqPQBQ7H3xuK0tDVdCAUMruOSlqAe9YRZwmiou1irpg9vZYwgTPA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c recipient: age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhSG94VE96TVlTNkEyclFE YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnUUlPS3ZPS1lSVmxwV0lq
SmdCSkJpeWlVWDFIMDhwUEkxL1RUTDJ5UG5FCjhKRDB6VGtwTVozdUVzbUxGL3BW WWE2QXltb1BzU3p3cFRuMzVieGJta05nWno4Cnk1RG1CUXZjM3dNQzlJK1ZGaHpZ
SnR3cmpSN2RxNnl4QmNvT2lkYmtoVFkKLS0tIHd2V2h2Wk5xOXlISzhjVzBsVkhz b3drMTB2L3hidkpoTXdMQ05vOUtTb28KLS0tIHhyUllpQnlzL093MCs4YWx4VlVn
VVpRenVnSVpHUWJqV0JHNXNWWXJOdW8Kv7PJSTDbwFOAcl7pynALaJiTXU/87bSF S3Q3QnpVNEx6ek54RGx5bmlqUk5hbVEKHVUvfk/xKtlOk2o73LodweIivSB4Oii/
F3HQllYOwOoibGzBCe18H2N+VxyNxoQL9OWe0TvOIR6bgHFIIF0/Dg== 6dwnE8wa+Qx/OU0vDJVelKOUE6JiLwckX/0u2YIUOybkx49n+tYs/Q==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn recipient: age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsMEZPUExRVHFFb3pSVHNO YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3MHlrTTdxYit0MTkxa293
cGJFN1ZzTDFVNEdneVpMZ253ekFJNjVtYkNnCnRGQjU2Q3dsRGRFV25LQ3pCbTJE eFFIcmRLMUpkZjd3eWdsOXYySEljNDB2NUdNCnAyanVkS0dmenZSa2dpSWZTSkZw
OTROaFBiT01xb200S1pUK0NYaTQ3R2sKLS0tIEdiQlZTbi9Vcm0zc2t6bHplZktF Tm5LMkRoWEk3L2ExVktnZDZMM09hZ0EKLS0tIGVGcHFuMjh4OTUxdHNJQSsrR1pq
ekRySENXcjBuR2psdHZSSUJrR0xUdjgKvBsmnC+cbq5TUDFjXCyImIoPKvh8wsjE WU1jVG5UN2hMbW9iQ3JmbHlwc2VveFUK4JC32glesv/9nMWBR0lFeX7MPgIt9sIo
7Shk7Act8Jayrhx0lXBDRmfpHRrB4L16rDSmqO0DTE48VhT3TiFyug== 9HsAkOMTxPTI0NZW8q79+G5hq8qG+QvQhwO6wQtwz/KEHPXntKDX+Q==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68 recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxNDlya3RmYzNhU2p6RkVw YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRbEZGNmtlUjJVZ2dlVGZC
cHJVWWY0Yi93cW1uZi9FWkVONmtpRWh5SzFzCmtWYTRIY3BkTWU5R2Jsa0ZJK3kz MGVEQXV1SFRpMjFqWks1Y2F6c29TY25oL1MwCjVFRk90Q2dJRXhUMmIrRVdHVFRD
SnZvZ3YwaGtoMVZ3V2laTlBBK3UyTmMKLS0tIFdpZWtqeGlacjlLbmFySHlSUUlj TWgvQS91VWdFWERJWEYrL3VnRXFraUkKLS0tIExUU01VRWlENlpBSGt3V1Izc1dS
RmRqQWVHK0FUT3VDbFhLbXQ5WDhLeEEKcDkgV34lUFJRIHRoLB8F2IOvGAM93sM+ RFBhSTVKTGV2KzFidDQ3b3lmUHZ3d0UKHWvWO/Om6ahwBkONreGeENqdFG9UI6JB
AkmaM4+WRcGeYWQKMG2x6cYCUKFaT1lDXuWZ9kI8Fd7b9gTSnQMs6w== fRA3gRHy98EorEiia1him6lGY+ewYkTtVNo6iEt0abncW5x5xMQywQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz recipient: age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjeFM4ajhHeVd4bzlqZmJ1 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dFFpeDU2Y2dtQU12cytZ
MUxYMUpZVjdjR0NGTzVteVB6WFBQeFI0ZkZrCk55TEt0Zjdwbk51RnhYclNzam1H WUJjKzZ5S2RiZmtvYVJScStYVHI5d2FvcmpzCm9COGZmRllyaElqb2NsczhKRjhO
cExKZXQxWFVLa1pDNFpkcGZzcnl6a0kKLS0tIFJ4ZEdJc3JVaEc5RU1aZk1uYm1l ZGhwZCtoaFcrWmNZbGVHMUNLMFQ3Ym8KLS0tIHJQR1JOblZlY0cydnhNUnBCT1Fh
d3RHS3hHSkRKRXFnN21FQmh0TlNtNmcKdc2G/1dhTJen6iT9kUWZM5OzCmDVprgx aHJiZVpMU3Y3NXd0ajc0MGMxSGp4MmcKky1oaS8yFydszcvZE2pAL90ZUKoN9MnJ
WN1Bl3JzYhLsNKn794887bVAICVqbXqkdpEZztNIS5n/Rw6geKsNvQ== 1TdqatDSMtmo3PWkdFeHbWxPjTizQGIcKEcsv6GmO/5KCAJDCHDNMg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp recipient: age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYa0gxNnBwNjNNTTgxKzgv YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrd3QwWE4yaVlIakdiYkNX
YkQwTjZxb25tQ053Ny9tSW4wNGVYYnlIaFZNCkVvOFNPOFkwQzhYZGxjb0FOZzJ5 bFVmR01yNjdWa0d6V29tcWh4M0hWRmQwR3pBCkFRV2gzSVdPQmQ5dXZYc0F1dW1r
ZjNXam1ZTWZrS1M4cGhhcHZaT1NjblkKLS0tIEdvL1dDTHpWcWF0S3ZqNkxrQW52 OFZNTkRwSzRFR0tlUXE2UzFxeU1Ld0kKLS0tIEZ6RlZkMFFiZytPZ05DSEI1Ni9X
VlYwa29sZVloOS9qajJWQWFzY2FKRmsKy074SLdttogXsWycaFX8xso4ek7Cbjph WEtpUlMycVhkMERsWDRGQ2lrNHdQcDAKRqVAOzQY4ZM1uH7LyZSqX1T6+1PwNhBY
MMEhZd/svmnSiYM81nmeaze7qXEUcsZXuSmZCYATTBEGtx/Srll8aA== 5M0RxSuI43E8YKC9axcppvKFJMQI1GAGvgfQxg2J0CEpnmPSVsO5DA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0 recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvb0xNMloyRW5tTUUvZGph
YWxIc0ZhSWI5RHNmaEIvdjZJNHFDM2xBMkNzCldlM2RUeFZCNE9tMVp0MFlWM2sr
Sm44MjkrWlNkY0U0eUlyNlE2ODl5SFkKLS0tIEdISElQa2VBZlduajhsV0ZaU3dZ
ci9COWE4bldUMS9lSVB1eEk4TG1sVjQKYu9zby36DP+41dTV8sRdtDil3Az40pf0
vIhqhCTxQymnJHEwEukiEzM/w3Z85R/W/MbOKyzEZmxzG0a5O5svhg==
-----END AGE ENCRYPTED FILE-----
recipient: age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZlFtOHI0NnlMeWdjZEsr
UnQ0UFJXY1pHTm4yLyszY0dBaldtUkIxdzJvCkNQbk9mN0wzK3BtN2pEa3JSWWFk
RXNoemFFbDZ1SlUxOEZMUm9mY1RaZW8KLS0tIEk3bCtyaUVGM3MwNUwxS3QyTjA4
YVpTTFlOdzljbmI2cnZUNG1nd1NrVFkKxARm3bpBL685Wky4Y06eA18VvUZRkAl3
BBYPJMn3lfDgiQPvfXINfhJI6O+bWyjt0WnfodCGFM6EJLARXiTaqA==
-----END AGE ENCRYPTED FILE-----
recipient: age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
lastmodified: "2026-07-23T21:15:41Z" lastmodified: "2026-07-23T21:15:41Z"
mac: ENC[AES256_GCM,data:qFhnPra6IE3wyKQ4WKweON0S0YtD5I0adGZVfA0m6BVilN6bX5oC/1j5NK2oHrsz920hSl0SOF8LrpqOrUyGjSRkPsN4kq8qr9bJcrX4URiktP0oRden5LLt6hf+ZRP7WmRXFqixPkPHJnZIoAvkNnTFce7cDq5NEAHkKUEKG7k=,iv:nyblUDGeu3TUfFivYylOn3C/HITj99qiPI2+mh8AGh4=,tag:FrtRzSCylC4wlIoqZdfx7w==,type:str] mac: ENC[AES256_GCM,data:qFhnPra6IE3wyKQ4WKweON0S0YtD5I0adGZVfA0m6BVilN6bX5oC/1j5NK2oHrsz920hSl0SOF8LrpqOrUyGjSRkPsN4kq8qr9bJcrX4URiktP0oRden5LLt6hf+ZRP7WmRXFqixPkPHJnZIoAvkNnTFce7cDq5NEAHkKUEKG7k=,iv:nyblUDGeu3TUfFivYylOn3C/HITj99qiPI2+mh8AGh4=,tag:FrtRzSCylC4wlIoqZdfx7w==,type:str]
unencrypted_suffix: _unencrypted unencrypted_suffix: _unencrypted
+26
View File
@@ -0,0 +1,26 @@
{
"data": "ENC[AES256_GCM,data://9IEHIVCfHjyMNao5sCu2zNlZ/CaW+JyxVpGpD/vab2qvURunCUY7eMfSOyvOx/2WPXnWWlkoVJPCR1ec/yUg09EaSMfxrvqlu4UJI3Sxvu9xNuDszMwMMD/sCulJDiMWFNLp8qaYt7UGzexp4+GlGiqWDxk3ZEu/iLmSApzBrpciTF0lfehT4qblDovo9QXG2KDWFhCt2SwEKmHJ61Yl3pVAQnPLyTWaNhwWD/mYL76mIiDVKq7DJlvi9MBxzi3aYh/ttuHgRCvnCL0C9oUvOyT2cljwra0LXuOrum7FhPIXXboA7WTEbTHJm1LB5yLfxDrnWCrGxQzFQAwNixVHIcjuvjjvA/LifTvbj5FYM8x7an+DwXPfPhruFrDew1paAWsEGNNYXSmAlju2QLI/OwLHFFLuDyLnBKQ6RLtZbCEAOUKf2h4iZj7nH86eb/aGU=,iv:rj76+MJBCpiYyx2Ogut5UxJj3Gn+bygvu2mtuY5hFLA=,tag:yQmTvWVu8ZLug/7fo6RnwA==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqcURBS2tIMGl1aWxLMHFQ\nS3ZGVTE0cnYzTmpnQ0daL1F3QnlpemRCMmtjClJiZzIrc2syN29sRXVoaXlPRUNF\nakw4RndmbEduTFg5ZVRKUTVwRWpGRjgKLS0tIHZ5ZFlyODlGWTJoNGpXR3RhY0ZH\nOGpPdDZQVmt6UWZXbHkxQTBoeW1JencKbsfH1V1lUj8mmHyLNj36VaRDgaBojcDU\ndoQWmSEXxjticJqdadbVKb3UABpvzAZxASCy81sa3wH0gT+7zLaNyQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrbFdaMmRjcHgyV3Y0eWZL\neG1uMEtpQVUxQmNTU0Z3aC95Zi9uQlgvMEhRCmJiVlRTR2NocGlnbVU1UmlLVVA2\neERiMXpiQlVPNVhRU09XVGxrY0Ixa2sKLS0tIFRuOHdoelJxOXNRRHBJNnlhSE9j\nUUZHZmQzOVFwRmhFV3pvdnpRMTMraGcKRHBuSUpbHaEzH2tuSBE5MsLJDCuH3vUx\nO0jnDldCWkCw7Wvr/tQAkaDI8axZcYVDUkCEk+xqAdxDozLCPhEO6g==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZMW5taXh2QkJkT0JjaVpq\nQlBjVFpFYUhXTHhOT3prbFcvUFB0YnRGUzMwCjhHT01lY3dPWDE2dDZWZnJza1hN\nS1AvZWIvdjZoYmRjWlliK1hiOEdrT00KLS0tIDljWWY0NlQveS9VR2hOSUNyTUtH\nK2gvbEVibmZSdktPemdEQ2p5U0Y4d2MKKitoTi2vbxJ41IoWMlj4vO91Ahpj0hHP\nrKCPyx7ws/IUMmKGyvDLpZ3pYqHD9jl5pLfB05Hh4Emhv4lA1qtwwQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3K2dVV1BHbDk2Z3FhNmdV\nZEN2Um94K0pma1k4b1V5ZUJxQ1hWU1grZjJNCi9iRVVqOVpEMmtBUi80NThlYldY\nTUtZOVErT2VSWk43NndpVzBlL1lQaTgKLS0tIEpCM3Q4NjM3N1lpUE56N04rTXN0\nZkZ6TkV1TU9OV3dpc25RNWRpVnprM00KItUzKBdShakOfX8Sr+k906nsvYPl8QLb\nge//1GA+ukGsaS9rcChOY89vFdm61JDmj1jXSJ0CN4wLMW9/eblZRw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c"
}
],
"lastmodified": "2026-07-28T01:44:28Z",
"mac": "ENC[AES256_GCM,data:efFXIqbauOURR7lrVpK7kqRIPgYjgWfenBYoX7mUrQ/thFc+ApcAx58Fi1zg4biwIs7NarJAgDqAxZi+yKb2Sll8H/wlsEjWDU4iQlLJdIQw7wey9eDW8pgBLd+6E7FXrgn1Vh49dS5GXlC6clAYk1lCiB4NvfzPDy5Ktpl+Chs=,iv:+zCqj5I1MLJfRRiIrqgocYB49PZnlV45PUTH4gYlfrQ=,tag:WxY1sF4kFOTEzbSFcfJFbQ==,type:str]",
"version": "3.13.2"
}
}
+22
View File
@@ -0,0 +1,22 @@
{
"data": "ENC[AES256_GCM,data:JwjmYg0qzoer+8/jv7KOfK/BxmpObHOn04GNsUFQd6hSFBWbQlRbODfluL54hnWQ4gsS9MHgwWh8nrLciBhye05kRQCqADB7Crm0CZYubNb65WzM/devh14jmerc3MYIp1M3VmDisML3x5IqhJkFMSqKM/VaNAC3f0otcVR8TTgZw/fJyvPlfqpPTjikGmo5xg++Yk7Cy8A1Dj70kYk10+EQjQ78jf4k/agLoaS+YvMmKXFk4EljtHg8Fe5H6Rn9nfMSSKGTZeAH3Riix2e+fIH+nWEZiPHC0UlPjJrB3SeBqqGRSUUJshVKHFIdKI1Tq2Ea+GDWrAyIRI0qO8e7WQ==,iv:mr39NxbZuYUDHDw0e+fJDshbt9R1hLgVqkUxUVX/+l0=,tag:u31xZRPfvH5j3Sw5U8/nqg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtVW5ENXlkUW83elc3ZnhE\nOEZ4MWIvQWZ4aXhuUmZGYXZNQnhJemsvOWlVCjIwNmFjS3pJOWEwRnhBOGNRamlM\ncXZid1RNYVBXOFZpMlpnU3ZnVE1scnMKLS0tIGFhWFZFd2JPTFNrV1VHSDVKVHRK\nbmhoeFh6YlUySU1ZRVhHbjZnM1IxMFkKo7aHkz2pEeV64m+OEkBZ2V1e+PUzoChu\nUc/Mnh37dNXSSJtg2KnocHdyzDGi1yQbA72xKTxx6QjYJWrAC/jurg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBReWRLb09reXUxS0YyMUVV\nQlR2bEdYY08rK0RoR2pIelpLVEtYL0pqNUVNCklRaDNoSjBWT20xbk1UdmNqWHNt\nR3JoSlVwODVMUVBzMERUNktHMVViZm8KLS0tIHc0SXRIS1IreDF1VCthb0FBR0Na\ncGFRWTROUHF3clFtTmY1azEwNUpPNzQKsgCND/BZcMTgBTAcnHunQcT6LG1jNrtO\n+W7Yx7bFtFBajWnRYiNpUZPibQJlv5SE9os47WDH1gs86xftgV+uyA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGbllNTlhxNVVkd01GSUNi\nR0lxQWZoRXI5NWxoTExRUnBSb2FPR1BONnpzCkxCN3pNOXVrTW5GR2J0WFhZUE9r\ncm1QTThINDVVVlZEalJiS1RXcHJRS28KLS0tIHV6aGh5QjB0M1VPWDVMT1k3cTBh\nNTNGY25NV0tWSUl3UHlJSExFQVdpVDQK3ARj8xhFRYU6oqYxNQ5+Ryza86ALNUwX\n7yK/8ATnquYC8/ZIYUbgTPxzsocmXX9lVT0+ktIALqjtG8PgNTlOow==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r"
}
],
"lastmodified": "2026-07-28T16:04:51Z",
"mac": "ENC[AES256_GCM,data:9X0dkAEGJiug16LDy/8//QOkHITTzt4zwTKiIU13tRgSRXRpt/7b27+3eDauscXDMamg5wbYo1YF0+VwLl21Ip4icnnwrZBHd6HWleP30HTgef8rmo21SstshnclZz6RH8SEGVDO/vjrMDChaZ3A2WD+nbcomw47DpK3cG3kBIk=,iv:G7CScSD5Z1Z9WMEMydyeGK/RD4W43xA0PlcpmTCxLOc=,tag:DZmby/xkxpinoJztZZRWwA==,type:str]",
"version": "3.13.3"
}
}
+19
View File
@@ -0,0 +1,19 @@
{
"data": "ENC[AES256_GCM,data:VpxnzrdX5FY2gfDcZpqg1f2wqL6mAawcrdE5A9A3fQOKr4BWXmCT/hjQEtt8qG+yVOTApdG+vF8SKDj1AKZ25KsZ253WS7W5C4j8oiT/xHo0CtvVani4fZqN60lB/9381h7aUZ+lcBMPaQcNn35zoQqYthuCGrC9PH1qQgD2VVHr0v8J6kceXKp0KTJldtDsQxky6ROC9Zyc44wwkrFPgPss4/yGaRWWqiojDvK5fK8238hmR8HyfD6Tf5KhgbqCsSGS2g6yt1muUhmegaZwtpi/KpoqBmf1XFaObuHGRPKvblq4Fa0x/sSjpPogBZyNuvRKIwRhla0FkKOH1bX4gC9bzOlgiUUfVcRJT6Nr9jocYuhIv9wvIT4TujGbgHyorYaDSQLFPJoJEUa3fgpGq+AC6p1DJJE3b/wqUSngNbdOtAPE/imaEuprOfnbS7cRx+Ti/Yu+ZYZuAlDtLkSVdXi1hQ4fLQ==,iv:rq/BzePXa/w/Gqewz8JfM/NdU+x4ShiH1OuQT7wRyCQ=,tag:IcO3ddoj4M57CkndHEmNkQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRdWFEUlRJMlZId2Yvc29F\nNnkwREkwQkllY25vZnRjV3Nob3BvNE1Zd21NCm56UVg5YUFGbEc2ak5MakxDbCs2\nTFVsRFhjVlRwMXdUbGpMKzQ1N2hBUUEKLS0tIEhmdm9hZEJqUEhRN0dsSk80bDY2\nY1JpajNOVFpEejhibjdKZlNHVjE1bGcKycyDfEBZ1WYc4EfAK2y5x/nKRqq7mnb4\nDlpR1Som4bSt7+B+OCZa48mC0Zv05HbP8PnoZdCe83swGNljvRONcQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0SGFVbXYycTE3amR3ZVhi\nVFJHVVhWNWVQQlJKZUJvZ2p4ditqdnYrWnd3ClZaaUVLUXFKaWlDdDQ4QW5SQUZY\nbG05c0RGNnZUenI1QTBlemVjT0szRncKLS0tIHBWTmNpd2FjWHc3MmR3cWt4SCtH\nWS8xZnVON1lGZVc4YzZFRmRBamdSVlUKdlQTlwiExTUVsiY05MXFG2/IQt1bZwKU\nlFDnuy6YCQzawBLQZuSUAp7WSGedupexlZKQKNUzPf+dis26XEYZoA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6"
}
],
"lastmodified": "2026-07-28T11:53:06Z",
"mac": "ENC[AES256_GCM,data:PMsPSffQkxRmqT6HwFezIoT4WKJ9WDlAcflWynsFLtzbqVTREN0YGOJgMwLd0yBR2sbGBWogG5xAuKjO16aiWUVpCFPRvP97um351679ZoXIlrOL+GlFn4NKwoLH9/eCp8BteB9mv5huhEEN631lqCr+jtYO6yS0ULk6QS4+7/c=,iv:oYWV4J1JUuGryvqntVXkj/HOzRDRr9E2C1RKNvr2bT4=,tag:e4p+I40y1LrgdlK9tLRicA==,type:str]",
"unencrypted_suffix": "_unencrypted",
"version": "3.13.3"
}
}
+25
View File
@@ -0,0 +1,25 @@
beszel-token: ENC[AES256_GCM,data:tT2a1I6AhVgXWfc=,iv:1iXoOMAakHXpI2lYiipxcVSzXODUdpqYmfrpNKZwljg=,tag:Vuxxz7pyIsZYbkg2x6GvgQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLT0oxSDR6WHJGT0VTY211
d3hYcU42SzRIQzhvdUxWSnZhRitoNGFlclVBClFEZ0xXVnU2OFV3TmRXZHJyOVp4
aDZjQ0g0RlV5bnlzU0Y4ZnYxWm4zK1UKLS0tIDJYZ1F6OG8vdUxnS2NuaG9SL1gv
bURjNjRWQXIvQlNyVVZieGpCblV6bmsKQrCQQz7b4+w2WB3E56lEZoZa8/Yx8mn5
vWtJa6nvqlE4miJdRpbjQWUeFHghr0A6FsYDg5yf+Mcpf2otTw8psQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5cVFzS3hFVkZpcGRWcUc3
YnRqVWUrSU1mUGRaaWhZTFQyRGRLMm9BSWdFCnhzYVhtWExGMFlRdU40K05TV0li
L09BK1h6SFlWSGRtQ2F1QW9iM3lDVWMKLS0tIGNQSE9aN0xHNytlOHBzM3MvMVpB
VEdyUFc0WWgzVEswRG1XSyt4dThTWGMKRkR+FifFDDKzrMrlJgZj/lbJDjfHsvjI
BGfJj1x0whWiozg/SZ6BsYYYlVkcCBwVTmPYx2iQHk4jLqHcMF36dw==
-----END AGE ENCRYPTED FILE-----
recipient: age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
lastmodified: "2026-07-28T06:44:32Z"
mac: ENC[AES256_GCM,data:rtDzbONqaNpbEKo1btGiHArAJVCVsStHhgvM8MTgkRMUNGU8p9W4tiuQLmobDpFh81kj/mDnkx4cCpScuiMhQojz9H74l7P9o9b1Wxy0fSiCtGoQhaq6JoIVJoKtEZoHfMStdIio5EBW9DBB5abvjPLTaC0afgHo5pNn3A9qr6g=,iv:SqtJhel/5qW0V7IV6V8dCRWqPnsK6QCzbC15/BKeq1k=,tag:vF7RJJccm0q7BSyNGYnplQ==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.3
+19
View File
@@ -0,0 +1,19 @@
{
"data": "ENC[AES256_GCM,data:Ea5AGwloFMyRgmQhJaDNTW19mpDvx0+dSc3ibENEXEniEIBV4Wv135mAwE6jQPbN8wF6f5ki3B5mDXHOMuhDQEqUXg6jcs4uEm2nwz7wAKYXEX4T8p0gUj7FPISiSQrw8O3jwzt4qF7AVcPsNhMtZLgrx9ssMtVmGct5do+E61Y7dFe0XbpJLynfzeiMKYvaBngTbHwQjv1mn9AMpX98apsp3tyzg8avLDR+WXRcMusds2mdov0jCoKuWPLG/srhMwjKOp1Z1Tu+aZCr9lHTT76mytpGIWGSbrP5l3UTp3oPLmy87Kwi/FJStZYdYikMnvmSo5qovePiYsLKVD74dS+d3oFGARyW4TILd9OQdXEjzdDRak5HcPv2/HhR638kvukVMdEdH0hpeJcqcV0h+BoLyIHe1e037iF1ZkCjuXE+A8LtlzRR0wqP202uzBIyg21ca4X5bRiHkkAOX4P3riaEcMgKfw==,iv:IXq4sHHsjnK6maVd3RRVZzhnxx1hOnxfsvYX8IKLn+4=,tag:jRZinAVLH8nGVf3gE4aSKA==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSByOTkrVVd0TXlBRzdkTktO\nalU3WlVOb2ltc1k5RG5sRE9EN2M2WUlnQVRnClFGZFYvSHdpVlhtNXExSXZMZnU3\ncnl6aHRzU3k2V3R3dU9RVDA3VWp3VkkKLS0tIHcwREM1dDREK0h0UFp0bU5JTW9Q\ncWdyanNMVjY0SnZUWUl3Y2R2SXNyc1UKI1FPIE66to19oQK2TwM5B4snGOVMLFfx\nl6JMEilDVSgy2RR/tSiovmo4NwOsv0hmmF1t34547rbZb+dB5KrMEQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnbWp2UWphSnFibnA4N0ZV\nRHlNUnBNYUNzMzNZakFVc3JXT1d0R3Yyakg0CkowMFlOcEhPcHE5dXd4ZUd3cDAx\nbjQ2T1NqejNRMnFvbTdNRGFGdW1pM2sKLS0tIDhGb1JBQ0RFQ3NxbmpkdXg2WVF6\nS25uZW1hMVo4Q1JGdGNQWWRyWE84b0EK0FmFZB6sN7uZynhzYU932x461zSZIUWU\n8oyDEKNoUuPqpWK8RktgjWlKHtB0oLXC+SLAOJEhtnhWF+GSqloGfw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r"
}
],
"lastmodified": "2026-07-28T11:53:32Z",
"mac": "ENC[AES256_GCM,data:EhWEEIamG62xHI6SqO4tzdK6gcEXxUU/UGcF25m+X11kUVXzHd6yLcfiM2DvqcwGLmwmOOf3SpFLABlQUzTka4QRzTMFIAJi7x1rzQV4J0YIGITHk9rgvy0V50TI0loONBb2du+vDt8IlTBNvF9WMbNCki+fHPAjIemyoRBdubo=,iv:wfsx0H4fD3L7WeGzz57SzZtajBr/xTRDljSpz22DD7Y=,tag:naDyGH0WTKkqgd954Ya8jA==,type:str]",
"unencrypted_suffix": "_unencrypted",
"version": "3.13.3"
}
}
+25
View File
@@ -0,0 +1,25 @@
beszel-token: ENC[AES256_GCM,data:0wooMaPiytDTGug=,iv:/WATBCEakrIvujt4oAeprw18W1n/WoK2zjpQdlVJl4Y=,tag:lhJa9hJgFqNFSB1Ogj5CtQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlMWJCTUNSVVA0Rk84VWl5
emJ5ck0rMTFRMmljUmVOTlpUT1dlM2VFekZnCmRZbWcvL214YjUzakNDYkovQm5q
cVJQZnJvVGpIMkpqUnZEenkxZHNDSkkKLS0tIE5tUFl4a0FtZFNNTXFudlFNbjcv
a1dEbnl1NTR2T1lWN2wweVZ2ZnErZEEKbn9bLqIsMwSGKBjR97kMfFDuh1l7Hu5o
zdLjnAHzCEAsTW6V6wPB1rvi++FO04lqq1RfEiWXJkB2KWSRf+jdTw==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBDYUhMZWZCVS9MbnlscVBQ
aENENGFLTjJ3MVg5SGJCeXZsNVZ2ZFM4aVZZClYyRUhYTFZENEkvQjJqNGp6b3Q2
SWthQ01EMDdQYkVBQ0ZRdmRIMTFnUE0KLS0tIEJHV2YzRmw1MDVsVWhHRW9LMnVH
cU9qdUQzcjlNWFdQZmVydHdpamdQQ2MKndSN6wQbwyGqnyS819tHJRqDLRryXonC
CPuBYrTwxosRPwgh+nyJto4lj1uZqCmFVMXGjn87em8ykY5arp9o8Q==
-----END AGE ENCRYPTED FILE-----
recipient: age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
lastmodified: "2026-07-28T06:45:50Z"
mac: ENC[AES256_GCM,data:f50JYTzMJiEjB9+fNIs734CXrkN1IRgJKcoJ0GHoI5p75cfLfPxuzUifty9kQ/4DbM7LbWa5w+9occoV/yNOa86BKpPtU0LmlzXf904/SgOCUKhyHqhyO5mp8w71GoyLxx3ld8CbqfBu8t5l4gwbQamftsmQRxnUdU/Oa/CzdZQ=,iv:EewXBjeihw89h8ypH5u/G4YfoKZzkHqRWgSSIXU3/a0=,tag:lEWMvAkebMexDGGpE+wqkg==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.3
+26
View File
@@ -0,0 +1,26 @@
{
"data": "ENC[AES256_GCM,data:z2bazlADvtWkGcY4an7NgZgIDSDq8KmYpSq8tcBleW/33HlElMNhdmje8m8hd7WxgrVvCQDdlAfodBLj2mjbgSwBptlPX2zbwcmJdm0g0Ope+YyQZucIio7NnBaYa+OK8meWqmxO2WEeOk+hnNfNPbt8MkzF/FMMbP8xv9wSTKP9CDdmO7s8rdg8G1JxVfn9dyaSDUTuA2JGNtbh+osd4CtNCV/bc84MjXDZTgTfT0W/uv8VH3iPNT7BhQbU9FIY4dsFRYBf758VtJ+3mTBiQkR38IevJBvhFB7Wiqn118LSBgjdTKJnBwmGo6lMdo0LRYHEgUnb2ejhWMNie19x+LEAErDki0RBmjc2BbnOF908kUlW2fUPo5fz/hlfqlhDLS1uOxESKhfnKt7s5qzWiMX/nohpknVfyMbmu9j/M571z9sgFnJAylpW3NLp53lX8N9mINwRGXsyL/ijbNE=,iv:M/MNquAGN+lIIyVVvFRgKR6DKAhLclc/OY1HV5Xwoog=,tag:eEuURMErSqsoh+l9ZaHY9Q==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqZlUxRG0zR3IzMlpDL3Vl\nNmRWQXUvc1NDek5wbWtKVEFTcTlxL3ZoWldNCmJVMkFtbDkxN1FGQ2VLTlhBUW5Q\nZ3VzL0UydUQ2YnlUMmtqa1Z0d0FSVFUKLS0tIGJFelc4Rm84T3ZHSGZDdnNUbUFM\nRXZHV3N3aVlSOXhUZk04UFRMWU5TbFUKhPU80PVYuDFUCxu1CA8+W8bqkr+Ne2fh\n+nBUPJbGxfN9TyD9tUC77AMbcL1R2L5x+SSAh0bEgSWE24/RjsuVKw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3M1o3S2FibHV6N2tIOFEx\nRytJRDFIUU05azNDbWlpaDdFQXFRWlBUNnlFCnBwNVB0SEVwL3NWNS9aeG4xSUI1\nV3NOcUFxUUhaODYyWXdFMEhWeFpvancKLS0tIHY4NHFQRGJwUURQbWZ2cGlSdm9s\nOFJodGZkekk3UnBRRUwxb2YrQ1ZPcU0K0tCapb1hfVHFSNpmESXexYa5k9OE9Tha\n51QpU4mZHKGrnWPK/kyj7rHiz95TLsmwUdA8Q2hOiYNSxEaVBjYDQw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkTmNrbUFoVFVGcytzQkk1\nMUxDdmJGL0xDcUNZaDFRQnUwNXoyMU90SEVvCi9EaTlVOGlMeUZtQzRJS1J2TTJY\neWdkMGNFMWZzQXVKOU5KcXVvalk1ekkKLS0tIDN2MlRQcG5aQnN3bUMrSDdvc09X\nM29hQ0pJRzk1amRHaS9mRlhrb2FMZG8KEWkSSP+MqGRU75qo7ctOCL7qHhyCM3l4\nL+ga1XOKxRsQkKQPozCi5Bm+k28W9hIDaC34Rw9difxICM9Z1rAbOA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1TUtxVjJ2STFpU3RNdWpJ\nVjcwUHVORVN6L1pKVC9TeWt4cTNoR2pPWVRBCkR4QVJ1bHFUa2Z5aWw1bG0vQWVy\ncjNGRDl5REZzbzdZWTFMRkU1eHFTc1kKLS0tIHNLVDFiRkRKTHhrSEx2S0J1R3cx\nSkdEQWJya1ptTTRqUjJZT3FFaFhkd0kKgecHrnzW9+Eb+b7c0z1yR+Y0Czsr9Kjh\nx1UUOfleHntJUCs8oYcOogKknEnhBJJoJ5oe/gWruRSwle1fs3cBqQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68"
}
],
"lastmodified": "2026-07-27T21:22:58Z",
"mac": "ENC[AES256_GCM,data:VG2ygV4X6yMxGlJgq3sN4GAzgVqiD06noyMxB2yb/FCmiFVYZAH/9LqF//G7MTInfIjFrTDyoHAny+m/ztUrJkHXI1Fzg0U8R/zYyQ6wj/GguL0gyhA70uriQhvHsRHefHPa7Km61Blo9cME6CBJWejtRvh1IMw9itAioVrSIZE=,iv:zjxNG6IYxkcNqLk08NADoDiIbS6at86y136SBzadW1U=,tag:sj4ZRR7ixU8h7aOUJmTCYg==,type:str]",
"version": "3.13.2"
}
}
+30
View File
@@ -0,0 +1,30 @@
{
"data": "ENC[AES256_GCM,data:apiijrtrqd77CTizITg0R35BfCi8PBnufpxIyC+hLYqwoBzP//3z/yjFyHPLG98m/c/qywoi3Kn+zsaTT7MjP++9OMhhX94YKlSHV1/cHB76OkwsNc+ClqWxl6vpaFX29Qvh3gFX9c/NR3xvYQutYwrIrQ9NR+t/M52IMC8hvtR1LQy0ak3VIuXJlSnG2r4kF2Ym1iP7phjuq39Gd245Axzw8OB7yGvOjNxSdTPxW/qL0fMlzNcMrjr9hw15WlqnZfWPOsB1+gZjHXpGfPD5BCbAAMoTRJd75vhKKXP/ERhIffewuuH2x/QHfSFvXVB3QyhBQMxd2b8QEEE5cjvcExOST3tkj6QARkzoUpRT7AE3jhl3XZ0uA2qu9SwyrSvbr0tBRKxCdK0g2E2/hqwcK/Tck5GB1eKb4aN+UkqxOblNDH+B1RfDoyNAuN+KEg==,iv:0p+ScrKpP4kQvO52gBAlwAis6oAzZ0EHFnU74hYPrn4=,tag:ON7qOjztF52xsJWAou7ogg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlYXB3cVAzb2xEZ2pGa1RJ\nbS9ZVTc4Ums5eUZJUjEvd1g1aGVyNUNramowCnptZXFOZVB3MFRFcUtzSXBEZk1B\neEtKcDdLS0h0b1h3VjRjRXRvV3V5V3MKLS0tIHBDemkyUnV6ZXhTeE5VOVVOMlky\nWWMzVGVzZlAxMjZYUGpQUCs5QmxiYkkKcuBshCgWX4TwfVlQ5lHikzvwWdLEXWD1\n/uSiy0J6yMSiu8u6cg2SxeFrlKJ3j47dDlT6WHCxS0PfeEA0bJb3LA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGa2VTc2NWdkFRNUJxelVR\nWFk4RWxoelYzNHo1UFVhU2ZkLzEySlRWN2xNCmNmcmJod2crL3NMRlVsSmpmVkU2\nMjlXMktjc3piUVNhUXlTdnVGTWJkUTQKLS0tIGQrMUxrNDlNTkRCSUtFWkxRdXgw\nRlV4ZmtYSGhPQU84eWtiQXVqTmxUK3cKk5fn72UZPH68t5ZappfAhZJwzpLkfKmT\ny9TbUPIr4Pbrexau6YiH43QIbDQFdwYPfkBjGkd57zCg8AVo1+MBRw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArNWkwQXFWT1RpaDNvbzYy\nQWc1aHhmNHFEUXVsQjZqb0EzM0wrV0dwN1hnCjFsUFJiT3REK05uSGRWTEw2SFE4\nY1FleE1XVjhBbndiMmZxTWNTYmhYeVEKLS0tIEJiZzJvS3BsYzB3cHIxa2k5N1Ro\nenFFZDVaODNnVGdBZTBOYWJwRjQzc1kKlXJgee8wTSN4Beq4P0t9cYbk0BWHCseQ\nyaWpiPT9aZBEGLFmuEd3zKABc8lrilX/ySTmOG49vRg6CPmr7cT0Wg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEa0JYenFMNmNzWnVmcXdz\ndG90ZUZ0WWlIU0FCZG9OaWpBM3ZDWnFhZFhNCjhuV1FTOTJ2WVJGa2RuNVV2MjR0\naVNXa3diaWxWUlJtdkNOQXZ2R2NsQkUKLS0tIDcyYXh3N3B2QmNiK3dzemFFMGV1\nNTZpTk5yNGV5YVo3cGswK0NLWFQxQlEKIe0N5OxooWXzt1cUViBmjihmGEe3G6/f\nkz2/IscnG78ZvNgYKjdoG1jlsyje/3zI4C8aWXLq2DnIyxUyAhPgsQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2b0NVVm9QNm9waUUzcXBi\nWnZWU0JETjZRZmprZVRUL1h6ZHB4cXkrdm5rCm9GZ0VnTXB3S1BYSmlGWFJVcDhJ\naUl3RjR0ak9BRmQvVk1GRnQxNmtYM00KLS0tIFlTU1p2OHhWUGlOWngwbE56NEhF\nRW5QSkVVUWZpdDZXWEIxZ1BkbzVwclEK2P25nBgf8255vaKW/+T97aNTecRgNjLu\nedIUiPdXbFATCe3v/YRo6sqzFwIsvM6Bl9yHh/SXo6Ftc7eWZZd8zQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs"
}
],
"lastmodified": "2026-07-28T01:44:24Z",
"mac": "ENC[AES256_GCM,data:J0D8bEs5mHLraLS6TvYuCgfiNU1xKM2Yfb5Y0f/q/4wM4LzXufNzv3+SWDHumTe328U8UnNXLqjNHEKL0bZi0coxpU5hVM+BvPcmqD72vscETzbQ2hnU05sfW+XjfZhcN8/ke0bpLt7nP0crD5hsZv3esV1E2UWvzjEiYtWzFHY=,iv:lcmXYG2H469UKBYDndWKMO+GP0mSGLztenm+kBaUdYI=,tag:ujiXbLM9CUsuoFwQQWI84Q==,type:str]",
"version": "3.13.2"
}
}
+22
View File
@@ -0,0 +1,22 @@
{
"data": "ENC[AES256_GCM,data:Q++XWxg9tvY7ugT8+8FWCC5jgOfQ1+LYLsnqN0/WGxTf5XT2VmoXvszcE/ow2BUOBG3qBTXS3OHYFPwmj1GgaBHB8rpdXX6+LveSBE2gmx1VR+NUDTxy+0/DBq411MK9n/J9eIYcybdFIE11biFSAob9EgfxBF5roCDXIPDPyVCSe6LyhNvqYPnGQsfHCbejSewLTcRQEiguP9BX96CpMPIpmaB9fHN25t5RWCgMI7MtacrRxRsyKg44+2FZstXdZrp2Wv9u86BxqdAFqtZE8qpPeGdrdzluZx9jhnw0wZPzHdKg5wS7/UrLCb0UxIQiDxDMDuuBuLGkKXfAhb6SGZU41wWAWYk548iGRUaG+79BO4HhRZNObOFvfsjpMXEfcH/Vbv/wHVY4OpJUPe/ZWK4wpL9YrY4nT0t62HH7MirOy1uhwLE50h2+6dHKj6ur5G7thkSqgzVWXQ==,iv:zaRBwS+gfXLhH30havn6Q2+oPWuLV3qBfbOj00kewlQ=,tag:Vr5iEQ2u+9YNahryhgzwSw==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArTGJnaVhleUtsWnlITE1s\nNmRCaW1QWTFNSG5LRFdmb1lzRlV5NWl3YTBNCmF6eUN6RkJBZzc0MmJJa0dKTW01\ncVh4K1VLR2lURUpKQXpxNGpQNnpSUlUKLS0tIFJONnJFWkNCR3pqalRsUW9POVBj\nQ0pFQ3ltKzBETTVXTW5sV1ppWTFJc1kKzxUboNZO+Nwn2eTWy11VP9w1pRswCHaJ\nE2dYU0oUOClVzc0oSuIJxraG6TPj1N4WGC24gS+UmpkmSuCiOeZBsw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBDSmxNR1I3dUFKT0xUVG9h\nWkRIakNVeWRQOEN3blNRVjZlWHF4K2NRa0hBClRiOXlmTTJ4T2JTUEw2c1l0R2N2\nMnRwdDA5bEZlQWJRTm9vUmNKclBSU1EKLS0tIFcyeDFjbTZyVEVDUjN1VzU1VHly\nNWNDMW9rTXY2bHNWYVR0SmtMckovUzQKhTWr6yFVW9am3okCiIswwqR5+/p9OLmB\nWCgPtwoFaBt1RjUXPK4/eS4LlucR2K6V/mNMn4xVsnkIl193U9632g==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2ME9CbmVGcENRWksyL0pW\nOEJyUGhpbGMwWlBhVXBSeXQ2MW1EWnFuR0E4CnMwU0pjdk1YMzF5ZEhTVFlBaHZq\nam94UWVGbjhZSEx6VHBmem9JRWgwYzAKLS0tIHFJZWRyRjRHNzhXdDJSYWN3bDlR\nYVp3eGJWWkh3Y09ZWElyclZQN1ZSVFkKjR32//EcFAdMjVlNgky5zvVkwXwEN68D\nrkTuHKjiO5aV7yAQGPkdNw0UM0oRGF0u4YF3oOUcZfSvnKgDeoi2Zw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz"
}
],
"lastmodified": "2026-07-28T00:27:46Z",
"mac": "ENC[AES256_GCM,data:AbJIHYcFpeanQsJ3x7RPL9Yjlg5BJgkepKax0fL9L/PpA03Antab93iUNG95Mp6k/duovp8Jm445lbuppDZq1dh9ij/deBa8GbzJ50wwEe9zMc3EwRKScpqZEhRPF7KlJsIjsHJyd8NkcI5ji49XkHb4Ae1//8zG5HpVgy+3b04=,iv:uxQCbMwMIfP5S1dbsvIx3F79YWEguxwox8T0YZvUBdc=,tag:3utjmBGDmPc8q4JjaXvCkA==,type:str]",
"version": "3.13.2"
}
}
+26
View File
@@ -0,0 +1,26 @@
{
"data": "ENC[AES256_GCM,data:aqlMnoVkGtH9z3fJRweeC0OYf7LGqJU2sWA9Q25dKK6NuNyJd4BvjPtpfeg/WhVtJsaOtcbwVm4WAhVK9FARE8g8j+vmq0f6BAU4s6mx0ZIhl+mP+/hIpt//LOdd+9YezelJxdpzUyZbdAngU99rsTluLRe2XmZ7Fquxd8yH/OHenSDY6dizp9+5jfEi8EU+EmuXvuWMPY59xnlnqYNPfSFxs43/pS402LzJoJ5H+cBPprddkUBVzy4cBQvMnrRFUSjnqp74ovZkfIWFqDWQ5YgSU2PjatBg18oulZ7wNRhQ6OLqj6gsu+xrMjNFwnp7rMlA3X//hIidxTkVcYITycXd8KzuMIaofUpnwoyT34fy6+H35/39iiEyG4LRTrOOKRDzXkY2rhJUxFSZ8GlhNhMd0RlkmLngVYrtjsswJ9meIwoAFLPYt7BC61PJf0TXdtk=,iv:mCA819J9LpAOj8QxFAkrHI9wFJIy8qVxv31D6IwWFnk=,tag:qqMtM0eyHbQEyl6ND/wf7g==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOZVpRYmNMQ1E4ZlFXMDVq\nbW1FcnYzWTBHQmdhSjlsZUtZK3dzNUZjY1FvCkV4VXU4MTcwNVZoZUpVVS9JeHE5\nbGk3UzE3YmpZQ1JISHV0RHJqSG9ZNXcKLS0tIHhBdnIvMjJkeWlVRE9Fb0FhWjNX\nUjBVWDltK2w0akkwOTJaTGNSWWNrRXMKqZRNnHiXvn1QBoSGdABp7vOqNlsEN6Xr\nDp3NByXow6PuRuWvQXHzd+WC+ADkwNaaiT6TUrbZcd/Pl8Ges9kcZg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2VFBxc2ZMUFp0YkJhVFhW\nc0ZNbnJUcGtkT1VLR2YrVGhRRFVTZFlyZXlJCmVnblNBNGxIdHh4Q0IweWR1b0dq\nRWFyOURuWkdkRE1RTnJRMEpXdk9HaUUKLS0tIFhKV3FoWW1zRk1pMlhuWWlhV2E5\nQzJTRHAyc0JtSjd2NHlJODZVbndaVDgKFA4565X/4FqNq/fZDZTg81/55hZi4c7b\nTti2AnyE3OcY/kurXJFHRinVMqURQf1fx9MxqUYRitiCz4qe5zFF+A==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwRWlKTFNocnBRN01pZ0hm\na3pyRUkxMWM3dE1iTkZZcGRYUFcyb1RDb2h3Ckt4blpGZHBjcnZ4SVE5UnF4cmY4\nWW0rZG5wUkZINVk5a2lmSzN2L3JJY1kKLS0tIExpWGlBRFZJVGpGbFhxRE5ZYjBo\nQVliTlFIZ1U5dE9xbDhHMUtxenpBSm8KY6sIFEfK8p+70IXsC4Jwb9Lm/pd9+V6K\n4JAzGrpA6mAuIwwSNnbdcA5j8FmBhCpK6nLBWmFhGm9Y+MRTaM7Jrw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoV0pIUXlyOFFTUHVTdllu\neG10VENOYWl5Wk0ycnpLUytTbnlrQ25pMkFRCnhyRU5xWW9vQXJJOHdFMGZqMkR1\nSktqT3lOdVVPRUN4YTlveGp0NXpqd0kKLS0tIGRaTktra0ZtSVpzSHBrY3VSSUph\nRVRZOSsxTTNmMmltMlJnVy9oT2VEWU0Krxf49B1BsrWn05fqg+cZ0k0PtfJJNfn0\nUL44RUWXWbK2igQHaIct9DfYe7DEonBJeROuxDYm8g7yNOv15S+P4Q==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp"
}
],
"lastmodified": "2026-07-28T01:44:26Z",
"mac": "ENC[AES256_GCM,data:eyOSn03dzHSgkshPzLVwc95382eEFaDQarHs9l83dtcsb1Ui9CjkKipl2DVSUb6bdMUH1qKYmXqJhwFnAZbFZFjT4VTKtutNtM+OkhVXfT+fJs+1+u7k+ZUYFL9HuxKA6AWpwX3eJ8vmJDJZaAayJbm4PRzOyJywvKeneQqdUS0=,iv:QRKltR4qzVofo/Elt2Us/lrHlD7BenX605X31x+Ng78=,tag:SZ97CcfgMJeu3yqNk8Y/cA==,type:str]",
"version": "3.13.2"
}
}
+26
View File
@@ -0,0 +1,26 @@
{
"data": "ENC[AES256_GCM,data:+/1AAVha+86abQIX8tebqPJ6BIcCCiTJgPe0OfFOwA2Mcx0NJdiPtQgyeo3G8fAolIQOFkk5reL+GUhlOz4iEbPOCWwFCckdH0tXiCiVcD35qQNMRurY8HmpM5FWFnyzBkKuyNZ8okPuo2+fA3NQh3gs94rpm2kBsfS18xvrlv9b8u1JvAxlH6GRMN1uUgFGmOXlpGAVjDUFiKUHN8tRSZpsxG0aKKPBZ82JdKgYfCiCQifS1366gIFrsjGriUC+P8wkadRnD1JE2ZAGSL7wJLaRmzA4LAMGXFGbitOsFxKxX3q8VWEXaaL+9h4rTe0WCrvmgDM9NUeVHpfRiJrLvQgrFZMUxuZF66vQVsLybjIWpYYJcjC6nfR0U3lMa9gjzOTMPxm8HdGKC53FhLM1HiPoe1a2Nno283fV0uaByMPmo257O6l1CXqb6KoZOGFccm7fKR/VnFAS1AInF+szI7/r+UcaTW6LwYeV3OwLE4a8b4OiqTLgCu6v,iv:kUlVXF4Yl5HGvoLwu9loiuuDVtz0kARRiUU3K+BPL2A=,tag:pVaGzQAX9Etoqc5XMGmNeg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB6OUVUdkoxM1M5THlYZ1ky\nYWJ5RjhtRGoxTldwRkI2T1p6dHR4Y3gwR1ZrCjRDS1hTM01lZ0FSalBYQ1B0bENs\nUytXNmlrUkV4TmJkMUM1QW1ZaExIcGMKLS0tIHdHKzdEcUxvM0ZYUEp0a2d6SHp5\nYXQydy9uNG15V2FhUUd4NEFTMTNNMEUKkIzKEoYzoVs+nhnpkHFgDkQqrWykatND\ntsNxcr1SXSKeEW1m/QpXZnn/aW3zSQR09PqOHf7PYU47/AdkwrUaAg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSByZENrck42dWJjR2hYUVBl\ncERtRjR6dTdING5nWXlBc3o3eXFhaSs5NkFRCk43QXFUeTBJR0U5dG9YTDRmQ0Yx\naEo2blUvUXNZY3dpbEZRTUx1elNzdkkKLS0tIHJwNjRNM3V6WktWZ1BlUUtLRTNI\nMTVEbCtYbllIbTdxTGozWUluS3pIRXMKZCruDIkD/JofdAHWgPuaaKTDsz408ZkY\n77mhO8J+kd03qwt6qhFC5KF1lyjhwEnqrOE195+R/8Yl7hA/DsL2ZQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxaC95RjYybzVlU2tYdHdU\naGJ4M2IvNkJwVlNIU0VCZEFUZk5VaksrVnpFCmJXS3luTG9WejR0Rk9DbHA3dngy\nRElFWjU1N2xuc3JaKzQ0L0U2Yktib1kKLS0tIFlrK3F0ZFh5Ync3ejVWMzRKTUx2\nZDhxdEMwa3B4TFZUcWdTdktDeGt1QUEKfgYnK2lW3cZuJGaw+bAKDipLuC4S5vK2\nxK2eJB5TP/xXrp0F3lx9sc2b1FOY9Vt9IQ7zVlBqJFkzJrcw8zYJJw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwOGFkSXNxTUIzLzR1Skk5\nMGZveHFDN3YwVmpwT3VTYkppWEU1aGVpdkhRCjZtRC92bVphd0ZFbzFxVzRUcGk4\nMUd1aWdEbFRaM01FT25JSVRoMTRsNUUKLS0tIDAybmcrYjFVWkFYaGY5TFZjcjFO\nYlZoQjJjN2lFNzd4RlJuSHFlRlNCOFEKgUIPnL2/OJgz9oMYt86/llHa7adTkhs6\n8yGYGV1wtU9aYtUMIR907SfYyZ6M4z8jH2wwzpQLbwQQMLgkejl1jg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m"
}
],
"lastmodified": "2026-07-27T23:14:51Z",
"mac": "ENC[AES256_GCM,data:CYhzR0Y29GXvWUBbb13s16v9hDQ4k4Xmd4FUpcyNtHe0L/IrDGkx1WHzAohNGZjGMNB9W3BgiEH85OoOo8r1F82El4/8s2prEJi8AARvQU3+2cmRjjhNCCYQWmJkoF/cZNpw6nVyWzZdfUpPvHjbuf6utlI3rtR0qiCpSo/32S0=,iv:XLKTU2Ute4jMkfRAbXiVGxqP9+fjSs6HwRv1JfTTe7w=,tag:oIqS/1HWQZ7mLWaO7GwLFA==,type:str]",
"version": "3.13.2"
}
}
+43
View File
@@ -0,0 +1,43 @@
beszel-token: ENC[AES256_GCM,data:qg2eb7Eq/w7vIuiiDeK/s6h9nmHWuooyrd93GvdLD2hGOvka,iv:FWRBQTIY9PzV2V8J7z5ftEb5dvjN/S0hP0H9JV+Uuq8=,tag:WP7DHOu4elfqdLoW7vo9qg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZmozK3VjcENGb3l2MDNj
QVE3VVI3Z3R0MjduSlJGL3BFWkg5NHZmZms0CkNBTWZKZFJVNkJMZ0tpNXhXWW5q
WndvbmhuOVh5ZHhJMmJwenh0M2dEVlUKLS0tIGpXQ3dhRXVWUDZzdGg4RHN4V3p0
WDZnK1lMUHVIS09hRUEzVHRHQ0EwcWsKEd5NtstHSPH8okXcFLW7BId5xPGmgyC3
dAodIGkLBlJlaL1bZuRy6Vaac+mJZL5KJYD/o1kYSXRiHJfNpaulTg==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXWWhYOFU3VHZ3S1VuRTF2
QlZvMkJUNDZmT1RDWWZsMkxnZjJldjdrRG44CkU0NEJTR0NhQUpjMEY4RXBLSVBJ
bDNtZ1UrRDNUbzA0NDJ1cWJKdlBBMWMKLS0tIE9vTmEzTUxpODF4SG1MQ1czM1N5
dVgrMERMRXJwVmhtMldTZUlHNUJVTEkKzUf9cETbo0K9bCPkrlbUkgkhRIPvjCHR
GFVwgb2fcaOzZHcDwlLDLaP6HbRPtzJfVbLpQ1XU0N/xijSr9ENB6g==
-----END AGE ENCRYPTED FILE-----
recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlOVdWN0tHNWZmaWlWQ2hJ
MVFYd0pMNmJTdTlJQ3h1c0dzbDE3U2VqdzNzCitCcFdhbjRPYnl3Q214WnRxa2Jq
eTFnc2VBcXlGcTVYdWpWTFZiWXBMUDAKLS0tIC9YdVkrTU5EeXNlNzliLzBlYUR2
Qk1ITW9tMElzUHF4MVJyWS9uRGs1QncKj8OgxWYabf8MccWD6zYLxl+FvYPAvLCW
p/FJ8uMa9YWp0RjICvWIwyO0gK7oaxTh/JzfBjABngEZnZazNYP7Uw==
-----END AGE ENCRYPTED FILE-----
recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsbzFKckFQVDg0VEpJMFlv
WSs1ODVoYlZZK3YrSXlMU1hIQjdLTUhNdHhJCkRnbHY5Z2NGQVMvOHV4R2hTd0xu
LzdheXdZVVBsTjhVOU8zZGFKN0lPWjQKLS0tIFdaaG5SeElDN3YvMjFxdHVLWlNX
M1BQQmtJTUdsK2x5TEo5TmNrMEZidkEKhi4jNKhrkBAPhJoeYNg1d60LQTqmJt+G
Z/2X5uWzh12zF5keL8cNHH6sD0z0MOChtd4UF37yr+lb/LFJkia6LQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
lastmodified: "2026-07-26T01:12:11Z"
mac: ENC[AES256_GCM,data:0r68y6XmdiW7p/h9QmTR5h37TxKp+0wF44LP8QfG6cGP0tvaasuvv+V9EC3msnREYjwJhlnHjRd8hJ31HGVkOHbYWI4V5cF4ZnaiGgrZJhsGcJvOJSroAvQwev+vIlXZpAPgZz3Z35FDvOpgb71l5c0so0l4EJhc70JOh2FxhvQ=,iv:Or6l/v/E01+9wtZwyl+pCttiuReR6V0kmBzE9KhsfgQ=,tag:cAEUgj/9tKDToKFr00g5UQ==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
+18
View File
@@ -0,0 +1,18 @@
{
"data": "ENC[AES256_GCM,data:QvViw/s1N7eIN3CoD27llEjriuSrohOou4Cv310nigcW8xMnA2SDN2id3H3AoKii1JlJ+qWpKn+gUmt5HOM0UlbeNe3os2RVwiX38O7eN9xFH9F5kA3TFs6Umqq3EoI586PwIVmB2LyxDnTeEEXVd7v5PFkBcfu7u8YIcNF7lpcj+6rOyHMB8uxPhrGep3yiKawFd9c9wWD0hlSSatV5tMHA1qmdK8VmDbCU/iuGwIoMzN1eZwGAXzG6LkCA63bUfdxU6yGuTboD+kN2Wbo+GZB0EACmiZoofl2wqlXuiw6qwTvlXkyauc9O5EG//PUkIECzDwiXcX+qSOM9DIBlZNth4ebhtic/PskyF09etL/gICz5YvV4ph5lyrWHq41KxljSU7QXOkGhzagruuMrYzhZb35wFH4Tie1ee2DXbGhreJr8V3Zse/zTMaD+iM57V8bvNcarTzOFXyKfp7Y=,iv:XBPhj2wT0k/yRCRHU4d+BQA/k00ZHWSKOucnZ5+PGys=,tag:Nh5MOxZIUXAYVZY5SZh/JA==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0aUtETXlBMmRlTC9vK3NF\nRW4zS1FrSmo1dDN0OGN4SmV5ODd5MTRVejFJCnpPQkM2MVJ5WFlYRW1NVjduV1hv\nUWUzY0hHV05LV1BROVZVUlM2NldsTlEKLS0tIHFsTzI2SVZzYUtJWTM0MmFiUlVQ\neHEvUXgzc1pxSU1OZFo0cXhSZDdGUVEKpLVfzQEnntluUGsblnkHZJ9Jezu8tFte\nxEoV96GVHxUca6TFWpTLMqdR7NtuQGCkx295W3i1tkp58DP1OzRTUA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA4ZW5ybVBvTkxibWJMWGRL\nZ1lnWEkxZ3l6d0FWbmxMY2VsTVlzOHJJM3owClNYd3BlQi9pL1lIdzJmekJUVEVi\ncWhSK3ZkZEMyeTNoZlpTT0NMUXZFWVEKLS0tIHVkVlhmQXdRQktTK3J5dXZ4aTNt\ncUw3WCt5dXJhTkdUbVpmeWoxWkoxNnMKj4XtdwmgFVOiVsIJs2Du7QJ09A9tv/Lf\nkFOq8y4tlZe0nCwRjq43sVz7hdCTdQ0rsaWjBGY90LLkJbOA+f+Wrw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2"
}
],
"lastmodified": "2026-07-28T01:44:30Z",
"mac": "ENC[AES256_GCM,data:zR2WLWX7NaHA15gi4kX0jDvzUIe9jtz5bMCAggbPW+IXOEedPrddAHZ8OfPErVMfx8O1pJKkAKSzoPTAbEle54FisSLMHXp8fI0297MByJrF9pOsMFpVcDy/L4Q+pBzmB7aS9r7+u7KRVVTZT3QwG1rFWZaDs5dFTP80RhtCbWQ=,iv:ZzPoO+h7ebS+jsSH7tWMx6QK8umpa2/HFQmx9dnJN+Y=,tag:vG9+ifxw4HaBE6YsmOwXcg==,type:str]",
"version": "3.13.2"
}
}
+79 -5
View File
@@ -4,8 +4,20 @@
homeDomain = "sweet.home"; # base LAN domain for service subdomains (pve., docker.) homeDomain = "sweet.home"; # base LAN domain for service subdomains (pve., docker.)
tailnetDomain = "tail13f623.ts.net"; # Tailscale MagicDNS suffix tailnetDomain = "tail13f623.ts.net"; # Tailscale MagicDNS suffix
lanCidr = "192.168.2.0/24"; # LAN subnet lanCidr = "192.168.2.0/24"; # LAN subnet
pxeServerIp = "192.168.2.247"; # pxe-boot host's LAN IP lanGateway = "192.168.2.254"; # LAN default gateway (router)
pbsIp = "192.168.2.108"; # Proxmox Backup Server LAN IP lanPrefixLength = 24; # LAN subnet prefix length (/24 = 255.255.255.0)
lxcLanInterface = "eth0"; # LAN NIC name in LXC containers (set by Proxmox --net0 name=eth0)
vmLanInterface = "ens18"; # LAN NIC name in Proxmox VMs (virtio, first NIC)
vmStorageInterface = "ens19"; # storage NIC name in HA server VMs (virtio, second NIC on vmbr1)
pxeServerIp = "192.168.2.223"; # pxe-boot LXC container LAN IP
nixCacheIp = "192.168.2.224"; # nix-cache LXC container LAN IP
tailscaleRouterIp = "192.168.2.222"; # tailscale-router LXC container LAN IP
torRelayIp = "192.168.2.221"; # tor-relay LXC container LAN IP
serverIp = "192.168.2.226"; # server (NFS/ZFS) Proxmox VM LAN IP
dockerIp = "192.168.2.225"; # docker Proxmox VM LAN IP
pbsIp = "192.168.2.244"; # Proxmox Backup Server LAN IP (not NixOS-managed)
domainControllerIp = "192.168.2.253"; # FreeIPA domain controller — authoritative DNS for sweet.home (not NixOS-managed)
ipaServer = "domain-controller.sweet.home"; # FreeIPA server hostname (used by security.ipa and Kerberos; must be a resolvable FQDN, not an IP)
# Cross-host references (LAN hostnames/users other hosts reach over the network) # Cross-host references (LAN hostnames/users other hosts reach over the network)
nixCacheHost = "nix-cache"; # substituter/remote-builder hostname nixCacheHost = "nix-cache"; # substituter/remote-builder hostname
@@ -68,6 +80,40 @@
# one-line change. # one-line change.
primaryUser = "nixos"; primaryUser = "nixos";
# Primary IPA/domain user. Home Manager is configured for this user on every
# IPA-enrolled host (see modules/ipa/client.nix) to manage the environment
# that IPA itself doesn't cover: dotfiles, user packages, session variables.
ipaUser = "wayne";
# GID of the IPA "docker-access" group (GID 50010 on the IPA server).
# The local "docker" group is pinned to this GID on every host that runs
# Docker so that IPA group membership alone grants docker socket access -
# no per-host users.groups.docker.members entry for the IPA user needed.
dockerAccessGid = 50010;
# HA file server cluster
# LAN IPs (vmbr0 / ens18) — client-facing: iSCSI initiators, NFS, management.
# Storage IPs (vmbr1 / ens19) — isolated internal bridge, used for DRBD
# replication and Corosync heartbeat only; never leaves pve1.
# haServerVip: floating virtual IP managed by Pacemaker's IPaddr2 resource;
# NFS and iSCSI clients connect here regardless of which node is Active.
haServer1Host = "ha-server-1";
haServer2Host = "ha-server-2";
haServer1Ip = "192.168.2.228"; # LAN IP, node 1
haServer2Ip = "192.168.2.227"; # LAN IP, node 2
haServerVip = "192.168.2.229"; # floating VIP (Pacemaker IPaddr2)
haServer1StorageIp = "192.168.4.228"; # storage-net IP, node 1 (vmbr1 / ens19)
haServer2StorageIp = "192.168.4.227"; # storage-net IP, node 2 (vmbr1 / ens19)
haStorageCidr = "192.168.4.0/29"; # storage subnet — internal to pve1 only
haStoragePrefixLength = 29; # storage subnet prefix length (/29)
haStorageRoot = "/srv/ha-data"; # XFS-over-DRBD mount point on the Active node
haIscsiIqn = "iqn.2026-01.home.sweet:ha-storage";
# DRBD backing disk — identified by SCSI controller path so it resolves to the
# correct block device regardless of OS-level naming (sda vs sdb can differ
# between Proxmox VMs depending on disk-add order). drive-scsi1 is always the
# dedicated data disk on all HA nodes; drive-scsi0 is the OS disk.
haServerDrbdDisk = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1";
# Storage # Storage
storageRoot = "/tank"; # ZFS pool root on `server` storageRoot = "/tank"; # ZFS pool root on `server`
@@ -83,6 +129,7 @@
# dataset or moving where it's mounted only needs changing it here — the # dataset or moving where it's mounted only needs changing it here — the
# export and every client reference follow automatically. # export and every client reference follow automatically.
nfsShares = { nfsShares = {
options = "(rw,sync,no_subtree_check,no_root_squash)";
dockerConfig = { dockerConfig = {
subpath = "docker/config"; subpath = "docker/config";
mountpoint = "/mnt/docker/config"; mountpoint = "/mnt/docker/config";
@@ -103,6 +150,18 @@
subpath = "raspi/volumes"; subpath = "raspi/volumes";
mountpoint = "/mnt/raspi-backup"; mountpoint = "/mnt/raspi-backup";
}; };
proxmoxIsos = {
subpath = "proxmox/iso";
mountpoint = "/mnt/iso";
};
proxmoxLxcImages = {
subpath = "proxmox/lxc";
mountpoint = "/mnt/lxc";
};
pxebootImages = {
subpath = "pxe-boot/images";
mountpoint = "/mnt/pxe-images";
};
}; };
# The Raspberry Pi's own NFS export — not under storageRoot/nfsServerHost, # The Raspberry Pi's own NFS export — not under storageRoot/nfsServerHost,
@@ -129,10 +188,25 @@
# (modules/build-types/pxe-boot.nix). # (modules/build-types/pxe-boot.nix).
pxeBootTftp = 69; pxeBootTftp = 69;
# `server`'s NFS exports need both the portmapper (rpcbind) and the # `server`'s NFS exports: portmapper (rpcbind), NFS data, and the
# NFS data port itself opened (modules/build-types/server.nix). # mountd RPC service (used by showmount/NFSv3 mount protocol).
# Mountd listens on a fixed port so the firewall can whitelist it
# explicitly rather than opening all of rpcbind's dynamic range.
# All three need both TCP and UDP (modules/build-types/server.nix and
# modules/build-types/ha-server.nix).
nfsRpcbind = 111; nfsRpcbind = 111;
nfsd = 2049; nfsd = 2049;
nfsMountd = 20048;
# HA cluster ports opened on ha-server-1 and ha-server-2
# (modules/build-types/ha-server.nix / modules/ha/cluster-config.nix).
haServerDrbd = 7789; # DRBD replication (TCP)
haServerIscsi = 3260; # iSCSI target (TCP)
haServerCorosync1 = 5404; # Corosync totem ring (UDP)
haServerCorosync2 = 5405; # Corosync totem ring (UDP)
haServerCorosyncCrypto = 5407; # Corosync crypto sync (UDP)
haServerPacemakerRemoted = 3121; # pacemaker-remoted (TCP)
haServerPcsd = 2224; # pcsd cluster daemon (TCP)
# Opened on the docker host's firewall for the Traefik-fronted # Opened on the docker host's firewall for the Traefik-fronted
# container stack (docker-compose config lives in the separate # container stack (docker-compose config lives in the separate
@@ -166,7 +240,7 @@
# build (modules/disko/proxmox.nix, config.system.build.diskoImagesScript # build (modules/disko/proxmox.nix, config.system.build.diskoImagesScript
# — see docs/proxmox-images.md). Root fills whatever's left after the ESP # — see docs/proxmox-images.md). Root fills whatever's left after the ESP
# and swap partitions within this total. # and swap partitions within this total.
proxmoxImageSize = "20G"; proxmoxImageSize = "50G";
# nix-cache's Nix store garbage collection retention # nix-cache's Nix store garbage collection retention
# (modules/nix-cache/server.nix). # (modules/nix-cache/server.nix).
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN7IABG7x9ejHSENmHsL/BF9MQadGuBnHZa1eUEFMYnH ha-server-1
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:EhYMNPzlMDyby6g3hfvBKisKJnU+YAxZjf326tPdflQ2PcmPTzZ0GjHqJOPrOWyLU6IyvnsLMBV0JF5/UUi2c/H3p2wvoBgsxDMaaMW0YUK2IZFo+PZpWoYJSMrx40sH90W0qL0VBw17JLwPU6CjeUZrn5+sCN9EhQvncD8NoJ6QwZpWKhWLn9/7mxd/PF1uxjEQXzHYFeuUuVpQdKSUfV/BSJIT3pMM/VdH2203jTMtwUbK1/7UtGKtDWdfFbkEvJ6qsV7hnN3EYU16WtdyAiHEZvvZ3gqg6YK2DemojWG7z7VQkQGbGm40yA6/mYGW2dVK5ULzjxALl05G6lo1vzxODbMHhzsfJZVw6ocGx3r8xwaasiX8S3loq8WNUWJ9QgjbxuEX9uGZja6A3nPhcwVGGZExbUoU+syXUk2m+WfIQ0epplaPDyAxrYO5ZPWLdMYoPRxbDx94rx5fMxA3jst++IEL2kQMTieGQYJy3R8veQIe3Jb074zY+Y79EmRxfo8ruR6ZMqORtbZzJOBcd0OObjUVLUqB4rnC,iv:XkCfOKtmZNz+UTZdElLm8L0PpymkDzCM24VkAa9Y/Qo=,tag:hmr9ik0V7mxEKR0DcDB3/w==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqQ3J3VDVwUVBvUjVXSWNz\nU3YrRXZlT2NLRThCNzd3YWNHVTlHeS9xMHpzCi9HVW5kMXZrdjE4bXdML3NhRnI1\nL0NQQkNQUmg2M0NBYzh2cG1vdHA0clkKLS0tIDJZYUwzK0RGNzJWckJJekI0SFg1\nUXE3N05xWThtdjdCZkdJTFJ2YUx6cm8KVANudVL54WBNc9DK9s9h4WQRLMewUqgN\ntu4LdMOmi5oV3LX06lbxhBq79dmsV5uos/qszhJkVGhEZwD1RGBoZA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-28T06:42:29Z",
"mac": "ENC[AES256_GCM,data:Xn3/CWhQJHWtd3QHoqpdwLuJdZTp+oxx43k38j4jspQSBPdVDSG6Ifi7Sb2r1g9YTxpjUto8mv7FH+BqI0wSYUkLfjaxI51xs3dq6e6kl5NWOsvWwIOGmBuhfIwusHGgsfBguxk0J6Ev8Irw674CRFc37jozXTVdpOOzFhPTvFs=,iv:/7OFE9NYkw2kYEetzeZRXah0KXuD5OXmfjQMIeqI75Q=,tag:VamuInZqYGYRwlyU8TtL9g==,type:str]",
"version": "3.13.3"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINWuFwPKW2jwQEj6c//xb95DbtandPkJ7M0ceyTNd35G ha-server-2
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:GEebcQrPPrQxLC6mWXLLtytGkIy6Q2FbcQ/Z+ImGa3gahI/nigdNyR/avIz1DAUsOibpoMVlrk0GQUBMuxiEXulIwUEzeTh7sMI3XTZiugujOFHPyJoFeynlgeRNBbl2EVVjGmEmOsR/cZJGyKaet9aNTHTPDJUhodY5d+D/sE3SgpHoch1DI02/DRnqlnlOYiBoujk++BW71EDHLVyz3n9NsurTDYDK4L0Ch5cMgdlvwIuxGuPKtkhU9z6PtGxPbNlYlNP8U+3jG4XWEEY5hZHBGTVHn+bd5FViGY+sMiigO6yaOmddbUqzJoblKeI6F/rfrL3kXETLgVP7uVm3gw9KPEl5JaTdVqkG4hQ9G9/LvRlyEh619mCp9IRXxabm5SQ4NWFzah1ZwqIKp6GoAtqeWRgZNVqijoF+bnb0YmNFmjwpSrSMZvKsrZb5geyGmqah+NJii0xqVebnufB5p8lW9eytBSNgC/XLjj0olwHMw7CWGmnY6Q7lxF1tNYAD/K3dkbxQYFM3kBnmAdweC8WYc+hDjtk0HJul,iv:H4BonLmf3VoW2S9+IUE+SzPb0qiWHuNuGpVIUgUYScs=,tag:5+bq7vC81DMmOk4jn3u8lg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrQitsaDlqTmlYM2RvR1Ew\nb3NzZTJzR3F2bjM5NHhsa3dCUzZ4TW1pa3pBCkxIeEJtbWdaUFMwMkFyRHpCZ1Bl\nbmM5dGV3b3BDcDEwVlJ4UWV5RFNvRkUKLS0tIFRDd0U3dHBHb0Y3WDhNV3hkd2Ew\nclRDVEZHYjZUSzNVZkFjWGt3SGVQTlUKOULuXiYD9k2uVUmhuC15Kgezrd69rc9P\n7SocPa8kBliffP9IuxxW2S/hPbK8rqEx/sh/Km85ZIi4pG0AUQJ3fA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-28T06:45:10Z",
"mac": "ENC[AES256_GCM,data:ZnsqhgELqEdNzlTKJx2+xzcZRMNGTme/0a+PVvpbUv5IApimtif0zkTEC1LJsyLPut3MmO3QGprVlb8ihoRdm/IVvLaQonc36f6vN5cxrpQadru/RWEjLbnge1E8VkN/PVvHqRfLyIOPSq6wtGjAPaC/iHW1scBi50pWr0XO628=,iv:FlqAL1It6JuvoBhZotHWdSz17nckBf2rXnwovl6ZJBA=,tag:tNwcaL1sJakXtgejsB/Sbg==,type:str]",
"version": "3.13.3"
}
}