Replace the static host list with dynamic discovery: workstation (nixos)
and pve1 are hard-wired first and second; remaining hosts are discovered
on every run by SSHing to pve1, listing running VMs/containers via
pct/qm list, and resolving their NixOS hostnames from a single flake eval.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Runs nix-collect-garbage -d on all deployed NixOS hosts and pve1 in
parallel, skipping nix-cache to avoid evicting shared cache paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nfs-utils changed ConditionPathExists from !/etc/krb5.keytab to
/etc/krb5.keytab, so on IPA-joined hosts the service now starts instead
of skipping. Exports use standard auth (no sec=krb5) so the nfs/ Kerberos
principal is never provisioned and the service fails.
enable = false masks the unit so nfs-server's Wants= can't pull it in.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
KEY (hub's SSH public key) is now stored in variables.nix as beszelHubKey
and set by enable-agent.nix — no host file ever needs to touch it.
TOKEN was already universal; this removes the last per-host beszel config.
Importing enable-agent.nix in a build type is now the only step needed to
add a new host to beszel monitoring.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five files contained only beszel-token and are deleted entirely.
nix-cache.yaml retains cache-priv-key; beszel-token removed from it.
The universal token in secrets/common.yaml replaces all of these.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace per-host host-token.nix imports with a single beszel-token secret
in secrets/common.yaml, wired once in enable-agent.nix. Host files now only
need services.beszel.agent.environment.KEY — no imports block required.
Delete modules/beszel/host-token.nix (no longer referenced anywhere).
Action needed: run `sops secrets/common.yaml` and add `beszel-token: <value>`
from the beszel hub UI before deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents how to wire beszel-agent into any flake build: the two-module
split (enable-agent.nix in build types, host-token.nix in host files),
sops secret creation, hub pairing flow, and optional EXTRA_FILESYSTEMS /
Docker monitoring knobs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three-phase online resize: Proxmox qm resize → guest block device
rescan → drbdadm resize + xfs_growfs. No downtime required.
Detects active node via crm_mon, pre-checks DRBD UpToDate/UpToDate,
auto-discovers VMIDs from qm list, and resolves the block device name
from the stable scsi-0QEMU... by-id symlink with a slot-index fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
crm_mon 2.x formats the Promoted line as " * Promoted: [ node ]" — the
asterisk bullet means the previous grep -E '^\s*(Promoted|Masters):' never
matched, so active-node detection silently returned empty in health.sh,
failover.sh, and acceptance-tests.sh.
Fix: pipe through grep -v Unpromoted first, then grep -E '(Promoted|Masters):'
without anchoring to start-of-line.
Also: remove the SSH_OPTS=-i ~/.ssh/... variable pattern in health.sh and
failover.sh; tilde is not expanded inside double-quoted strings, so $SSH_OPTS
was passing a literal ~ to SSH. Inline the key path in each function
definition instead (same as acceptance-tests.sh already did).
Also: drop the 2>/dev/null embedded in the crm_mon argument string — the
outer 2>/dev/null on the n1/n2 call already suppresses SSH stderr; the
embedded one was harmless but noisy to reason about.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
failover.sh: graceful active→standby resource migration with pre-checks
(DRBD UpToDate/UpToDate, quorum), spinner wait for XFS mount on target,
rollback on timeout, --to/--force/--dry-run/--timeout flags.
health.sh: read-only status panel — node reachability, quorum, DRBD
role/dstate/cs per node, Pacemaker resources, failure history, XFS mount
usage, and service port checks (NFS 2049, iSCSI 3260) via VIP.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Even with both nodes in standby, Pacemaker's monitor operations keep running.
When the monitor sees DRBD is Primary on a standby node (that it didn't start),
it triggers a stop action — killing the initial sync after ~10 s.
Enable maintenance-mode after standby stops the existing resources but before
DRBD is manually brought up for the sync. Maintenance-mode suspends all
start/stop/monitor actions so Pacemaker is completely hands-off during the
sync. Disable it alongside crm_standby -v off once UpToDate/UpToDate is
confirmed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The metadata creation check used grep -q "UpToDate" which matches when the
resource is DOWN — drbdadm dstate reads metadata directly and returns just
"UpToDate" (no slash) for a down-but-previously-synced resource. This caused
metadata creation to be silently skipped, leaving mismatched UUIDs from an
interrupted sync which cause DRBD to go WFConnection→StandAlone immediately.
Fix: require exact "UpToDate/UpToDate" (with slash, only possible when the
resource is UP and both disks are confirmed synced) before skipping.
Also give the StandAlone detection a 5-iteration (15 s) grace period before
dying, to avoid false-positive on the first few ticks while the peer is still
coming up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clearing crm_standby before the initial sync finished caused Pacemaker's OCF
DRBD agent to race with the manual drbdadm up/primary calls. The agent saw
DRBD in WFConnection or SyncSource and tore it down, driving the resource back
to StandAlone and killing the sync in seconds.
Move the crm_standby -v off calls to immediately after the sync-complete
break, so Pacemaker only resumes once DRBD is UpToDate/UpToDate and safe
to hand back.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If the DRBD peer connection drops during the initial sync wait (cs:StandAlone),
the loop would spin forever printing "waiting for sync progress" with no
indication of what's wrong. Now parses cs: from /proc/drbd each tick and
dies with a clear error if StandAlone is detected. Also shows the cs: field
in the non-syncing message so the actual connection state is always visible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the DRBD kernel module invokes fence-peer via the User Mode Helper
mechanism it provides a minimal PATH that omits /run/current-system/sw/bin.
crm-fence-peer.sh calls cibadmin, crm_mon etc.; without those in PATH a
pipeline breaks with SIGPIPE. A signal-killed process has WEXITSTATUS()==0,
so the kernel sees exit code 0, logs "fence-peer helper broken, returned 0",
and retries forever — blocking Pacemaker failover completely.
Fix: use pkgs.writeShellScript to create thin PATH-fixing wrappers in the
Nix store. The wrappers prepend /run/current-system/sw/bin before exec-ing
the real crm-fence-peer.sh / crm-unfence-peer.sh, giving them a working
Pacemaker toolchain regardless of what PATH the kernel provides.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The TTY check ([[ -t 1 ]]) evaluated false under sudo, causing each update
to print on a new line via log(). Since \r worked correctly in the original
code (user confirmed), drop the branching and always use printf \r.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The sync progress grep was silently failing because /proc/drbd formats the
percentage line as "sync'ed: 5.2%" (two spaces after colon), but the
pattern matched only one. Use [[:space:]]+ throughout to tolerate any
whitespace.
Also make the display TTY-aware: interactive sessions get the spinning
\r in-place update; piped/logged sessions get a plain log line printed
once per unique percentage point (avoiding scroll spam).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cluster-init.sh: replace the fixed 300-iteration sync wait with an
indefinite loop that prints an in-place progress line (%done, ETA, speed)
from /proc/drbd every 3 s. Clears the line with printf \r before logging
completion, so the output stays clean alongside the [cluster-init] log
lines.
acceptance-tests.sh: add a pre-flight check that hard-exits if
drbdadm dstate is not UpToDate/UpToDate, with a hint to the watch
command for monitoring progress. Tests cannot give accurate results
while the initial full-sync is in progress.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
acceptance-tests.sh: replace drbdadm-role-based Active node detection with
crm_mon-based detection. DRBD role briefly reads Secondary during Pacemaker
transitions, causing T2/T3/T4 to target the wrong node. crm_mon Promoted:/
Masters: lines are authoritative; wait up to 90 s for Pacemaker to settle
before falling back to NODE1.
cluster-init.sh: add crm_resource --cleanup before the VIP wait loop so
stale migration-threshold failure counters from previous cluster-init runs
are cleared before Pacemaker decides resource placement.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/dev/sda and /dev/sdb are assigned by the OS based on Proxmox disk-add
order, which is not consistent across VMs. Use the SCSI controller path
instead — drive-scsi1 is always the dedicated data disk on all HA nodes
regardless of which sda/sdb it gets assigned to.
variables.nix: replace per-node haServer{1,2}DrbdDisk with a single
haServerDrbdDisk using /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1.
cluster-config.nix: revert to shared volume block (per-node block was
needed for /dev/sd* but the by-id path is identical on both nodes).
cluster-init.sh:
- single DRBD_DISK variable (matching haServerDrbdDisk)
- robust /etc/drbd.conf patch: NixOS manages this file as a symlink to
a read-only Nix store path; cp --remove-destination breaks the symlink
before sed -i so the edit actually takes effect
- scp helper script to NODE2 rather than bash -c over SSH to avoid
quoting complexity
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Proxmox VM disk ordering differs between the two HA nodes:
ha-server-1: sda=OS (50G), sdb=DRBD data (32G)
ha-server-2: sda=DRBD data (32G), sdb=OS (50G)
The DRBD resource config was using a shared disk=/dev/sdb which targeted
the OS disk on ha-server-2, causing drbdmeta and drbdadm up to operate
on the mounted root filesystem (hence "Device or resource busy").
Changes:
- variables.nix: add haServer1DrbdDisk/haServer2DrbdDisk
- cluster-config.nix: move volume block inside per-host on{} sections
so each node uses the correct backing disk
- cluster-init.sh: use NODE1_DRBD_DISK/NODE2_DRBD_DISK variables;
add runtime check that patches /etc/drbd.d/*.res on the running nodes
if the deployed config points to the wrong disk (workaround for VMs
built before this fix; redeploy with --force-rebuild to make permanent)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drbdmeta is in /run/current-system/sw/bin which is already on sudo's
PATH on the HA nodes; the bash -c wrapper with PATH manipulation was
adding complexity without benefit and introduced quoting hazards that
likely caused write-dev-uuid to fail or run in the wrong context.
Use direct n2_ssh calls instead — the same pattern that works when
tested manually on NODE2.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drbdmeta lives in the same Nix store dir as drbdadm but sudo doesn't
inherit the full PATH, so drbdmeta was not found (exit 127) even though
drbdadm was. Resolve drbdmeta's directory from drbdadm's location and
prepend it to PATH.
Replace openssl rand for UUID generation with /proc/sys/kernel/random/uuid
— openssl is not guaranteed to be on PATH in a minimal NixOS root
environment, but /proc/sys/kernel/random/uuid is always present.
Apply the same PATH fix on NODE2 inline in the bash -c invocations that
call drbdmeta over SSH.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cluster-init.sh: replace 'drbdadm create-md --force' with direct drbdmeta
calls using --force on both steps. drbdadm create-md --force passes --force
to the create-md sub-call but NOT to write-dev-uuid, which then fails when
/dev/sdb is busy and stdin is not a TTY ("stdin not a TTY, not waiting for
confirmation" → exit 20). write-dev-uuid failing means DRBD has no UUID,
so the peer can't identify the device → connection fails → no sync → wrong
Active node detected by acceptance tests.
acceptance-tests.sh T4: grep -c returns exit code 1 when the count is 0
(no matches), triggering '|| echo "0"' and producing "0\n0" which breaks
[[ "$IQN_COUNT" -ge 1 ]] with "arithmetic syntax error". Fixed by running
the pipe inside bash -c with '|| true' on the grep to suppress the
non-zero exit code. Same fix applied to T5's IQN_ON_STANDBY check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause of recurring sops failures on new VM boots: disko builds raw
disk images, and create-proxmox-resource.sh only syncs the clan-var SSH
host key to the Proxmox node (for proxmox.nix to bake into the image) when
it actually builds — reusing a cached image skips sync_remote_host_keys, so
destroy+recreate reuses a stale image with the wrong or randomly-generated
key baked in. On first boot the VM gets a different key than what .sops.yaml
was encrypted for, and sops fails permanently.
Fix 1 — deploy.sh Phase 3: always pass --force-rebuild so every VM creation
rebuilds the disko image fresh with the current clan-var key baked in via
NIXOS_HOST_KEYS_DIR (proxmox.nix already reads this under --impure).
Fix 2 — deploy.sh Phase 5.5: after VMs boot, scan their actual ed25519 host
keys and, if they drift from clan vars, update the clan var pub-key files,
rewrite the .sops.yaml age anchors, and re-encrypt all affected sops files.
Defence-in-depth: normally a no-op after Fix 1, but catches any residual
mismatch (e.g. --skip-create-vms reuse of an older image).
Fix 3 — cluster-init.sh: add crm_standby -v on for both nodes before DRBD
metadata init. Without this, Pacemaker's OCF DRBD agent races: it sees
drbdadm down as a failure and immediately calls drbdadm up again, leaving
/dev/sdb busy when create-md / write-dev-uuid runs (drbdmeta exits 20 with
"stdin not a TTY, not waiting for confirmation"). Standby suppresses
resource scheduling during init; crm_standby -v off restores it after DRBD
is up on both nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
beatzaplentyandClaude Sonnet 4.6