Three-phase online resize: Proxmox qm resize → guest block device
rescan → drbdadm resize + xfs_growfs. No downtime required.
Detects active node via crm_mon, pre-checks DRBD UpToDate/UpToDate,
auto-discovers VMIDs from qm list, and resolves the block device name
from the stable scsi-0QEMU... by-id symlink with a slot-index fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
crm_mon 2.x formats the Promoted line as " * Promoted: [ node ]" — the
asterisk bullet means the previous grep -E '^\s*(Promoted|Masters):' never
matched, so active-node detection silently returned empty in health.sh,
failover.sh, and acceptance-tests.sh.
Fix: pipe through grep -v Unpromoted first, then grep -E '(Promoted|Masters):'
without anchoring to start-of-line.
Also: remove the SSH_OPTS=-i ~/.ssh/... variable pattern in health.sh and
failover.sh; tilde is not expanded inside double-quoted strings, so $SSH_OPTS
was passing a literal ~ to SSH. Inline the key path in each function
definition instead (same as acceptance-tests.sh already did).
Also: drop the 2>/dev/null embedded in the crm_mon argument string — the
outer 2>/dev/null on the n1/n2 call already suppresses SSH stderr; the
embedded one was harmless but noisy to reason about.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
failover.sh: graceful active→standby resource migration with pre-checks
(DRBD UpToDate/UpToDate, quorum), spinner wait for XFS mount on target,
rollback on timeout, --to/--force/--dry-run/--timeout flags.
health.sh: read-only status panel — node reachability, quorum, DRBD
role/dstate/cs per node, Pacemaker resources, failure history, XFS mount
usage, and service port checks (NFS 2049, iSCSI 3260) via VIP.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Even with both nodes in standby, Pacemaker's monitor operations keep running.
When the monitor sees DRBD is Primary on a standby node (that it didn't start),
it triggers a stop action — killing the initial sync after ~10 s.
Enable maintenance-mode after standby stops the existing resources but before
DRBD is manually brought up for the sync. Maintenance-mode suspends all
start/stop/monitor actions so Pacemaker is completely hands-off during the
sync. Disable it alongside crm_standby -v off once UpToDate/UpToDate is
confirmed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The metadata creation check used grep -q "UpToDate" which matches when the
resource is DOWN — drbdadm dstate reads metadata directly and returns just
"UpToDate" (no slash) for a down-but-previously-synced resource. This caused
metadata creation to be silently skipped, leaving mismatched UUIDs from an
interrupted sync which cause DRBD to go WFConnection→StandAlone immediately.
Fix: require exact "UpToDate/UpToDate" (with slash, only possible when the
resource is UP and both disks are confirmed synced) before skipping.
Also give the StandAlone detection a 5-iteration (15 s) grace period before
dying, to avoid false-positive on the first few ticks while the peer is still
coming up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clearing crm_standby before the initial sync finished caused Pacemaker's OCF
DRBD agent to race with the manual drbdadm up/primary calls. The agent saw
DRBD in WFConnection or SyncSource and tore it down, driving the resource back
to StandAlone and killing the sync in seconds.
Move the crm_standby -v off calls to immediately after the sync-complete
break, so Pacemaker only resumes once DRBD is UpToDate/UpToDate and safe
to hand back.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If the DRBD peer connection drops during the initial sync wait (cs:StandAlone),
the loop would spin forever printing "waiting for sync progress" with no
indication of what's wrong. Now parses cs: from /proc/drbd each tick and
dies with a clear error if StandAlone is detected. Also shows the cs: field
in the non-syncing message so the actual connection state is always visible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the DRBD kernel module invokes fence-peer via the User Mode Helper
mechanism it provides a minimal PATH that omits /run/current-system/sw/bin.
crm-fence-peer.sh calls cibadmin, crm_mon etc.; without those in PATH a
pipeline breaks with SIGPIPE. A signal-killed process has WEXITSTATUS()==0,
so the kernel sees exit code 0, logs "fence-peer helper broken, returned 0",
and retries forever — blocking Pacemaker failover completely.
Fix: use pkgs.writeShellScript to create thin PATH-fixing wrappers in the
Nix store. The wrappers prepend /run/current-system/sw/bin before exec-ing
the real crm-fence-peer.sh / crm-unfence-peer.sh, giving them a working
Pacemaker toolchain regardless of what PATH the kernel provides.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The TTY check ([[ -t 1 ]]) evaluated false under sudo, causing each update
to print on a new line via log(). Since \r worked correctly in the original
code (user confirmed), drop the branching and always use printf \r.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The sync progress grep was silently failing because /proc/drbd formats the
percentage line as "sync'ed: 5.2%" (two spaces after colon), but the
pattern matched only one. Use [[:space:]]+ throughout to tolerate any
whitespace.
Also make the display TTY-aware: interactive sessions get the spinning
\r in-place update; piped/logged sessions get a plain log line printed
once per unique percentage point (avoiding scroll spam).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cluster-init.sh: replace the fixed 300-iteration sync wait with an
indefinite loop that prints an in-place progress line (%done, ETA, speed)
from /proc/drbd every 3 s. Clears the line with printf \r before logging
completion, so the output stays clean alongside the [cluster-init] log
lines.
acceptance-tests.sh: add a pre-flight check that hard-exits if
drbdadm dstate is not UpToDate/UpToDate, with a hint to the watch
command for monitoring progress. Tests cannot give accurate results
while the initial full-sync is in progress.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
acceptance-tests.sh: replace drbdadm-role-based Active node detection with
crm_mon-based detection. DRBD role briefly reads Secondary during Pacemaker
transitions, causing T2/T3/T4 to target the wrong node. crm_mon Promoted:/
Masters: lines are authoritative; wait up to 90 s for Pacemaker to settle
before falling back to NODE1.
cluster-init.sh: add crm_resource --cleanup before the VIP wait loop so
stale migration-threshold failure counters from previous cluster-init runs
are cleared before Pacemaker decides resource placement.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/dev/sda and /dev/sdb are assigned by the OS based on Proxmox disk-add
order, which is not consistent across VMs. Use the SCSI controller path
instead — drive-scsi1 is always the dedicated data disk on all HA nodes
regardless of which sda/sdb it gets assigned to.
variables.nix: replace per-node haServer{1,2}DrbdDisk with a single
haServerDrbdDisk using /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1.
cluster-config.nix: revert to shared volume block (per-node block was
needed for /dev/sd* but the by-id path is identical on both nodes).
cluster-init.sh:
- single DRBD_DISK variable (matching haServerDrbdDisk)
- robust /etc/drbd.conf patch: NixOS manages this file as a symlink to
a read-only Nix store path; cp --remove-destination breaks the symlink
before sed -i so the edit actually takes effect
- scp helper script to NODE2 rather than bash -c over SSH to avoid
quoting complexity
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Proxmox VM disk ordering differs between the two HA nodes:
ha-server-1: sda=OS (50G), sdb=DRBD data (32G)
ha-server-2: sda=DRBD data (32G), sdb=OS (50G)
The DRBD resource config was using a shared disk=/dev/sdb which targeted
the OS disk on ha-server-2, causing drbdmeta and drbdadm up to operate
on the mounted root filesystem (hence "Device or resource busy").
Changes:
- variables.nix: add haServer1DrbdDisk/haServer2DrbdDisk
- cluster-config.nix: move volume block inside per-host on{} sections
so each node uses the correct backing disk
- cluster-init.sh: use NODE1_DRBD_DISK/NODE2_DRBD_DISK variables;
add runtime check that patches /etc/drbd.d/*.res on the running nodes
if the deployed config points to the wrong disk (workaround for VMs
built before this fix; redeploy with --force-rebuild to make permanent)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drbdmeta is in /run/current-system/sw/bin which is already on sudo's
PATH on the HA nodes; the bash -c wrapper with PATH manipulation was
adding complexity without benefit and introduced quoting hazards that
likely caused write-dev-uuid to fail or run in the wrong context.
Use direct n2_ssh calls instead — the same pattern that works when
tested manually on NODE2.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drbdmeta lives in the same Nix store dir as drbdadm but sudo doesn't
inherit the full PATH, so drbdmeta was not found (exit 127) even though
drbdadm was. Resolve drbdmeta's directory from drbdadm's location and
prepend it to PATH.
Replace openssl rand for UUID generation with /proc/sys/kernel/random/uuid
— openssl is not guaranteed to be on PATH in a minimal NixOS root
environment, but /proc/sys/kernel/random/uuid is always present.
Apply the same PATH fix on NODE2 inline in the bash -c invocations that
call drbdmeta over SSH.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cluster-init.sh: replace 'drbdadm create-md --force' with direct drbdmeta
calls using --force on both steps. drbdadm create-md --force passes --force
to the create-md sub-call but NOT to write-dev-uuid, which then fails when
/dev/sdb is busy and stdin is not a TTY ("stdin not a TTY, not waiting for
confirmation" → exit 20). write-dev-uuid failing means DRBD has no UUID,
so the peer can't identify the device → connection fails → no sync → wrong
Active node detected by acceptance tests.
acceptance-tests.sh T4: grep -c returns exit code 1 when the count is 0
(no matches), triggering '|| echo "0"' and producing "0\n0" which breaks
[[ "$IQN_COUNT" -ge 1 ]] with "arithmetic syntax error". Fixed by running
the pipe inside bash -c with '|| true' on the grep to suppress the
non-zero exit code. Same fix applied to T5's IQN_ON_STANDBY check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause of recurring sops failures on new VM boots: disko builds raw
disk images, and create-proxmox-resource.sh only syncs the clan-var SSH
host key to the Proxmox node (for proxmox.nix to bake into the image) when
it actually builds — reusing a cached image skips sync_remote_host_keys, so
destroy+recreate reuses a stale image with the wrong or randomly-generated
key baked in. On first boot the VM gets a different key than what .sops.yaml
was encrypted for, and sops fails permanently.
Fix 1 — deploy.sh Phase 3: always pass --force-rebuild so every VM creation
rebuilds the disko image fresh with the current clan-var key baked in via
NIXOS_HOST_KEYS_DIR (proxmox.nix already reads this under --impure).
Fix 2 — deploy.sh Phase 5.5: after VMs boot, scan their actual ed25519 host
keys and, if they drift from clan vars, update the clan var pub-key files,
rewrite the .sops.yaml age anchors, and re-encrypt all affected sops files.
Defence-in-depth: normally a no-op after Fix 1, but catches any residual
mismatch (e.g. --skip-create-vms reuse of an older image).
Fix 3 — cluster-init.sh: add crm_standby -v on for both nodes before DRBD
metadata init. Without this, Pacemaker's OCF DRBD agent races: it sees
drbdadm down as a failure and immediately calls drbdadm up again, leaving
/dev/sdb busy when create-md / write-dev-uuid runs (drbdmeta exits 20 with
"stdin not a TTY, not waiting for confirmation"). Standby suppresses
resource scheduling during init; crm_standby -v off restores it after DRBD
is up on both nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drbd.service runs drbdadm up all at activation time, but Pacemaker's
OCF drbd agent manages the resource lifecycle (up/down/promote/demote).
When both run simultaneously the device is busy, causing drbdmeta
apply-al to abort with exit 20.
Set wantedBy = [] so the kernel module and config are still present
(via services.drbd.enable = true) but systemd doesn't auto-start the
service. Pacemaker's OCF agent calls drbdadm directly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
At first boot, pacemaker fails with 'Dependency failed' because ipa-activation
fails (IPA not enrolled yet) before corosync/authkey are ready, causing a
systemd boot-ordering race. The service recovers fine when started manually.
Add an explicit 'systemctl start pacemaker' on both nodes immediately after
restarting corosync so cluster-init doesn't time out waiting for it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
The _patch_targetctl runtime dropin was a workaround for the wrong targetctl
binary path in the original iscsi-target.nix (used pkgs.targetcli-fb, but
targetctl is in rtslib-fb's python3 env). The module is now fixed to use
\${python3}/bin/targetctl, so new VM images have the correct ExecStart/ExecStop
baked in and the patch finds nothing — failing with exit 1 and killing the deploy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Adds nextcloud-client, chromium, git, vim, claude-code, fish, sops,
the Nextcloud tray service, SOPS_AGE_KEY_FILE, and Proxmox/PBS desktop
shortcuts to wayne's HM config on GUI hosts. Uses lib.mkDefault on the
EDITOR setting in ipa/client.nix so the GUI layer can override to vim.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both secrets/ha-server-{1,2}.keytab had unresolved conflict markers from
the stash/merge interaction. Keep the upstream version (2 age recipients:
admin + the host's own key) so each host can decrypt its own keytab on boot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
The HA branch merge left conflict markers in the ha-server-{1,2}.keytab
path_regex entries. Keep the upstream side (adds *proxmox-ha-server-{1,2}
as recipients) so each host can decrypt its own IPA keytab on boot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Deploy/init fixes:
- iscsi-target.nix: targetctl binary is in rtslib-fb (python3 env), not
targetcli-fb — fixes ExecStart and ExecStop for the targetctl.service
- deploy.sh: _patch_targetctl() applies runtime dropin to both nodes before
cluster-init so Pacemaker can manage the iSCSI target from first start
- cluster-init.sh: replace crm configure heredoc with cibadmin --replace XML
(pacemaker-4.0 schema: globally-unique in meta_attributes, promoted-max/
promoted-node-max, Promoted role in constraints); force_unmount=true on
xfs-data; DRBD promote timeout 240s
- cluster-config.nix: add crm-fence-peer.sh/crm-unfence-peer.sh handlers;
update fencing comment to reflect resource-only + Pacemaker-aware handler
replacing STONITH during testing phase
- ha-server.nix: add openiscsi to systemPackages for T4 iscsiadm availability
Acceptance test fixes:
- acceptance-tests.sh: fix ((PASS++)) set -e bug → PASS=$((PASS+1));
detect Active/Standby dynamically via drbdadm role (Pacemaker can promote
either node); T4 bash TCP probe instead of iscsiadm; T5 timeout 120s;
T6 echo|sudo tee for root-owned XFS write (bash -c redirect runs as nixos
not sudo — permission denied); use ns cat / ns rm for root-owned reads
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
beatzaplenty
