When the DRBD kernel module invokes fence-peer via the User Mode Helper
mechanism it provides a minimal PATH that omits /run/current-system/sw/bin.
crm-fence-peer.sh calls cibadmin, crm_mon etc.; without those in PATH a
pipeline breaks with SIGPIPE. A signal-killed process has WEXITSTATUS()==0,
so the kernel sees exit code 0, logs "fence-peer helper broken, returned 0",
and retries forever — blocking Pacemaker failover completely.
Fix: use pkgs.writeShellScript to create thin PATH-fixing wrappers in the
Nix store. The wrappers prepend /run/current-system/sw/bin before exec-ing
the real crm-fence-peer.sh / crm-unfence-peer.sh, giving them a working
Pacemaker toolchain regardless of what PATH the kernel provides.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/dev/sda and /dev/sdb are assigned by the OS based on Proxmox disk-add
order, which is not consistent across VMs. Use the SCSI controller path
instead — drive-scsi1 is always the dedicated data disk on all HA nodes
regardless of which sda/sdb it gets assigned to.
variables.nix: replace per-node haServer{1,2}DrbdDisk with a single
haServerDrbdDisk using /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1.
cluster-config.nix: revert to shared volume block (per-node block was
needed for /dev/sd* but the by-id path is identical on both nodes).
cluster-init.sh:
- single DRBD_DISK variable (matching haServerDrbdDisk)
- robust /etc/drbd.conf patch: NixOS manages this file as a symlink to
a read-only Nix store path; cp --remove-destination breaks the symlink
before sed -i so the edit actually takes effect
- scp helper script to NODE2 rather than bash -c over SSH to avoid
quoting complexity
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Proxmox VM disk ordering differs between the two HA nodes:
ha-server-1: sda=OS (50G), sdb=DRBD data (32G)
ha-server-2: sda=DRBD data (32G), sdb=OS (50G)
The DRBD resource config was using a shared disk=/dev/sdb which targeted
the OS disk on ha-server-2, causing drbdmeta and drbdadm up to operate
on the mounted root filesystem (hence "Device or resource busy").
Changes:
- variables.nix: add haServer1DrbdDisk/haServer2DrbdDisk
- cluster-config.nix: move volume block inside per-host on{} sections
so each node uses the correct backing disk
- cluster-init.sh: use NODE1_DRBD_DISK/NODE2_DRBD_DISK variables;
add runtime check that patches /etc/drbd.d/*.res on the running nodes
if the deployed config points to the wrong disk (workaround for VMs
built before this fix; redeploy with --force-rebuild to make permanent)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drbd.service runs drbdadm up all at activation time, but Pacemaker's
OCF drbd agent manages the resource lifecycle (up/down/promote/demote).
When both run simultaneously the device is busy, causing drbdmeta
apply-al to abort with exit 20.
Set wantedBy = [] so the kernel module and config are still present
(via services.drbd.enable = true) but systemd doesn't auto-start the
service. Pacemaker's OCF agent calls drbdadm directly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Deploy/init fixes:
- iscsi-target.nix: targetctl binary is in rtslib-fb (python3 env), not
targetcli-fb — fixes ExecStart and ExecStop for the targetctl.service
- deploy.sh: _patch_targetctl() applies runtime dropin to both nodes before
cluster-init so Pacemaker can manage the iSCSI target from first start
- cluster-init.sh: replace crm configure heredoc with cibadmin --replace XML
(pacemaker-4.0 schema: globally-unique in meta_attributes, promoted-max/
promoted-node-max, Promoted role in constraints); force_unmount=true on
xfs-data; DRBD promote timeout 240s
- cluster-config.nix: add crm-fence-peer.sh/crm-unfence-peer.sh handlers;
update fencing comment to reflect resource-only + Pacemaker-aware handler
replacing STONITH during testing phase
- ha-server.nix: add openiscsi to systemPackages for T4 iscsiadm availability
Acceptance test fixes:
- acceptance-tests.sh: fix ((PASS++)) set -e bug → PASS=$((PASS+1));
detect Active/Standby dynamically via drbdadm role (Pacemaker can promote
either node); T4 bash TCP probe instead of iscsiadm; T5 timeout 120s;
T6 echo|sudo tee for root-owned XFS write (bash -c redirect runs as nixos
not sudo — permission denied); use ns cat / ns rm for root-owned reads
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- ha-server.nix: add xfsprogs to systemPackages so mkfs.xfs is on PATH for
root (needed by cluster-init.sh during initial setup)
- cluster-config.nix: create /var/lib/drbd via tmpfiles to silence
lk_bdev_save warnings from drbd-utils
- cluster-init.sh: dynamically find xfsprogs in /nix/store if not on PATH
(fallback for running VMs before xfsprogs is in the system profile)
- cluster-init.sh: fix DRBD metadata check on node2 — broken regex now uses
grep -E for ERE alternation to correctly skip create-md when DRBD is already
set up (previous regex would have triggered create-md on a live secondary)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Root SSH was failing because only the RSA admin key was authorized but the
local dev box only has an ed25519 key. Fix:
- cluster-config.nix: add ed25519 keys to root (same set as nixos user) so
future deployments work without the temp-key workaround
- deploy.sh/acceptance-tests.sh: SSH as nixos user with sudo instead of root@
- cluster-init.sh: HA_USER/HA_KEY env vars + n2_ssh()/n2_scp() helpers so
inter-node SSH works regardless of whether root-to-root is available
- deploy.sh Phase 6: generate temp keypair, authorize on node2, place on node1
for root to use during cluster-init, clean up afterward
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Add root SSH key + passwordless sudo to ha-server nodes (needed for
deploy script to run cluster-init.sh via SSH as root on node1)
- Fix cluster-init.sh: correct default IPs (228/227/229 per variables.nix),
use \${VAR:-default} for all config so deploy.sh can override via env
- Fix acceptance-tests.sh: same IP corrections, add -i flag to SSH calls,
use \${VAR:-default} pattern
- Fix deploy.sh dry-run bugs: pve_check() always runs SSH for read-only
probes so bridge existence check is accurate; wait_for_ssh skips in
dry-run instead of timing out
- Fix cluster-init invocation: upload script via scp and run via SSH as
root on node1 (was incorrectly trying to run the script locally)
- Fix acceptance-tests invocation: pass IP env vars from deploy.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Adds proxmox-ha-server-1 and proxmox-ha-server-2 as real mkTarget entries
alongside the existing proxmox-server, backed by a new ha-server build type.
New modules
modules/ha/cluster-config.nix — DRBD resource + corosync nodelist sourced
from vars (haServer1Host/Ip, haServer2Host/Ip); resource-only fencing for
production STONITH; HA port firewall rules for DRBD, iSCSI, Corosync, pcsd
modules/build-types/ha-server.nix — imports pacemaker-stack + iscsi-target
+ cluster-config + beszel; NFS exports from vars.haStorageRoot (XFS-over-DRBD
mount); nfs-server.service.wantedBy force-cleared so Pacemaker controls
start/stop on the Active node only
New hosts
hosts/ha-server-{1,2}/host.nix — static IP from vars, unique hostId; sops
secrets (beszel, corosync authkey) are TODOs pending sync-host-keys.sh
variables.nix
haServer1/2Host, haServer1/2Ip, haServerVip, haStorageRoot, haIscsiIqn
ports.haServerDrbd/Iscsi/Corosync{1,2,Crypto}/PacemakerRemoted/Pcsd
scripts/ha/ (migrated + updated from test-lab/ha/)
cluster-init.sh — generates corosync authkey, initialises DRBD/XFS/iSCSI,
creates NFS dataset dirs, configures Pacemaker with DRBD + XFS + iSCSI
+ nfs-server + VIP; STONITH disabled initially (enable separately)
cluster-enable-stonith.sh — enables fence_pve_ssh STONITH after key deploy
fence-pve-ssh.py — Proxmox SSH fence agent (node names updated to ha-server-1/2)
acceptance-tests.sh — T1–T7 production acceptance tests
test-lab/ha/ removed — all Nix config moved to modules/ha/ and
modules/build-types/; scripts moved to scripts/ha/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
All 7 acceptance tests pass on live NixOS 25.11 VMs (VMIDs 200/201 on
pve1). Failover completes in ~5 s with data integrity verified.
modules/ha/pacemaker-stack.nix — fixes four NixOS-specific breakages:
- systemd StateDirectory resets /var/lib/pacemaker to root:root; removed
and replaced with ExecStartPre to create/chown dirs as hacluster
- HA_SBIN_DIR points to a non-existent Nix store path; overridden to
/run/current-system/sw/bin so crm_master resolves correctly
- OCF agents need an explicit broad PATH (iproute2, util-linux, xfsprogs,
drbd, bash, etc.) — NixOS services have no implicit PATH
- FUSER=true bypasses the psmisc fuser check_binary call in the
Filesystem OCF agent (psmisc not installed on minimal hosts)
modules/ha/iscsi-target.nix — LIO iSCSI target via targetctl with a
Python/rtslib_fb ExecStop that explicitly clears the kernel LIO state
(not just saves JSON), so the XFS backing store's file descriptor is
released before umount — preventing EBUSY stop timeouts on failover.
Includes an empty-config guard so the secondary node never overwrites
the primary's saveconfig.json with an empty one.
test-lab/ha/common.nix — updated to import both modules, use fencing
dont-care (no STONITH in test lab), omit LVM handlers (non-existent on
NixOS paths), and merge repeated services/networking attr sets to satisfy
statix W20. test-lab/ha/acceptance-tests.sh — final v4 with crm_standby
fix (pacemaker 3.x API).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
beatzaplentyandClaude Sonnet 4.6