nixos/docs/ha.md

HA File-Server Cluster

Two proxmox-ha-server-{1,2} VMs form an active/passive file-server cluster: DRBD replicates a block device between nodes; Corosync + Pacemaker manage failover; XFS, LIO iSCSI, and NFS are brought up as a collocated resource group on whichever node holds the DRBD Primary role.

NixOS modules: modules/ha/. Lifecycle scripts: scripts/ha/. Cluster-wide constants: variables.nix (haServer* vars).

---

Network layout

Three subnets — all internal to pve1 (vmbr0/vmbr1/vmbr2):

Subnet	VLAN	CIDR	Bridge	Purpose
LAN	2	192.168.2.0/24	vmbr0	Management, LAN NFS
Cluster	10	192.168.10.224/29	vmbr1	Corosync ring0 + DRBD replication
Storage-client	20	192.168.20.0/24	vmbr2	NFS + iSCSI for docker/swarm

Each HA VM has three NICs: ens18 (LAN/vmbr0), ens19 (cluster/vmbr1), ens20 (storage-client/vmbr2). See docs/ip-addressing.md for all IPs.

Corosync ring0 uses the cluster NIC; ring1 (backup heartbeat) uses the LAN NIC. DRBD replicates over the cluster NIC. No storage traffic crosses the LAN.

---

Pacemaker resources

All resources run collocated on whichever node is Primary, in this order:

ms-drbd0 (promotable DRBD clone)
  → xfs-data   (XFS mount on /dev/drbd0 → /srv/ha-data)
  → iscsi-target (targetctl)
  → nfs-server  (nfs-server.service)
  → vip-lan     (192.168.2.229/24 on vmbr0 — NFS for LAN clients)
  → vip-storage (192.168.20.229/24 on vmbr2 — NFS + iSCSI for VLAN 20)

vip-lan serves pxe-boot and other LAN-only NFS clients. vip-storage serves docker and any future swarm nodes; iSCSI is available on VLAN 20 but NFS is preferred for multi-host volume sharing.

---

DRBD fencing

fencing resource-only with crm-fence-peer.sh/crm-unfence-peer.sh wrappers (modules/ha/cluster-config.nix). The DRBD kernel module invokes these via the User Mode Helper with a minimal PATH; the wrappers prepend /run/current-system/sw/bin before exec-ing the real handlers so Pacemaker tools (cibadmin, crm_mon, etc.) are found.

STONITH is initially disabled (stonith-enabled: false, no-quorum-policy: ignore). Enable it once the fence_pve_ssh fence agent (scripts/ha/fence-pve-ssh.py) is deployed and authorised:

scripts/ha/cluster-enable-stonith.sh   # run as root on ha-server-1

---

Deploying the cluster from scratch

Use scripts/ha/deploy.sh — it orchestrates all phases:

# Against pve-test (safe — Claude's default target):
scripts/ha/deploy.sh --node "$PVE_TEST_HOST" [--dry-run]

# Against pve1 (production — requires explicit operator go-ahead):
scripts/ha/deploy.sh --node "$PVE1_HOST"

Phases (each skippable with --skip-<phase>):

1. ensure-bridge — creates vmbr1/vmbr2 on the Proxmox node if absent
2. sync-keys — generates SSH host keys for both nodes; registers sops recipients
3. create-vms — builds disk images, creates VMs via create-proxmox-resource.sh
4. add-hardware — attaches storage NIC and DRBD data disk to each VM
5. init-cluster — runs scripts/ha/cluster-init.sh on ha-server-1

--destroy runs the teardown sequence.

---

Day-to-day operations

# Read-only health check (safe from workstation):
scripts/ha/health.sh

# Graceful failover (prompts for confirmation):
scripts/ha/failover.sh [--to node1|node2]

# Online data-disk growth (no downtime):
scripts/ha/resize-data-disk.sh --size +20G

# Acceptance tests (run after any significant change):
scripts/ha/acceptance-tests.sh

---

Adding FreeIPA host accounts

IPA host registration is automated:

scripts/ipa/create-nixos-ipa-host-account.sh <hostname>

This runs ipa host-add, fetches a keytab from the domain controller, and writes a sops-encrypted secrets/<hostname>.keytab in one step. The module modules/ipa/client.nix (imported by every host via modules/common/configuration.nix) consumes the keytab via sops-nix.

---

Storage layout

/srv/ha-data/
  docker/
    config/          NFS → docker:/mnt/docker/config
    databases/       NFS → docker:/mnt/docker/databases
    volumes/         NFS → docker:/mnt/docker/volumes
    nextcloud-data/  NFS → docker:/mnt/docker/nextcloud-data
  proxmox/
    iso/             NFS → pve1 ISO storage
    lxc/             NFS → pve1 CT template storage
  pxe-boot/
    images/          NFS → pxe-boot:/srv/pxe/http/images (PXE assets)
  raspi/
    volumes/         NFS → raspi NFS mounts
  iscsi-lun.img      iSCSI fileio backstore (VLAN 20 only, not in active use)

All shares are defined in variables.nix (vars.nfsShares.*). The NFS export list lives in modules/ha/nfs-exports.nix.

---

Key variables

Variable	Description
vars.haServer1Ip / vars.haServer2Ip	LAN management IPs
vars.haServer1StorageIp / vars.haServer2StorageIp	Cluster NIC IPs (DRBD/Corosync ring0)
vars.haServerLanVip	Pacemaker vip-lan — NFS for LAN (192.168.2.229)
vars.haServerVip	Pacemaker vip-storage — NFS + iSCSI for VLAN 20 (192.168.20.229)
vars.haLanNfsFqdn	FQDN of vip-lan: ha-vip-lan.sweet.home
vars.haStorageRoot	XFS mount point: /srv/ha-data
vars.haServerDrbdDisk	Block device for DRBD backing store
vars.haStorageCidr	Cluster subnet CIDR (192.168.10.224/29)
vars.haClientCidr	Storage-client subnet CIDR (192.168.20.0/24)