Archived
beatzaplentyandClaude Sonnet 4.6
724d9a45af
All 7 acceptance tests pass on live NixOS 25.11 VMs (VMIDs 200/201 on
pve1). Failover completes in ~5 s with data integrity verified.
modules/ha/pacemaker-stack.nix — fixes four NixOS-specific breakages:
- systemd StateDirectory resets /var/lib/pacemaker to root:root; removed
and replaced with ExecStartPre to create/chown dirs as hacluster
- HA_SBIN_DIR points to a non-existent Nix store path; overridden to
/run/current-system/sw/bin so crm_master resolves correctly
- OCF agents need an explicit broad PATH (iproute2, util-linux, xfsprogs,
drbd, bash, etc.) — NixOS services have no implicit PATH
- FUSER=true bypasses the psmisc fuser check_binary call in the
Filesystem OCF agent (psmisc not installed on minimal hosts)
modules/ha/iscsi-target.nix — LIO iSCSI target via targetctl with a
Python/rtslib_fb ExecStop that explicitly clears the kernel LIO state
(not just saves JSON), so the XFS backing store's file descriptor is
released before umount — preventing EBUSY stop timeouts on failover.
Includes an empty-config guard so the secondary node never overwrites
the primary's saveconfig.json with an empty one.
test-lab/ha/common.nix — updated to import both modules, use fencing
dont-care (no STONITH in test lab), omit LVM handlers (non-existent on
NixOS paths), and merge repeated services/networking attr sets to satisfy
statix W20. test-lab/ha/acceptance-tests.sh — final v4 with crm_standby
fix (pacemaker 3.x API).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
95 lines
3.3 KiB
Nix
95 lines
3.3 KiB
Nix
# Pacemaker + Corosync HA stack for NixOS with known-good workarounds.
|
|
#
|
|
# Issues fixed here (confirmed through live testing on NixOS 25.11):
|
|
#
|
|
# 1. StateDirectory ownership reset: systemd's StateDirectory=pacemaker
|
|
# creates /var/lib/pacemaker owned root:root. pacemaker-based (the CIB
|
|
# daemon) runs as the hacluster user and calls pcmk__daemon_can_write,
|
|
# which requires the CIB directory to be owned by hacluster or be
|
|
# group-writable by haclient. Workaround: remove StateDirectory and let
|
|
# ExecStartPre create every required subdirectory with correct ownership.
|
|
#
|
|
# 2. HA_SBIN_DIR wrong path: ocf-shellfuncs sets HA_SBIN_DIR to the Nix
|
|
# store path of the resource-agents derivation's /sbin, which doesn't
|
|
# exist. The DRBD OCF agent uses ${HA_SBIN_DIR}/crm_master, so it exits
|
|
# 127 without this override. Fix: export HA_SBIN_DIR=/run/current-system/sw/bin.
|
|
#
|
|
# 3. Broad PATH for OCF agents: the resource executor (pacemaker-execd) runs
|
|
# OCF agent scripts as children. NixOS provides no implicit PATH for
|
|
# system services; without an explicit PATH the agents can't find ip, ss,
|
|
# mount, umount, drbdadm, etc.
|
|
#
|
|
# 4. FUSER=true: the Filesystem OCF agent calls check_binary $FUSER (default:
|
|
# fuser from psmisc), which is not installed. Setting FUSER=true makes
|
|
# check_binary succeed (true is always in PATH) and the subsequent
|
|
# "$FUSER -km $mountpoint" becomes a no-op. Pair with force_unmount=false
|
|
# on each Filesystem resource unless you want lazy unmount behaviour.
|
|
{ lib, pkgs, ... }:
|
|
|
|
let
|
|
ocfBinPath = lib.concatStringsSep ":" [
|
|
"${pkgs.iproute2}/bin"
|
|
"${pkgs.iproute2}/sbin"
|
|
"${pkgs.iputils}/bin"
|
|
"${pkgs.util-linux}/bin"
|
|
"${pkgs.util-linux}/sbin"
|
|
"${pkgs.gawk}/bin"
|
|
"${pkgs.gnugrep}/bin"
|
|
"${pkgs.gnused}/bin"
|
|
"${pkgs.coreutils}/bin"
|
|
"${pkgs.bash}/bin"
|
|
"${pkgs.procps}/bin"
|
|
"${pkgs.xfsprogs}/bin"
|
|
"${pkgs.drbd}/bin"
|
|
"${pkgs.python3}/bin"
|
|
"/run/current-system/sw/bin"
|
|
"/run/current-system/sw/sbin"
|
|
"/usr/local/sbin"
|
|
"/usr/local/bin"
|
|
"/usr/sbin"
|
|
"/usr/bin"
|
|
"/sbin"
|
|
"/bin"
|
|
];
|
|
|
|
# Single pre-start script: schemas symlink + directory ownership.
|
|
# Runs before pacemakerd so pacemaker-based finds hacluster-owned dirs.
|
|
preStartCmd = "${pkgs.bash}/bin/bash -c '"
|
|
+ "ln -sfn ${pkgs.pacemaker}/share/pacemaker /var/lib/pacemaker/schemas; "
|
|
+ "for d in /var/lib/pacemaker /var/lib/pacemaker/cib /var/lib/pacemaker/cores "
|
|
+ "/var/lib/pacemaker/pengine /var/lib/pacemaker/blackbox "
|
|
+ "/var/lib/pacemaker/hostcache; do "
|
|
+ "mkdir -p \"\\$d\" && chown hacluster:pacemaker \"\\$d\" && chmod 2770 \"\\$d\"; "
|
|
+ "done'";
|
|
|
|
ocfEnv = {
|
|
PATH = lib.mkForce ocfBinPath;
|
|
OCF_ROOT = "${pkgs.ocf-resource-agents}/usr/lib/ocf";
|
|
HA_SBIN_DIR = "/run/current-system/sw/bin";
|
|
FUSER = "true";
|
|
};
|
|
in
|
|
{
|
|
users.groups.haclient = { };
|
|
|
|
services.corosync.enable = true;
|
|
services.pacemaker.enable = true;
|
|
|
|
systemd.services = {
|
|
pacemaker = {
|
|
serviceConfig = {
|
|
StateDirectory = lib.mkForce "";
|
|
ExecStartPre = lib.mkBefore [ preStartCmd ];
|
|
};
|
|
environment = ocfEnv;
|
|
};
|
|
pacemaker-execd.environment = ocfEnv;
|
|
};
|
|
|
|
environment.systemPackages = with pkgs; [
|
|
corosync
|
|
pacemaker
|
|
ocf-resource-agents
|
|
];
|
|
}
|