Worktree ha file server test #93

2026-07-28T09:08:58Z

beatzaplenty commented

2026-07-28 09:08:58 +00:00

No description provided.

beatzaplenty added 12 commits 2026-07-28 09:08:59 +00:00

feat(ha): pre-stage IPA enrollment for ha-server-1 and ha-server-2 dc83333526

- Add stub keytab files (secrets/ha-server-{1,2}.keytab) so modules/ipa/client.nix
  activates for both nodes — IPA security.ipa.enable = true confirmed by eval
- Add .sops.yaml creation rules for both keytabs (admin-only until
  sync-host-keys.sh registers the host age keys)

Eval: both proxmox-ha-server-1 and proxmox-ha-server-2 build cleanly
with IPA enabled, beszel agent enabled, corosync authkey + beszel-token
+ ipa-host-keytab all wired via sops-nix.

Bootstrap order before first deploy:
  1. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
  2. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
  3. sops updatekeys secrets/common.yaml
  4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
  5. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
  6. sops secrets/ha-server-1.yaml  (set beszel-token)
  7. sops secrets/ha-server-2.yaml  (set beszel-token)
  8. On node1 after first boot: corosync-keygen, then
     sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey
  9. Set services.beszel.agent.environment.KEY in each host.nix after hub pairing
 10. git add/commit the real secrets, then nixos-rebuild switch on both nodes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

feat(ha): implement clan vars and fully encrypt all HA secrets 59854a0229

- Run sync-host-keys.sh for proxmox-ha-server-{1,2}: generates SSH host
  key pairs in vars/per-machine/, registers age anchors in .sops.yaml,
  adds both hosts as recipients for common.yaml, ha-corosync-authkey,
  and per-host secrets/keytab files
- Re-encrypt secrets/common.yaml with both new host keys
- Convert all stub secrets to real sops-encrypted files:
    secrets/ha-server-{1,2}.yaml    (YAML, beszel-token = PLACEHOLDER)
    secrets/ha-server-{1,2}.keytab  (binary, stub text encrypted)
    secrets/ha-corosync-authkey      (binary, stub text encrypted)
- Add scripts/ha/deploy.sh: full lifecycle script (bridge setup, VM
  creation, DRBD disk + storage NIC attachment, boot wait, cluster-init,
  acceptance tests, --destroy)

Bootstrap order (operator runs these before first deploy):
  1. bash scripts/ha/deploy.sh            # deploys, tests
  # Post-deploy secret replacement:
  2. sops secrets/ha-server-{1,2}.yaml   (set real beszel-token)
  3. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
  4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
  5. Set services.beszel.agent.environment.KEY in host.nix after hub pairing
  6. nixos-rebuild switch on both nodes to pick up real secrets

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

fix(ha): fix deploy script bugs and correct cluster configuration b76d54e702

- Add root SSH key + passwordless sudo to ha-server nodes (needed for
  deploy script to run cluster-init.sh via SSH as root on node1)
- Fix cluster-init.sh: correct default IPs (228/227/229 per variables.nix),
  use \${VAR:-default} for all config so deploy.sh can override via env
- Fix acceptance-tests.sh: same IP corrections, add -i flag to SSH calls,
  use \${VAR:-default} pattern
- Fix deploy.sh dry-run bugs: pve_check() always runs SSH for read-only
  probes so bridge existence check is accurate; wait_for_ssh skips in
  dry-run instead of timing out
- Fix cluster-init invocation: upload script via scp and run via SSH as
  root on node1 (was incorrectly trying to run the script locally)
- Fix acceptance-tests invocation: pass IP env vars from deploy.sh

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

fix(ha/deploy): fix /nix ownership on pve1 before codex-setup a8d8b1465c

pve1 has a pre-existing /nix store owned by a different UID; wayne's IPA
UID (50002) can't write to it.  Add a pre-phase-3 check that uses wayne's
passwordless sudo to chown -R the store before create-proxmox-resource.sh
runs codex-setup.sh on the node.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

fix(ha/deploy): ensure remote clone is on correct branch before building 1263c7540c

When create-proxmox-resource.sh clones the repo to pve1, it stays on
whatever branch was checked out. Add a pre-build phase that detects
branch mismatch and switches the remote clone to the current local
branch before building, so the Proxmox node always builds from the
same commits we're deploying.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

fix(ha/deploy): use non-sudo SSH for /nix writability check 3f9b968a41

pve_check() always uses sudo, so "sudo test -w /nix" passes as root
regardless of whether the SSH user can actually write there.  Use a
direct non-sudo SSH command for the writability probe so the check
reflects wayne's own access, not root's.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

fix(ha/deploy): use nixos+sudo instead of root SSH; temp key for inter-node comms acebbdbe26

Root SSH was failing because only the RSA admin key was authorized but the
local dev box only has an ed25519 key. Fix:

- cluster-config.nix: add ed25519 keys to root (same set as nixos user) so
  future deployments work without the temp-key workaround
- deploy.sh/acceptance-tests.sh: SSH as nixos user with sudo instead of root@
- cluster-init.sh: HA_USER/HA_KEY env vars + n2_ssh()/n2_scp() helpers so
  inter-node SSH works regardless of whether root-to-root is available
- deploy.sh Phase 6: generate temp keypair, authorize on node2, place on node1
  for root to use during cluster-init, clean up afterward

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

fix(ha): add xfsprogs to system packages; fix cluster-init PATH and DRBD check c76698efb7

- ha-server.nix: add xfsprogs to systemPackages so mkfs.xfs is on PATH for
  root (needed by cluster-init.sh during initial setup)
- cluster-config.nix: create /var/lib/drbd via tmpfiles to silence
  lk_bdev_save warnings from drbd-utils
- cluster-init.sh: dynamically find xfsprogs in /nix/store if not on PATH
  (fallback for running VMs before xfsprogs is in the system profile)
- cluster-init.sh: fix DRBD metadata check on node2 — broken regex now uses
  grep -E for ERE alternation to correctly skip create-md when DRBD is already
  set up (previous regex would have triggered create-md on a live secondary)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

fix(ha/cluster-init): fix LIO teardown, mount idempotency, VIP portal d31d9fa584

- Remove VIP-specific portal binding — the VIP doesn't exist until Pacemaker
  assigns it; the default all-IPs portal (::0:3260) is correct for Pacemaker
  to manage
- Clear existing LIO targets before re-running targetcli (idempotent on
  partial failures)
- Tear down LIO kernel objects after saveconfig so umount succeeds (LIO holds
  the backing file open otherwise)
- Guard mount with mountpoint check so re-runs don't fail when already mounted
- Use --replace for cibadmin constraints (idempotent vs --create)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

fix(ha/cluster-init): use targetcli delete commands for LIO teardown 79cde50e27

clearconfig does not reliably clear kernel LIO configfs state — the kernel
still holds backing-file references, blocking umount.  Replace clearconfig
with explicit targetcli delete commands (/iscsi delete, /backstores/fileio
delete) which do release kernel state.  Also make DRBD secondary demote
idempotent (skip if already Secondary).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

fix(ha/cluster-init): use crm configure instead of cibadmin XML for resources da4d808c6a

cibadmin raw XML fails schema validation under pacemaker-4.0: globally-unique
is not a valid direct <clone> attribute, and master-max/master-node-max are
renamed.  Switch to crm configure commands which are schema-version-aware:
- promotable clone with promoted-max/promoted-node-max
- order/colocation constraints using Promoted role (Pacemaker 4.0 naming)
- crm configure handles schema differences automatically

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

fix(ha): all 7 acceptance tests pass — targetctl, fencing, failover, data integrity

Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s

Details

d740064a35

Deploy/init fixes:
- iscsi-target.nix: targetctl binary is in rtslib-fb (python3 env), not
  targetcli-fb — fixes ExecStart and ExecStop for the targetctl.service
- deploy.sh: _patch_targetctl() applies runtime dropin to both nodes before
  cluster-init so Pacemaker can manage the iSCSI target from first start
- cluster-init.sh: replace crm configure heredoc with cibadmin --replace XML
  (pacemaker-4.0 schema: globally-unique in meta_attributes, promoted-max/
  promoted-node-max, Promoted role in constraints); force_unmount=true on
  xfs-data; DRBD promote timeout 240s
- cluster-config.nix: add crm-fence-peer.sh/crm-unfence-peer.sh handlers;
  update fencing comment to reflect resource-only + Pacemaker-aware handler
  replacing STONITH during testing phase
- ha-server.nix: add openiscsi to systemPackages for T4 iscsiadm availability

Acceptance test fixes:
- acceptance-tests.sh: fix ((PASS++)) set -e bug → PASS=$((PASS+1));
  detect Active/Standby dynamically via drbdadm role (Pacemaker can promote
  either node); T4 bash TCP probe instead of iscsiadm; T5 timeout 120s;
  T6 echo|sudo tee for root-owned XFS write (bash -c redirect runs as nixos
  not sudo — permission denied); use ns cat / ns rm for root-owned reads

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8

beatzaplenty merged commit 8194707478 into main

2026-07-28 09:09:05 +00:00

This repo is archived. You cannot comment on pull requests.

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: beatzaplenty/nixos#93