Root cause of recurring sops failures on new VM boots: disko builds raw
disk images, and create-proxmox-resource.sh only syncs the clan-var SSH
host key to the Proxmox node (for proxmox.nix to bake into the image) when
it actually builds — reusing a cached image skips sync_remote_host_keys, so
destroy+recreate reuses a stale image with the wrong or randomly-generated
key baked in. On first boot the VM gets a different key than what .sops.yaml
was encrypted for, and sops fails permanently.
Fix 1 — deploy.sh Phase 3: always pass --force-rebuild so every VM creation
rebuilds the disko image fresh with the current clan-var key baked in via
NIXOS_HOST_KEYS_DIR (proxmox.nix already reads this under --impure).
Fix 2 — deploy.sh Phase 5.5: after VMs boot, scan their actual ed25519 host
keys and, if they drift from clan vars, update the clan var pub-key files,
rewrite the .sops.yaml age anchors, and re-encrypt all affected sops files.
Defence-in-depth: normally a no-op after Fix 1, but catches any residual
mismatch (e.g. --skip-create-vms reuse of an older image).
Fix 3 — cluster-init.sh: add crm_standby -v on for both nodes before DRBD
metadata init. Without this, Pacemaker's OCF DRBD agent races: it sees
drbdadm down as a failure and immediately calls drbdadm up again, leaving
/dev/sdb busy when create-md / write-dev-uuid runs (drbdmeta exits 20 with
"stdin not a TTY, not waiting for confirmation"). Standby suppresses
resource scheduling during init; crm_standby -v off restores it after DRBD
is up on both nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The _patch_targetctl runtime dropin was a workaround for the wrong targetctl
binary path in the original iscsi-target.nix (used pkgs.targetcli-fb, but
targetctl is in rtslib-fb's python3 env). The module is now fixed to use
\${python3}/bin/targetctl, so new VM images have the correct ExecStart/ExecStop
baked in and the patch finds nothing — failing with exit 1 and killing the deploy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Deploy/init fixes:
- iscsi-target.nix: targetctl binary is in rtslib-fb (python3 env), not
targetcli-fb — fixes ExecStart and ExecStop for the targetctl.service
- deploy.sh: _patch_targetctl() applies runtime dropin to both nodes before
cluster-init so Pacemaker can manage the iSCSI target from first start
- cluster-init.sh: replace crm configure heredoc with cibadmin --replace XML
(pacemaker-4.0 schema: globally-unique in meta_attributes, promoted-max/
promoted-node-max, Promoted role in constraints); force_unmount=true on
xfs-data; DRBD promote timeout 240s
- cluster-config.nix: add crm-fence-peer.sh/crm-unfence-peer.sh handlers;
update fencing comment to reflect resource-only + Pacemaker-aware handler
replacing STONITH during testing phase
- ha-server.nix: add openiscsi to systemPackages for T4 iscsiadm availability
Acceptance test fixes:
- acceptance-tests.sh: fix ((PASS++)) set -e bug → PASS=$((PASS+1));
detect Active/Standby dynamically via drbdadm role (Pacemaker can promote
either node); T4 bash TCP probe instead of iscsiadm; T5 timeout 120s;
T6 echo|sudo tee for root-owned XFS write (bash -c redirect runs as nixos
not sudo — permission denied); use ns cat / ns rm for root-owned reads
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Root SSH was failing because only the RSA admin key was authorized but the
local dev box only has an ed25519 key. Fix:
- cluster-config.nix: add ed25519 keys to root (same set as nixos user) so
future deployments work without the temp-key workaround
- deploy.sh/acceptance-tests.sh: SSH as nixos user with sudo instead of root@
- cluster-init.sh: HA_USER/HA_KEY env vars + n2_ssh()/n2_scp() helpers so
inter-node SSH works regardless of whether root-to-root is available
- deploy.sh Phase 6: generate temp keypair, authorize on node2, place on node1
for root to use during cluster-init, clean up afterward
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
pve_check() always uses sudo, so "sudo test -w /nix" passes as root
regardless of whether the SSH user can actually write there. Use a
direct non-sudo SSH command for the writability probe so the check
reflects wayne's own access, not root's.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
When create-proxmox-resource.sh clones the repo to pve1, it stays on
whatever branch was checked out. Add a pre-build phase that detects
branch mismatch and switches the remote clone to the current local
branch before building, so the Proxmox node always builds from the
same commits we're deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
pve1 has a pre-existing /nix store owned by a different UID; wayne's IPA
UID (50002) can't write to it. Add a pre-phase-3 check that uses wayne's
passwordless sudo to chown -R the store before create-proxmox-resource.sh
runs codex-setup.sh on the node.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Add root SSH key + passwordless sudo to ha-server nodes (needed for
deploy script to run cluster-init.sh via SSH as root on node1)
- Fix cluster-init.sh: correct default IPs (228/227/229 per variables.nix),
use \${VAR:-default} for all config so deploy.sh can override via env
- Fix acceptance-tests.sh: same IP corrections, add -i flag to SSH calls,
use \${VAR:-default} pattern
- Fix deploy.sh dry-run bugs: pve_check() always runs SSH for read-only
probes so bridge existence check is accurate; wait_for_ssh skips in
dry-run instead of timing out
- Fix cluster-init invocation: upload script via scp and run via SSH as
root on node1 (was incorrectly trying to run the script locally)
- Fix acceptance-tests invocation: pass IP env vars from deploy.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Run sync-host-keys.sh for proxmox-ha-server-{1,2}: generates SSH host
key pairs in vars/per-machine/, registers age anchors in .sops.yaml,
adds both hosts as recipients for common.yaml, ha-corosync-authkey,
and per-host secrets/keytab files
- Re-encrypt secrets/common.yaml with both new host keys
- Convert all stub secrets to real sops-encrypted files:
secrets/ha-server-{1,2}.yaml (YAML, beszel-token = PLACEHOLDER)
secrets/ha-server-{1,2}.keytab (binary, stub text encrypted)
secrets/ha-corosync-authkey (binary, stub text encrypted)
- Add scripts/ha/deploy.sh: full lifecycle script (bridge setup, VM
creation, DRBD disk + storage NIC attachment, boot wait, cluster-init,
acceptance tests, --destroy)
Bootstrap order (operator runs these before first deploy):
1. bash scripts/ha/deploy.sh # deploys, tests
# Post-deploy secret replacement:
2. sops secrets/ha-server-{1,2}.yaml (set real beszel-token)
3. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
5. Set services.beszel.agent.environment.KEY in host.nix after hub pairing
6. nixos-rebuild switch on both nodes to pick up real secrets
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
beatzaplentyandClaude Sonnet 4.6