fix(ha): pre-seed SSH host key in disko image; fix DRBD init race

Root cause of recurring sops failures on new VM boots: disko builds raw
disk images, and create-proxmox-resource.sh only syncs the clan-var SSH
host key to the Proxmox node (for proxmox.nix to bake into the image) when
it actually builds — reusing a cached image skips sync_remote_host_keys, so
destroy+recreate reuses a stale image with the wrong or randomly-generated
key baked in.  On first boot the VM gets a different key than what .sops.yaml
was encrypted for, and sops fails permanently.

Fix 1 — deploy.sh Phase 3: always pass --force-rebuild so every VM creation
rebuilds the disko image fresh with the current clan-var key baked in via
NIXOS_HOST_KEYS_DIR (proxmox.nix already reads this under --impure).

Fix 2 — deploy.sh Phase 5.5: after VMs boot, scan their actual ed25519 host
keys and, if they drift from clan vars, update the clan var pub-key files,
rewrite the .sops.yaml age anchors, and re-encrypt all affected sops files.
Defence-in-depth: normally a no-op after Fix 1, but catches any residual
mismatch (e.g. --skip-create-vms reuse of an older image).

Fix 3 — cluster-init.sh: add crm_standby -v on for both nodes before DRBD
metadata init.  Without this, Pacemaker's OCF DRBD agent races: it sees
drbdadm down as a failure and immediately calls drbdadm up again, leaving
/dev/sdb busy when create-md / write-dev-uuid runs (drbdmeta exits 20 with
"stdin not a TTY, not waiting for confirmation").  Standby suppresses
resource scheduling during init; crm_standby -v off restores it after DRBD
is up on both nodes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

scripts/ha/cluster-init.sh

@@ -135,14 +135,33 @@ for i in $(seq 1 30); do
 done
 
 # ── 2. DRBD initialisation ────────────────────────────────────────────────
-# Down DRBD first on both nodes before (re-)initialising metadata.
-# This ensures /dev/sdb is not held open by the kernel module, which would
-# trigger a drbdmeta TTY-confirmation prompt ("stdin not a TTY, not waiting
-# for confirmation") during the write-dev-uuid step even when --force is set.
-log "Detaching DRBD on $NODE1 (idempotent pre-init clean-up)..."
+# Put both nodes in Pacemaker standby before touching DRBD metadata.
+# Without this, the OCF DRBD agent races: it sees drbdadm-down as a failure
+# and immediately calls drbdadm-up again, leaving /dev/sdb busy when
+# create-md / write-dev-uuid runs.  On a fresh cluster with no resources
+# configured this is a no-op; on a re-run it stops the race.
+log "Setting both nodes to Pacemaker standby for DRBD metadata init..."
+crm_standby -N "$NODE1" -v on 2>/dev/null || true
+crm_standby -N "$NODE2" -v on 2>/dev/null || true
+
+# Wait for Pacemaker to actually stop DRBD (if it was managing it).
+log "Waiting for DRBD to stop under Pacemaker control..."
+for i in $(seq 1 30); do
+  n1_role=$(drbdadm role ha-data 2>/dev/null || echo "Unconfigured")
+  n2_role=$(n2_ssh "drbdadm role ha-data 2>/dev/null" 2>/dev/null || echo "Unconfigured")
+  if [[ "$n1_role" == "Unconfigured" ]] && [[ "$n2_role" == "Unconfigured" ]]; then
+    log "DRBD stopped on both nodes"
+    break
+  fi
+  [[ $i -eq 30 ]] && warn "DRBD still active after 60s standby — forcing down anyway"
+  sleep 2
+done
+
+log "Detaching DRBD on $NODE1 (belt-and-suspenders after standby)..."
 drbdadm down ha-data 2>/dev/null || true
-log "Detaching DRBD on $NODE2 (idempotent pre-init clean-up)..."
+log "Detaching DRBD on $NODE2..."
 n2_ssh "drbdadm down ha-data 2>/dev/null || true"
+sleep 2
 
 log "Initialising DRBD metadata on $NODE1..."
 if ! drbdadm dstate ha-data 2>/dev/null | grep -q "UpToDate\|Inconsistent\|Diskless"; then
@@ -157,6 +176,10 @@ log "Bringing up DRBD on both nodes..."
 drbdadm up ha-data 2>/dev/null || true
 n2_ssh "drbdadm up ha-data" 2>/dev/null || true
 
+log "Clearing Pacemaker standby — DRBD is up, letting Pacemaker resume..."
+crm_standby -N "$NODE1" -v off 2>/dev/null || true
+crm_standby -N "$NODE2" -v off 2>/dev/null || true
+
 log "Forcing $NODE1 to DRBD Primary for initial sync..."
 drbdadm primary ha-data --force