feat(ha): add resize-data-disk.sh for online data disk expansion

Three-phase online resize: Proxmox qm resize → guest block device rescan → drbdadm resize + xfs_growfs. No downtime required. Detects active node via crm_mon, pre-checks DRBD UpToDate/UpToDate, auto-discovers VMIDs from qm list, and resolves the block device name from the stable scsi-0QEMU... by-id symlink with a slot-index fallback. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
remove stray pihole backup
2026-07-29 09:59:39 +10:00 · 2026-07-29 09:50:50 +10:00 · 2026-07-29 02:36:22 +10:00 · 2026-07-29 02:23:47 +10:00 · 2026-07-29 02:04:51 +10:00
6 changed files with 722 additions and 9 deletions
@@ -63,9 +63,10 @@ echo "Detecting Active/Standby nodes (waiting for Pacemaker to settle)..."
 ACTIVE_NODE=""
 for i in $(seq 1 30); do
  # crm_mon -1 output contains "Promoted: [ <node> ]" for the DRBD master.
-  CRM_OUT=$(n1 "crm_mon -1 2>/dev/null" 2>/dev/null || n2 "crm_mon -1 2>/dev/null" 2>/dev/null || true)
-  # Match "Promoted:" (Pacemaker 2.x) or "Masters:" (Pacemaker 1.x), not "Unpromoted:"/"Slaves:"
-  ACTIVE_NODE=$(echo "$CRM_OUT" | grep -E '^\s*(Promoted|Masters):' | grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
+  CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || n2 "crm_mon -1" 2>/dev/null || true)
+  # crm_mon 2.x formats Promoted lines as "    * Promoted: [ node ]" — the * bullet
+  # means ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
+  ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | grep -E '(Promoted|Masters):' | grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
  [[ -n "$ACTIVE_NODE" ]] && break
  sleep 3
 done
@@ -0,0 +1,256 @@
+#!/usr/bin/env bash
+# failover.sh — graceful HA cluster failover
+#
+# Detects which node is active and moves all resources to the other node by
+# putting the active node into Pacemaker standby.  Waits for the XFS mount to
+# appear on the target before returning.
+#
+# Usage:
+#   scripts/ha/failover.sh [--to node1|node2] [--force] [--timeout <s>] [--dry-run]
+#
+#   --to node1|node2  target node (default: the node that is NOT currently active)
+#   --force           skip the interactive confirmation prompt
+#   --timeout <s>     seconds to wait for resources to move (default: 120)
+#   --dry-run         show what would be done without changing anything
+set -euo pipefail
+
+# ── Configuration ─────────────────────────────────────────────────────────
+NODE1="${NODE1:-ha-server-1}"
+NODE2="${NODE2:-ha-server-2}"
+NODE1_IP="${NODE1_IP:-192.168.2.228}"
+NODE2_IP="${NODE2_IP:-192.168.2.227}"
+VIP="${VIP:-192.168.2.229}"
+XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"
+HA_USER="${HA_USER:-nixos}"
+# ──────────────────────────────────────────────────────────────────────────
+
+n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
+n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
+
+# ── Argument parsing ───────────────────────────────────────────────────────
+TARGET_NODE=""
+FORCE=false
+DRY_RUN=false
+TIMEOUT=120
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --to)
+      shift
+      case "${1:-}" in
+        node1|ha-server-1) TARGET_NODE="$NODE1" ;;
+        node2|ha-server-2) TARGET_NODE="$NODE2" ;;
+        *) echo "ERROR: --to must be node1, node2, ha-server-1, or ha-server-2"; exit 1 ;;
+      esac
+      ;;
+    --force)   FORCE=true ;;
+    --dry-run) DRY_RUN=true ;;
+    --timeout) shift; TIMEOUT="${1:?--timeout requires a value}" ;;
+    *) echo "Unknown argument: $1"; echo "Usage: $0 [--to node1|node2] [--force] [--timeout <s>] [--dry-run]"; exit 1 ;;
+  esac
+  shift
+done
+
+DRY_PREFIX=""
+$DRY_RUN && DRY_PREFIX="[dry-run] "
+
+echo "════════════════════════════════════════════════════"
+echo " HA Cluster Failover — $(date '+%Y-%m-%d %H:%M:%S')"
+$DRY_RUN && echo " MODE: dry-run — no changes will be made"
+echo "════════════════════════════════════════════════════"
+
+# ── Detect active node ─────────────────────────────────────────────────────
+echo ""
+echo "Detecting active node..."
+
+CRM_OUT=""
+if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" true 2>/dev/null; then
+  CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
+fi
+if [[ -z "$CRM_OUT" ]]; then
+  if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" true 2>/dev/null; then
+    CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
+  fi
+fi
+
+# crm_mon 2.x formats Promoted lines as "    * Promoted: [ node ]" — the * bullet
+# means ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
+ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | \
+  grep -E '(Promoted|Masters):' | \
+  grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
+
+if [[ -z "$ACTIVE_NODE" ]]; then
+  echo ""
+  echo "ERROR: could not determine active node from crm_mon."
+  echo "  Is Pacemaker still settling?  Try running scripts/ha/health.sh first."
+  echo "  If Pacemaker is down on both nodes, manual recovery is required."
+  exit 1
+fi
+
+if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
+  ACTIVE_IP="$NODE1_IP"
+  STANDBY_NODE="$NODE2"
+  STANDBY_IP="$NODE2_IP"
+  na() { n1 "$@"; }
+  ns() { n2 "$@"; }
+else
+  ACTIVE_IP="$NODE2_IP"
+  STANDBY_NODE="$NODE1"
+  STANDBY_IP="$NODE1_IP"
+  na() { n2 "$@"; }
+  ns() { n1 "$@"; }
+fi
+
+echo "  Active:  $ACTIVE_NODE ($ACTIVE_IP)"
+echo "  Standby: $STANDBY_NODE ($STANDBY_IP)"
+
+# ── Validate target ────────────────────────────────────────────────────────
+if [[ -n "$TARGET_NODE" ]]; then
+  if [[ "$TARGET_NODE" == "$ACTIVE_NODE" ]]; then
+    echo ""
+    echo "ERROR: $TARGET_NODE is already the active node — nothing to do."
+    exit 1
+  fi
+  echo "  Target:  $TARGET_NODE (as requested)"
+else
+  echo "  Target:  $STANDBY_NODE (auto — the other node)"
+fi
+
+# ── Pre-checks ─────────────────────────────────────────────────────────────
+echo ""
+echo "Pre-checks..."
+
+DRBD_DSTATE=$(na "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
+if ! echo "$DRBD_DSTATE" | grep -q "UpToDate/UpToDate"; then
+  echo ""
+  echo "  WARNING: DRBD dstate is '$DRBD_DSTATE' (not UpToDate/UpToDate)."
+  echo "  Failing over with a partially-synced disk risks split-brain."
+  if ! $FORCE; then
+    echo "  Use --force to proceed anyway (not recommended)."
+    exit 1
+  fi
+  echo "  --force specified — proceeding despite non-ideal DRBD state."
+else
+  echo "  DRBD dstate: $DRBD_DSTATE — OK"
+fi
+
+QUORUM_OK=$(na "corosync-quorumtool -s 2>/dev/null | grep -c 'Quorate:.*Yes'" 2>/dev/null || echo "0")
+if [[ "$QUORUM_OK" -lt 1 ]]; then
+  echo "  ERROR: cluster does not have quorum — failover would be unsafe."
+  exit 1
+fi
+echo "  Quorum: OK"
+
+# ── Confirm ────────────────────────────────────────────────────────────────
+if ! $FORCE && ! $DRY_RUN; then
+  echo ""
+  echo "  This will move all resources from $ACTIVE_NODE → $STANDBY_NODE."
+  echo "  VIP and services will be unreachable for ~10–30 seconds."
+  printf "  Proceed? [y/N] "
+  read -r ANSWER
+  [[ "${ANSWER,,}" == "y" || "${ANSWER,,}" == "yes" ]] || { echo "Aborted."; exit 0; }
+fi
+
+# ── Capture active node's crm_node name ───────────────────────────────────
+# crm_node -n returns the node name as registered in Pacemaker (may differ
+# from hostname if Pacemaker was configured with explicit node names).
+ACTIVE_CRMD_NAME=$(na "crm_node -n 2>/dev/null" 2>/dev/null || echo "$ACTIVE_NODE")
+
+# ── Perform failover ───────────────────────────────────────────────────────
+echo ""
+echo "${DRY_PREFIX}Putting $ACTIVE_NODE into standby (resources will migrate to $STANDBY_NODE)..."
+if ! $DRY_RUN; then
+  na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v on" 2>/dev/null || true
+fi
+
+# ── Wait for resources to move ─────────────────────────────────────────────
+echo "${DRY_PREFIX}Waiting up to ${TIMEOUT}s for XFS to mount on $STANDBY_NODE..."
+MOVED=false
+SPIN_CHARS=('|' '/' '-' '\')
+SPIN_I=0
+
+if $DRY_RUN; then
+  echo "  [dry-run] would wait for mountpoint $XFS_MOUNT on $STANDBY_NODE"
+  MOVED=true
+else
+  for i in $(seq 1 "$TIMEOUT"); do
+    if ns "mountpoint -q '${XFS_MOUNT}' 2>/dev/null" 2>/dev/null; then
+      printf "\r%-80s\r" ""
+      echo "  Resources moved in ${i}s"
+      MOVED=true
+      break
+    fi
+    SPIN_I=$(( SPIN_I + 1 ))
+    SC="${SPIN_CHARS[$((SPIN_I % 4))]}"
+    printf "\r  [%s] waiting... (%ds)     " "$SC" "$i"
+    sleep 1
+  done
+fi
+
+if ! $MOVED; then
+  echo ""
+  echo "ERROR: XFS did not mount on $STANDBY_NODE within ${TIMEOUT}s."
+  echo ""
+  echo "  Current resource state:"
+  na "crm_mon -1 2>/dev/null" 2>/dev/null | grep -E 'Started|Stopped|Promoted|Unpromoted|FAILED' | sed 's/^/  /' || true
+  echo ""
+  echo "  Clearing standby to restore $ACTIVE_NODE (undo the failover attempt)..."
+  na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
+  na "crm_resource --cleanup" 2>/dev/null || true
+  exit 1
+fi
+
+# ── Clear failure history ──────────────────────────────────────────────────
+echo "${DRY_PREFIX}Clearing Pacemaker failure history..."
+if ! $DRY_RUN; then
+  ns "crm_resource --cleanup 2>/dev/null" 2>/dev/null || true
+fi
+
+# ── Re-enable original active node as standby ─────────────────────────────
+echo "${DRY_PREFIX}Re-enabling $ACTIVE_NODE (now standby — will not claim resources)..."
+if ! $DRY_RUN; then
+  na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
+fi
+
+# ── Wait briefly for DRBD resync to begin ─────────────────────────────────
+if ! $DRY_RUN; then
+  sleep 5
+fi
+
+# ── Final state ────────────────────────────────────────────────────────────
+echo ""
+echo "Failover complete.  Final state:"
+echo ""
+
+CRM_OUT_AFTER=""
+if ! $DRY_RUN; then
+  CRM_OUT_AFTER=$(ns "crm_mon -1" 2>/dev/null || na "crm_mon -1" 2>/dev/null || true)
+else
+  CRM_OUT_AFTER="$CRM_OUT"
+fi
+
+NEW_ACTIVE=$(echo "$CRM_OUT_AFTER" | grep -v 'Unpromoted\|Unmanaged' | \
+  grep -E '(Promoted|Masters):' | \
+  grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
+
+if [[ -n "$NEW_ACTIVE" ]]; then
+  if [[ "$NEW_ACTIVE" == "$ACTIVE_NODE" ]]; then
+    echo "  WARNING: $ACTIVE_NODE is still showing as active in crm_mon."
+    echo "  Pacemaker may still be settling — check again in a few seconds."
+  else
+    echo "  Active:  $NEW_ACTIVE"
+    echo "  Standby: $ACTIVE_NODE"
+  fi
+fi
+
+echo ""
+RESOURCES_AFTER=$(echo "$CRM_OUT_AFTER" | awk '/Full List of Resources/,0' | tail -n +2 || true)
+[[ -z "$RESOURCES_AFTER" ]] && RESOURCES_AFTER=$(echo "$CRM_OUT_AFTER" | \
+  grep -E 'Started|Stopped|Promoted|Unpromoted|FAILED|Master|Slave' || true)
+[[ -n "$RESOURCES_AFTER" ]] && echo "$RESOURCES_AFTER" | sed 's/^/  /'
+
+echo ""
+echo "  (DRBD resync of $ACTIVE_NODE may take a moment; monitor with:"
+echo "   ssh nixos@${ACTIVE_IP} 'sudo watch -n3 cat /proc/drbd')"
+echo ""
+echo "════════════════════════════════════════════════════"
@@ -0,0 +1,180 @@
+#!/usr/bin/env bash
+# health.sh — HA cluster health snapshot (read-only, non-destructive)
+#
+# Prints a compact status panel across both nodes: SSH reachability, quorum,
+# DRBD state, Pacemaker resources, and service ports via the VIP.
+# Run from any host with SSH access to the HA nodes.
+set -euo pipefail
+
+# ── Configuration ─────────────────────────────────────────────────────────
+NODE1="${NODE1:-ha-server-1}"
+NODE2="${NODE2:-ha-server-2}"
+NODE1_IP="${NODE1_IP:-192.168.2.228}"   # vars.haServer1Ip
+NODE2_IP="${NODE2_IP:-192.168.2.227}"   # vars.haServer2Ip
+VIP="${VIP:-192.168.2.229}"             # vars.haServerVip
+XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"  # vars.haStorageRoot
+HA_USER="${HA_USER:-nixos}"
+# ──────────────────────────────────────────────────────────────────────────
+
+REACHABLE_1=false
+REACHABLE_2=false
+
+n1()  { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
+n2()  { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
+
+probe_node() {
+  local ip=$1
+  ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${ip}" true 2>/dev/null && echo "ONLINE" || echo "OFFLINE"
+}
+
+section() { echo ""; echo "── $* ──"; }
+
+echo "════════════════════════════════════════════════════"
+echo " HA Cluster Health — $(date '+%Y-%m-%d %H:%M:%S')"
+echo "════════════════════════════════════════════════════"
+
+# ── Node reachability ──────────────────────────────────────────────────────
+section "Nodes"
+N1_STATUS=$(probe_node "$NODE1_IP")
+N2_STATUS=$(probe_node "$NODE2_IP")
+[[ "$N1_STATUS" == "ONLINE" ]] && REACHABLE_1=true
+[[ "$N2_STATUS" == "ONLINE" ]] && REACHABLE_2=true
+
+if ! $REACHABLE_1 && ! $REACHABLE_2; then
+  echo "  ERROR: both nodes unreachable — cannot continue."
+  exit 1
+fi
+
+# ── Detect active node ─────────────────────────────────────────────────────
+# crm_mon 2.x formats the Promoted line as "    * Promoted: [ node ]" — the
+# bullet * means ^\s*(Promoted|Masters): never matches.  Filter out Unpromoted
+# first, then match anywhere on the line.
+ACTIVE_NODE=""
+CRM_OUT=""
+if $REACHABLE_1; then
+  CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
+elif $REACHABLE_2; then
+  CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
+fi
+ACTIVE_NODE=$(echo "${CRM_OUT:-}" | grep -v 'Unpromoted\|Unmanaged' | \
+  grep -E '(Promoted|Masters):' | \
+  grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
+
+STANDBY_NODE=""
+if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
+  STANDBY_NODE="$NODE2"
+elif [[ "$ACTIVE_NODE" == "$NODE2" ]]; then
+  STANDBY_NODE="$NODE1"
+fi
+
+n1_tag=""; n2_tag=""
+[[ "$ACTIVE_NODE" == "$NODE1" ]] && n1_tag=" [ACTIVE]"  || n1_tag=" [STANDBY]"
+[[ "$ACTIVE_NODE" == "$NODE2" ]] && n2_tag=" [ACTIVE]"  || n2_tag=" [STANDBY]"
+[[ -z "$ACTIVE_NODE" ]] && { n1_tag=""; n2_tag=""; }
+
+printf "  %-14s  [%s]%s\n" "$NODE1" "$N1_STATUS" "$n1_tag"
+printf "  %-14s  [%s]%s\n" "$NODE2" "$N2_STATUS" "$n2_tag"
+
+if [[ -z "$ACTIVE_NODE" ]]; then
+  echo ""
+  echo "  WARNING: could not determine active node from crm_mon."
+  echo "  Pacemaker may still be settling, or both nodes may be in standby."
+fi
+
+# ── Quorum ─────────────────────────────────────────────────────────────────
+section "Quorum"
+if $REACHABLE_1; then
+  QUORUM=$(n1 "corosync-quorumtool -s 2>/dev/null" 2>/dev/null || echo "")
+elif $REACHABLE_2; then
+  QUORUM=$(n2 "corosync-quorumtool -s 2>/dev/null" 2>/dev/null || echo "")
+fi
+
+if [[ -z "${QUORUM:-}" ]]; then
+  echo "  corosync-quorumtool: unavailable"
+else
+  QUORATE=$(echo "$QUORUM" | grep "Quorate:" | awk '{print $2}' || echo "?")
+  VOTES=$(echo "$QUORUM" | grep "Total votes:" | awk '{print $3}' || echo "?")
+  NEEDED=$(echo "$QUORUM" | grep "Quorum votes:" | awk '{print $3}' || echo "?")
+  echo "  Quorate: $QUORATE   Votes: $VOTES / Expected: $NEEDED"
+fi
+
+# ── DRBD ───────────────────────────────────────────────────────────────────
+section "DRBD (ha-data)"
+
+drbd_info_from() {
+  local node=$1 run=$2
+  local role dstate cs pct
+  role=$($run "drbdadm role ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
+  dstate=$($run "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
+  cs=$($run "grep -oE 'cs:[A-Za-z]+' /proc/drbd 2>/dev/null | head -1 | sed 's/cs://'" 2>/dev/null || echo "unknown")
+  pct=$($run "grep -oE \"sync'ed:[[:space:]]+[0-9.]+\" /proc/drbd 2>/dev/null | grep -oE '[0-9.]+$' | head -1" 2>/dev/null || echo "")
+  printf "  %-14s  role: %-22s  dstate: %-25s  cs: %s" \
+    "$node" "${role:-unknown}" "${dstate:-unknown}" "${cs:-unknown}"
+  [[ -n "$pct" ]] && printf "  syncing: %s%%" "$pct"
+  echo ""
+}
+
+$REACHABLE_1 && drbd_info_from "$NODE1" n1 || echo "  $NODE1  [OFFLINE]"
+$REACHABLE_2 && drbd_info_from "$NODE2" n2 || echo "  $NODE2  [OFFLINE]"
+
+# ── Pacemaker (full crm_mon output) ───────────────────────────────────────
+section "Pacemaker"
+if [[ -n "${CRM_OUT:-}" ]]; then
+  echo "$CRM_OUT" | sed 's/^/  /'
+else
+  echo "  crm_mon returned no output — trying again without suppression:"
+  if $REACHABLE_1; then
+    n1 "crm_mon -1" || true
+  elif $REACHABLE_2; then
+    n2 "crm_mon -1" || true
+  fi
+fi
+
+# ── XFS mount ─────────────────────────────────────────────────────────────
+section "XFS Mount ($XFS_MOUNT)"
+check_mount() {
+  local node=$1 run=$2
+  local status
+  if $run "mountpoint -q '$XFS_MOUNT' 2>/dev/null" 2>/dev/null; then
+    local usage
+    usage=$($run "df -h '$XFS_MOUNT' 2>/dev/null | tail -1 | awk '{print \$3\"/\"\$2\" used (\"\$5\")\";}'" 2>/dev/null || echo "")
+    status="mounted"
+    [[ -n "$usage" ]] && status="mounted  $usage"
+  else
+    status="not mounted"
+  fi
+  printf "  %-14s  %s\n" "$node" "$status"
+}
+
+$REACHABLE_1 && check_mount "$NODE1" n1 || echo "  $NODE1  [OFFLINE]"
+$REACHABLE_2 && check_mount "$NODE2" n2 || echo "  $NODE2  [OFFLINE]"
+
+# ── Service ports via VIP ──────────────────────────────────────────────────
+section "Services via VIP ($VIP)"
+
+check_port() {
+  local name=$1 port=$2
+  if bash -c "echo >/dev/tcp/${VIP}/${port}" 2>/dev/null; then
+    printf "  %-10s  port %-5s  OK\n" "$name" "$port"
+  else
+    printf "  %-10s  port %-5s  UNREACHABLE\n" "$name" "$port"
+  fi
+}
+
+if ping -c1 -W2 "$VIP" >/dev/null 2>&1; then
+  echo "  Ping       OK"
+else
+  echo "  Ping       UNREACHABLE"
+fi
+check_port "NFS"    2049
+check_port "iSCSI"  3260
+
+echo ""
+echo "════════════════════════════════════════════════════"
+if [[ -n "$ACTIVE_NODE" ]]; then
+  echo " Active: $ACTIVE_NODE   Standby: $STANDBY_NODE"
+else
+  echo " Active: unknown (Pacemaker not settled)"
+fi
+echo "════════════════════════════════════════════════════"
+echo ""
@@ -0,0 +1,276 @@
+#!/usr/bin/env bash
+# resize-data-disk.sh — online resize of the HA cluster data disk
+#
+# Three-phase process (all online-safe, no downtime required):
+#   1. Proxmox: grow scsi1 on both HA VMs   (qm resize)
+#   2. Guest:   rescan block device on both nodes so the kernel sees the new size
+#   3. DRBD + XFS: drbdadm resize, then xfs_growfs — active node only
+#
+# Usage:
+#   scripts/ha/resize-data-disk.sh --size +20G [--force] [--dry-run]
+#
+#   --size +NNg    amount to grow scsi1 by, e.g. +20G, +50G (required)
+#                  XFS and DRBD cannot shrink; only positive deltas accepted
+#   --force        skip the interactive confirmation prompt
+#   --dry-run      show what would be done without changing anything
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=../env.sh
+source "${SCRIPT_DIR}/../env.sh"
+
+# ── Configuration ─────────────────────────────────────────────────────────
+NODE1="${NODE1:-ha-server-1}"
+NODE2="${NODE2:-ha-server-2}"
+NODE1_IP="${NODE1_IP:-192.168.2.228}"
+NODE2_IP="${NODE2_IP:-192.168.2.227}"
+XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"
+DRBD_RESOURCE="${DRBD_RESOURCE:-ha-data}"
+DATA_DISK_SLOT="${DATA_DISK_SLOT:-scsi1}"   # Proxmox disk name (scsi1 = data disk)
+HA_USER="${HA_USER:-nixos}"
+PVE_HOST="${PVE_HOST:-${PVE1_HOST}}"
+PVE_SSH_USER="${PVE_SSH_USER:-${PROXMOX_SSH_USER}}"
+# By-id symlink for the data disk; basename resolves to the raw block device.
+# matches variables.nix's haServerDrbdDisk.
+DATA_DISK_BYID="${DATA_DISK_BYID:-scsi-0QEMU_QEMU_HARDDISK_drive-${DATA_DISK_SLOT}}"
+# ──────────────────────────────────────────────────────────────────────────
+
+pve() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
+          "${PVE_SSH_USER}@${PVE_HOST}" "$@"; }
+n1()  { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
+          "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
+n2()  { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
+          "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
+
+# ── Argument parsing ───────────────────────────────────────────────────────
+SIZE=""
+FORCE=false
+DRY_RUN=false
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --size)    shift; SIZE="${1:?--size requires a value (e.g. +20G)}" ;;
+    --force)   FORCE=true ;;
+    --dry-run) DRY_RUN=true ;;
+    *) echo "Unknown argument: $1"
+       echo "Usage: $0 --size +NNg [--force] [--dry-run]"
+       exit 1 ;;
+  esac
+  shift
+done
+
+if [[ -z "$SIZE" ]]; then
+  echo "ERROR: --size is required (e.g. --size +20G)"
+  echo "Usage: $0 --size +NNg [--force] [--dry-run]"
+  exit 1
+fi
+
+# Only positive deltas — qm resize, DRBD, and XFS all refuse to shrink.
+if [[ ! "$SIZE" =~ ^\+[0-9]+(G|M|T)$ ]]; then
+  echo "ERROR: --size must be a positive delta like +20G, +50G, +500M, +2T"
+  echo "       (qm resize, drbdadm resize, and xfs_growfs can only grow, not shrink)"
+  exit 1
+fi
+
+DRY_PREFIX=""
+$DRY_RUN && DRY_PREFIX="[dry-run] "
+
+echo "════════════════════════════════════════════════════"
+echo " HA Data Disk Resize — $(date '+%Y-%m-%d %H:%M:%S')"
+echo " Size delta: $SIZE  •  Proxmox: $PVE_HOST"
+$DRY_RUN && echo " MODE: dry-run — no changes will be made"
+echo "════════════════════════════════════════════════════"
+
+# ── Detect active node ─────────────────────────────────────────────────────
+echo ""
+echo "Detecting active node..."
+
+CRM_OUT=""
+if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
+     "${HA_USER}@${NODE1_IP}" true 2>/dev/null; then
+  CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
+fi
+if [[ -z "$CRM_OUT" ]]; then
+  if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
+       "${HA_USER}@${NODE2_IP}" true 2>/dev/null; then
+    CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
+  fi
+fi
+
+# crm_mon 2.x formats Promoted lines as "    * Promoted: [ node ]" (bullet *),
+# so ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
+ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | \
+  grep -E '(Promoted|Masters):' | \
+  grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
+
+if [[ -z "$ACTIVE_NODE" ]]; then
+  echo "ERROR: could not determine active node from crm_mon."
+  echo "  Is Pacemaker still settling?  Try running scripts/ha/health.sh first."
+  exit 1
+fi
+
+if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
+  ACTIVE_IP="$NODE1_IP"
+  na() { n1 "$@"; }
+else
+  ACTIVE_IP="$NODE2_IP"
+  na() { n2 "$@"; }
+fi
+
+echo "  Active node: $ACTIVE_NODE ($ACTIVE_IP)"
+
+# ── Pre-check DRBD state ───────────────────────────────────────────────────
+echo ""
+echo "Pre-checks..."
+
+DRBD_DSTATE=$(na "drbdadm dstate ${DRBD_RESOURCE} 2>/dev/null" 2>/dev/null || echo "unknown")
+if ! echo "$DRBD_DSTATE" | grep -q "UpToDate/UpToDate"; then
+  echo "  WARNING: DRBD dstate is '$DRBD_DSTATE' (expected UpToDate/UpToDate)."
+  echo "  Resizing with a partially-synced disk may cause issues."
+  if ! $FORCE; then
+    echo "  Use --force to proceed anyway."
+    exit 1
+  fi
+  echo "  --force specified — proceeding despite non-ideal DRBD state."
+else
+  echo "  DRBD dstate: $DRBD_DSTATE — OK"
+fi
+
+# ── Find VMIDs on Proxmox ─────────────────────────────────────────────────
+echo ""
+echo "Looking up VM IDs on ${PVE_HOST}..."
+
+QM_LIST=$(pve "qm list 2>/dev/null" || true)
+VMID1=$(echo "$QM_LIST" | awk -v name="$NODE1" '$0 ~ name {print $1}' | head -1)
+VMID2=$(echo "$QM_LIST" | awk -v name="$NODE2" '$0 ~ name {print $1}' | head -1)
+
+if [[ -z "$VMID1" ]]; then
+  echo "  ERROR: could not find VMID for $NODE1 on $PVE_HOST"
+  echo "  qm list output:"
+  echo "$QM_LIST" | sed 's/^/    /'
+  exit 1
+fi
+if [[ -z "$VMID2" ]]; then
+  echo "  ERROR: could not find VMID for $NODE2 on $PVE_HOST"
+  echo "  qm list output:"
+  echo "$QM_LIST" | sed 's/^/    /'
+  exit 1
+fi
+
+echo "  $NODE1: VMID $VMID1"
+echo "  $NODE2: VMID $VMID2"
+
+# ── Confirm ────────────────────────────────────────────────────────────────
+if ! $FORCE && ! $DRY_RUN; then
+  echo ""
+  echo "  Plan:"
+  echo "    Phase 1 — qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE}  (on $PVE_HOST)"
+  echo "              qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE}  (on $PVE_HOST)"
+  echo "    Phase 2 — block device rescan on $NODE1 and $NODE2"
+  echo "    Phase 3 — drbdadm resize + xfs_growfs on $ACTIVE_NODE"
+  echo "  No downtime required (all operations are online-safe)."
+  printf "  Proceed? [y/N] "
+  read -r ANSWER
+  [[ "${ANSWER,,}" == "y" || "${ANSWER,,}" == "yes" ]] || { echo "Aborted."; exit 0; }
+fi
+
+# ═══════════════════════════════════════════════════════════════
+# Phase 1 — Resize both VM data disks in Proxmox
+# ═══════════════════════════════════════════════════════════════
+echo ""
+echo "── Phase 1 — Proxmox disk resize (${DATA_DISK_SLOT} ${SIZE} on both VMs) ──"
+
+echo "  ${DRY_PREFIX}qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE}  ($NODE1 on ${PVE_HOST})"
+if ! $DRY_RUN; then
+  pve "qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE}"
+fi
+
+echo "  ${DRY_PREFIX}qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE}  ($NODE2 on ${PVE_HOST})"
+if ! $DRY_RUN; then
+  pve "qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE}"
+fi
+
+echo "  Phase 1 done."
+
+# ═══════════════════════════════════════════════════════════════
+# Phase 2 — Rescan block device on both guest nodes
+# ═══════════════════════════════════════════════════════════════
+echo ""
+echo "── Phase 2 — Block device rescan (both nodes) ──"
+
+rescan_node() {
+  local node_name=$1 run_fn=$2
+
+  # Resolve block device name from the stable by-id symlink on the guest.
+  local blk_dev=""
+  if ! $DRY_RUN; then
+    blk_dev=$($run_fn "bash -c 'basename \$(readlink -f /dev/disk/by-id/${DATA_DISK_BYID})'" 2>/dev/null || true)
+  fi
+  if [[ -z "$blk_dev" ]]; then
+    # Fall back: scsi1 → index 1 → sdb, scsi2 → sdc, etc.
+    local slot_idx
+    slot_idx=$(echo "$DATA_DISK_SLOT" | grep -oE '[0-9]+$' || echo "1")
+    blk_dev=$(printf "sd%s" "$(echo "abcdefghij" | cut -c$((slot_idx + 1)))")
+    echo "  WARNING: could not resolve /dev/disk/by-id/${DATA_DISK_BYID} on $node_name;" \
+         "falling back to /dev/${blk_dev}" >&2
+  fi
+
+  echo "  ${DRY_PREFIX}Rescanning /dev/${blk_dev} on ${node_name}..."
+  if ! $DRY_RUN; then
+    $run_fn "bash -c 'echo 1 > /sys/block/${blk_dev}/device/rescan'" 2>/dev/null
+    local new_size
+    new_size=$($run_fn "lsblk -nd -o SIZE /dev/${blk_dev} 2>/dev/null" 2>/dev/null || echo "unknown")
+    echo "    /dev/${blk_dev} on $node_name now reports: $new_size"
+  fi
+}
+
+rescan_node "$NODE1" n1
+rescan_node "$NODE2" n2
+
+echo "  Phase 2 done."
+
+# ═══════════════════════════════════════════════════════════════
+# Phase 3 — Grow DRBD metadata, then XFS (active node only)
+# ═══════════════════════════════════════════════════════════════
+echo ""
+echo "── Phase 3 — DRBD resize + XFS grow (on active node: $ACTIVE_NODE) ──"
+
+echo "  ${DRY_PREFIX}drbdadm resize ${DRBD_RESOURCE}"
+if ! $DRY_RUN; then
+  na "drbdadm resize ${DRBD_RESOURCE}"
+fi
+
+echo "  ${DRY_PREFIX}xfs_growfs ${XFS_MOUNT}"
+if ! $DRY_RUN; then
+  na "xfs_growfs ${XFS_MOUNT}"
+fi
+
+echo "  Phase 3 done."
+
+# ── Verify ────────────────────────────────────────────────────────────────
+echo ""
+echo "── Verify ──"
+
+if ! $DRY_RUN; then
+  DF_OUT=$(na "df -h '${XFS_MOUNT}' 2>/dev/null" 2>/dev/null || echo "")
+  if [[ -n "$DF_OUT" ]]; then
+    echo "  ${XFS_MOUNT}:"
+    echo "$DF_OUT" | sed 's/^/    /'
+  fi
+
+  DRBD_DSTATE_AFTER=$(na "drbdadm dstate ${DRBD_RESOURCE} 2>/dev/null" 2>/dev/null || echo "unknown")
+  echo "  DRBD dstate: $DRBD_DSTATE_AFTER"
+  if ! echo "$DRBD_DSTATE_AFTER" | grep -q "UpToDate/UpToDate"; then
+    echo "  NOTE: DRBD is resyncing — normal immediately after resize."
+    echo "        Monitor: ssh nixos@${ACTIVE_IP} 'sudo watch -n3 cat /proc/drbd'"
+  fi
+else
+  echo "  [dry-run] would verify df -h ${XFS_MOUNT} and drbdadm dstate on $ACTIVE_NODE"
+fi
+
+echo ""
+echo "════════════════════════════════════════════════════"
+echo " Resize complete."
+echo " Active node: $ACTIVE_NODE"
+echo "════════════════════════════════════════════════════"
+echo ""
@@ -1,22 +1,22 @@
 {
-	"data": "ENC[AES256_GCM,data:ePLME5cokqR9hNA3nlCzLuOc0ykalnBD7qD3krkUn2Sq3KrlM4ytbx+su9zz3TActsmRUOBDMfTllgtk63brw6IJE7KTF63VVGbg8tZq8JLLhTbzfUkV2u8AR0ILKdx/ic02x5dBdbyTemyJP2rsaqW2z2Iojw8UdBx5rg7+T9uZnav2hVXQyTm5OuyZCIgZlxp9qPFg7L0Z9pN9Cch/7lqnt2l311C4K7xaYlDDd0pCoH/UsqrUyBHVTsXJzAlIjru8vb/dGf9OoYBN91NrvxIxT1RrWTar/C4ErUcy/wLqWq4O2ktlmK0Zlt9S5GuVfT2zvHdudbC0wYnfr8bChg==,iv:FELIArqUxYs1XH0oKa+TItiOKi5pCnG+GVGL+vnHk34=,tag:DSDOZvYEQKYlS7CsOkVP6Q==,type:str]",
+	"data": "ENC[AES256_GCM,data:JwjmYg0qzoer+8/jv7KOfK/BxmpObHOn04GNsUFQd6hSFBWbQlRbODfluL54hnWQ4gsS9MHgwWh8nrLciBhye05kRQCqADB7Crm0CZYubNb65WzM/devh14jmerc3MYIp1M3VmDisML3x5IqhJkFMSqKM/VaNAC3f0otcVR8TTgZw/fJyvPlfqpPTjikGmo5xg++Yk7Cy8A1Dj70kYk10+EQjQ78jf4k/agLoaS+YvMmKXFk4EljtHg8Fe5H6Rn9nfMSSKGTZeAH3Riix2e+fIH+nWEZiPHC0UlPjJrB3SeBqqGRSUUJshVKHFIdKI1Tq2Ea+GDWrAyIRI0qO8e7WQ==,iv:mr39NxbZuYUDHDw0e+fJDshbt9R1hLgVqkUxUVX/+l0=,tag:u31xZRPfvH5j3Sw5U8/nqg==,type:str]",
 	"sops": {
 		"age": [
 			{
-				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAySGJmSkRVdnRnVWY0MktP\nR2RDdmlHTXBra2NvOTByOXZ5WDlGRHZTWWpFCnQxaldqMzFxWStmd0VzODNFdG8v\nZkN3Q2tUZW10bWVPTUxqWkJ5djF0LzgKLS0tIHVUR0R6L0ZlNzZyK0hrUkRWQXdY\naU1UTGlOUytnbWxoWGFPdW1pRmxWdnMKtz5kp/BvpGFxbx0z6N/GX9UMz/xKwtEQ\neqUAj8eMbZfJblpkTds0he8kjp0Z7jKvMMbKWiRaUixr/ZpufPAbgQ==\n-----END AGE ENCRYPTED FILE-----\n",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtVW5ENXlkUW83elc3ZnhE\nOEZ4MWIvQWZ4aXhuUmZGYXZNQnhJemsvOWlVCjIwNmFjS3pJOWEwRnhBOGNRamlM\ncXZid1RNYVBXOFZpMlpnU3ZnVE1scnMKLS0tIGFhWFZFd2JPTFNrV1VHSDVKVHRK\nbmhoeFh6YlUySU1ZRVhHbjZnM1IxMFkKo7aHkz2pEeV64m+OEkBZ2V1e+PUzoChu\nUc/Mnh37dNXSSJtg2KnocHdyzDGi1yQbA72xKTxx6QjYJWrAC/jurg==\n-----END AGE ENCRYPTED FILE-----\n",
 				"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
 			},
 			{
-				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkcWdXQ0RqUFk0N2JpUDBZ\nbXFYU0FMYWVGbW5yUDFYemhPeGcyTFM4OTFrCm1pM2U1WExmYkF5VTIxdk51anpB\nNVhsck5sSnlDTFdzSDlUNllzN3dtZTgKLS0tIEJRQjdmUktkREVHQTZhZDRvRjhQ\nTHNzeVlMWDVkUFgvTVhjejM4Y2FqQU0K4Zbcc8hJObX/S5OnBwh/AJbTYTXzbYsB\n0P9jSaQj2nuoJoJvvLJHr0bA+LpvxGq6sAJBfiMkT/GLAB6PMawwmQ==\n-----END AGE ENCRYPTED FILE-----\n",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBReWRLb09reXUxS0YyMUVV\nQlR2bEdYY08rK0RoR2pIelpLVEtYL0pqNUVNCklRaDNoSjBWT20xbk1UdmNqWHNt\nR3JoSlVwODVMUVBzMERUNktHMVViZm8KLS0tIHc0SXRIS1IreDF1VCthb0FBR0Na\ncGFRWTROUHF3clFtTmY1azEwNUpPNzQKsgCND/BZcMTgBTAcnHunQcT6LG1jNrtO\n+W7Yx7bFtFBajWnRYiNpUZPibQJlv5SE9os47WDH1gs86xftgV+uyA==\n-----END AGE ENCRYPTED FILE-----\n",
 				"recipient": "age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6"
 			},
 			{
-				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWdGViZkVweHYxeWp2M1Ro\ncDZuVElkZVRwQ1o3SzI2RFNhWUw4YWJmSXg0CkwzaEdLa2JjYVNrVnFOQ05YUXlE\nK1cybGM3RkxnUFFEU1ZKVk5lQjRoNVUKLS0tICtIS3ZsbndFajJnRW9SZVdiRmps\nMks3amJSZ3BqQlRUME1pRGZmUmhJTUkKHU6UXlgN4dAyJS9nweOoEG1XaMOUja2C\nbjOrq3t//L3Ftg3wWpXeAtzzDvjIkQtLo8vv4fPBqC9nYI0AR5O02w==\n-----END AGE ENCRYPTED FILE-----\n",
+				"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGbllNTlhxNVVkd01GSUNi\nR0lxQWZoRXI5NWxoTExRUnBSb2FPR1BONnpzCkxCN3pNOXVrTW5GR2J0WFhZUE9r\ncm1QTThINDVVVlZEalJiS1RXcHJRS28KLS0tIHV6aGh5QjB0M1VPWDVMT1k3cTBh\nNTNGY25NV0tWSUl3UHlJSExFQVdpVDQK3ARj8xhFRYU6oqYxNQ5+Ryza86ALNUwX\n7yK/8ATnquYC8/ZIYUbgTPxzsocmXX9lVT0+ktIALqjtG8PgNTlOow==\n-----END AGE ENCRYPTED FILE-----\n",
 				"recipient": "age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r"
 			}
 		],
-		"lastmodified": "2026-07-28T15:11:23Z",
-		"mac": "ENC[AES256_GCM,data:5hMLSLf6vynndO/pDH7OxKwsuP2hCcaKxRUUMqnM8W4jLPHnu1zCkfaADw6bRESzzBdcRM+jwKyg5DsNYgqgFU+c9EThDtMUiC5sjx3KRWPRt+Y4ebDxfkEKUiay14fWp0I6WWEF5qjF4Pl8gkzDtaBxW3vnHrn7cg0RlLKtqEw=,iv:pX0++fkhQ4gAydJUrc7Br4zT3lylWk2QFSxAv8RxTX0=,tag:jbMIEfAorHf8Cg8732K8mg==,type:str]",
+		"lastmodified": "2026-07-28T16:04:51Z",
+		"mac": "ENC[AES256_GCM,data:9X0dkAEGJiug16LDy/8//QOkHITTzt4zwTKiIU13tRgSRXRpt/7b27+3eDauscXDMamg5wbYo1YF0+VwLl21Ip4icnnwrZBHd6HWleP30HTgef8rmo21SstshnclZz6RH8SEGVDO/vjrMDChaZ3A2WD+nbcomw47DpK3cG3kBIk=,iv:G7CScSD5Z1Z9WMEMydyeGK/RD4W43xA0PlcpmTCxLOc=,tag:DZmby/xkxpinoJztZZRWwA==,type:str]",
 		"version": "3.13.3"
 	}
 }
Author	SHA1	Message	Date
beatzaplentyandClaude Sonnet 4.6	e18b605706	feat(ha): add resize-data-disk.sh for online data disk expansion Check NixOS configurations / eval-hosts (pull_request) Successful in 10m27s Details Three-phase online resize: Proxmox qm resize → guest block device rescan → drbdadm resize + xfs_growfs. No downtime required. Detects active node via crm_mon, pre-checks DRBD UpToDate/UpToDate, auto-discovers VMIDs from qm list, and resolves the block device name from the stable scsi-0QEMU... by-id symlink with a slot-index fallback. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 09:59:39 +10:00
beatzaplenty	ee2451d87c	remove stray pihole backup Check NixOS configurations / eval-hosts (push) Successful in 10m25s Details	2026-07-29 09:50:50 +10:00
beatzaplentyandClaude Sonnet 4.6	71a6c4738c	fix(ha): fix crm_mon active-node detection in all three scripts Check NixOS configurations / eval-hosts (push) Successful in 10m23s Details crm_mon 2.x formats the Promoted line as " * Promoted: [ node ]" — the asterisk bullet means the previous grep -E '^\s*(Promoted\|Masters):' never matched, so active-node detection silently returned empty in health.sh, failover.sh, and acceptance-tests.sh. Fix: pipe through grep -v Unpromoted first, then grep -E '(Promoted\|Masters):' without anchoring to start-of-line. Also: remove the SSH_OPTS=-i ~/.ssh/... variable pattern in health.sh and failover.sh; tilde is not expanded inside double-quoted strings, so $SSH_OPTS was passing a literal ~ to SSH. Inline the key path in each function definition instead (same as acceptance-tests.sh already did). Also: drop the 2>/dev/null embedded in the crm_mon argument string — the outer 2>/dev/null on the n1/n2 call already suppresses SSH stderr; the embedded one was harmless but noisy to reason about. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 02:36:22 +10:00
beatzaplentyandClaude Sonnet 4.6	793a2b5924	feat(ha): add failover.sh and health.sh operator scripts Check NixOS configurations / eval-hosts (push) Successful in 10m23s Details failover.sh: graceful active→standby resource migration with pre-checks (DRBD UpToDate/UpToDate, quorum), spinner wait for XFS mount on target, rollback on timeout, --to/--force/--dry-run/--timeout flags. health.sh: read-only status panel — node reachability, quorum, DRBD role/dstate/cs per node, Pacemaker resources, failure history, XFS mount usage, and service port checks (NFS 2049, iSCSI 3260) via VIP. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 02:23:47 +10:00
beatzaplenty	98b5429fb9	secrets(ha): encrypt corosync authkey generated by cluster-init	2026-07-29 02:04:51 +10:00