refactor: full repo sweep — variables, docs, and comment cleanup

- variables.nix: switch to rec {}, extract giteaDomain/giteaRepoPath,
  extraAdminSshKeys, haLanNfsFqdn, tailscaleResolverIp, ports.dhcp,
  ports.dns; ipaServer now derives from homeDomain ref; section headers
- modules: use new vars throughout (pxe-boot, ts-dns-forwarder,
  cluster-config, configuration.nix, mount-pxe-images) — eval unchanged
- docs: delete ephemeral planning docs (AUDIT_REPORT, ha-network-audit,
  network-cutover); add docs/ha.md; drop migration reference table from
  ip-addressing.md; remove stale server example from beszel.md
- CLAUDE.md/README.md/AGENTS.md: fix build types (tailscale-router,
  ha-server, drop server); document scripts/ha/, scripts/ipa/, and
  all previously undocumented top-level and lib scripts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

CLAUDE.md

@@ -252,6 +252,14 @@ instead of copying it.
   silent skip rather than a failure) only reports drift; the no-flags form
   updates both files in place. Declarative clients still need a rebuild to
   pick up the fix.
+- `scripts/secrets/push-host-keys.sh [--all | <target>] [--dry-run]
+  [--skip-git-check]` — pushes newly-generated SSH host keys from
+  `host-keys/` to already-running NixOS hosts, so they can decrypt sops
+  secrets after a rebuild following `sync-host-keys.sh
+  --regenerate-all-keys`. Verifies that `.sops.yaml` and `secrets/*.yaml`
+  are committed and pushed to the remote first (hosts rebuild from the
+  remote Gitea flake, so recipient changes must land there before any key
+  push).
 
 ### `scripts/proxmox/`
 
@@ -273,6 +281,16 @@ instead of copying it.
   failure just falls back to building from source / `cache.nixos.org`) so
   the node substitutes from and can offload builds to nix-cache on every
   subsequent run, not just this one.
+- `scripts/proxmox/clone-pve1-to-pve-test.sh <vmid> [--new-vmid <id>]
+  [--mode snapshot|suspend|stop] [--dry-run]` — ad-hoc clone of a single
+  VM or CT from pve1 (production) to pve-test (sandbox) via vzdump +
+  qmrestore/pct restore. Streams the archive directly between nodes (no
+  local staging copy). Always restores with `--unique 1` (fresh MAC
+  addresses) since the original is still running on the LAN. Cleans up
+  the vzdump archive from both nodes after a successful restore. The
+  script's own default is pve1 → pve-test, matching CLAUDE.md's policy
+  (unlike `create-proxmox-resource.sh`, which defaults to production for
+  the operator's own unqualified use).
 - `scripts/proxmox/configure-nix-cache-client.sh [--dry-run]
   [--no-remote-builder] [--no-restart]` — the non-NixOS equivalent of
   `modules/nix-cache/client.nix`/`remote-builder-client.nix`, for a plain
@@ -288,6 +306,50 @@ instead of copying it.
   marked block rather than duplicating it); restarts `nix-daemon` by
   default so the change takes effect immediately.
 
+### `scripts/ha/`
+
+HA cluster lifecycle and operational scripts. All mutate real cluster state
+when run for real — always run against pve-test first unless the operator
+explicitly targets pve1.
+
+- `scripts/ha/deploy.sh [--skip-*] [--destroy] [--dry-run]` — full
+  lifecycle manager: phases through bridge creation, key sync, VM creation
+  (via `create-proxmox-resource.sh`), NIC/disk attachment, and cluster
+  initialisation. `--destroy` tears it back down. Safe to rerun
+  idempotently; each phase can be individually skipped.
+- `scripts/ha/cluster-init.sh` — one-time cluster bootstrap run **as root
+  on ha-server-1** after both VMs are booted. Generates/distributes the
+  Corosync authkey, initialises DRBD metadata, creates XFS on `/dev/drbd0`,
+  configures LIO iSCSI, and registers all Pacemaker resources (DRBD → XFS
+  → iSCSI → NFS → VIPs).
+- `scripts/ha/health.sh` — read-only cluster health snapshot: SSH
+  reachability, quorum, DRBD state, Pacemaker resources, and VIP port
+  reachability. Safe to run from the workstation at any time.
+- `scripts/ha/failover.sh [--to node1|node2] [--force] [--timeout <s>]
+  [--dry-run]` — graceful failover by putting the active node into
+  Pacemaker standby and waiting for resources to appear on the target.
+- `scripts/ha/acceptance-tests.sh` — T1–T7 acceptance tests (failover,
+  NFS/iSCSI connectivity, DRBD sync, etc.) that must all pass before the
+  cluster is considered production-ready.
+- `scripts/ha/resize-data-disk.sh --size +NNg [--force] [--dry-run]` —
+  online data-disk resize: `qm resize` on both VMs, guest block-device
+  rescan, `drbdadm resize`, `xfs_growfs`. No downtime required.
+- `scripts/ha/cluster-enable-stonith.sh` — enables the `fence_pve_ssh`
+  STONITH resource after the fence SSH key is deployed to both nodes and
+  authorised on the Proxmox host. Run once after `cluster-init.sh`.
+- `scripts/ha/fence-pve-ssh.py` — Python STONITH fence agent for Pacemaker.
+  Deploy to `/etc/pacemaker/fence_pve_ssh` on both HA nodes (`chmod +x`).
+  SSHes to the Proxmox host and runs `qm stop/start <vmid>`.
+
+### `scripts/ipa/`
+
+- `scripts/ipa/create-nixos-ipa-host-account.sh [options] <hostname>` —
+  adds a NixOS host to the FreeIPA domain and produces a sops-encrypted
+  keytab at `secrets/<hostname>.keytab`, ready for `modules/ipa/client.nix`.
+  Replaces three error-prone manual steps: `ipa host-add`, `ipa-getkeytab`
+  (run on the DC, SCP'd back), and `sops encrypt` in the correct location
+  (must be at `secrets/<hostname>.keytab` for the creation rule to match).
+
 ### `scripts/lib/`
 
 Sourced by the scripts above, never run directly:
@@ -297,6 +359,15 @@ Sourced by the scripts above, never run directly:
   `create-proxmox-resource.sh` runs over SSH.
 - `nix-eval.sh` — `NIX_EVAL_FLAGS` plus `list_flake_targets`/
   `flake_target_hostname` flake-introspection helpers.
+- `nix-parallel.sh` — `run_nix_parallel`: fans out independent `nix eval`/
+  `nix build --dry-run` calls across up to `NIX_PARALLEL_JOBS` processes,
+  capped by available memory (~1 GB/job) rather than raw `nproc` to avoid
+  OOM on constrained CI runners. Used by `codex-maintenance.sh`.
+- `clan-vars.sh` — helpers for reading/writing SSH host keys stored as clan
+  vars (`vars/per-machine/<target>/openssh/`, sops-encrypted) instead of
+  the gitignored `host-keys/` directory. Sourced by
+  `create-proxmox-resource.sh` and `sync-host-keys.sh`; depends on
+  `sops-age.sh` and `ssh-host-keys.sh` being sourced first.
 - `ssh-host-keys.sh` — `generate_host_ed25519_key`/`ssh_pubkey_to_age`,
   shared by `sync-host-keys.sh` and `prepare-host-key.sh`.
 - `sops-age.sh` — `age_pubkey_from_identity_file`/`sops_yaml_admin_pubkey`/
@@ -316,6 +387,17 @@ Sourced by the scripts above, never run directly:
   default cores/memory, `NIX_CACHE_HOST`, `LAN_DOMAIN`) sourced by
   `create-proxmox-resource.sh` and `scripts/installer/auto-install.sh`. Add
   new cross-script config here instead of duplicating it per-script.
+- `scripts/recover-hosts.sh [<hostname> ...]` — fixes sops/SSH-key/GitHub-token
+  issues on deployed NixOS hosts and triggers a `Switch-nix` rebuild on each.
+  With no args discovers every known hostname; with args checks only those.
+  Fixes applied automatically (prompts before rebuilding): SSH host key drift
+  (restores the registered key) and stale GitHub access tokens (empties the
+  rendered `nix-github-token.conf` so Nix falls back to unauthenticated requests
+  until sops-nix re-renders the correct token after the next successful rebuild).
+- `scripts/gc-hosts.sh [--dry-run]` — runs `nix-collect-garbage -d` on all live
+  NixOS hosts (workstation first, then pve1, then all Proxmox guests). Excludes
+  `nix-cache` (gc-ing the shared binary cache evicts store paths other hosts
+  depend on). Uses passwordless sudo where available; falls back to user-level gc.
 - `scripts/bump-nixpkgs-release.sh` — bumps `flake.nix`'s `nixpkgs.url`/
   `home-manager.url` in place. Exists because flake input URLs can't
   reference `variables.nix` (confirmed empirically — `nix flake metadata`
@@ -354,12 +436,13 @@ nixosSystem {
 ```
 
 Platforms: `linode`, `proxmox`, `lxc`, `baremetal`. Build types: `minimal`,
-`nix-cache`, `server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`,
-`tor-relay`. Not every combination is built — e.g. `pxe-boot` has no `linode`
+`nix-cache`, `docker`, `gui`, `pxe-boot`, `tailscale-router`, `tor-relay`,
+`ha-server`. Not every combination is built — e.g. `pxe-boot` has no `linode`
 variant (PXE/DHCP/TFTP need LAN L2 adjacency a Linode VPS doesn't have),
-`tor-relay` currently only exists as `lxc-tor-relay`, and `baremetal`
-currently only exists as `baremetal-gui` (the real gui-host hardware —
-see `hosts/nixos/host.nix` and `modules/platforms/baremetal.nix`). Treat
+`tor-relay` only exists as `lxc-tor-relay`, `ha-server` only exists as
+`proxmox-ha-server-{1,2}`, and `baremetal` only exists as `baremetal-gui`
+(the real gui-host hardware — see `hosts/nixos/host.nix` and
+`modules/platforms/baremetal.nix`). Treat
 `flake.nix`'s
 `generatedTargets` as the source
 of truth for which hosts exist — `README.md`, `AGENTS.md`,
@@ -391,7 +474,7 @@ removing a host.
   `vzdump` backup-archive metadata this doesn't have), no install step —
   see `docs/auto-installer.md`.
 - `modules/build-types/*.nix` — what a system is for:
-  minimal/server/docker/gui/pxe-boot/nix-cache/tailscale-exit-node/tor-relay.
+  minimal/docker/gui/pxe-boot/nix-cache/tailscale-router/tor-relay/ha-server.
 - `modules/common/configuration.nix` — base NixOS config imported by every
   host: locale, users, nix settings, git.
 - `modules/common/home.nix` / `hosts/nixos/home.nix` — Home Manager config for
@@ -417,7 +500,7 @@ removing a host.
   `modules/platforms/baremetal.nix` also imports
   `modules/services/zfs/enable-service.nix` for this (the `zfs_unstable`
   package, autoScrub/autoSnapshot/trim) — the only other importer today is
-  `server`'s NFS data pool, an unrelated non-root ZFS use.
+  `ha-server`'s NFS data pool, an unrelated non-root ZFS use.
 - `modules/boot/efi.nix` — systemd-boot + EFI vars, paired with the disko module.
 - `modules/installer/` — the auto-installer environment (ISO, also served as
   PXE netboot): `common.nix` (shared config + the generated
@@ -431,14 +514,21 @@ removing a host.
   substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the
   full design (per-host local stores, no shared `/nix/store`, and how the
   `nixremote` signing/SSH keys fit together).
+- `modules/ha/` — HA cluster NixOS modules: `cluster-config.nix` (DRBD,
+  Corosync, Pacemaker, firewall rules, cluster-wide NFS/iSCSI port
+  authorisation — shared by both ha-server nodes), `pacemaker-stack.nix`
+  (Pacemaker + Corosync service enablement), and supporting modules. See
+  `docs/ha.md` for the cluster operational guide.
+- `modules/ipa/client.nix` — FreeIPA client enrollment: sssd, Kerberos keytab,
+  and IPA host registration; imported by every real host via
+  `modules/common/configuration.nix`.
 - `modules/beszel/enable-agent.nix` — enables beszel-agent, sets `HUB_URL`,
   fixes the upstream `StateDirectory` bug, and wires the universal
   `beszel-token` sops secret (from `secrets/common.yaml`) into the agent's
   `environmentFile`; see `docs/beszel.md` for the full setup guide.
 - `modules/tailscale/`, `modules/docker/`, `modules/networking/`,
   `modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose,
-  single-host
-  feature modules (e.g. `docker/enable-service.nix`,
+  single-host feature modules (e.g. `docker/enable-service.nix`,
   `services/zfs/enable-service.nix`). Grep `modules/build-types/*.nix` for
   each build type's `imports` list to see which modules apply where.
 
@@ -462,3 +552,5 @@ duplicating config.
 - `docs/flake-lock-automation.md` — how `flake.lock` updates flow through CI
   (scheduled `nix flake update` PR + host-eval-on-PR workflow) and why hosts
   should track the committed lock file rather than `nixos-rebuild --upgrade-all`.
+- `docs/ha.md` — HA file-server cluster: DRBD + XFS + LIO iSCSI + NFS managed
+  by Corosync + Pacemaker; network topology; lifecycle scripts in `scripts/ha/`.