# HA File-Server Cluster

Two `proxmox-ha-server-{1,2}` VMs form an active/passive file-server cluster:
DRBD replicates a block device between nodes; Corosync + Pacemaker manage
failover; XFS, LIO iSCSI, and NFS are brought up as a collocated resource
group on whichever node holds the DRBD Primary role.

NixOS modules: `modules/ha/`. Lifecycle scripts: `scripts/ha/`.
Cluster-wide constants: `variables.nix` (`haServer*` vars).

---

## Network layout

Three subnets — all internal to pve1 (`vmbr0`/`vmbr1`/`vmbr2`):

| Subnet | VLAN | CIDR | Bridge | Purpose |
|---|---|---|---|---|
| LAN | 2 | `192.168.2.0/24` | `vmbr0` | Management, LAN NFS |
| Cluster | 10 | `192.168.10.224/29` | `vmbr1` | Corosync ring0 + DRBD replication |
| Storage-client | 20 | `192.168.20.0/24` | `vmbr2` | NFS + iSCSI for docker/swarm |

Each HA VM has three NICs: `ens18` (LAN/vmbr0), `ens19` (cluster/vmbr1),
`ens20` (storage-client/vmbr2). See `docs/ip-addressing.md` for all IPs.

Corosync ring0 uses the cluster NIC; ring1 (backup heartbeat) uses the LAN
NIC. DRBD replicates over the cluster NIC. No storage traffic crosses the LAN.

---

## Pacemaker resources

All resources run collocated on whichever node is Primary, in this order:

```
ms-drbd0 (promotable DRBD clone)
  → xfs-data   (XFS mount on /dev/drbd0 → /srv/ha-data)
  → iscsi-target (targetctl)
  → nfs-server  (nfs-server.service)
  → vip-lan     (192.168.2.229/24 on vmbr0 — NFS for LAN clients)
  → vip-storage (192.168.20.229/24 on vmbr2 — NFS + iSCSI for VLAN 20)
```

`vip-lan` serves pxe-boot and other LAN-only NFS clients.
`vip-storage` serves docker and any future swarm nodes; iSCSI is available on
VLAN 20 but NFS is preferred for multi-host volume sharing.

---

## DRBD fencing

`fencing resource-only` with `crm-fence-peer.sh`/`crm-unfence-peer.sh`
wrappers (`modules/ha/cluster-config.nix`). The DRBD kernel module invokes
these via the User Mode Helper with a minimal PATH; the wrappers prepend
`/run/current-system/sw/bin` before exec-ing the real handlers so Pacemaker
tools (`cibadmin`, `crm_mon`, etc.) are found.

STONITH is initially disabled (`stonith-enabled: false`,
`no-quorum-policy: ignore`). Enable it once the `fence_pve_ssh` fence agent
(`scripts/ha/fence-pve-ssh.py`) is deployed and authorised:

```bash
scripts/ha/cluster-enable-stonith.sh   # run as root on ha-server-1
```

---

## Deploying the cluster from scratch

Use `scripts/ha/deploy.sh` — it orchestrates all phases:

```bash
# Against pve-test (safe — Claude's default target):
scripts/ha/deploy.sh --node "$PVE_TEST_HOST" [--dry-run]

# Against pve1 (production — requires explicit operator go-ahead):
scripts/ha/deploy.sh --node "$PVE1_HOST"
```

Phases (each skippable with `--skip-<phase>`):
1. `ensure-bridge` — creates `vmbr1`/`vmbr2` on the Proxmox node if absent
2. `sync-keys` — generates SSH host keys for both nodes; registers sops recipients
3. `create-vms` — builds disk images, creates VMs via `create-proxmox-resource.sh`
4. `add-hardware` — attaches storage NIC and DRBD data disk to each VM
5. `init-cluster` — runs `scripts/ha/cluster-init.sh` on ha-server-1

`--destroy` runs the teardown sequence.

---

## Day-to-day operations

```bash
# Read-only health check (safe from workstation):
scripts/ha/health.sh

# Graceful failover (prompts for confirmation):
scripts/ha/failover.sh [--to node1|node2]

# Online data-disk growth (no downtime):
scripts/ha/resize-data-disk.sh --size +20G

# Acceptance tests (run after any significant change):
scripts/ha/acceptance-tests.sh
```

---

## Adding FreeIPA host accounts

IPA host registration is automated:

```bash
scripts/ipa/create-nixos-ipa-host-account.sh <hostname>
```

This runs `ipa host-add`, fetches a keytab from the domain controller, and
writes a sops-encrypted `secrets/<hostname>.keytab` in one step. The module
`modules/ipa/client.nix` (imported by every host via
`modules/common/configuration.nix`) consumes the keytab via sops-nix.

---

## Storage layout

```
/srv/ha-data/
  docker/
    config/          NFS → docker:/mnt/docker/config
    databases/       NFS → docker:/mnt/docker/databases
    volumes/         NFS → docker:/mnt/docker/volumes
    nextcloud-data/  NFS → docker:/mnt/docker/nextcloud-data
  proxmox/
    iso/             NFS → pve1 ISO storage
    lxc/             NFS → pve1 CT template storage
  pxe-boot/
    images/          NFS → pxe-boot:/srv/pxe/http/images (PXE assets)
  raspi/
    volumes/         NFS → raspi NFS mounts
  iscsi-lun.img      iSCSI fileio backstore (VLAN 20 only, not in active use)
```

All shares are defined in `variables.nix` (`vars.nfsShares.*`). The NFS
export list lives in `modules/ha/nfs-exports.nix`.

---

## Key variables

| Variable | Description |
|---|---|
| `vars.haServer1Ip` / `vars.haServer2Ip` | LAN management IPs |
| `vars.haServer1StorageIp` / `vars.haServer2StorageIp` | Cluster NIC IPs (DRBD/Corosync ring0) |
| `vars.haServerLanVip` | Pacemaker `vip-lan` — NFS for LAN (192.168.2.229) |
| `vars.haServerVip` | Pacemaker `vip-storage` — NFS + iSCSI for VLAN 20 (192.168.20.229) |
| `vars.haLanNfsFqdn` | FQDN of `vip-lan`: `ha-vip-lan.sweet.home` |
| `vars.haStorageRoot` | XFS mount point: `/srv/ha-data` |
| `vars.haServerDrbdDisk` | Block device for DRBD backing store |
| `vars.haStorageCidr` | Cluster subnet CIDR (`192.168.10.224/29`) |
| `vars.haClientCidr` | Storage-client subnet CIDR (`192.168.20.0/24`) |