Archived
beatzaplentyandClaude Sonnet 4.6
cded77919d
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
- variables.nix: switch to rec {}, extract giteaDomain/giteaRepoPath,
extraAdminSshKeys, haLanNfsFqdn, tailscaleResolverIp, ports.dhcp,
ports.dns; ipaServer now derives from homeDomain ref; section headers
- modules: use new vars throughout (pxe-boot, ts-dns-forwarder,
cluster-config, configuration.nix, mount-pxe-images) — eval unchanged
- docs: delete ephemeral planning docs (AUDIT_REPORT, ha-network-audit,
network-cutover); add docs/ha.md; drop migration reference table from
ip-addressing.md; remove stale server example from beszel.md
- CLAUDE.md/README.md/AGENTS.md: fix build types (tailscale-router,
ha-server, drop server); document scripts/ha/, scripts/ipa/, and
all previously undocumented top-level and lib scripts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
161 lines
5.3 KiB
Markdown
161 lines
5.3 KiB
Markdown
# HA File-Server Cluster
|
|
|
|
Two `proxmox-ha-server-{1,2}` VMs form an active/passive file-server cluster:
|
|
DRBD replicates a block device between nodes; Corosync + Pacemaker manage
|
|
failover; XFS, LIO iSCSI, and NFS are brought up as a collocated resource
|
|
group on whichever node holds the DRBD Primary role.
|
|
|
|
NixOS modules: `modules/ha/`. Lifecycle scripts: `scripts/ha/`.
|
|
Cluster-wide constants: `variables.nix` (`haServer*` vars).
|
|
|
|
---
|
|
|
|
## Network layout
|
|
|
|
Three subnets — all internal to pve1 (`vmbr0`/`vmbr1`/`vmbr2`):
|
|
|
|
| Subnet | VLAN | CIDR | Bridge | Purpose |
|
|
|---|---|---|---|---|
|
|
| LAN | 2 | `192.168.2.0/24` | `vmbr0` | Management, LAN NFS |
|
|
| Cluster | 10 | `192.168.10.224/29` | `vmbr1` | Corosync ring0 + DRBD replication |
|
|
| Storage-client | 20 | `192.168.20.0/24` | `vmbr2` | NFS + iSCSI for docker/swarm |
|
|
|
|
Each HA VM has three NICs: `ens18` (LAN/vmbr0), `ens19` (cluster/vmbr1),
|
|
`ens20` (storage-client/vmbr2). See `docs/ip-addressing.md` for all IPs.
|
|
|
|
Corosync ring0 uses the cluster NIC; ring1 (backup heartbeat) uses the LAN
|
|
NIC. DRBD replicates over the cluster NIC. No storage traffic crosses the LAN.
|
|
|
|
---
|
|
|
|
## Pacemaker resources
|
|
|
|
All resources run collocated on whichever node is Primary, in this order:
|
|
|
|
```
|
|
ms-drbd0 (promotable DRBD clone)
|
|
→ xfs-data (XFS mount on /dev/drbd0 → /srv/ha-data)
|
|
→ iscsi-target (targetctl)
|
|
→ nfs-server (nfs-server.service)
|
|
→ vip-lan (192.168.2.229/24 on vmbr0 — NFS for LAN clients)
|
|
→ vip-storage (192.168.20.229/24 on vmbr2 — NFS + iSCSI for VLAN 20)
|
|
```
|
|
|
|
`vip-lan` serves pxe-boot and other LAN-only NFS clients.
|
|
`vip-storage` serves docker and any future swarm nodes; iSCSI is available on
|
|
VLAN 20 but NFS is preferred for multi-host volume sharing.
|
|
|
|
---
|
|
|
|
## DRBD fencing
|
|
|
|
`fencing resource-only` with `crm-fence-peer.sh`/`crm-unfence-peer.sh`
|
|
wrappers (`modules/ha/cluster-config.nix`). The DRBD kernel module invokes
|
|
these via the User Mode Helper with a minimal PATH; the wrappers prepend
|
|
`/run/current-system/sw/bin` before exec-ing the real handlers so Pacemaker
|
|
tools (`cibadmin`, `crm_mon`, etc.) are found.
|
|
|
|
STONITH is initially disabled (`stonith-enabled: false`,
|
|
`no-quorum-policy: ignore`). Enable it once the `fence_pve_ssh` fence agent
|
|
(`scripts/ha/fence-pve-ssh.py`) is deployed and authorised:
|
|
|
|
```bash
|
|
scripts/ha/cluster-enable-stonith.sh # run as root on ha-server-1
|
|
```
|
|
|
|
---
|
|
|
|
## Deploying the cluster from scratch
|
|
|
|
Use `scripts/ha/deploy.sh` — it orchestrates all phases:
|
|
|
|
```bash
|
|
# Against pve-test (safe — Claude's default target):
|
|
scripts/ha/deploy.sh --node "$PVE_TEST_HOST" [--dry-run]
|
|
|
|
# Against pve1 (production — requires explicit operator go-ahead):
|
|
scripts/ha/deploy.sh --node "$PVE1_HOST"
|
|
```
|
|
|
|
Phases (each skippable with `--skip-<phase>`):
|
|
1. `ensure-bridge` — creates `vmbr1`/`vmbr2` on the Proxmox node if absent
|
|
2. `sync-keys` — generates SSH host keys for both nodes; registers sops recipients
|
|
3. `create-vms` — builds disk images, creates VMs via `create-proxmox-resource.sh`
|
|
4. `add-hardware` — attaches storage NIC and DRBD data disk to each VM
|
|
5. `init-cluster` — runs `scripts/ha/cluster-init.sh` on ha-server-1
|
|
|
|
`--destroy` runs the teardown sequence.
|
|
|
|
---
|
|
|
|
## Day-to-day operations
|
|
|
|
```bash
|
|
# Read-only health check (safe from workstation):
|
|
scripts/ha/health.sh
|
|
|
|
# Graceful failover (prompts for confirmation):
|
|
scripts/ha/failover.sh [--to node1|node2]
|
|
|
|
# Online data-disk growth (no downtime):
|
|
scripts/ha/resize-data-disk.sh --size +20G
|
|
|
|
# Acceptance tests (run after any significant change):
|
|
scripts/ha/acceptance-tests.sh
|
|
```
|
|
|
|
---
|
|
|
|
## Adding FreeIPA host accounts
|
|
|
|
IPA host registration is automated:
|
|
|
|
```bash
|
|
scripts/ipa/create-nixos-ipa-host-account.sh <hostname>
|
|
```
|
|
|
|
This runs `ipa host-add`, fetches a keytab from the domain controller, and
|
|
writes a sops-encrypted `secrets/<hostname>.keytab` in one step. The module
|
|
`modules/ipa/client.nix` (imported by every host via
|
|
`modules/common/configuration.nix`) consumes the keytab via sops-nix.
|
|
|
|
---
|
|
|
|
## Storage layout
|
|
|
|
```
|
|
/srv/ha-data/
|
|
docker/
|
|
config/ NFS → docker:/mnt/docker/config
|
|
databases/ NFS → docker:/mnt/docker/databases
|
|
volumes/ NFS → docker:/mnt/docker/volumes
|
|
nextcloud-data/ NFS → docker:/mnt/docker/nextcloud-data
|
|
proxmox/
|
|
iso/ NFS → pve1 ISO storage
|
|
lxc/ NFS → pve1 CT template storage
|
|
pxe-boot/
|
|
images/ NFS → pxe-boot:/srv/pxe/http/images (PXE assets)
|
|
raspi/
|
|
volumes/ NFS → raspi NFS mounts
|
|
iscsi-lun.img iSCSI fileio backstore (VLAN 20 only, not in active use)
|
|
```
|
|
|
|
All shares are defined in `variables.nix` (`vars.nfsShares.*`). The NFS
|
|
export list lives in `modules/ha/nfs-exports.nix`.
|
|
|
|
---
|
|
|
|
## Key variables
|
|
|
|
| Variable | Description |
|
|
|---|---|
|
|
| `vars.haServer1Ip` / `vars.haServer2Ip` | LAN management IPs |
|
|
| `vars.haServer1StorageIp` / `vars.haServer2StorageIp` | Cluster NIC IPs (DRBD/Corosync ring0) |
|
|
| `vars.haServerLanVip` | Pacemaker `vip-lan` — NFS for LAN (192.168.2.229) |
|
|
| `vars.haServerVip` | Pacemaker `vip-storage` — NFS + iSCSI for VLAN 20 (192.168.20.229) |
|
|
| `vars.haLanNfsFqdn` | FQDN of `vip-lan`: `ha-vip-lan.sweet.home` |
|
|
| `vars.haStorageRoot` | XFS mount point: `/srv/ha-data` |
|
|
| `vars.haServerDrbdDisk` | Block device for DRBD backing store |
|
|
| `vars.haStorageCidr` | Cluster subnet CIDR (`192.168.10.224/29`) |
|
|
| `vars.haClientCidr` | Storage-client subnet CIDR (`192.168.20.0/24`) |
|