Archived
refactor: full repo sweep — variables, docs, and comment cleanup
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
- variables.nix: switch to rec {}, extract giteaDomain/giteaRepoPath,
extraAdminSshKeys, haLanNfsFqdn, tailscaleResolverIp, ports.dhcp,
ports.dns; ipaServer now derives from homeDomain ref; section headers
- modules: use new vars throughout (pxe-boot, ts-dns-forwarder,
cluster-config, configuration.nix, mount-pxe-images) — eval unchanged
- docs: delete ephemeral planning docs (AUDIT_REPORT, ha-network-audit,
network-cutover); add docs/ha.md; drop migration reference table from
ip-addressing.md; remove stale server example from beszel.md
- CLAUDE.md/README.md/AGENTS.md: fix build types (tailscale-router,
ha-server, drop server); document scripts/ha/, scripts/ipa/, and
all previously undocumented top-level and lib scripts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
Claude Sonnet 4.6
+160
@@ -0,0 +1,160 @@
|
||||
# HA File-Server Cluster
|
||||
|
||||
Two `proxmox-ha-server-{1,2}` VMs form an active/passive file-server cluster:
|
||||
DRBD replicates a block device between nodes; Corosync + Pacemaker manage
|
||||
failover; XFS, LIO iSCSI, and NFS are brought up as a collocated resource
|
||||
group on whichever node holds the DRBD Primary role.
|
||||
|
||||
NixOS modules: `modules/ha/`. Lifecycle scripts: `scripts/ha/`.
|
||||
Cluster-wide constants: `variables.nix` (`haServer*` vars).
|
||||
|
||||
---
|
||||
|
||||
## Network layout
|
||||
|
||||
Three subnets — all internal to pve1 (`vmbr0`/`vmbr1`/`vmbr2`):
|
||||
|
||||
| Subnet | VLAN | CIDR | Bridge | Purpose |
|
||||
|---|---|---|---|---|
|
||||
| LAN | 2 | `192.168.2.0/24` | `vmbr0` | Management, LAN NFS |
|
||||
| Cluster | 10 | `192.168.10.224/29` | `vmbr1` | Corosync ring0 + DRBD replication |
|
||||
| Storage-client | 20 | `192.168.20.0/24` | `vmbr2` | NFS + iSCSI for docker/swarm |
|
||||
|
||||
Each HA VM has three NICs: `ens18` (LAN/vmbr0), `ens19` (cluster/vmbr1),
|
||||
`ens20` (storage-client/vmbr2). See `docs/ip-addressing.md` for all IPs.
|
||||
|
||||
Corosync ring0 uses the cluster NIC; ring1 (backup heartbeat) uses the LAN
|
||||
NIC. DRBD replicates over the cluster NIC. No storage traffic crosses the LAN.
|
||||
|
||||
---
|
||||
|
||||
## Pacemaker resources
|
||||
|
||||
All resources run collocated on whichever node is Primary, in this order:
|
||||
|
||||
```
|
||||
ms-drbd0 (promotable DRBD clone)
|
||||
→ xfs-data (XFS mount on /dev/drbd0 → /srv/ha-data)
|
||||
→ iscsi-target (targetctl)
|
||||
→ nfs-server (nfs-server.service)
|
||||
→ vip-lan (192.168.2.229/24 on vmbr0 — NFS for LAN clients)
|
||||
→ vip-storage (192.168.20.229/24 on vmbr2 — NFS + iSCSI for VLAN 20)
|
||||
```
|
||||
|
||||
`vip-lan` serves pxe-boot and other LAN-only NFS clients.
|
||||
`vip-storage` serves docker and any future swarm nodes; iSCSI is available on
|
||||
VLAN 20 but NFS is preferred for multi-host volume sharing.
|
||||
|
||||
---
|
||||
|
||||
## DRBD fencing
|
||||
|
||||
`fencing resource-only` with `crm-fence-peer.sh`/`crm-unfence-peer.sh`
|
||||
wrappers (`modules/ha/cluster-config.nix`). The DRBD kernel module invokes
|
||||
these via the User Mode Helper with a minimal PATH; the wrappers prepend
|
||||
`/run/current-system/sw/bin` before exec-ing the real handlers so Pacemaker
|
||||
tools (`cibadmin`, `crm_mon`, etc.) are found.
|
||||
|
||||
STONITH is initially disabled (`stonith-enabled: false`,
|
||||
`no-quorum-policy: ignore`). Enable it once the `fence_pve_ssh` fence agent
|
||||
(`scripts/ha/fence-pve-ssh.py`) is deployed and authorised:
|
||||
|
||||
```bash
|
||||
scripts/ha/cluster-enable-stonith.sh # run as root on ha-server-1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deploying the cluster from scratch
|
||||
|
||||
Use `scripts/ha/deploy.sh` — it orchestrates all phases:
|
||||
|
||||
```bash
|
||||
# Against pve-test (safe — Claude's default target):
|
||||
scripts/ha/deploy.sh --node "$PVE_TEST_HOST" [--dry-run]
|
||||
|
||||
# Against pve1 (production — requires explicit operator go-ahead):
|
||||
scripts/ha/deploy.sh --node "$PVE1_HOST"
|
||||
```
|
||||
|
||||
Phases (each skippable with `--skip-<phase>`):
|
||||
1. `ensure-bridge` — creates `vmbr1`/`vmbr2` on the Proxmox node if absent
|
||||
2. `sync-keys` — generates SSH host keys for both nodes; registers sops recipients
|
||||
3. `create-vms` — builds disk images, creates VMs via `create-proxmox-resource.sh`
|
||||
4. `add-hardware` — attaches storage NIC and DRBD data disk to each VM
|
||||
5. `init-cluster` — runs `scripts/ha/cluster-init.sh` on ha-server-1
|
||||
|
||||
`--destroy` runs the teardown sequence.
|
||||
|
||||
---
|
||||
|
||||
## Day-to-day operations
|
||||
|
||||
```bash
|
||||
# Read-only health check (safe from workstation):
|
||||
scripts/ha/health.sh
|
||||
|
||||
# Graceful failover (prompts for confirmation):
|
||||
scripts/ha/failover.sh [--to node1|node2]
|
||||
|
||||
# Online data-disk growth (no downtime):
|
||||
scripts/ha/resize-data-disk.sh --size +20G
|
||||
|
||||
# Acceptance tests (run after any significant change):
|
||||
scripts/ha/acceptance-tests.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding FreeIPA host accounts
|
||||
|
||||
IPA host registration is automated:
|
||||
|
||||
```bash
|
||||
scripts/ipa/create-nixos-ipa-host-account.sh <hostname>
|
||||
```
|
||||
|
||||
This runs `ipa host-add`, fetches a keytab from the domain controller, and
|
||||
writes a sops-encrypted `secrets/<hostname>.keytab` in one step. The module
|
||||
`modules/ipa/client.nix` (imported by every host via
|
||||
`modules/common/configuration.nix`) consumes the keytab via sops-nix.
|
||||
|
||||
---
|
||||
|
||||
## Storage layout
|
||||
|
||||
```
|
||||
/srv/ha-data/
|
||||
docker/
|
||||
config/ NFS → docker:/mnt/docker/config
|
||||
databases/ NFS → docker:/mnt/docker/databases
|
||||
volumes/ NFS → docker:/mnt/docker/volumes
|
||||
nextcloud-data/ NFS → docker:/mnt/docker/nextcloud-data
|
||||
proxmox/
|
||||
iso/ NFS → pve1 ISO storage
|
||||
lxc/ NFS → pve1 CT template storage
|
||||
pxe-boot/
|
||||
images/ NFS → pxe-boot:/srv/pxe/http/images (PXE assets)
|
||||
raspi/
|
||||
volumes/ NFS → raspi NFS mounts
|
||||
iscsi-lun.img iSCSI fileio backstore (VLAN 20 only, not in active use)
|
||||
```
|
||||
|
||||
All shares are defined in `variables.nix` (`vars.nfsShares.*`). The NFS
|
||||
export list lives in `modules/ha/nfs-exports.nix`.
|
||||
|
||||
---
|
||||
|
||||
## Key variables
|
||||
|
||||
| Variable | Description |
|
||||
|---|---|
|
||||
| `vars.haServer1Ip` / `vars.haServer2Ip` | LAN management IPs |
|
||||
| `vars.haServer1StorageIp` / `vars.haServer2StorageIp` | Cluster NIC IPs (DRBD/Corosync ring0) |
|
||||
| `vars.haServerLanVip` | Pacemaker `vip-lan` — NFS for LAN (192.168.2.229) |
|
||||
| `vars.haServerVip` | Pacemaker `vip-storage` — NFS + iSCSI for VLAN 20 (192.168.20.229) |
|
||||
| `vars.haLanNfsFqdn` | FQDN of `vip-lan`: `ha-vip-lan.sweet.home` |
|
||||
| `vars.haStorageRoot` | XFS mount point: `/srv/ha-data` |
|
||||
| `vars.haServerDrbdDisk` | Block device for DRBD backing store |
|
||||
| `vars.haStorageCidr` | Cluster subnet CIDR (`192.168.10.224/29`) |
|
||||
| `vars.haClientCidr` | Storage-client subnet CIDR (`192.168.20.0/24`) |
|
||||
Reference in New Issue
Block a user