diff --git a/docs/ha-network-audit.md b/docs/ha-network-audit.md index 630486e..64db39d 100644 --- a/docs/ha-network-audit.md +++ b/docs/ha-network-audit.md @@ -3,6 +3,18 @@ **Date:** 2026-07-29 **Scope:** Read-only discovery of pve1.sweet.home host networking, HA cluster VMs (200/201), and Docker CT (105). No changes made. +> **Implementation status — 2026-07-29:** All recommendations from this audit have been +> implemented in the same session. See `docs/ip-addressing.md` for the current state. +> Key decisions that diverged from the original recommendations: +> - VLAN IDs renumbered: cluster → VLAN 10 (192.168.10.x), storage-client → VLAN 20 (192.168.20.x) +> - Two Pacemaker VIPs: `vip-lan` (192.168.2.229, NFS for LAN) and `vip-storage` (192.168.20.229, NFS + iSCSI for VLAN 20) +> - NFS served on **both** VIPs (each firewalled to its own subnet); iSCSI available on VLAN 20 but NFS is preferred for docker to support future Docker Swarm multi-host access +> - `corosync.conf` ring1 added using LAN IPs (RF-1 resolved) +> - `vmbr2` created and NICs added to HA VMs and docker CT (RF-6/RF-7 resolved) +> - iSCSI portal remains on `[::0]`; firewall enforces VLAN 20 restriction (RF-4 mitigated) +> - STONITH still disabled (RF-3 deferred — accepted risk during development phase) +> - iSCSI ACLs not configured (RF-5 deferred — iSCSI not in active use) + --- ## 1. Current State Summary diff --git a/docs/ip-addressing.md b/docs/ip-addressing.md index 0c85d61..8614408 100644 --- a/docs/ip-addressing.md +++ b/docs/ip-addressing.md @@ -6,7 +6,7 @@ |---|---|---|---|---| | LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) | | Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink | -| Storage client | 20 | `192.168.20.0/24` | HA file server iSCSI — docker and server access VIP here | No — internal `vmbr2` only, no uplink | +| Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink | The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux bridges with no physical port attached; traffic between guests on each bridge stays in-kernel. @@ -16,8 +16,12 @@ VLAN 20 → 192.168.**20**.x). The host octet is consistent across all subnets is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20.228` (storage client). **Protocol separation** (enforced by firewall on HA nodes): -- iSCSI (port 3260): storage-client subnet (VLAN 20) only — docker and server -- NFS (ports 111, 2049, 20048): LAN subnet (VLAN 2) only — pxe-boot and other LAN clients +- NFS (ports 111, 2049, 20048): both subnets, each restricted to its own CIDR + - VLAN 2 only → `vip-lan` (192.168.2.229) — pxe-boot and other LAN clients + - VLAN 20 only → `vip-storage` (192.168.20.229) — docker, future swarm nodes +- iSCSI (port 3260): VLAN 20 only — available but not in active use; NFS is preferred + for multi-host access (shared volumes across a Docker Swarm require a shared filesystem, + not per-host block devices) --- @@ -105,16 +109,18 @@ Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached. | IP | Hostname | Interface / role | |---|---|---| -| `192.168.20.229` | ha-vip-storage | Pacemaker floating VIP — iSCSI endpoint (port 3260) | +| `192.168.20.229` | ha-vip-storage | Pacemaker floating VIP — NFS + iSCSI endpoint | | `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) | | `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) | | `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — temporary, for final rsync before decommission | -| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — iSCSI initiator | +| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client | | — | no gateway | Isolated — not routed to LAN or internet | -iSCSI initiators connect to the VIP (`192.168.20.229:3260`) so sessions survive failover -transparently. Portal listens on `[::0]:3260`; firewall restricts port 3260 to -`192.168.20.0/24` source only — LAN clients cannot reach iSCSI. +NFS clients mount from `192.168.20.229` (surviving failover transparently via the VIP). +Firewall on each HA node restricts NFS and iSCSI ports to `192.168.20.0/24` — LAN hosts +cannot reach either service on this VIP. The `vip-storage` endpoint is not reachable +from the workstation directly (internal bridge only); health checks proxy through the +active HA node. --- diff --git a/docs/network-cutover.md b/docs/network-cutover.md index 631250c..294d8c1 100644 --- a/docs/network-cutover.md +++ b/docs/network-cutover.md @@ -336,7 +336,85 @@ temporarily use either. --- -## Stage 6 — Final cleanup +## Stage 6 — HA storage cutover (docker NFS remount) + +> **Prerequisite:** HA cluster fully deployed and `vip-storage` (192.168.20.229) serving +> NFS. docker CT must have eth1 on vmbr2 (already done). Final rsync from server.sweet.home +> to `/srv/ha-data` must be complete before this stage. + +docker.sweet.home currently NFS-mounts its persistent volumes from `server.sweet.home` +(`192.168.2.226:/tank/docker/...`). This stage moves those mounts to the HA cluster's +storage VIP so server can be decommissioned. + +### 6a. Final rsync from server to HA cluster + +Run from server.sweet.home (or over SSH from the workstation) to sync any data written +since the initial rsync: + +```bash +# Confirm active HA node and mount point +ssh wayne@192.168.2.228 'sudo findmnt /srv/ha-data' # check which node is active + +# rsync each dataset (adjust source paths to match /tank layout on server) +sudo rsync -av --delete /tank/docker/config/ wayne@:/srv/ha-data/docker/config/ +sudo rsync -av --delete /tank/docker/databases/ wayne@:/srv/ha-data/docker/databases/ +sudo rsync -av --delete /tank/docker/volumes/ wayne@:/srv/ha-data/docker/volumes/ +sudo rsync -av --delete /tank/docker/nextcloud-data/ wayne@:/srv/ha-data/docker/nextcloud-data/ +``` + +### 6b. Update docker NixOS config to mount from vip-storage + +In `hosts/docker/host.nix` (or wherever the NFS mount fileSystems are declared), change +the NFS server from `server.sweet.home` / `192.168.2.226` to `192.168.20.229`: + +```nix +# Before: +fileSystems."/mnt/docker/config" = { + device = "server:/tank/docker/config"; # or 192.168.2.226:... + ... +}; + +# After: +fileSystems."/mnt/docker/config" = { + device = "192.168.20.229:/srv/ha-data/docker/config"; + ... +}; +``` + +Repeat for all four docker shares (`config`, `databases`, `volumes`, `nextcloud-data`). +Then rebuild docker: + +```bash +# On the workstation — or via Switch-nix on docker itself +sudo nixos-rebuild switch --no-write-lock-file --refresh \ + --flake "git+https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos.git#lxc-docker" +``` + +### 6c. Verify mounts and container health + +```bash +ssh wayne@192.168.2.225 'findmnt | grep 192.168.20' # mounts should show vip-storage +ssh wayne@192.168.2.225 'docker ps' # all containers running +``` + +Spot-check Nextcloud, Traefik, and any database containers for connectivity. + +### 6d. Decommission server.sweet.home + +Once docker is confirmed healthy on the HA NFS mounts: + +```bash +# Stop server VM on pve1 +ssh wayne@192.168.2.245 'sudo qm stop 101' + +# (Optional) Archive the ZFS pool snapshot before destroying +# Then after a settling period: +ssh wayne@192.168.2.245 'sudo qm destroy 101 --destroy-unreferenced-disks 1' +``` + +--- + +## Stage 7 — Final cleanup Once all hosts are at their new IPs and verified: