Archived
docs(ha): update docs to reflect implemented network segmentation
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
ip-addressing.md:
- Storage-client network purpose updated: NFS primary, iSCSI available
- Protocol separation box updated: NFS on both VIPs (each CIDR-restricted);
iSCSI on VLAN 20 only but not in active use
- docker role updated to NFS client (not iSCSI initiator)
- vip-storage description updated to NFS + iSCSI endpoint
ha-network-audit.md:
- Implementation status header added noting all audit items resolved
- Records decisions that diverged from original recommendations
(VLAN renumbering, dual VIP, NFS-over-iSCSI for docker/swarm)
- Notes deferred items (STONITH, iSCSI ACLs)
network-cutover.md:
- New Stage 6: HA storage cutover — docker NFS remount from
server.sweet.home to vip-storage (192.168.20.229), final rsync
steps, and server.sweet.home decommission procedure
- Former Stage 6 (final cleanup) renumbered to Stage 7
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
This commit is contained in:
@@ -3,6 +3,18 @@
|
|||||||
**Date:** 2026-07-29
|
**Date:** 2026-07-29
|
||||||
**Scope:** Read-only discovery of pve1.sweet.home host networking, HA cluster VMs (200/201), and Docker CT (105). No changes made.
|
**Scope:** Read-only discovery of pve1.sweet.home host networking, HA cluster VMs (200/201), and Docker CT (105). No changes made.
|
||||||
|
|
||||||
|
> **Implementation status — 2026-07-29:** All recommendations from this audit have been
|
||||||
|
> implemented in the same session. See `docs/ip-addressing.md` for the current state.
|
||||||
|
> Key decisions that diverged from the original recommendations:
|
||||||
|
> - VLAN IDs renumbered: cluster → VLAN 10 (192.168.10.x), storage-client → VLAN 20 (192.168.20.x)
|
||||||
|
> - Two Pacemaker VIPs: `vip-lan` (192.168.2.229, NFS for LAN) and `vip-storage` (192.168.20.229, NFS + iSCSI for VLAN 20)
|
||||||
|
> - NFS served on **both** VIPs (each firewalled to its own subnet); iSCSI available on VLAN 20 but NFS is preferred for docker to support future Docker Swarm multi-host access
|
||||||
|
> - `corosync.conf` ring1 added using LAN IPs (RF-1 resolved)
|
||||||
|
> - `vmbr2` created and NICs added to HA VMs and docker CT (RF-6/RF-7 resolved)
|
||||||
|
> - iSCSI portal remains on `[::0]`; firewall enforces VLAN 20 restriction (RF-4 mitigated)
|
||||||
|
> - STONITH still disabled (RF-3 deferred — accepted risk during development phase)
|
||||||
|
> - iSCSI ACLs not configured (RF-5 deferred — iSCSI not in active use)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## 1. Current State Summary
|
## 1. Current State Summary
|
||||||
|
|||||||
+14
-8
@@ -6,7 +6,7 @@
|
|||||||
|---|---|---|---|---|
|
|---|---|---|---|---|
|
||||||
| LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
|
| LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
|
||||||
| Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
|
| Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
|
||||||
| Storage client | 20 | `192.168.20.0/24` | HA file server iSCSI — docker and server access VIP here | No — internal `vmbr2` only, no uplink |
|
| Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink |
|
||||||
|
|
||||||
The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
|
The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
|
||||||
bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
|
bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
|
||||||
@@ -16,8 +16,12 @@ VLAN 20 → 192.168.**20**.x). The host octet is consistent across all subnets
|
|||||||
is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20.228` (storage client).
|
is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20.228` (storage client).
|
||||||
|
|
||||||
**Protocol separation** (enforced by firewall on HA nodes):
|
**Protocol separation** (enforced by firewall on HA nodes):
|
||||||
- iSCSI (port 3260): storage-client subnet (VLAN 20) only — docker and server
|
- NFS (ports 111, 2049, 20048): both subnets, each restricted to its own CIDR
|
||||||
- NFS (ports 111, 2049, 20048): LAN subnet (VLAN 2) only — pxe-boot and other LAN clients
|
- VLAN 2 only → `vip-lan` (192.168.2.229) — pxe-boot and other LAN clients
|
||||||
|
- VLAN 20 only → `vip-storage` (192.168.20.229) — docker, future swarm nodes
|
||||||
|
- iSCSI (port 3260): VLAN 20 only — available but not in active use; NFS is preferred
|
||||||
|
for multi-host access (shared volumes across a Docker Swarm require a shared filesystem,
|
||||||
|
not per-host block devices)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
@@ -105,16 +109,18 @@ Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached.
|
|||||||
|
|
||||||
| IP | Hostname | Interface / role |
|
| IP | Hostname | Interface / role |
|
||||||
|---|---|---|
|
|---|---|---|
|
||||||
| `192.168.20.229` | ha-vip-storage | Pacemaker floating VIP — iSCSI endpoint (port 3260) |
|
| `192.168.20.229` | ha-vip-storage | Pacemaker floating VIP — NFS + iSCSI endpoint |
|
||||||
| `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
|
| `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
|
||||||
| `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
|
| `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
|
||||||
| `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — temporary, for final rsync before decommission |
|
| `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — temporary, for final rsync before decommission |
|
||||||
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — iSCSI initiator |
|
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client |
|
||||||
| — | no gateway | Isolated — not routed to LAN or internet |
|
| — | no gateway | Isolated — not routed to LAN or internet |
|
||||||
|
|
||||||
iSCSI initiators connect to the VIP (`192.168.20.229:3260`) so sessions survive failover
|
NFS clients mount from `192.168.20.229` (surviving failover transparently via the VIP).
|
||||||
transparently. Portal listens on `[::0]:3260`; firewall restricts port 3260 to
|
Firewall on each HA node restricts NFS and iSCSI ports to `192.168.20.0/24` — LAN hosts
|
||||||
`192.168.20.0/24` source only — LAN clients cannot reach iSCSI.
|
cannot reach either service on this VIP. The `vip-storage` endpoint is not reachable
|
||||||
|
from the workstation directly (internal bridge only); health checks proxy through the
|
||||||
|
active HA node.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|||||||
+79
-1
@@ -336,7 +336,85 @@ temporarily use either.
|
|||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
## Stage 6 — Final cleanup
|
## Stage 6 — HA storage cutover (docker NFS remount)
|
||||||
|
|
||||||
|
> **Prerequisite:** HA cluster fully deployed and `vip-storage` (192.168.20.229) serving
|
||||||
|
> NFS. docker CT must have eth1 on vmbr2 (already done). Final rsync from server.sweet.home
|
||||||
|
> to `/srv/ha-data` must be complete before this stage.
|
||||||
|
|
||||||
|
docker.sweet.home currently NFS-mounts its persistent volumes from `server.sweet.home`
|
||||||
|
(`192.168.2.226:/tank/docker/...`). This stage moves those mounts to the HA cluster's
|
||||||
|
storage VIP so server can be decommissioned.
|
||||||
|
|
||||||
|
### 6a. Final rsync from server to HA cluster
|
||||||
|
|
||||||
|
Run from server.sweet.home (or over SSH from the workstation) to sync any data written
|
||||||
|
since the initial rsync:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Confirm active HA node and mount point
|
||||||
|
ssh wayne@192.168.2.228 'sudo findmnt /srv/ha-data' # check which node is active
|
||||||
|
|
||||||
|
# rsync each dataset (adjust source paths to match /tank layout on server)
|
||||||
|
sudo rsync -av --delete /tank/docker/config/ wayne@<active-node-ip>:/srv/ha-data/docker/config/
|
||||||
|
sudo rsync -av --delete /tank/docker/databases/ wayne@<active-node-ip>:/srv/ha-data/docker/databases/
|
||||||
|
sudo rsync -av --delete /tank/docker/volumes/ wayne@<active-node-ip>:/srv/ha-data/docker/volumes/
|
||||||
|
sudo rsync -av --delete /tank/docker/nextcloud-data/ wayne@<active-node-ip>:/srv/ha-data/docker/nextcloud-data/
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6b. Update docker NixOS config to mount from vip-storage
|
||||||
|
|
||||||
|
In `hosts/docker/host.nix` (or wherever the NFS mount fileSystems are declared), change
|
||||||
|
the NFS server from `server.sweet.home` / `192.168.2.226` to `192.168.20.229`:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
# Before:
|
||||||
|
fileSystems."/mnt/docker/config" = {
|
||||||
|
device = "server:/tank/docker/config"; # or 192.168.2.226:...
|
||||||
|
...
|
||||||
|
};
|
||||||
|
|
||||||
|
# After:
|
||||||
|
fileSystems."/mnt/docker/config" = {
|
||||||
|
device = "192.168.20.229:/srv/ha-data/docker/config";
|
||||||
|
...
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
Repeat for all four docker shares (`config`, `databases`, `volumes`, `nextcloud-data`).
|
||||||
|
Then rebuild docker:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On the workstation — or via Switch-nix on docker itself
|
||||||
|
sudo nixos-rebuild switch --no-write-lock-file --refresh \
|
||||||
|
--flake "git+https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos.git#lxc-docker"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6c. Verify mounts and container health
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh wayne@192.168.2.225 'findmnt | grep 192.168.20' # mounts should show vip-storage
|
||||||
|
ssh wayne@192.168.2.225 'docker ps' # all containers running
|
||||||
|
```
|
||||||
|
|
||||||
|
Spot-check Nextcloud, Traefik, and any database containers for connectivity.
|
||||||
|
|
||||||
|
### 6d. Decommission server.sweet.home
|
||||||
|
|
||||||
|
Once docker is confirmed healthy on the HA NFS mounts:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Stop server VM on pve1
|
||||||
|
ssh wayne@192.168.2.245 'sudo qm stop 101'
|
||||||
|
|
||||||
|
# (Optional) Archive the ZFS pool snapshot before destroying
|
||||||
|
# Then after a settling period:
|
||||||
|
ssh wayne@192.168.2.245 'sudo qm destroy 101 --destroy-unreferenced-disks 1'
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Stage 7 — Final cleanup
|
||||||
|
|
||||||
Once all hosts are at their new IPs and verified:
|
Once all hosts are at their new IPs and verified:
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user