docs(ha): add network segmentation audit report

Read-only discovery of pve1 bridges, HA cluster VMs (200/201), and Docker CT (105) — current state, risk flags, recommended vmbr2 storage bridge layout, gap list, and migration notes. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 15:38:21 +10:00
parent a5fb0404e2
commit 8e58226d2b
1 changed files with 249 additions and 0 deletions
@@ -0,0 +1,249 @@
+# Storage/Cluster Network Segmentation Audit — pve1.sweet.home
+
+**Date:** 2026-07-29  
+**Scope:** Read-only discovery of pve1.sweet.home host networking, HA cluster VMs (200/201), and Docker CT (105). No changes made.
+
+---
+
+## 1. Current State Summary
+
+### pve1.sweet.home Host — Physical NICs
+
+| Interface | Speed/Duplex | Notes |
+|-----------|-------------|-------|
+| `nic0` | 2500 Mb/s / Full (2.5GbE) | Only active physical NIC; sole bridge port for vmbr0 |
+| `nic1` | (not connected / no data) | Present in config, not UP |
+| `wlp4s0` | DOWN | WiFi, unused |
+
+No bonding configured. Every guest's traffic ultimately funnels through the single 2.5GbE `nic0`.
+
+### Proxmox Bridges
+
+| Bridge | Physical NIC | Host IP | Subnet | VLAN-aware | Purpose (current) |
+|--------|-------------|---------|--------|------------|-------------------|
+| `vmbr0` | `nic0` (2.5GbE) | 192.168.2.245/24 | 192.168.2.0/24 | No | General LAN, management, **iSCSI/NFS VIP** |
+| `vmbr1` | **none** (internal-only) | — | 192.168.4.224/29 | No | Corosync heartbeat + DRBD replication |
+
+`vmbr1` has `bridge-ports none` in `/etc/network/interfaces.d/vmbr1.conf` — it is a purely software bridge with zero physical uplink. All traffic on it stays inside the hypervisor's memory.
+
+### Guest NIC Assignments
+
+| Guest | VMID | Role | NIC | Bridge | IP | Traffic type |
+|-------|------|------|-----|--------|----|--------------|
+| nix-cache | CT 102 | Build cache | eth0 | vmbr0 | DHCP/LAN | LAN |
+| pxe-boot | CT 103 | PXE/TFTP | eth0 | vmbr0 | DHCP/LAN | LAN |
+| tor-relay | CT 104 | Tor | eth0 | vmbr0 | DHCP/LAN | LAN |
+| **docker** | **CT 105** | **Docker host** | **eth0** | **vmbr0** | **192.168.2.225/24** | **LAN only** |
+| pdm | CT 106 | Proxmox mgmt | eth0 | vmbr0 | 192.168.2.220/24 | LAN |
+| server | VM 101 | General server | net0 | vmbr0 | DHCP/LAN | LAN |
+| tailscale-router | VM 107 | Tailscale exit | net0 | vmbr0 | DHCP/LAN | LAN |
+| domain-controller | VM 108 | FreeIPA | net0 | vmbr0 | 192.168.2.253/24 | LAN |
+| **ha-server-1** | **VM 200** | **HA primary** | **net0** | **vmbr0** | **192.168.2.228/24 + VIP 192.168.2.229/24** | **LAN + VIP** |
+| **ha-server-1** | **VM 200** | **HA primary** | **net1** | **vmbr1** | **192.168.4.228/29** | **Corosync + DRBD** |
+| **ha-server-2** | **VM 201** | **HA secondary** | **net0** | **vmbr0** | **192.168.2.227/24** | **LAN** |
+| **ha-server-2** | **VM 201** | **HA secondary** | **net1** | **vmbr1** | **192.168.4.227/29** | **Corosync + DRBD** |
+
+### Corosync/Pacemaker State
+
+- **Transport:** knet/UDP
+- **Rings:** 1 only — ring0 on `192.168.4.228` / `192.168.4.227` (vmbr1/ens19)
+- **Cluster status:** Both nodes online, DC = ha-server-1, quorum achieved
+- **STONITH:** `stonith-enabled: false`
+- **no-quorum-policy:** `ignore`
+- **Resources (all active on ha-server-1):**
+  - `ms-drbd0` — promotable DRBD clone (Primary: ha-server-1, Secondary: ha-server-2)
+  - `xfs-data` — XFS on `/dev/drbd0` → `/srv/ha-data`
+  - `iscsi-target` — targetctl service
+  - `nfs-server` — nfs-server service
+  - `vip` — IPaddr2 at **192.168.2.229/24** (no `nic=` parameter specified; floats to ens18/vmbr0 automatically based on subnet match)
+- **Resource ordering:** ha-group starts only after DRBD is promoted; collocated with Promoted DRBD clone.
+
+### DRBD State
+
+- **Resource:** `ha-data` (DRBD 8.4.11 kernel module, config in `/etc/drbd.conf`)
+- **Protocol:** C (synchronous)
+- **Replication endpoints:**
+  - ha-server-1: `192.168.4.228:7789` (ens19 / vmbr1)
+  - ha-server-2: `192.168.4.227:7789` (ens19 / vmbr1)
+- **State at audit time:** Initial sync in progress — ~20% complete, ~40 MB/s, ~34 min remaining (100 GB disk)
+- **Fencing config:** `fencing resource-only` + fence-peer/unfence-peer handlers
+
+### iSCSI Target
+
+- **IQN:** `iqn.2026-01.home.sweet:ha-storage`
+- **Portal:** `[::0]:3260` — confirmed listening on all interfaces (`ss -tnlp` shows `*:3260 *:*`)
+- **LUN 0:** fileio backstore — `/srv/ha-data/iscsi-lun.img` (10 GiB, write-thru)
+- **ACLs:** **None** (`no-gen-acls`, `no-auth`)
+- **VIP (intended portal):** 192.168.2.229 — on vmbr0/LAN, no storage-NIC-specific binding
+
+### NFS Exports
+
+Served from the same `ha-group` as iSCSI (starts/stops together):
+
+| Export path | Client subnet |
+|------------|---------------|
+| `/srv/ha-data/docker/{config,databases,volumes,nextcloud-data}` | 192.168.2.0/24 |
+| `/srv/ha-data/proxmox/{iso,lxc}` | 192.168.2.0/24 |
+| `/srv/ha-data/pxe-boot/images` | 192.168.2.0/24 |
+| `/srv/ha-data/raspi/volumes` | 192.168.2.0/24 |
+
+All NFS exports restrict to 192.168.2.0/24 and are served via the VIP at 192.168.2.229. `rw,sync,no_subtree_check,no_root_squash`.
+
+### Docker Host Current State
+
+- **Single NIC:** eth0 on vmbr0, 192.168.2.225/24, gateway 192.168.2.254
+- **Route to storage network (192.168.4.x):** none — no NIC and no route
+- **iSCSI sessions:** none
+- **iSCSI nodes discovered:** none
+- **Docker networks:** several active compose-project networks (core_traefik, core_nextcloud, core_passbolt, core_gramps, core_docker-socket-proxy, plus CI isolation networks)
+
+---
+
+## 2. Risk Flags
+
+### RF-1: Corosync has only one ring (no heartbeat path redundancy)
+
+`corosync.conf` defines only `ring0_addr` for each node, using 192.168.4.x on vmbr1. No `ring1_addr` / second knet link is configured. On a single Proxmox host, vmbr1 is a software bridge (no physical NIC), so physical link failure is not the concern — but a kernel network stack hiccup, a `pveproxy` restart dropping bridge state, or vmbr1 getting disrupted during heavy DRBD sync all leave corosync with zero fallback path. Missed heartbeats on a two-node cluster with `no-quorum-policy: ignore` do not cause a clean shutdown; they cause a false failover or split-brain.
+
+Adding the LAN addresses (192.168.2.228 / 192.168.2.227 via ens18/vmbr0) as a second knet link would provide a backup path with no infrastructure changes needed.
+
+### RF-2: Corosync heartbeat and DRBD replication share vmbr1 — no isolation between them
+
+Both corosync (knet/UDP, ~1 kB heartbeat packets every ~100 ms) and DRBD replication (protocol C, synchronous, syncing at ~40 MB/s on a 100 GB initial fill at audit time) traverse the same `vmbr1` virtual bridge and terminate on the same ens19 NIC pair inside each HA VM. Under heavy DRBD write load, the guest-kernel scheduler's NIC transmit queue processes both flows together. While corosync's heartbeat is tiny, the absence of QoS/priority marking on vmbr1 means a DRBD burst can delay a heartbeat enough to trigger a ring fault warning. This is a latent risk that grows under high-write workloads.
+
+### RF-3: STONITH disabled — split-brain protection relies solely on DRBD's resource-only fencing
+
+`stonith-enabled: false` in the CIB. With `no-quorum-policy: ignore`, both nodes will continue running if corosync loses communication. DRBD's `fencing resource-only` does call `fence-peer` before allowing a Primary promotion, which provides some protection, but there is no hard external power fence to guarantee the other node actually stops. In a real split-brain (both nodes believe they are Primary), data corruption on the shared XFS filesystem is possible. **This is the highest-severity risk in the current setup.**
+
+Getting STONITH to work on Proxmox-hosted VMs requires either a `fence_pve` agent (Proxmox API fencing) or `fence_virtd` (QEMU guest agent fencing). Neither is configured.
+
+### RF-4: iSCSI portal bound to `[::0]:3260` — listens on every interface, not just the VIP
+
+The targetcli portal is `[::0]:3260` (confirmed: `*:3260 *:*` in ss). This means the target is reachable on:
+- 192.168.2.229 (VIP — correct, failover-safe)
+- 192.168.2.228 (ha-server-1 LAN IP — does **not** move during failover; an initiator session connecting here would break on failover)
+- 192.168.4.228 (storage NIC — not reachable by the Docker host today, but unintentionally exposed)
+
+Binding the portal explicitly to the VIP IP instead of wildcard eliminates the non-VIP reachability risks.
+
+### RF-5: iSCSI has zero ACLs and no authentication
+
+`targetcli ls` shows `acls: 0`, `no-gen-acls`, `no-auth`. Any host that can reach port 3260 on any of the above IPs can log into the LUN with no credentials. The Docker host is not yet configured as an initiator — but neither is it blocked.
+
+### RF-6: Docker host has no path to the storage network — iSCSI would traverse vmbr0/nic0
+
+CT 105 (docker, 192.168.2.225) has one NIC, on vmbr0. To reach the VIP at 192.168.2.229, iSCSI traffic would travel:
+
+```
+docker (eth0/vmbr0) → nic0 (2.5GbE) → vmbr0 → tap200i0 (VM 200 net0/ens18)
+```
+
+All of the following share this same path over vmbr0 → nic0:
+- Docker container traffic (outbound and inter-container)
+- CI/CD runner traffic (Gitea Actions jobs visible in `docker network ls`)
+- NFS mounts from pxe-boot, proxmox host itself, and other LAN clients
+- iSCSI block traffic (protocol-sensitive to latency and retransmit)
+
+A Nextcloud upload or a CI `nix build` job can saturate nic0 and starve the iSCSI session, causing command timeouts and filesystem errors on the Docker host.
+
+### RF-7: VIP is on the LAN interface with no storage-specific binding
+
+The Pacemaker `vip` resource specifies `ip=192.168.2.229, cidr_netmask=24` with no `nic=` parameter. Pacemaker's IPaddr2 agent selects the interface by longest-prefix match, landing it on ens18 (vmbr0/LAN). There is no way to keep this VIP from competing with general LAN traffic on nic0 without moving the VIP to a separate subnet on a different virtual bridge.
+
+---
+
+## 3. Recommended Target Layout
+
+### Design constraints
+
+- Single Proxmox host: all traffic ultimately shares nic0's bandwidth. The goal is QoS partitioning via separate bridges and subnets, not true physical isolation.
+- Future physical split: bridge/VLAN IDs chosen here should map cleanly to physical uplink VLAN tags when the HA nodes move to separate hardware.
+
+### Proposed bridge layout
+
+| Bridge | Physical port | VLAN tag (future) | Subnet | Purpose |
+|--------|-------------|-------------------|--------|---------|
+| `vmbr0` | nic0 | untagged / VLAN 1 | 192.168.2.0/24 | **LAN/management only** — no storage traffic |
+| `vmbr1` | (none / VLAN 10 on future trunk) | VLAN 10 | 192.168.4.224/29 | **Corosync heartbeat + DRBD replication** (current, keep) |
+| `vmbr2` *(new)* | (none / VLAN 20 on future trunk) | VLAN 20 | 192.168.5.0/24 | **Storage: iSCSI + NFS client access** |
+
+This is the minimum-disruption path: vmbr1 stays as-is (no DRBD reconfiguration needed), and the new vmbr2 gives the Docker host a direct path to the storage VIP without crossing vmbr0.
+
+If stricter isolation is later desired, DRBD can be migrated from vmbr1 to vmbr2 in a separate maintenance window (see §5), leaving vmbr1 as corosync-only.
+
+### Per-guest NIC assignments in target layout
+
+| Guest | VMID | NIC | Bridge | Proposed IP | Purpose |
+|-------|------|-----|--------|-------------|---------|
+| ha-server-1 | VM 200 | net0 | vmbr0 | 192.168.2.228/24 | LAN/management (keep) |
+| ha-server-1 | VM 200 | net1 | vmbr1 | 192.168.4.228/29 | Corosync + DRBD (keep) |
+| ha-server-1 | VM 200 | **net2 (new)** | **vmbr2** | **192.168.5.1/24** | iSCSI + NFS storage client |
+| ha-server-2 | VM 201 | net0 | vmbr0 | 192.168.2.227/24 | LAN/management (keep) |
+| ha-server-2 | VM 201 | net1 | vmbr1 | 192.168.4.227/29 | Corosync + DRBD (keep) |
+| ha-server-2 | VM 201 | **net2 (new)** | **vmbr2** | **192.168.5.2/24** | iSCSI + NFS storage client |
+| docker | CT 105 | net0 | vmbr0 | 192.168.2.225/24 | LAN/management (keep) |
+| docker | CT 105 | **net1 (new)** | **vmbr2** | **192.168.5.10/24** | iSCSI + NFS |
+
+**VIP target:** `192.168.5.100/24` on vmbr2. The Pacemaker `vip` resource changes from `ip=192.168.2.229` to `ip=192.168.5.100, nic=<ens20>` (whichever name the new NIC gets inside the HA VMs). The existing `192.168.2.229` LAN VIP can optionally be retained as a separate static alias on ens18 for management-plane access, but should not be the iSCSI portal target.
+
+**iSCSI portal:** Bind to `192.168.5.100:3260` instead of `[::0]:3260`. In targetcli: remove the wildcard portal, add `portals/ create 192.168.5.100`.
+
+**NFS exports:** NFS is a file-level protocol and is fine being accessed over a routed path. After the VIP moves, non-docker LAN clients (proxmox host, pxe-boot, raspi) can reach NFS either via a static route to 192.168.5.0/24 or by keeping a secondary static alias at 192.168.2.229 on ens18 dedicated to NFS. Either approach works — NFS handles reconnect gracefully in ways iSCSI block I/O cannot.
+
+**Corosync second ring (independent, low-disruption improvement):**
+
+Add a second knet link using the LAN addresses as a backup heartbeat path. Edit `corosync.conf` on both nodes:
+
+```
+node { ring0_addr: 192.168.4.228; ring1_addr: 192.168.2.228; name: ha-server-1; nodeid: 1; }
+node { ring0_addr: 192.168.4.227; ring1_addr: 192.168.2.227; name: ha-server-2; nodeid: 2; }
+```
+
+Requires a corosync service restart (brief cluster pause, ~5 seconds), no interface or bridge changes.
+
+**Future physical-host split:**  
+When ha-server-1 and ha-server-2 move to separate physical machines, vmbr1 and vmbr2 become VLAN-tagged sub-interfaces on a physical trunk (e.g. VLAN 10 → cluster, VLAN 20 → storage). The bridge/subnet/IP layout above is designed so the tag numbers can be layered onto the existing addresses without renumbering.
+
+---
+
+## 4. Gap List
+
+| Gap | Action needed |
+|-----|--------------|
+| `vmbr2` does not exist on pve1 | Create internal bridge: `/etc/network/interfaces.d/vmbr2.conf` with `bridge-ports none`, `inet manual` |
+| VM 200 and VM 201 have no net2 | `qm set 200 --net2 virtio,bridge=vmbr2` / `qm set 201 --net2 virtio,bridge=vmbr2` (hot-plug, no reboot needed) |
+| CT 105 has no net1 | `pct set 105 --net1 name=eth1,bridge=vmbr2,ip=192.168.5.10/24` |
+| HA VMs have no OS config for the new NIC | NixOS `networking.interfaces.<ens20>` with `ipv4.addresses = [{address="192.168.5.1"; prefixLength=24;}]` per host (name may differ — check `ip link` after hotplug) |
+| VIP needs to move to 192.168.5.100 on vmbr2 | `pcs resource update vip ip=192.168.5.100 cidr_netmask=24 nic=<ens20>` |
+| iSCSI portal bound to `[::0]` | `targetcli /iscsi/iqn.2026-01.home.sweet:ha-storage/tpg1/portals delete ::0 3260` then `create 192.168.5.100`; save and restart via `pcs resource restart iscsi-target` |
+| iSCSI ACLs empty | Get Docker initiator IQN via `iscsiadm -m iface` on CT 105, then add via targetcli `acls/ create <iqn>` |
+| Docker host has no iSCSI initiator config | `iscsiadm -m discoverydb -t sendtargets -p 192.168.5.100 -D` then `iscsiadm -m node -l` once ACLs are set |
+| Corosync single ring | Add `ring1_addr` entries in `corosync.conf` using LAN IPs; restart corosync cluster-wide (one node at a time) |
+| STONITH not configured | Evaluate `fence_pve` (Proxmox API agent); document accepted risk if deferred |
+| DRBD still on vmbr1 (optional, separate window) | Stop ms-drbd0 via pcs, edit `/etc/drbd.conf` on both nodes (change `192.168.4.x` → `192.168.5.x`), restart DRBD, re-enable via pcs |
+
+---
+
+## 5. Migration Notes
+
+### Non-disruptive (no service impact)
+
+- **Create vmbr2 on pve1:** Bridge definition edit only; no effect on existing bridges or guests.
+- **Hot-add net2 to VMs 200/201:** Proxmox allows adding a NIC without reboot (`qm set 200 --net2 ...`). The NIC appears inside the VM immediately via QEMU hotplug but will be unconfigured (down) inside NixOS until the NixOS config is deployed — no impact on running services.
+- **Add net1 to CT 105:** LXC NIC hotplug works similarly; CT does not need to restart.
+- **Add corosync ring1:** Requires `systemctl restart corosync` on both nodes (one at a time). Pacemaker briefly sees corosync go offline and recover; with two nodes and `wait_for_all: 0`, this typically completes in under 5 seconds and resources stay running.
+
+### Disruptive — requires maintenance window
+
+- **Move VIP from 192.168.2.229 to 192.168.5.100:** `pcs resource update vip ip=192.168.5.100` causes Pacemaker to immediately stop the old VIP and start the new one. Any NFS mounts referencing 192.168.2.229 will stall until remounted at the new address (or a static alias is added at 192.168.2.229 on ens18). No iSCSI sessions exist yet, so no iSCSI disruption.
+
+- **Change iSCSI portal from `[::0]` to VIP-specific:** Requires `pcs resource restart iscsi-target` after the targetcli portal change — brief target unavailability. Any initiator sessions (once configured) will need to re-login.
+
+- **Migrate DRBD replication from 192.168.4.x to 192.168.5.x** (optional — only needed to give corosync sole ownership of vmbr1):
+  1. `pcs resource disable ms-drbd0` — demotes DRBD Primary, stops ha-group (unmounts XFS, stops iSCSI + NFS + VIP)
+  2. `drbdadm down ha-data` on both nodes
+  3. Edit `/etc/drbd.conf` on both nodes (change `address` lines)
+  4. `drbdadm up ha-data` on both nodes
+  5. `pcs resource enable ms-drbd0` — Pacemaker re-promotes, mounts, starts services
+
+  DRBD does **not** require a full resync when only the address changes — the disk data and metadata are unchanged; only the TCP connection endpoint changes. However, the initial sync was in progress at audit time (~20% at ~40 MB/s). Recommend waiting for that sync to complete before scheduling this migration.