nixos/docs/ha-network-audit.md

Storage/Cluster Network Segmentation Audit — pve1.sweet.home

Date: 2026-07-29
Scope: Read-only discovery of pve1.sweet.home host networking, HA cluster VMs (200/201), and Docker CT (105). No changes made.

---

1. Current State Summary

pve1.sweet.home Host — Physical NICs

Interface	Speed/Duplex	Notes
nic0	2500 Mb/s / Full (2.5GbE)	Only active physical NIC; sole bridge port for vmbr0
nic1	(not connected / no data)	Present in config, not UP
wlp4s0	DOWN	WiFi, unused

No bonding configured. Every guest's traffic ultimately funnels through the single 2.5GbE nic0.

Proxmox Bridges

Bridge	Physical NIC	Host IP	Subnet	VLAN-aware	Purpose (current)
vmbr0	nic0 (2.5GbE)	192.168.2.245/24	192.168.2.0/24	No	General LAN, management, iSCSI/NFS VIP
vmbr1	none (internal-only)	—	192.168.4.224/29	No	Corosync heartbeat + DRBD replication

vmbr1 has bridge-ports none in /etc/network/interfaces.d/vmbr1.conf — it is a purely software bridge with zero physical uplink. All traffic on it stays inside the hypervisor's memory.

Guest NIC Assignments

Guest	VMID	Role	NIC	Bridge	IP	Traffic type
nix-cache	CT 102	Build cache	eth0	vmbr0	DHCP/LAN	LAN
pxe-boot	CT 103	PXE/TFTP	eth0	vmbr0	DHCP/LAN	LAN
tor-relay	CT 104	Tor	eth0	vmbr0	DHCP/LAN	LAN
docker	CT 105	Docker host	eth0	vmbr0	192.168.2.225/24	LAN only
pdm	CT 106	Proxmox mgmt	eth0	vmbr0	192.168.2.220/24	LAN
server	VM 101	General server	net0	vmbr0	DHCP/LAN	LAN
tailscale-router	VM 107	Tailscale exit	net0	vmbr0	DHCP/LAN	LAN
domain-controller	VM 108	FreeIPA	net0	vmbr0	192.168.2.253/24	LAN
ha-server-1	VM 200	HA primary	net0	vmbr0	192.168.2.228/24 + VIP 192.168.2.229/24	LAN + VIP
ha-server-1	VM 200	HA primary	net1	vmbr1	192.168.4.228/29	Corosync + DRBD
ha-server-2	VM 201	HA secondary	net0	vmbr0	192.168.2.227/24	LAN
ha-server-2	VM 201	HA secondary	net1	vmbr1	192.168.4.227/29	Corosync + DRBD

Corosync/Pacemaker State

• Transport: knet/UDP
• Rings: 1 only — ring0 on 192.168.4.228 / 192.168.4.227 (vmbr1/ens19)
• Cluster status: Both nodes online, DC = ha-server-1, quorum achieved
• STONITH: stonith-enabled: false
• no-quorum-policy: ignore
• Resources (all active on ha-server-1):
  • ms-drbd0 — promotable DRBD clone (Primary: ha-server-1, Secondary: ha-server-2)
  • xfs-data — XFS on /dev/drbd0 → /srv/ha-data
  • iscsi-target — targetctl service
  • nfs-server — nfs-server service
  • vip — IPaddr2 at 192.168.2.229/24 (no nic= parameter specified; floats to ens18/vmbr0 automatically based on subnet match)
• Resource ordering: ha-group starts only after DRBD is promoted; collocated with Promoted DRBD clone.

DRBD State

• Resource: ha-data (DRBD 8.4.11 kernel module, config in /etc/drbd.conf)
• Protocol: C (synchronous)
• Replication endpoints:
  • ha-server-1: 192.168.4.228:7789 (ens19 / vmbr1)
  • ha-server-2: 192.168.4.227:7789 (ens19 / vmbr1)
• State at audit time: Initial sync in progress — ~20% complete, ~40 MB/s, ~34 min remaining (100 GB disk)
• Fencing config: fencing resource-only + fence-peer/unfence-peer handlers

iSCSI Target

• IQN: iqn.2026-01.home.sweet:ha-storage
• Portal: [::0]:3260 — confirmed listening on all interfaces (ss -tnlp shows *:3260 *:*)
• LUN 0: fileio backstore — /srv/ha-data/iscsi-lun.img (10 GiB, write-thru)
• ACLs: None (no-gen-acls, no-auth)
• VIP (intended portal): 192.168.2.229 — on vmbr0/LAN, no storage-NIC-specific binding

NFS Exports

Served from the same ha-group as iSCSI (starts/stops together):

Export path	Client subnet
/srv/ha-data/docker/{config,databases,volumes,nextcloud-data}	192.168.2.0/24
/srv/ha-data/proxmox/{iso,lxc}	192.168.2.0/24
/srv/ha-data/pxe-boot/images	192.168.2.0/24
/srv/ha-data/raspi/volumes	192.168.2.0/24

All NFS exports restrict to 192.168.2.0/24 and are served via the VIP at 192.168.2.229. rw,sync,no_subtree_check,no_root_squash.

Docker Host Current State

• Single NIC: eth0 on vmbr0, 192.168.2.225/24, gateway 192.168.2.254
• Route to storage network (192.168.4.x): none — no NIC and no route
• iSCSI sessions: none
• iSCSI nodes discovered: none
• Docker networks: several active compose-project networks (core_traefik, core_nextcloud, core_passbolt, core_gramps, core_docker-socket-proxy, plus CI isolation networks)

---

2. Risk Flags

RF-1: Corosync has only one ring (no heartbeat path redundancy)

corosync.conf defines only ring0_addr for each node, using 192.168.4.x on vmbr1. No ring1_addr / second knet link is configured. On a single Proxmox host, vmbr1 is a software bridge (no physical NIC), so physical link failure is not the concern — but a kernel network stack hiccup, a pveproxy restart dropping bridge state, or vmbr1 getting disrupted during heavy DRBD sync all leave corosync with zero fallback path. Missed heartbeats on a two-node cluster with no-quorum-policy: ignore do not cause a clean shutdown; they cause a false failover or split-brain.

Adding the LAN addresses (192.168.2.228 / 192.168.2.227 via ens18/vmbr0) as a second knet link would provide a backup path with no infrastructure changes needed.

RF-2: Corosync heartbeat and DRBD replication share vmbr1 — no isolation between them

Both corosync (knet/UDP, ~1 kB heartbeat packets every ~100 ms) and DRBD replication (protocol C, synchronous, syncing at ~40 MB/s on a 100 GB initial fill at audit time) traverse the same vmbr1 virtual bridge and terminate on the same ens19 NIC pair inside each HA VM. Under heavy DRBD write load, the guest-kernel scheduler's NIC transmit queue processes both flows together. While corosync's heartbeat is tiny, the absence of QoS/priority marking on vmbr1 means a DRBD burst can delay a heartbeat enough to trigger a ring fault warning. This is a latent risk that grows under high-write workloads.

RF-3: STONITH disabled — split-brain protection relies solely on DRBD's resource-only fencing

stonith-enabled: false in the CIB. With no-quorum-policy: ignore, both nodes will continue running if corosync loses communication. DRBD's fencing resource-only does call fence-peer before allowing a Primary promotion, which provides some protection, but there is no hard external power fence to guarantee the other node actually stops. In a real split-brain (both nodes believe they are Primary), data corruption on the shared XFS filesystem is possible. This is the highest-severity risk in the current setup.

Getting STONITH to work on Proxmox-hosted VMs requires either a fence_pve agent (Proxmox API fencing) or fence_virtd (QEMU guest agent fencing). Neither is configured.

RF-4: iSCSI portal bound to [::0]:3260 — listens on every interface, not just the VIP

The targetcli portal is [::0]:3260 (confirmed: *:3260 *:* in ss). This means the target is reachable on:

• 192.168.2.229 (VIP — correct, failover-safe)
• 192.168.2.228 (ha-server-1 LAN IP — does not move during failover; an initiator session connecting here would break on failover)
• 192.168.4.228 (storage NIC — not reachable by the Docker host today, but unintentionally exposed)

Binding the portal explicitly to the VIP IP instead of wildcard eliminates the non-VIP reachability risks.

RF-5: iSCSI has zero ACLs and no authentication

targetcli ls shows acls: 0, no-gen-acls, no-auth. Any host that can reach port 3260 on any of the above IPs can log into the LUN with no credentials. The Docker host is not yet configured as an initiator — but neither is it blocked.

RF-6: Docker host has no path to the storage network — iSCSI would traverse vmbr0/nic0

CT 105 (docker, 192.168.2.225) has one NIC, on vmbr0. To reach the VIP at 192.168.2.229, iSCSI traffic would travel:

docker (eth0/vmbr0) → nic0 (2.5GbE) → vmbr0 → tap200i0 (VM 200 net0/ens18)

All of the following share this same path over vmbr0 → nic0:

• Docker container traffic (outbound and inter-container)
• CI/CD runner traffic (Gitea Actions jobs visible in docker network ls)
• NFS mounts from pxe-boot, proxmox host itself, and other LAN clients
• iSCSI block traffic (protocol-sensitive to latency and retransmit)

A Nextcloud upload or a CI nix build job can saturate nic0 and starve the iSCSI session, causing command timeouts and filesystem errors on the Docker host.

RF-7: VIP is on the LAN interface with no storage-specific binding

The Pacemaker vip resource specifies ip=192.168.2.229, cidr_netmask=24 with no nic= parameter. Pacemaker's IPaddr2 agent selects the interface by longest-prefix match, landing it on ens18 (vmbr0/LAN). There is no way to keep this VIP from competing with general LAN traffic on nic0 without moving the VIP to a separate subnet on a different virtual bridge.

---

3. Recommended Target Layout

Design constraints

• Single Proxmox host: all traffic ultimately shares nic0's bandwidth. The goal is QoS partitioning via separate bridges and subnets, not true physical isolation.
• Future physical split: bridge/VLAN IDs chosen here should map cleanly to physical uplink VLAN tags when the HA nodes move to separate hardware.

Proposed bridge layout

Bridge	Physical port	VLAN tag (future)	Subnet	Purpose
vmbr0	nic0	untagged / VLAN 1	192.168.2.0/24	LAN/management only — no storage traffic
vmbr1	(none / VLAN 10 on future trunk)	VLAN 10	192.168.4.224/29	Corosync heartbeat + DRBD replication (current, keep)
vmbr2 (new)	(none / VLAN 20 on future trunk)	VLAN 20	192.168.5.0/24	Storage: iSCSI + NFS client access

This is the minimum-disruption path: vmbr1 stays as-is (no DRBD reconfiguration needed), and the new vmbr2 gives the Docker host a direct path to the storage VIP without crossing vmbr0.

If stricter isolation is later desired, DRBD can be migrated from vmbr1 to vmbr2 in a separate maintenance window (see §5), leaving vmbr1 as corosync-only.

Per-guest NIC assignments in target layout

Guest	VMID	NIC	Bridge	Proposed IP	Purpose
ha-server-1	VM 200	net0	vmbr0	192.168.2.228/24	LAN/management (keep)
ha-server-1	VM 200	net1	vmbr1	192.168.4.228/29	Corosync + DRBD (keep)
ha-server-1	VM 200	net2 (new)	vmbr2	192.168.5.1/24	iSCSI + NFS storage client
ha-server-2	VM 201	net0	vmbr0	192.168.2.227/24	LAN/management (keep)
ha-server-2	VM 201	net1	vmbr1	192.168.4.227/29	Corosync + DRBD (keep)
ha-server-2	VM 201	net2 (new)	vmbr2	192.168.5.2/24	iSCSI + NFS storage client
docker	CT 105	net0	vmbr0	192.168.2.225/24	LAN/management (keep)
docker	CT 105	net1 (new)	vmbr2	192.168.5.10/24	iSCSI + NFS

VIP target: 192.168.5.100/24 on vmbr2. The Pacemaker vip resource changes from ip=192.168.2.229 to ip=192.168.5.100, nic=<ens20> (whichever name the new NIC gets inside the HA VMs). The existing 192.168.2.229 LAN VIP can optionally be retained as a separate static alias on ens18 for management-plane access, but should not be the iSCSI portal target.

iSCSI portal: Bind to 192.168.5.100:3260 instead of [::0]:3260. In targetcli: remove the wildcard portal, add portals/ create 192.168.5.100.

NFS exports: NFS is a file-level protocol and is fine being accessed over a routed path. After the VIP moves, non-docker LAN clients (proxmox host, pxe-boot, raspi) can reach NFS either via a static route to 192.168.5.0/24 or by keeping a secondary static alias at 192.168.2.229 on ens18 dedicated to NFS. Either approach works — NFS handles reconnect gracefully in ways iSCSI block I/O cannot.

Corosync second ring (independent, low-disruption improvement):

Add a second knet link using the LAN addresses as a backup heartbeat path. Edit corosync.conf on both nodes:

node { ring0_addr: 192.168.4.228; ring1_addr: 192.168.2.228; name: ha-server-1; nodeid: 1; }
node { ring0_addr: 192.168.4.227; ring1_addr: 192.168.2.227; name: ha-server-2; nodeid: 2; }

Requires a corosync service restart (brief cluster pause, ~5 seconds), no interface or bridge changes.

Future physical-host split:
When ha-server-1 and ha-server-2 move to separate physical machines, vmbr1 and vmbr2 become VLAN-tagged sub-interfaces on a physical trunk (e.g. VLAN 10 → cluster, VLAN 20 → storage). The bridge/subnet/IP layout above is designed so the tag numbers can be layered onto the existing addresses without renumbering.

---

4. Gap List

Gap	Action needed
vmbr2 does not exist on pve1	Create internal bridge: /etc/network/interfaces.d/vmbr2.conf with bridge-ports none, inet manual
VM 200 and VM 201 have no net2	qm set 200 --net2 virtio,bridge=vmbr2 / qm set 201 --net2 virtio,bridge=vmbr2 (hot-plug, no reboot needed)
CT 105 has no net1	pct set 105 --net1 name=eth1,bridge=vmbr2,ip=192.168.5.10/24
HA VMs have no OS config for the new NIC	NixOS networking.interfaces.<ens20> with ipv4.addresses = [{address="192.168.5.1"; prefixLength=24;}] per host (name may differ — check ip link after hotplug)
VIP needs to move to 192.168.5.100 on vmbr2	pcs resource update vip ip=192.168.5.100 cidr_netmask=24 nic=<ens20>
iSCSI portal bound to [::0]	targetcli /iscsi/iqn.2026-01.home.sweet:ha-storage/tpg1/portals delete ::0 3260 then create 192.168.5.100; save and restart via pcs resource restart iscsi-target
iSCSI ACLs empty	Get Docker initiator IQN via iscsiadm -m iface on CT 105, then add via targetcli acls/ create <iqn>
Docker host has no iSCSI initiator config	iscsiadm -m discoverydb -t sendtargets -p 192.168.5.100 -D then iscsiadm -m node -l once ACLs are set
Corosync single ring	Add ring1_addr entries in corosync.conf using LAN IPs; restart corosync cluster-wide (one node at a time)
STONITH not configured	Evaluate fence_pve (Proxmox API agent); document accepted risk if deferred
DRBD still on vmbr1 (optional, separate window)	Stop ms-drbd0 via pcs, edit /etc/drbd.conf on both nodes (change 192.168.4.x → 192.168.5.x), restart DRBD, re-enable via pcs

---

5. Migration Notes

Non-disruptive (no service impact)

• Create vmbr2 on pve1: Bridge definition edit only; no effect on existing bridges or guests.
• Hot-add net2 to VMs 200/201: Proxmox allows adding a NIC without reboot (qm set 200 --net2 ...). The NIC appears inside the VM immediately via QEMU hotplug but will be unconfigured (down) inside NixOS until the NixOS config is deployed — no impact on running services.
• Add net1 to CT 105: LXC NIC hotplug works similarly; CT does not need to restart.
• Add corosync ring1: Requires systemctl restart corosync on both nodes (one at a time). Pacemaker briefly sees corosync go offline and recover; with two nodes and wait_for_all: 0, this typically completes in under 5 seconds and resources stay running.

Disruptive — requires maintenance window

• Move VIP from 192.168.2.229 to 192.168.5.100: pcs resource update vip ip=192.168.5.100 causes Pacemaker to immediately stop the old VIP and start the new one. Any NFS mounts referencing 192.168.2.229 will stall until remounted at the new address (or a static alias is added at 192.168.2.229 on ens18). No iSCSI sessions exist yet, so no iSCSI disruption.

• Change iSCSI portal from [::0] to VIP-specific: Requires pcs resource restart iscsi-target after the targetcli portal change — brief target unavailability. Any initiator sessions (once configured) will need to re-login.

• Migrate DRBD replication from 192.168.4.x to 192.168.5.x (optional — only needed to give corosync sole ownership of vmbr1):

  1. pcs resource disable ms-drbd0 — demotes DRBD Primary, stops ha-group (unmounts XFS, stops iSCSI + NFS + VIP)
  2. drbdadm down ha-data on both nodes
  3. Edit /etc/drbd.conf on both nodes (change address lines)
  4. drbdadm up ha-data on both nodes
  5. pcs resource enable ms-drbd0 — Pacemaker re-promotes, mounts, starts services

  DRBD does not require a full resync when only the address changes — the disk data and metadata are unchanged; only the TCP connection endpoint changes. However, the initial sync was in progress at audit time (~20% at ~40 MB/s). Recommend waiting for that sync to complete before scheduling this migration.