beatzaplenty/nixos
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(ha): renumber cluster→VLAN10, storage→VLAN20; iSCSI/NFS protocol separation
Check NixOS configurations / eval-hosts (push) Failing after 17m31s
Details

Browse Source 
Renumber internal HA bridges so VLAN ID matches IP third octet:
  vmbr1 (cluster): 192.168.4.x → 192.168.10.x (VLAN 10, /29)
  vmbr2 (storage): 192.168.5.x → 192.168.20.x (VLAN 20, /24)

Enforce protocol separation at the firewall on both HA nodes:
  - iSCSI (3260/tcp): storage-client subnet (192.168.20.0/24) only
  - NFS (111,2049,20048 tcp+udp): LAN subnet (192.168.2.0/24) only
  - Cluster subnet (192.168.10.224/29) accepted wholesale (DRBD+Corosync)
Moves from allowedTCPPorts/allowedUDPPorts to source-restricted extraCommands
rules on the nixos-fw chain.

NFS exports reduced to LAN-only (drop haClientCidr lines from mkNfsExports).
Storage-client network clients use iSCSI only; no NFS on VLAN 20.

All HA script VIP/storage IP defaults updated to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
This commit is contained in:
beatzaplenty
2026-07-29 18:19:07 +10:00
co-authored by Claude Sonnet 4.6
parent 5eea38d3ca
commit bf4836efac
9 changed files with 73 additions and 63 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ip-addressing.md
+28 -22
View File
@@ -2,17 +2,22 @@
## Subnets
| Subnet | CIDR | Purpose | Routed? |
|---|---|---|---|
| LAN | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
| Cluster | `192.168.4.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
| Storage client | `192.168.5.0/24` | HA file server iSCSI + NFS — docker and server access VIP here | No — internal `vmbr2` only, no uplink |
| Subnet | VLAN | CIDR | Purpose | Routed? |
|---|---|---|---|---|
| LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
| Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
| Storage client | 20 | `192.168.20.0/24` | HA file server iSCSI — docker and server access VIP here | No — internal `vmbr2` only, no uplink |
The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
The host octet is consistent across all subnets — e.g. ha-node1 is always `.228`:
`192.168.2.228` (LAN), `192.168.4.228` (cluster), `192.168.5.228` (storage client).
VLAN IDs match the third octet of each subnet (VLAN 2 → 192.168.**2**.x, VLAN 10 → 192.168.**10**.x,
VLAN 20 → 192.168.**20**.x). The host octet is consistent across all subnets — e.g. ha-node1
is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20.228` (storage client).
**Protocol separation** (enforced by firewall on HA nodes):
- iSCSI (port 3260): storage-client subnet (VLAN 20) only — docker and server
- NFS (ports 111, 2049, 20048): LAN subnet (VLAN 2) only — pxe-boot and other LAN clients
---
@@ -79,14 +84,14 @@ MAC address churn.
---
## Cluster network — 192.168.4.224/29
## Cluster network — VLAN 10 — 192.168.10.224/29
Internal to pve1 only. Proxmox bridge `vmbr1`, no physical NIC attached.
| IP | Hostname | Interface role |
|---|---|---|
| `192.168.4.228` | ha-node1 | DRBD replication + Corosync ring0 (primary heartbeat) |
| `192.168.4.227` | ha-node2 | DRBD replication + Corosync ring0 (primary heartbeat) |
| `192.168.10.228` | ha-node1 | DRBD replication + Corosync ring0 (primary heartbeat) |
| `192.168.10.227` | ha-node2 | DRBD replication + Corosync ring0 (primary heartbeat) |
| — | no gateway | Isolated — not routed to LAN or internet |
Corosync ring1 (backup heartbeat only) uses the LAN IPs (`192.168.2.228` / `192.168.2.227`)
@@ -94,21 +99,22 @@ over `vmbr0` — no additional bridge needed, and DRBD traffic never crosses rin
---
## Storage-client network — 192.168.5.0/24
## Storage-client network — VLAN 20 — 192.168.20.0/24
Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached.
| IP | Hostname | Interface / role |
|---|---|---|
| `192.168.5.229` | ha-vip | Pacemaker floating VIP — iSCSI portal + NFS endpoint |
| `192.168.5.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
| `192.168.5.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
| `192.168.5.226` | server | Storage-client NIC (ens19 / vmbr2) — for final rsync before cutover |
| `192.168.5.225` | docker | Storage-client NIC (eth1 / vmbr2) — iSCSI initiator + NFS client |
| `192.168.20.229` | ha-vip-storage | Pacemaker floating VIP — iSCSI endpoint (port 3260) |
| `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
| `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
| `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — temporary, for final rsync before decommission |
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — iSCSI initiator |
| — | no gateway | Isolated — not routed to LAN or internet |
iSCSI initiators and NFS clients connect exclusively to the VIP (`192.168.5.229`) so sessions
survive failover transparently. The portal is bound to the VIP only (not `[::0]`).
iSCSI initiators connect to the VIP (`192.168.20.229:3260`) so sessions survive failover
transparently. Portal listens on `[::0]:3260`; firewall restricts port 3260 to
`192.168.20.0/24` source only — LAN clients cannot reach iSCSI.
---
@@ -123,10 +129,10 @@ Current → target IP for every host being renumbered.
| pve1 | `192.168.2.250` | `192.168.2.245` | `/etc/network/interfaces` on Proxmox host |
| pbs | `192.168.2.108` | `192.168.2.244` | static config on PBS host |
| nixos workstation | `192.168.2.119` | `192.168.2.243` | `networking.interfaces` / NetworkManager on guest |
| ha-node1 | — | `192.168.2.228` (LAN), `192.168.4.228` (cluster), `192.168.5.228` (storage) | active |
| ha-node2 | — | `192.168.2.227` (LAN), `192.168.4.227` (cluster), `192.168.5.227` (storage) | active |
| ha-vip-lan | — | `192.168.2.229` (vmbr0 / Pacemaker `vip-lan`) — LAN iSCSI + NFS | active |
| ha-vip-storage | — | `192.168.5.229` (vmbr2 / Pacemaker `vip-storage`) — storage-client iSCSI + NFS | active |
| ha-node1 | — | `192.168.2.228` (LAN), `192.168.10.228` (cluster/VLAN 10), `192.168.20.228` (storage/VLAN 20) | active |
| ha-node2 | — | `192.168.2.227` (LAN), `192.168.10.227` (cluster/VLAN 10), `192.168.20.227` (storage/VLAN 20) | active |
| ha-vip-lan | — | `192.168.2.229` (vmbr0 / Pacemaker `vip-lan`) — NFS endpoint for LAN clients | active |
| ha-vip-storage | — | `192.168.20.229` (vmbr2 / Pacemaker `vip-storage`) — iSCSI endpoint for VLAN 20 clients | active |
| server | `192.168.2.252` | `192.168.2.226` | static config on guest |
| docker | `192.168.2.249` | `192.168.2.225` | static config on guest |
| nix-cache | `192.168.2.120` | `192.168.2.224` | static config on guest |