Archived
feat(ha): add vmbr2 storage-client network and corosync ring1
Check NixOS configurations / eval-hosts (push) Successful in 10m39s
Check NixOS configurations / eval-hosts (push) Successful in 10m39s
Infrastructure changes already applied to pve1:
- vmbr2 internal bridge created (192.168.5.0/24, no physical uplink)
- VM 200 (ha-server-1): net2 added → vmbr2 (ens20)
- VM 201 (ha-server-2): net2 added → vmbr2 (ens20)
- CT 105 (docker): net1 added → vmbr2 (eth1)
- VM 101 (server): net1 added → vmbr2 (ens19) — needs reboot to activate
NixOS config (deploy to ha nodes to complete; docker/server at cutover):
- ha-server-{1,2}/host.nix: ens20 with 192.168.5.{228,227}/24
- docker/host.nix: eth1 with 192.168.5.225/24
- server/host.nix: ens19 with 192.168.5.226/24
- cluster-config.nix: corosync ring1 on LAN IPs as backup heartbeat path
- cluster-config.nix: allow haClientCidr (192.168.5.0/24) in iptables
- ha-server.nix: NFS exports now allow both lanCidr and haClientCidr
- VIP moves from 192.168.2.229 (vmbr0/LAN) to 192.168.5.229 (vmbr2)
- iSCSI portal to be rebound from [::0] to 192.168.5.229 at cutover
variables.nix: haStorageCidr corrected to 192.168.4.224/29; new vars:
vmStorageClientInterface, lxcStorageInterface, haServer{1,2}ClientIp,
dockerStorageIp, serverStorageIp, haClientCidr/PrefixLength; haServerVip
updated to 192.168.5.229.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
+36
-14
@@ -5,13 +5,14 @@
|
||||
| Subnet | CIDR | Purpose | Routed? |
|
||||
|---|---|---|---|
|
||||
| LAN | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
|
||||
| Storage | `192.168.4.0/29` | HA file server DRBD replication | No — internal `vmbr1` only, no uplink |
|
||||
| Cluster | `192.168.4.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
|
||||
| Storage client | `192.168.5.0/24` | HA file server iSCSI + NFS — docker and server access VIP here | No — internal `vmbr2` only, no uplink |
|
||||
|
||||
The storage subnet never leaves pve1. `vmbr1` is a Proxmox Linux bridge with no physical port
|
||||
attached; traffic between the two HA file server VMs stays in-kernel.
|
||||
The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
|
||||
bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
|
||||
|
||||
The host octet is consistent across subnets for any host that has multiple interfaces — e.g.
|
||||
ha-node1 is always `.228` (LAN: `192.168.2.228`, storage: `192.168.4.228`).
|
||||
The host octet is consistent across all subnets — e.g. ha-node1 is always `.228`:
|
||||
`192.168.2.228` (LAN), `192.168.4.228` (cluster), `192.168.5.228` (storage client).
|
||||
|
||||
---
|
||||
|
||||
@@ -56,9 +57,9 @@ All VMs and LXC containers run on pve1.
|
||||
|
||||
| IP | Hostname | Role | Status |
|
||||
|---|---|---|---|
|
||||
| `192.168.2.229` | ha-vip | HA file server iSCSI floating VIP (Pacemaker) | Future |
|
||||
| `192.168.2.228` | ha-node1 | HA file server node 1 (DRBD + XFS + iSCSI) | Future |
|
||||
| `192.168.2.227` | ha-node2 | HA file server node 2 (DRBD + XFS + iSCSI) | Future |
|
||||
| `192.168.2.229` | — | Was planned as ha-vip; VIP moved to `192.168.5.229` (vmbr2) | Unassigned |
|
||||
| `192.168.2.228` | ha-node1 | HA file server node 1 — management NIC | Active |
|
||||
| `192.168.2.227` | ha-node2 | HA file server node 2 — management NIC | Active |
|
||||
| `192.168.2.226` | server | Current NFS/ZFS file server — retires when HA is live | Retiring |
|
||||
| `192.168.2.225` | docker | Docker / Traefik stack | Active |
|
||||
| `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
|
||||
@@ -78,16 +79,37 @@ MAC address churn.
|
||||
|
||||
---
|
||||
|
||||
## Storage network — 192.168.4.0/29
|
||||
## Cluster network — 192.168.4.224/29
|
||||
|
||||
Internal to pve1 only. Proxmox bridge `vmbr1`, no physical NIC attached.
|
||||
|
||||
| IP | Hostname | Interface role |
|
||||
|---|---|---|
|
||||
| `192.168.4.228` | ha-node1 | DRBD replication NIC |
|
||||
| `192.168.4.227` | ha-node2 | DRBD replication NIC |
|
||||
| `192.168.4.228` | ha-node1 | DRBD replication + Corosync ring0 (primary heartbeat) |
|
||||
| `192.168.4.227` | ha-node2 | DRBD replication + Corosync ring0 (primary heartbeat) |
|
||||
| — | no gateway | Isolated — not routed to LAN or internet |
|
||||
|
||||
Corosync ring1 (backup heartbeat only) uses the LAN IPs (`192.168.2.228` / `192.168.2.227`)
|
||||
over `vmbr0` — no additional bridge needed, and DRBD traffic never crosses ring1.
|
||||
|
||||
---
|
||||
|
||||
## Storage-client network — 192.168.5.0/24
|
||||
|
||||
Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached.
|
||||
|
||||
| IP | Hostname | Interface / role |
|
||||
|---|---|---|
|
||||
| `192.168.5.229` | ha-vip | Pacemaker floating VIP — iSCSI portal + NFS endpoint |
|
||||
| `192.168.5.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
|
||||
| `192.168.5.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
|
||||
| `192.168.5.226` | server | Storage-client NIC (ens19 / vmbr2) — for final rsync before cutover |
|
||||
| `192.168.5.225` | docker | Storage-client NIC (eth1 / vmbr2) — iSCSI initiator + NFS client |
|
||||
| — | no gateway | Isolated — not routed to LAN or internet |
|
||||
|
||||
iSCSI initiators and NFS clients connect exclusively to the VIP (`192.168.5.229`) so sessions
|
||||
survive failover transparently. The portal is bound to the VIP only (not `[::0]`).
|
||||
|
||||
---
|
||||
|
||||
## Migration reference
|
||||
@@ -101,9 +123,9 @@ Current → target IP for every host being renumbered.
|
||||
| pve1 | `192.168.2.250` | `192.168.2.245` | `/etc/network/interfaces` on Proxmox host |
|
||||
| pbs | `192.168.2.108` | `192.168.2.244` | static config on PBS host |
|
||||
| nixos workstation | `192.168.2.119` | `192.168.2.243` | `networking.interfaces` / NetworkManager on guest |
|
||||
| ha-node1 | — | `192.168.2.228` | future |
|
||||
| ha-node2 | — | `192.168.2.227` | future |
|
||||
| ha-vip | — | `192.168.2.229` | future (Pacemaker resource) |
|
||||
| ha-node1 | — | `192.168.2.228` (LAN), `192.168.4.228` (cluster), `192.168.5.228` (storage) | active |
|
||||
| ha-node2 | — | `192.168.2.227` (LAN), `192.168.4.227` (cluster), `192.168.5.227` (storage) | active |
|
||||
| ha-vip | — | `192.168.5.229` (vmbr2 / Pacemaker IPaddr2) | active |
|
||||
| server | `192.168.2.252` | `192.168.2.226` | static config on guest |
|
||||
| docker | `192.168.2.249` | `192.168.2.225` | static config on guest |
|
||||
| nix-cache | `192.168.2.120` | `192.168.2.224` | static config on guest |
|
||||
|
||||
Reference in New Issue
Block a user