Archived
beatzaplentyandClaude Sonnet 4.6
bf4836efac
Check NixOS configurations / eval-hosts (push) Failing after 17m31s
Renumber internal HA bridges so VLAN ID matches IP third octet: vmbr1 (cluster): 192.168.4.x → 192.168.10.x (VLAN 10, /29) vmbr2 (storage): 192.168.5.x → 192.168.20.x (VLAN 20, /24) Enforce protocol separation at the firewall on both HA nodes: - iSCSI (3260/tcp): storage-client subnet (192.168.20.0/24) only - NFS (111,2049,20048 tcp+udp): LAN subnet (192.168.2.0/24) only - Cluster subnet (192.168.10.224/29) accepted wholesale (DRBD+Corosync) Moves from allowedTCPPorts/allowedUDPPorts to source-restricted extraCommands rules on the nixos-fw chain. NFS exports reduced to LAN-only (drop haClientCidr lines from mkNfsExports). Storage-client network clients use iSCSI only; no NFS on VLAN 20. All HA script VIP/storage IP defaults updated to match. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
158 lines
7.7 KiB
Markdown
158 lines
7.7 KiB
Markdown
# IP Addressing Scheme
|
||
|
||
## Subnets
|
||
|
||
| Subnet | VLAN | CIDR | Purpose | Routed? |
|
||
|---|---|---|---|---|
|
||
| LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
|
||
| Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
|
||
| Storage client | 20 | `192.168.20.0/24` | HA file server iSCSI — docker and server access VIP here | No — internal `vmbr2` only, no uplink |
|
||
|
||
The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
|
||
bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
|
||
|
||
VLAN IDs match the third octet of each subnet (VLAN 2 → 192.168.**2**.x, VLAN 10 → 192.168.**10**.x,
|
||
VLAN 20 → 192.168.**20**.x). The host octet is consistent across all subnets — e.g. ha-node1
|
||
is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20.228` (storage client).
|
||
|
||
**Protocol separation** (enforced by firewall on HA nodes):
|
||
- iSCSI (port 3260): storage-client subnet (VLAN 20) only — docker and server
|
||
- NFS (ports 111, 2049, 20048): LAN subnet (VLAN 2) only — pxe-boot and other LAN clients
|
||
|
||
---
|
||
|
||
## LAN — 192.168.2.0/24
|
||
|
||
### Address map
|
||
|
||
| Range | Purpose |
|
||
|---|---|
|
||
| .1–.9 | Reserved, never assign |
|
||
| .10–.59 | Client DHCP pool (router-assigned) |
|
||
| .60–.219 | Unallocated buffer |
|
||
| .220–.229 | Virtual nodes (VMs / LXC containers) |
|
||
| .230–.239 | Expansion buffer (reserved, unallocated) |
|
||
| .240–.249 | Physical nodes (bare-metal hosts) |
|
||
| .250–.253 | Network services |
|
||
| .254 | Router / gateway |
|
||
|
||
### Network services (.250–.253)
|
||
|
||
| IP | Hostname | Role |
|
||
|---|---|---|
|
||
| `192.168.2.254` | router | Gateway (TP-Link) |
|
||
| `192.168.2.253` | domain-controller | FreeIPA — authoritative DNS for `sweet.home`, Kerberos, LDAP |
|
||
| `192.168.2.250`–`.252` | — | Reserved for future network services |
|
||
|
||
### Physical nodes (.240–.249)
|
||
|
||
| IP | Hostname | Role |
|
||
|---|---|---|
|
||
| `192.168.2.245` | pve1 | Proxmox VE hypervisor |
|
||
| `192.168.2.244` | pbs | Proxmox Backup Server |
|
||
| `192.168.2.243` | nixos | Bare-metal workstation (`baremetal-gui`) |
|
||
| `192.168.2.246`–`.249` | — | Reserved — second Proxmox node and associated services |
|
||
| `192.168.2.240`–`.242` | — | Reserved |
|
||
|
||
pve1 sits mid-range deliberately so a second Proxmox node can slot in on either side.
|
||
|
||
### Virtual nodes (.220–.229)
|
||
|
||
All VMs and LXC containers run on pve1.
|
||
|
||
| IP | Hostname | Role | Status |
|
||
|---|---|---|---|
|
||
| `192.168.2.229` | ha-vip-lan | HA file server LAN floating VIP (Pacemaker `vip-lan`) — LAN iSCSI + NFS | Active |
|
||
| `192.168.2.228` | ha-node1 | HA file server node 1 — management NIC | Active |
|
||
| `192.168.2.227` | ha-node2 | HA file server node 2 — management NIC | Active |
|
||
| `192.168.2.226` | server | Current NFS/ZFS file server — retires when HA is live | Retiring |
|
||
| `192.168.2.225` | docker | Docker / Traefik stack | Active |
|
||
| `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
|
||
| `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active |
|
||
| `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active |
|
||
| `192.168.2.221` | tor-relay | Tor relay | Active |
|
||
| `192.168.2.220` | pdm | Proxmox Deploy Manager | Active |
|
||
|
||
### Client DHCP pool (.10–.59)
|
||
|
||
Assigned by the router. DNS option points to `192.168.2.253` (domain-controller).
|
||
|
||
Devices in this range: phones, laptops, IoT, Canon printer, any non-infrastructure host.
|
||
No static reservations for infrastructure hosts — all infra uses static IP configuration
|
||
on the guest itself (not DHCP reservations), so IPs survive VM recreation regardless of
|
||
MAC address churn.
|
||
|
||
---
|
||
|
||
## Cluster network — VLAN 10 — 192.168.10.224/29
|
||
|
||
Internal to pve1 only. Proxmox bridge `vmbr1`, no physical NIC attached.
|
||
|
||
| IP | Hostname | Interface role |
|
||
|---|---|---|
|
||
| `192.168.10.228` | ha-node1 | DRBD replication + Corosync ring0 (primary heartbeat) |
|
||
| `192.168.10.227` | ha-node2 | DRBD replication + Corosync ring0 (primary heartbeat) |
|
||
| — | no gateway | Isolated — not routed to LAN or internet |
|
||
|
||
Corosync ring1 (backup heartbeat only) uses the LAN IPs (`192.168.2.228` / `192.168.2.227`)
|
||
over `vmbr0` — no additional bridge needed, and DRBD traffic never crosses ring1.
|
||
|
||
---
|
||
|
||
## Storage-client network — VLAN 20 — 192.168.20.0/24
|
||
|
||
Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached.
|
||
|
||
| IP | Hostname | Interface / role |
|
||
|---|---|---|
|
||
| `192.168.20.229` | ha-vip-storage | Pacemaker floating VIP — iSCSI endpoint (port 3260) |
|
||
| `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
|
||
| `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
|
||
| `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — temporary, for final rsync before decommission |
|
||
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — iSCSI initiator |
|
||
| — | no gateway | Isolated — not routed to LAN or internet |
|
||
|
||
iSCSI initiators connect to the VIP (`192.168.20.229:3260`) so sessions survive failover
|
||
transparently. Portal listens on `[::0]:3260`; firewall restricts port 3260 to
|
||
`192.168.20.0/24` source only — LAN clients cannot reach iSCSI.
|
||
|
||
---
|
||
|
||
## Migration reference
|
||
|
||
Current → target IP for every host being renumbered.
|
||
|
||
| Host | Current IP | New IP | Config location |
|
||
|---|---|---|---|
|
||
| router | `192.168.2.254` | `192.168.2.254` | unchanged |
|
||
| domain-controller | `192.168.2.138` | `192.168.2.253` | `/etc/sysconfig/network-scripts/ifcfg-eth0` on guest |
|
||
| pve1 | `192.168.2.250` | `192.168.2.245` | `/etc/network/interfaces` on Proxmox host |
|
||
| pbs | `192.168.2.108` | `192.168.2.244` | static config on PBS host |
|
||
| nixos workstation | `192.168.2.119` | `192.168.2.243` | `networking.interfaces` / NetworkManager on guest |
|
||
| ha-node1 | — | `192.168.2.228` (LAN), `192.168.10.228` (cluster/VLAN 10), `192.168.20.228` (storage/VLAN 20) | active |
|
||
| ha-node2 | — | `192.168.2.227` (LAN), `192.168.10.227` (cluster/VLAN 10), `192.168.20.227` (storage/VLAN 20) | active |
|
||
| ha-vip-lan | — | `192.168.2.229` (vmbr0 / Pacemaker `vip-lan`) — NFS endpoint for LAN clients | active |
|
||
| ha-vip-storage | — | `192.168.20.229` (vmbr2 / Pacemaker `vip-storage`) — iSCSI endpoint for VLAN 20 clients | active |
|
||
| server | `192.168.2.252` | `192.168.2.226` | static config on guest |
|
||
| docker | `192.168.2.249` | `192.168.2.225` | static config on guest |
|
||
| nix-cache | `192.168.2.120` | `192.168.2.224` | static config on guest |
|
||
| pxe-boot | `192.168.2.247` | `192.168.2.223` | static config on guest; update `vars.pxeServerIp` in `variables.nix` ✓ |
|
||
| tailscale-router | `192.168.2.121` | `192.168.2.222` | static config on guest |
|
||
| tor-relay | `192.168.2.107` | `192.168.2.221` | static config on guest |
|
||
| pdm | `192.168.2.248` | `192.168.2.220` | static config on guest |
|
||
|
||
### Cutover notes
|
||
|
||
- **Do domain-controller first** — it becomes the DNS server; everything else depends on it
|
||
having its new IP and FreeIPA DNS configured before Pi-hole is retired.
|
||
- **pve1 last among physical hosts** — changing the Proxmox management IP drops the web UI
|
||
briefly; all guests keep running.
|
||
- **Update Pi-hole custom.list / FreeIPA DNS A records** to new IPs before flipping any host,
|
||
so name resolution stays valid throughout the migration.
|
||
- **variables.nix already updated** for `pxeServerIp` (.247→.223), `pbsIp` (.108→.244), and
|
||
new `domainControllerIp` (.253). Rebuild affected hosts after renumbering.
|
||
- **Router DHCP**: once domain-controller is at .253 and FreeIPA DNS is serving `sweet.home`,
|
||
switch router DHCP on with pool .10–.59 and DNS option pointing to .253; retire Pi-hole CT.
|
||
- **Pi-hole's iPXE dnsmasq config** (`99-ipxe-chainload.conf`) moves to the pxe-boot CT as a
|
||
dnsmasq proxy-mode config before Pi-hole is decommissioned.
|