beatzaplenty/nixos
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
This repository has been archived on 2026-07-30. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
bf4836efacf6a9ee7f7c95ed44b0f7d8a164ff62
nixos/docs/ip-addressing.md
T
beatzaplentyandClaude Sonnet 4.6 bf4836efac
Check NixOS configurations / eval-hosts (push) Failing after 17m31s
Details
refactor(ha): renumber cluster→VLAN10, storage→VLAN20; iSCSI/NFS protocol separation 
Renumber internal HA bridges so VLAN ID matches IP third octet:
  vmbr1 (cluster): 192.168.4.x → 192.168.10.x (VLAN 10, /29)
  vmbr2 (storage): 192.168.5.x → 192.168.20.x (VLAN 20, /24)

Enforce protocol separation at the firewall on both HA nodes:
  - iSCSI (3260/tcp): storage-client subnet (192.168.20.0/24) only
  - NFS (111,2049,20048 tcp+udp): LAN subnet (192.168.2.0/24) only
  - Cluster subnet (192.168.10.224/29) accepted wholesale (DRBD+Corosync)
Moves from allowedTCPPorts/allowedUDPPorts to source-restricted extraCommands
rules on the nixos-fw chain.

NFS exports reduced to LAN-only (drop haClientCidr lines from mkNfsExports).
Storage-client network clients use iSCSI only; no NFS on VLAN 20.

All HA script VIP/storage IP defaults updated to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 18:19:07 +10:00

7.7 KiB
Raw Blame History

IP Addressing Scheme

Subnets

Subnet VLAN CIDR Purpose Routed?
LAN 2 (native/untagged) 192.168.2.0/24 General LAN — clients and infrastructure Yes (gateway .254)
Cluster 10 192.168.10.224/29 HA file server DRBD replication + Corosync heartbeat No — internal vmbr1 only, no uplink
Storage client 20 192.168.20.0/24 HA file server iSCSI — docker and server access VIP here No — internal vmbr2 only, no uplink

The cluster and storage-client subnets never leave pve1. vmbr1 and vmbr2 are Proxmox Linux bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.

VLAN IDs match the third octet of each subnet (VLAN 2 → 192.168.2.x, VLAN 10 → 192.168.10.x, VLAN 20 → 192.168.20.x). The host octet is consistent across all subnets — e.g. ha-node1 is always .228: 192.168.2.228 (LAN), 192.168.10.228 (cluster), 192.168.20.228 (storage client).

Protocol separation (enforced by firewall on HA nodes):

  • iSCSI (port 3260): storage-client subnet (VLAN 20) only — docker and server
  • NFS (ports 111, 2049, 20048): LAN subnet (VLAN 2) only — pxe-boot and other LAN clients

LAN — 192.168.2.0/24

Address map

Range Purpose
.1–.9 Reserved, never assign
.10–.59 Client DHCP pool (router-assigned)
.60–.219 Unallocated buffer
.220–.229 Virtual nodes (VMs / LXC containers)
.230–.239 Expansion buffer (reserved, unallocated)
.240–.249 Physical nodes (bare-metal hosts)
.250–.253 Network services
.254 Router / gateway

Network services (.250–.253)

IP Hostname Role
192.168.2.254 router Gateway (TP-Link)
192.168.2.253 domain-controller FreeIPA — authoritative DNS for sweet.home, Kerberos, LDAP
192.168.2.250–.252 — Reserved for future network services

Physical nodes (.240–.249)

IP Hostname Role
192.168.2.245 pve1 Proxmox VE hypervisor
192.168.2.244 pbs Proxmox Backup Server
192.168.2.243 nixos Bare-metal workstation (baremetal-gui)
192.168.2.246–.249 — Reserved — second Proxmox node and associated services
192.168.2.240–.242 — Reserved

pve1 sits mid-range deliberately so a second Proxmox node can slot in on either side.

Virtual nodes (.220–.229)

All VMs and LXC containers run on pve1.

IP Hostname Role Status
192.168.2.229 ha-vip-lan HA file server LAN floating VIP (Pacemaker vip-lan) — LAN iSCSI + NFS Active
192.168.2.228 ha-node1 HA file server node 1 — management NIC Active
192.168.2.227 ha-node2 HA file server node 2 — management NIC Active
192.168.2.226 server Current NFS/ZFS file server — retires when HA is live Retiring
192.168.2.225 docker Docker / Traefik stack Active
192.168.2.224 nix-cache Nix binary cache + remote builder Active
192.168.2.223 pxe-boot PXE / TFTP / HTTP netboot server Active
192.168.2.222 tailscale-router Tailscale exit node / router Active
192.168.2.221 tor-relay Tor relay Active
192.168.2.220 pdm Proxmox Deploy Manager Active

Client DHCP pool (.10–.59)

Assigned by the router. DNS option points to 192.168.2.253 (domain-controller).

Devices in this range: phones, laptops, IoT, Canon printer, any non-infrastructure host. No static reservations for infrastructure hosts — all infra uses static IP configuration on the guest itself (not DHCP reservations), so IPs survive VM recreation regardless of MAC address churn.

Cluster network — VLAN 10 — 192.168.10.224/29

Internal to pve1 only. Proxmox bridge vmbr1, no physical NIC attached.

IP Hostname Interface role
192.168.10.228 ha-node1 DRBD replication + Corosync ring0 (primary heartbeat)
192.168.10.227 ha-node2 DRBD replication + Corosync ring0 (primary heartbeat)
— no gateway Isolated — not routed to LAN or internet

Corosync ring1 (backup heartbeat only) uses the LAN IPs (192.168.2.228 / 192.168.2.227) over vmbr0 — no additional bridge needed, and DRBD traffic never crosses ring1.

Storage-client network — VLAN 20 — 192.168.20.0/24

Internal to pve1 only. Proxmox bridge vmbr2, no physical NIC attached.

IP Hostname Interface / role
192.168.20.229 ha-vip-storage Pacemaker floating VIP — iSCSI endpoint (port 3260)
192.168.20.228 ha-node1 Storage-client NIC (ens20 / vmbr2)
192.168.20.227 ha-node2 Storage-client NIC (ens20 / vmbr2)
192.168.20.226 server Storage-client NIC (ens19 / vmbr2) — temporary, for final rsync before decommission
192.168.20.225 docker Storage-client NIC (eth1 / vmbr2) — iSCSI initiator
— no gateway Isolated — not routed to LAN or internet

iSCSI initiators connect to the VIP (192.168.20.229:3260) so sessions survive failover transparently. Portal listens on [::0]:3260; firewall restricts port 3260 to 192.168.20.0/24 source only — LAN clients cannot reach iSCSI.

Migration reference

Current → target IP for every host being renumbered.

Host Current IP New IP Config location
router 192.168.2.254 192.168.2.254 unchanged
domain-controller 192.168.2.138 192.168.2.253 /etc/sysconfig/network-scripts/ifcfg-eth0 on guest
pve1 192.168.2.250 192.168.2.245 /etc/network/interfaces on Proxmox host
pbs 192.168.2.108 192.168.2.244 static config on PBS host
nixos workstation 192.168.2.119 192.168.2.243 networking.interfaces / NetworkManager on guest
ha-node1 — 192.168.2.228 (LAN), 192.168.10.228 (cluster/VLAN 10), 192.168.20.228 (storage/VLAN 20) active
ha-node2 — 192.168.2.227 (LAN), 192.168.10.227 (cluster/VLAN 10), 192.168.20.227 (storage/VLAN 20) active
ha-vip-lan — 192.168.2.229 (vmbr0 / Pacemaker vip-lan) — NFS endpoint for LAN clients active
ha-vip-storage — 192.168.20.229 (vmbr2 / Pacemaker vip-storage) — iSCSI endpoint for VLAN 20 clients active
server 192.168.2.252 192.168.2.226 static config on guest
docker 192.168.2.249 192.168.2.225 static config on guest
nix-cache 192.168.2.120 192.168.2.224 static config on guest
pxe-boot 192.168.2.247 192.168.2.223 static config on guest; update vars.pxeServerIp in variables.nix ✓
tailscale-router 192.168.2.121 192.168.2.222 static config on guest
tor-relay 192.168.2.107 192.168.2.221 static config on guest
pdm 192.168.2.248 192.168.2.220 static config on guest

Cutover notes

  • Do domain-controller first — it becomes the DNS server; everything else depends on it having its new IP and FreeIPA DNS configured before Pi-hole is retired.
  • pve1 last among physical hosts — changing the Proxmox management IP drops the web UI briefly; all guests keep running.
  • Update Pi-hole custom.list / FreeIPA DNS A records to new IPs before flipping any host, so name resolution stays valid throughout the migration.
  • variables.nix already updated for pxeServerIp (.247→.223), pbsIp (.108→.244), and new domainControllerIp (.253). Rebuild affected hosts after renumbering.
  • Router DHCP: once domain-controller is at .253 and FreeIPA DNS is serving sweet.home, switch router DHCP on with pool .10–.59 and DNS option pointing to .253; retire Pi-hole CT.
  • Pi-hole's iPXE dnsmasq config (99-ipxe-chainload.conf) moves to the pxe-boot CT as a dnsmasq proxy-mode config before Pi-hole is decommissioned.
Reference in New Issue View Git Blame Copy Permalink