nixos/docs/network-cutover.md

Network Cutover Plan

Moves the LAN from the current flat/Pi-hole-managed state to the new IP scheme defined in docs/ip-addressing.md. Works in five independent stages — each stage is safe to pause after and resume later. Rollback steps are given at every point where something can break.

Before starting anything: confirm you have

• SSH access to 192.168.2.138 (domain-controller, current IP)
• SSH access to 192.168.2.250 (pve1)
• Browser access to Pi-hole admin at http://192.168.2.253
• Browser access to router admin at http://192.168.2.254
• The FreeIPA admin password to hand

---

Stage 1 — Prepare FreeIPA DNS (zero downtime)

Everything here is additive. Pi-hole keeps running. Nothing breaks if you stop mid-stage.

1a. Add NextDNS forwarders

ssh wayne@192.168.2.138
kinit admin          # enter FreeIPA admin password when prompted
ipa dnsconfig-mod \
  --forwarder=45.90.28.142 \
  --forwarder=45.90.30.142 \
  --forward-policy=only

Verify external resolution works through FreeIPA before continuing:

dig @127.0.0.1 google.com +short   # must return an IP, not SERVFAIL

1b. Add A records for every host at their CURRENT IPs

These represent the live state now. You'll update each record to the new IP when you renumber that host in Stage 5.

ipa dnsrecord-add sweet.home pve1             --a-rec 192.168.2.250
ipa dnsrecord-add sweet.home pbs              --a-rec 192.168.2.108
ipa dnsrecord-add sweet.home nixos            --a-rec 192.168.2.119
ipa dnsrecord-add sweet.home server           --a-rec 192.168.2.252
ipa dnsrecord-add sweet.home docker           --a-rec 192.168.2.249
ipa dnsrecord-add sweet.home nix-cache        --a-rec 192.168.2.120
ipa dnsrecord-add sweet.home pxe-boot         --a-rec 192.168.2.247
ipa dnsrecord-add sweet.home tailscale-router --a-rec 192.168.2.121
ipa dnsrecord-add sweet.home tor-relay        --a-rec 192.168.2.107
ipa dnsrecord-add sweet.home pdm              --a-rec 192.168.2.248
ipa dnsrecord-add sweet.home router           --a-rec 192.168.2.254

1c. Clean up stale reverse-zone PTR records

FreeIPA already has PTR records from an earlier import but some are wrong. Fix them now so reverse DNS is accurate from day one.

# Remove stale "win11" entry at .250 (should be pve1)
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec win11.
ipa dnsrecord-add 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.

# Fix unqualified PTR records (missing .sweet.home. suffix)
ipa dnsrecord-mod 2.168.192.in-addr.arpa 108 --ptr-rec pbs.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 248 --ptr-rec pdm.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 249 --ptr-rec docker.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 252 --ptr-rec server.sweet.home.

# Add any missing PTR records
ipa dnsrecord-add 2.168.192.in-addr.arpa 119 --ptr-rec nixos.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 120 --ptr-rec nix-cache.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 121 --ptr-rec tailscale-router.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 247 --ptr-rec pxe-boot.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 254 --ptr-rec router.sweet.home.

1d. Point domain-controller's own DNS at itself

sudo nmcli connection modify "System eth0" ipv4.dns "127.0.0.1"
sudo nmcli connection up "System eth0"

Verify:

dig pve1.sweet.home +short          # must return 192.168.2.250
dig google.com +short               # must return an IP (NextDNS forwarding)

Rollback 1d: sudo nmcli connection modify "System eth0" ipv4.dns "192.168.2.253" && sudo nmcli connection up "System eth0"

---

Stage 2 — Move pxe-boot DHCP options off Pi-hole (zero downtime)

Pi-hole's dnsmasq currently serves the iPXE boot options via 99-ipxe-chainload.conf. Before Pi-hole is retired, that config must move to the pxe-boot CT running dnsmasq in proxy mode so PXE boot keeps working.

2a. Add dnsmasq proxy config to the pxe-boot NixOS module

In modules/build-types/pxe-boot.nix, add:

services.dnsmasq = {
  enable = true;
  settings = {
    # Proxy mode: respond only to PXE DHCP requests, leave normal leases to router
    dhcp-range = [ "192.168.2.0,proxy" ];
    # iPXE client detection
    dhcp-match = [
      "set:ipxe,175"
      "set:efi64,option:client-arch,7"
      "set:efi64,option:client-arch,9"
    ];
    dhcp-userclass = "set:ipxe,iPXE";
    # Boot file selection
    dhcp-boot = [
      "tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
      "tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
      "tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
      "undionly.kpxe,,${vars.pxeServerIp}"
    ];
  };
};

2b. Rebuild and deploy the pxe-boot CT

# On pve1 — build the new tarball
nix build .#lxc-pxe-boot.config.system.build.tarball

# Verify dnsmasq starts correctly in the CT after deploy
ssh nixos@192.168.2.247 systemctl status dnsmasq

2c. Remove the iPXE config from Pi-hole

In the Pi-hole CT, remove /etc/dnsmasq.d/99-ipxe-chainload.conf and restart the FTL service:

ssh wayne@pve1.sweet.home \
  "sudo pct exec 100 -- bash -c 'rm /etc/dnsmasq.d/99-ipxe-chainload.conf && systemctl restart pihole-FTL'"

Verify: PXE boot a test machine — it should still get an iPXE response and reach the boot menu.

Rollback 2c: restore the file from the Pi-hole config backup at /etc/pihole/config_backups/ and restart pihole-FTL.

---

Stage 3 — DHCP migration: Pi-hole → router (brief maintenance window)

Do this in the evening. Existing DHCP leases stay valid during the switchover so connected devices don't drop — only new lease requests fail during the gap, which is under 60 seconds if you follow the steps in order.

The key: configure the router's DHCP DNS option to point at .253 (Pi-hole's current IP). This way, all new leases issued by the router still get the same DNS server address — clients never need to change their DNS config. When Pi-hole is retired and the DC takes .253 in Stage 4, .253 just starts answering differently. No client reconfiguration.

3a. Pre-configure router DHCP (do not enable yet)

Log into http://192.168.2.254, find the DHCP settings and fill in — but leave DHCP disabled until step 3b:

Setting	Value
Start IP	192.168.2.10
End IP	192.168.2.59
Subnet mask	255.255.255.0
Gateway	192.168.2.254
Primary DNS	192.168.2.253
Secondary DNS	(leave blank)
Lease time	24h

Save without enabling.

3b. Switchover (do steps in quick succession)

1. Disable Pi-hole DHCP: Pi-hole admin UI → Settings → DHCP → uncheck "DHCP server enabled" → Save
2. Enable router DHCP immediately after step 1

3c. Verify router DHCP is working

On a phone or laptop, disconnect from WiFi and reconnect (or run sudo dhclient -r && sudo dhclient on a Linux host):

ip addr show        # IP should be in 192.168.2.10–59 range
dig google.com      # should resolve (Pi-hole DNS still running at .253)
dig pve1.sweet.home # should resolve via FreeIPA at .138 (relayed via Pi-hole)

Wait 10–15 minutes for the most active devices to renew their leases. There's no need to wait for all leases to expire before proceeding.

Rollback 3b: Re-enable Pi-hole DHCP. Disable router DHCP. Done — existing leases remain valid so most devices are unaffected.

---

Stage 4 — Move domain-controller from .138 to .253

Pi-hole lives at .253. The DC must take .253 the moment Pi-hole stops so clients that still have .253 as their DNS server don't notice the change. Script these commands in advance and run them in rapid succession.

Pre-stage: have this SSH command ready before running step 4a:

ssh wayne@192.168.2.138 "
  sudo nmcli connection modify 'System eth0' \
    ipv4.addresses '192.168.2.253/24' \
    ipv4.gateway '192.168.2.254' \
    ipv4.dns '127.0.0.1' \
    ipv4.method manual && \
  sudo nmcli connection up 'System eth0'
"

Also update the Proxmox VM config to match (run from pve1):

sudo qm set 108 \
  --ipconfig0 ip=192.168.2.253/24,gw=192.168.2.254 \
  --nameserver 192.168.2.253

4a. Stop Pi-hole

ssh wayne@pve1.sweet.home "sudo pct stop 100"

4b. Immediately: change DC's IP to .253

Run the pre-staged SSH command from above. You have ~30 seconds before any client notices Pi-hole is gone. If SSH to .138 refuses (the IP is already changing), open a Proxmox console to VM 108 and run the nmcli commands there.

4c. Update Proxmox VM config

Run the pre-staged qm set 108 command from above.

4d. Verify

ssh wayne@192.168.2.253   # must connect (new DC IP)
dig @192.168.2.253 pve1.sweet.home +short   # must return 192.168.2.250
dig @192.168.2.253 google.com +short        # must return an IP

From a client device that renewed its DHCP lease in Stage 3:

cat /etc/resolv.conf    # should show 192.168.2.253
dig pve1.sweet.home     # should resolve

Rollback 4: ssh wayne@pve1.sweet.home "sudo pct start 100". Change DC IP back to .138 via Proxmox console. This restores full Pi-hole DNS/DHCP service. Leave Pi-hole CT stopped-but-intact for 48 hours before deleting it.

---

Stage 5 — Host renumbering (one at a time, any order)

For each host:

1. Update FreeIPA DNS A record and PTR record to the new IP
2. Change the static IP on the host itself
3. Verify SSH to new IP
4. Update variables.nix if that host has an IP variable (pxe-boot, pbs — already done in this PR)

FreeIPA record update template (run as admin on domain-controller):

ipa dnsrecord-mod sweet.home <hostname> --a-rec <new-ip>
ipa dnsrecord-del 2.168.192.in-addr.arpa <old-last-octet> --ptr-rec <hostname>.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa <new-last-octet> --ptr-rec <hostname>.sweet.home.

Renumbering order

#	Host	Old IP	New IP	How to change IP
1	nixos workstation	.119	.243	NetworkManager on guest; or nmcli connection modify
2	nix-cache	.120	.224	pct set 102 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.224/24,gw=192.168.2.254 then pct reboot 102
3	tailscale-router	.121	.222	Static config on guest; check Tailscale ACLs if IP is referenced there
4	tor-relay	.107	.221	pct set 104 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.221/24,gw=192.168.2.254 then pct reboot 104
5	pdm	.248	.220	pct set 106 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.220/24,gw=192.168.2.254 then pct reboot 106
6	pxe-boot	.247	.223	pct set 103 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.223/24,gw=192.168.2.254 then rebuild NixOS (already updated in variables.nix)
7	server	.252	.226	Static config on guest; NFS clients (docker) lose mounts briefly — they remount automatically
8	docker	.249	.225	Static config on guest; do this after server is at .226
9	pbs	.108	.244	Static config on PBS host itself; update in pbsIp already done in variables.nix
10	pve1	.250	.245	Edit /etc/network/interfaces on the Proxmox host — see below

pve1 renumber (step 10 — do last)

All guests keep running; only the Proxmox web UI is briefly unreachable.

ssh wayne@pve1.sweet.home

# Edit /etc/network/interfaces: change address from .250 to .245
sudo nano /etc/network/interfaces
# Change:  address 192.168.2.250/24
# To:      address 192.168.2.245/24

sudo systemctl restart networking
# SSH will drop here — reconnect to new IP

ssh wayne@192.168.2.245    # verify

Update FreeIPA DNS:

ipa dnsrecord-mod sweet.home pve1 --a-rec 192.168.2.245
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 245 --ptr-rec pve1.sweet.home.

Rollback any step 5 host: change the IP back on the guest and update the FreeIPA record back to the old IP. The old IP is unoccupied so you can temporarily use either.

---

Stage 6 — HA storage cutover (docker NFS remount)

Prerequisites:

• HA cluster fully deployed and vip-storage (nfs.storage.home → 192.168.20.229) serving NFS ✓
• DNS configured: storage.home zone populated, nfs.storage.home resolves to 192.168.20.229 ✓
• docker CT has eth1 on vmbr2 (docker.storage.home → 192.168.20.225) ✓
• Final rsync from server.sweet.home to /srv/ha-data complete before step 6b

docker.sweet.home currently NFS-mounts its persistent volumes from server.sweet.home (192.168.2.226:/tank/docker/...). This stage moves those mounts to the HA cluster's storage VIP so server can be decommissioned.

6a. Final rsync from server to HA cluster

Run from server.sweet.home (or over SSH from the workstation) to sync any data written since the initial rsync:

# Confirm active HA node and mount point
ssh wayne@192.168.2.228 'sudo findmnt /srv/ha-data'  # check which node is active

# rsync each dataset (adjust source paths to match /tank layout on server)
sudo rsync -av --delete /tank/docker/config/    wayne@<active-node-ip>:/srv/ha-data/docker/config/
sudo rsync -av --delete /tank/docker/databases/ wayne@<active-node-ip>:/srv/ha-data/docker/databases/
sudo rsync -av --delete /tank/docker/volumes/   wayne@<active-node-ip>:/srv/ha-data/docker/volumes/
sudo rsync -av --delete /tank/docker/nextcloud-data/ wayne@<active-node-ip>:/srv/ha-data/docker/nextcloud-data/

6b. Update docker NixOS config to mount from vip-storage

In hosts/docker/host.nix (or wherever the NFS mount fileSystems are declared), change the NFS server from server.sweet.home / 192.168.2.226 to nfs.storage.home:

# Before:
fileSystems."/mnt/docker/config" = {
  device = "server:/tank/docker/config";   # or 192.168.2.226:...
  ...
};

# After:
fileSystems."/mnt/docker/config" = {
  device = "nfs.storage.home:/srv/ha-data/docker/config";
  ...
};

Using the DNS name (nfs.storage.home) rather than the VIP IP means the mount config survives a future VIP renumber without touching the NixOS config. Repeat for all four docker shares (config, databases, volumes, nextcloud-data). Then rebuild docker:

# On the workstation — or via Switch-nix on docker itself
sudo nixos-rebuild switch --no-write-lock-file --refresh \
  --flake "git+https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos.git#lxc-docker"

6c. Verify mounts and container health

ssh wayne@192.168.2.225 'findmnt | grep 192.168.20'   # mounts should show vip-storage
ssh wayne@192.168.2.225 'docker ps'                    # all containers running

Spot-check Nextcloud, Traefik, and any database containers for connectivity.

6d. Decommission server.sweet.home

Once docker is confirmed healthy on the HA NFS mounts:

# Stop server VM on pve1
ssh wayne@192.168.2.245 'sudo qm stop 101'

# (Optional) Archive the ZFS pool snapshot before destroying
# Then after a settling period:
ssh wayne@192.168.2.245 'sudo qm destroy 101 --destroy-unreferenced-disks 1'

---

Stage 7 — Final cleanup

Once all hosts are at their new IPs and verified:

# Delete the Pi-hole CT (already stopped since Stage 4)
ssh wayne@pve1.sweet.home "sudo pct destroy 100"

# Remove stale FreeIPA records for retired addresses
ipa dnsrecord-del sweet.home pihole --del-all
ipa dnsrecord-del 2.168.192.in-addr.arpa 253 --ptr-rec pihole.sweet.home.

# Rebuild any NixOS hosts that reference pbsIp or pxeServerIp to pick up
# the updated variables.nix values (pxe-boot mandatory; others as convenient)

---

Rollback summary

What broke	How to roll back
FreeIPA DNS not resolving	Check systemctl status named on DC; restart if failed
FreeIPA DNS unreachable	pct start 100 on pve1 (restores Pi-hole)
Router DHCP not handing out leases	Re-enable Pi-hole DHCP; disable router DHCP
DC unreachable after IP change	Proxmox console on VM 108 → nmcli connection up "System eth0" with old IP
Host unreachable after renumber	Proxmox console → revert IP; or pct set <id> --net0 ... old IP and reboot CT
pve1 web UI gone after renumber	SSH to .245 and check /etc/network/interfaces; if wrong, fix and restart networking