Archived
beatzaplentyandClaude Sonnet 4.6
944af9597d
Check NixOS configurations / eval-hosts (push) Failing after 14m52s
Adds a DNS Zones section to ip-addressing.md covering the three FreeIPA zones (sweet.home / cluster.home / storage.home) with all A records and PTR records as configured. Updates Stage 6 in network-cutover.md to reference nfs.storage.home by DNS name rather than the raw VIP IP, and marks DNS as a completed prerequisite for the HA storage cutover. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
449 lines
16 KiB
Markdown
449 lines
16 KiB
Markdown
# Network Cutover Plan
|
||
|
||
Moves the LAN from the current flat/Pi-hole-managed state to the new IP scheme
|
||
defined in `docs/ip-addressing.md`. Works in five independent stages — each
|
||
stage is safe to pause after and resume later. Rollback steps are given at
|
||
every point where something can break.
|
||
|
||
**Before starting anything:** confirm you have
|
||
- SSH access to `192.168.2.138` (domain-controller, current IP)
|
||
- SSH access to `192.168.2.250` (pve1)
|
||
- Browser access to Pi-hole admin at `http://192.168.2.253`
|
||
- Browser access to router admin at `http://192.168.2.254`
|
||
- The FreeIPA `admin` password to hand
|
||
|
||
---
|
||
|
||
## Stage 1 — Prepare FreeIPA DNS (zero downtime)
|
||
|
||
Everything here is additive. Pi-hole keeps running. Nothing breaks if you stop
|
||
mid-stage.
|
||
|
||
### 1a. Add NextDNS forwarders
|
||
|
||
```bash
|
||
ssh wayne@192.168.2.138
|
||
kinit admin # enter FreeIPA admin password when prompted
|
||
ipa dnsconfig-mod \
|
||
--forwarder=45.90.28.142 \
|
||
--forwarder=45.90.30.142 \
|
||
--forward-policy=only
|
||
```
|
||
|
||
**Verify external resolution works through FreeIPA before continuing:**
|
||
```bash
|
||
dig @127.0.0.1 google.com +short # must return an IP, not SERVFAIL
|
||
```
|
||
|
||
### 1b. Add A records for every host at their CURRENT IPs
|
||
|
||
These represent the live state now. You'll update each record to the new IP
|
||
when you renumber that host in Stage 5.
|
||
|
||
```bash
|
||
ipa dnsrecord-add sweet.home pve1 --a-rec 192.168.2.250
|
||
ipa dnsrecord-add sweet.home pbs --a-rec 192.168.2.108
|
||
ipa dnsrecord-add sweet.home nixos --a-rec 192.168.2.119
|
||
ipa dnsrecord-add sweet.home server --a-rec 192.168.2.252
|
||
ipa dnsrecord-add sweet.home docker --a-rec 192.168.2.249
|
||
ipa dnsrecord-add sweet.home nix-cache --a-rec 192.168.2.120
|
||
ipa dnsrecord-add sweet.home pxe-boot --a-rec 192.168.2.247
|
||
ipa dnsrecord-add sweet.home tailscale-router --a-rec 192.168.2.121
|
||
ipa dnsrecord-add sweet.home tor-relay --a-rec 192.168.2.107
|
||
ipa dnsrecord-add sweet.home pdm --a-rec 192.168.2.248
|
||
ipa dnsrecord-add sweet.home router --a-rec 192.168.2.254
|
||
```
|
||
|
||
### 1c. Clean up stale reverse-zone PTR records
|
||
|
||
FreeIPA already has PTR records from an earlier import but some are wrong.
|
||
Fix them now so reverse DNS is accurate from day one.
|
||
|
||
```bash
|
||
# Remove stale "win11" entry at .250 (should be pve1)
|
||
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec win11.
|
||
ipa dnsrecord-add 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
|
||
|
||
# Fix unqualified PTR records (missing .sweet.home. suffix)
|
||
ipa dnsrecord-mod 2.168.192.in-addr.arpa 108 --ptr-rec pbs.sweet.home.
|
||
ipa dnsrecord-mod 2.168.192.in-addr.arpa 248 --ptr-rec pdm.sweet.home.
|
||
ipa dnsrecord-mod 2.168.192.in-addr.arpa 249 --ptr-rec docker.sweet.home.
|
||
ipa dnsrecord-mod 2.168.192.in-addr.arpa 252 --ptr-rec server.sweet.home.
|
||
|
||
# Add any missing PTR records
|
||
ipa dnsrecord-add 2.168.192.in-addr.arpa 119 --ptr-rec nixos.sweet.home.
|
||
ipa dnsrecord-add 2.168.192.in-addr.arpa 120 --ptr-rec nix-cache.sweet.home.
|
||
ipa dnsrecord-add 2.168.192.in-addr.arpa 121 --ptr-rec tailscale-router.sweet.home.
|
||
ipa dnsrecord-add 2.168.192.in-addr.arpa 247 --ptr-rec pxe-boot.sweet.home.
|
||
ipa dnsrecord-add 2.168.192.in-addr.arpa 254 --ptr-rec router.sweet.home.
|
||
```
|
||
|
||
### 1d. Point domain-controller's own DNS at itself
|
||
|
||
```bash
|
||
sudo nmcli connection modify "System eth0" ipv4.dns "127.0.0.1"
|
||
sudo nmcli connection up "System eth0"
|
||
```
|
||
|
||
**Verify:**
|
||
```bash
|
||
dig pve1.sweet.home +short # must return 192.168.2.250
|
||
dig google.com +short # must return an IP (NextDNS forwarding)
|
||
```
|
||
|
||
**Rollback 1d:** `sudo nmcli connection modify "System eth0" ipv4.dns "192.168.2.253" && sudo nmcli connection up "System eth0"`
|
||
|
||
---
|
||
|
||
## Stage 2 — Move pxe-boot DHCP options off Pi-hole (zero downtime)
|
||
|
||
Pi-hole's dnsmasq currently serves the iPXE boot options via
|
||
`99-ipxe-chainload.conf`. Before Pi-hole is retired, that config must move to
|
||
the pxe-boot CT running dnsmasq in proxy mode so PXE boot keeps working.
|
||
|
||
### 2a. Add dnsmasq proxy config to the pxe-boot NixOS module
|
||
|
||
In `modules/build-types/pxe-boot.nix`, add:
|
||
|
||
```nix
|
||
services.dnsmasq = {
|
||
enable = true;
|
||
settings = {
|
||
# Proxy mode: respond only to PXE DHCP requests, leave normal leases to router
|
||
dhcp-range = [ "192.168.2.0,proxy" ];
|
||
# iPXE client detection
|
||
dhcp-match = [
|
||
"set:ipxe,175"
|
||
"set:efi64,option:client-arch,7"
|
||
"set:efi64,option:client-arch,9"
|
||
];
|
||
dhcp-userclass = "set:ipxe,iPXE";
|
||
# Boot file selection
|
||
dhcp-boot = [
|
||
"tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
|
||
"tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
|
||
"tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
|
||
"undionly.kpxe,,${vars.pxeServerIp}"
|
||
];
|
||
};
|
||
};
|
||
```
|
||
|
||
### 2b. Rebuild and deploy the pxe-boot CT
|
||
|
||
```bash
|
||
# On pve1 — build the new tarball
|
||
nix build .#lxc-pxe-boot.config.system.build.tarball
|
||
|
||
# Verify dnsmasq starts correctly in the CT after deploy
|
||
ssh nixos@192.168.2.247 systemctl status dnsmasq
|
||
```
|
||
|
||
### 2c. Remove the iPXE config from Pi-hole
|
||
|
||
In the Pi-hole CT, remove `/etc/dnsmasq.d/99-ipxe-chainload.conf` and
|
||
restart the FTL service:
|
||
|
||
```bash
|
||
ssh wayne@pve1.sweet.home \
|
||
"sudo pct exec 100 -- bash -c 'rm /etc/dnsmasq.d/99-ipxe-chainload.conf && systemctl restart pihole-FTL'"
|
||
```
|
||
|
||
**Verify:** PXE boot a test machine — it should still get an iPXE response and
|
||
reach the boot menu.
|
||
|
||
**Rollback 2c:** restore the file from the Pi-hole config backup at
|
||
`/etc/pihole/config_backups/` and restart pihole-FTL.
|
||
|
||
---
|
||
|
||
## Stage 3 — DHCP migration: Pi-hole → router (brief maintenance window)
|
||
|
||
**Do this in the evening.** Existing DHCP leases stay valid during the
|
||
switchover so connected devices don't drop — only new lease requests fail
|
||
during the gap, which is under 60 seconds if you follow the steps in order.
|
||
|
||
The key: configure the router's DHCP DNS option to point at `.253` (Pi-hole's
|
||
current IP). This way, all new leases issued by the router still get the same
|
||
DNS server address — clients never need to change their DNS config. When Pi-hole
|
||
is retired and the DC takes `.253` in Stage 4, `.253` just starts answering
|
||
differently. No client reconfiguration.
|
||
|
||
### 3a. Pre-configure router DHCP (do not enable yet)
|
||
|
||
Log into `http://192.168.2.254`, find the DHCP settings and fill in — but
|
||
leave DHCP **disabled** until step 3b:
|
||
|
||
| Setting | Value |
|
||
|---|---|
|
||
| Start IP | 192.168.2.10 |
|
||
| End IP | 192.168.2.59 |
|
||
| Subnet mask | 255.255.255.0 |
|
||
| Gateway | 192.168.2.254 |
|
||
| Primary DNS | 192.168.2.253 |
|
||
| Secondary DNS | *(leave blank)* |
|
||
| Lease time | 24h |
|
||
|
||
Save without enabling.
|
||
|
||
### 3b. Switchover (do steps in quick succession)
|
||
|
||
1. **Disable Pi-hole DHCP:** Pi-hole admin UI → Settings → DHCP → uncheck
|
||
"DHCP server enabled" → Save
|
||
2. **Enable router DHCP** immediately after step 1
|
||
|
||
### 3c. Verify router DHCP is working
|
||
|
||
On a phone or laptop, disconnect from WiFi and reconnect (or run
|
||
`sudo dhclient -r && sudo dhclient` on a Linux host):
|
||
|
||
```bash
|
||
ip addr show # IP should be in 192.168.2.10–59 range
|
||
dig google.com # should resolve (Pi-hole DNS still running at .253)
|
||
dig pve1.sweet.home # should resolve via FreeIPA at .138 (relayed via Pi-hole)
|
||
```
|
||
|
||
Wait 10–15 minutes for the most active devices to renew their leases. There's
|
||
no need to wait for all leases to expire before proceeding.
|
||
|
||
**Rollback 3b:** Re-enable Pi-hole DHCP. Disable router DHCP. Done — existing
|
||
leases remain valid so most devices are unaffected.
|
||
|
||
---
|
||
|
||
## Stage 4 — Move domain-controller from .138 to .253
|
||
|
||
Pi-hole lives at `.253`. The DC must take `.253` the moment Pi-hole stops so
|
||
clients that still have `.253` as their DNS server don't notice the change.
|
||
Script these commands in advance and run them in rapid succession.
|
||
|
||
**Pre-stage: have this SSH command ready before running step 4a:**
|
||
```bash
|
||
ssh wayne@192.168.2.138 "
|
||
sudo nmcli connection modify 'System eth0' \
|
||
ipv4.addresses '192.168.2.253/24' \
|
||
ipv4.gateway '192.168.2.254' \
|
||
ipv4.dns '127.0.0.1' \
|
||
ipv4.method manual && \
|
||
sudo nmcli connection up 'System eth0'
|
||
"
|
||
```
|
||
|
||
**Also update the Proxmox VM config to match (run from pve1):**
|
||
```bash
|
||
sudo qm set 108 \
|
||
--ipconfig0 ip=192.168.2.253/24,gw=192.168.2.254 \
|
||
--nameserver 192.168.2.253
|
||
```
|
||
|
||
### 4a. Stop Pi-hole
|
||
|
||
```bash
|
||
ssh wayne@pve1.sweet.home "sudo pct stop 100"
|
||
```
|
||
|
||
### 4b. Immediately: change DC's IP to .253
|
||
|
||
Run the pre-staged SSH command from above. You have ~30 seconds before any
|
||
client notices Pi-hole is gone. If SSH to `.138` refuses (the IP is already
|
||
changing), open a Proxmox console to VM 108 and run the `nmcli` commands
|
||
there.
|
||
|
||
### 4c. Update Proxmox VM config
|
||
|
||
Run the pre-staged `qm set 108` command from above.
|
||
|
||
### 4d. Verify
|
||
|
||
```bash
|
||
ssh wayne@192.168.2.253 # must connect (new DC IP)
|
||
dig @192.168.2.253 pve1.sweet.home +short # must return 192.168.2.250
|
||
dig @192.168.2.253 google.com +short # must return an IP
|
||
```
|
||
|
||
From a client device that renewed its DHCP lease in Stage 3:
|
||
```bash
|
||
cat /etc/resolv.conf # should show 192.168.2.253
|
||
dig pve1.sweet.home # should resolve
|
||
```
|
||
|
||
**Rollback 4:** `ssh wayne@pve1.sweet.home "sudo pct start 100"`. Change DC IP
|
||
back to .138 via Proxmox console. This restores full Pi-hole DNS/DHCP service.
|
||
Leave Pi-hole CT stopped-but-intact for 48 hours before deleting it.
|
||
|
||
---
|
||
|
||
## Stage 5 — Host renumbering (one at a time, any order)
|
||
|
||
For each host:
|
||
1. Update FreeIPA DNS A record and PTR record to the new IP
|
||
2. Change the static IP on the host itself
|
||
3. Verify SSH to new IP
|
||
4. Update `variables.nix` if that host has an IP variable (pxe-boot, pbs — already done in this PR)
|
||
|
||
**FreeIPA record update template** (run as admin on domain-controller):
|
||
```bash
|
||
ipa dnsrecord-mod sweet.home <hostname> --a-rec <new-ip>
|
||
ipa dnsrecord-del 2.168.192.in-addr.arpa <old-last-octet> --ptr-rec <hostname>.sweet.home.
|
||
ipa dnsrecord-add 2.168.192.in-addr.arpa <new-last-octet> --ptr-rec <hostname>.sweet.home.
|
||
```
|
||
|
||
### Renumbering order
|
||
|
||
| # | Host | Old IP | New IP | How to change IP |
|
||
|---|---|---|---|---|
|
||
| 1 | nixos workstation | .119 | .243 | NetworkManager on guest; or `nmcli connection modify` |
|
||
| 2 | nix-cache | .120 | .224 | `pct set 102 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.224/24,gw=192.168.2.254` then `pct reboot 102` |
|
||
| 3 | tailscale-router | .121 | .222 | Static config on guest; check Tailscale ACLs if IP is referenced there |
|
||
| 4 | tor-relay | .107 | .221 | `pct set 104 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.221/24,gw=192.168.2.254` then `pct reboot 104` |
|
||
| 5 | pdm | .248 | .220 | `pct set 106 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.220/24,gw=192.168.2.254` then `pct reboot 106` |
|
||
| 6 | pxe-boot | .247 | .223 | `pct set 103 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.223/24,gw=192.168.2.254` then rebuild NixOS (already updated in variables.nix) |
|
||
| 7 | server | .252 | .226 | Static config on guest; NFS clients (docker) lose mounts briefly — they remount automatically |
|
||
| 8 | docker | .249 | .225 | Static config on guest; do this after server is at .226 |
|
||
| 9 | pbs | .108 | .244 | Static config on PBS host itself; update in `pbsIp` already done in variables.nix |
|
||
| 10 | pve1 | .250 | .245 | Edit `/etc/network/interfaces` on the Proxmox host — see below |
|
||
|
||
### pve1 renumber (step 10 — do last)
|
||
|
||
All guests keep running; only the Proxmox web UI is briefly unreachable.
|
||
|
||
```bash
|
||
ssh wayne@pve1.sweet.home
|
||
|
||
# Edit /etc/network/interfaces: change address from .250 to .245
|
||
sudo nano /etc/network/interfaces
|
||
# Change: address 192.168.2.250/24
|
||
# To: address 192.168.2.245/24
|
||
|
||
sudo systemctl restart networking
|
||
# SSH will drop here — reconnect to new IP
|
||
```
|
||
|
||
```bash
|
||
ssh wayne@192.168.2.245 # verify
|
||
```
|
||
|
||
Update FreeIPA DNS:
|
||
```bash
|
||
ipa dnsrecord-mod sweet.home pve1 --a-rec 192.168.2.245
|
||
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
|
||
ipa dnsrecord-add 2.168.192.in-addr.arpa 245 --ptr-rec pve1.sweet.home.
|
||
```
|
||
|
||
**Rollback any step 5 host:** change the IP back on the guest and update the
|
||
FreeIPA record back to the old IP. The old IP is unoccupied so you can
|
||
temporarily use either.
|
||
|
||
---
|
||
|
||
## Stage 6 — HA storage cutover (docker NFS remount)
|
||
|
||
> **Prerequisites:**
|
||
> - HA cluster fully deployed and `vip-storage` (`nfs.storage.home` → 192.168.20.229) serving NFS ✓
|
||
> - DNS configured: `storage.home` zone populated, `nfs.storage.home` resolves to 192.168.20.229 ✓
|
||
> - docker CT has eth1 on vmbr2 (`docker.storage.home` → 192.168.20.225) ✓
|
||
> - Final rsync from server.sweet.home to `/srv/ha-data` complete before step 6b
|
||
|
||
docker.sweet.home currently NFS-mounts its persistent volumes from `server.sweet.home`
|
||
(`192.168.2.226:/tank/docker/...`). This stage moves those mounts to the HA cluster's
|
||
storage VIP so server can be decommissioned.
|
||
|
||
### 6a. Final rsync from server to HA cluster
|
||
|
||
Run from server.sweet.home (or over SSH from the workstation) to sync any data written
|
||
since the initial rsync:
|
||
|
||
```bash
|
||
# Confirm active HA node and mount point
|
||
ssh wayne@192.168.2.228 'sudo findmnt /srv/ha-data' # check which node is active
|
||
|
||
# rsync each dataset (adjust source paths to match /tank layout on server)
|
||
sudo rsync -av --delete /tank/docker/config/ wayne@<active-node-ip>:/srv/ha-data/docker/config/
|
||
sudo rsync -av --delete /tank/docker/databases/ wayne@<active-node-ip>:/srv/ha-data/docker/databases/
|
||
sudo rsync -av --delete /tank/docker/volumes/ wayne@<active-node-ip>:/srv/ha-data/docker/volumes/
|
||
sudo rsync -av --delete /tank/docker/nextcloud-data/ wayne@<active-node-ip>:/srv/ha-data/docker/nextcloud-data/
|
||
```
|
||
|
||
### 6b. Update docker NixOS config to mount from vip-storage
|
||
|
||
In `hosts/docker/host.nix` (or wherever the NFS mount fileSystems are declared), change
|
||
the NFS server from `server.sweet.home` / `192.168.2.226` to `nfs.storage.home`:
|
||
|
||
```nix
|
||
# Before:
|
||
fileSystems."/mnt/docker/config" = {
|
||
device = "server:/tank/docker/config"; # or 192.168.2.226:...
|
||
...
|
||
};
|
||
|
||
# After:
|
||
fileSystems."/mnt/docker/config" = {
|
||
device = "nfs.storage.home:/srv/ha-data/docker/config";
|
||
...
|
||
};
|
||
```
|
||
|
||
Using the DNS name (`nfs.storage.home`) rather than the VIP IP means the mount
|
||
config survives a future VIP renumber without touching the NixOS config.
|
||
Repeat for all four docker shares (`config`, `databases`, `volumes`, `nextcloud-data`).
|
||
Then rebuild docker:
|
||
|
||
```bash
|
||
# On the workstation — or via Switch-nix on docker itself
|
||
sudo nixos-rebuild switch --no-write-lock-file --refresh \
|
||
--flake "git+https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos.git#lxc-docker"
|
||
```
|
||
|
||
### 6c. Verify mounts and container health
|
||
|
||
```bash
|
||
ssh wayne@192.168.2.225 'findmnt | grep 192.168.20' # mounts should show vip-storage
|
||
ssh wayne@192.168.2.225 'docker ps' # all containers running
|
||
```
|
||
|
||
Spot-check Nextcloud, Traefik, and any database containers for connectivity.
|
||
|
||
### 6d. Decommission server.sweet.home
|
||
|
||
Once docker is confirmed healthy on the HA NFS mounts:
|
||
|
||
```bash
|
||
# Stop server VM on pve1
|
||
ssh wayne@192.168.2.245 'sudo qm stop 101'
|
||
|
||
# (Optional) Archive the ZFS pool snapshot before destroying
|
||
# Then after a settling period:
|
||
ssh wayne@192.168.2.245 'sudo qm destroy 101 --destroy-unreferenced-disks 1'
|
||
```
|
||
|
||
---
|
||
|
||
## Stage 7 — Final cleanup
|
||
|
||
Once all hosts are at their new IPs and verified:
|
||
|
||
```bash
|
||
# Delete the Pi-hole CT (already stopped since Stage 4)
|
||
ssh wayne@pve1.sweet.home "sudo pct destroy 100"
|
||
|
||
# Remove stale FreeIPA records for retired addresses
|
||
ipa dnsrecord-del sweet.home pihole --del-all
|
||
ipa dnsrecord-del 2.168.192.in-addr.arpa 253 --ptr-rec pihole.sweet.home.
|
||
|
||
# Rebuild any NixOS hosts that reference pbsIp or pxeServerIp to pick up
|
||
# the updated variables.nix values (pxe-boot mandatory; others as convenient)
|
||
```
|
||
|
||
---
|
||
|
||
## Rollback summary
|
||
|
||
| What broke | How to roll back |
|
||
|---|---|
|
||
| FreeIPA DNS not resolving | Check `systemctl status named` on DC; restart if failed |
|
||
| FreeIPA DNS unreachable | `pct start 100` on pve1 (restores Pi-hole) |
|
||
| Router DHCP not handing out leases | Re-enable Pi-hole DHCP; disable router DHCP |
|
||
| DC unreachable after IP change | Proxmox console on VM 108 → `nmcli connection up "System eth0"` with old IP |
|
||
| Host unreachable after renumber | Proxmox console → revert IP; or `pct set <id> --net0 ...` old IP and reboot CT |
|
||
| pve1 web UI gone after renumber | SSH to .245 and check `/etc/network/interfaces`; if wrong, fix and restart networking |
|