fix(server): prevent zfs-init-tank from wiping pool on udev race

zpool create -f was called if `zpool import -d /dev/disk/by-id` failed,
which could happen due to a race with systemd-udev-settle. The disk
would then be visible by the time zpool create ran, silently destroying
all data on an otherwise-intact pool.

Fix: locate the data disk first, retry the import directly against it
as a fallback, then check zdb -l for existing ZFS label metadata before
concluding the disk is blank. Remove -f so zpool create refuses rather
than overwrites if a pool is present.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ

docs/network-cutover.md

@@ -0,0 +1,366 @@
+# Network Cutover Plan
+
+Moves the LAN from the current flat/Pi-hole-managed state to the new IP scheme
+defined in `docs/ip-addressing.md`. Works in five independent stages — each
+stage is safe to pause after and resume later. Rollback steps are given at
+every point where something can break.
+
+**Before starting anything:** confirm you have
+- SSH access to `192.168.2.138` (domain-controller, current IP)
+- SSH access to `192.168.2.250` (pve1)
+- Browser access to Pi-hole admin at `http://192.168.2.253`
+- Browser access to router admin at `http://192.168.2.254`
+- The FreeIPA `admin` password to hand
+
+---
+
+## Stage 1 — Prepare FreeIPA DNS (zero downtime)
+
+Everything here is additive. Pi-hole keeps running. Nothing breaks if you stop
+mid-stage.
+
+### 1a. Add NextDNS forwarders
+
+```bash
+ssh wayne@192.168.2.138
+kinit admin          # enter FreeIPA admin password when prompted
+ipa dnsconfig-mod \
+  --forwarder=45.90.28.142 \
+  --forwarder=45.90.30.142 \
+  --forward-policy=only
+```
+
+**Verify external resolution works through FreeIPA before continuing:**
+```bash
+dig @127.0.0.1 google.com +short   # must return an IP, not SERVFAIL
+```
+
+### 1b. Add A records for every host at their CURRENT IPs
+
+These represent the live state now. You'll update each record to the new IP
+when you renumber that host in Stage 5.
+
+```bash
+ipa dnsrecord-add sweet.home pve1             --a-rec 192.168.2.250
+ipa dnsrecord-add sweet.home pbs              --a-rec 192.168.2.108
+ipa dnsrecord-add sweet.home nixos            --a-rec 192.168.2.119
+ipa dnsrecord-add sweet.home server           --a-rec 192.168.2.252
+ipa dnsrecord-add sweet.home docker           --a-rec 192.168.2.249
+ipa dnsrecord-add sweet.home nix-cache        --a-rec 192.168.2.120
+ipa dnsrecord-add sweet.home pxe-boot         --a-rec 192.168.2.247
+ipa dnsrecord-add sweet.home tailscale-router --a-rec 192.168.2.121
+ipa dnsrecord-add sweet.home tor-relay        --a-rec 192.168.2.107
+ipa dnsrecord-add sweet.home pdm              --a-rec 192.168.2.248
+ipa dnsrecord-add sweet.home router           --a-rec 192.168.2.254
+```
+
+### 1c. Clean up stale reverse-zone PTR records
+
+FreeIPA already has PTR records from an earlier import but some are wrong.
+Fix them now so reverse DNS is accurate from day one.
+
+```bash
+# Remove stale "win11" entry at .250 (should be pve1)
+ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec win11.
+ipa dnsrecord-add 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
+
+# Fix unqualified PTR records (missing .sweet.home. suffix)
+ipa dnsrecord-mod 2.168.192.in-addr.arpa 108 --ptr-rec pbs.sweet.home.
+ipa dnsrecord-mod 2.168.192.in-addr.arpa 248 --ptr-rec pdm.sweet.home.
+ipa dnsrecord-mod 2.168.192.in-addr.arpa 249 --ptr-rec docker.sweet.home.
+ipa dnsrecord-mod 2.168.192.in-addr.arpa 252 --ptr-rec server.sweet.home.
+
+# Add any missing PTR records
+ipa dnsrecord-add 2.168.192.in-addr.arpa 119 --ptr-rec nixos.sweet.home.
+ipa dnsrecord-add 2.168.192.in-addr.arpa 120 --ptr-rec nix-cache.sweet.home.
+ipa dnsrecord-add 2.168.192.in-addr.arpa 121 --ptr-rec tailscale-router.sweet.home.
+ipa dnsrecord-add 2.168.192.in-addr.arpa 247 --ptr-rec pxe-boot.sweet.home.
+ipa dnsrecord-add 2.168.192.in-addr.arpa 254 --ptr-rec router.sweet.home.
+```
+
+### 1d. Point domain-controller's own DNS at itself
+
+```bash
+sudo nmcli connection modify "System eth0" ipv4.dns "127.0.0.1"
+sudo nmcli connection up "System eth0"
+```
+
+**Verify:**
+```bash
+dig pve1.sweet.home +short          # must return 192.168.2.250
+dig google.com +short               # must return an IP (NextDNS forwarding)
+```
+
+**Rollback 1d:** `sudo nmcli connection modify "System eth0" ipv4.dns "192.168.2.253" && sudo nmcli connection up "System eth0"`
+
+---
+
+## Stage 2 — Move pxe-boot DHCP options off Pi-hole (zero downtime)
+
+Pi-hole's dnsmasq currently serves the iPXE boot options via
+`99-ipxe-chainload.conf`. Before Pi-hole is retired, that config must move to
+the pxe-boot CT running dnsmasq in proxy mode so PXE boot keeps working.
+
+### 2a. Add dnsmasq proxy config to the pxe-boot NixOS module
+
+In `modules/build-types/pxe-boot.nix`, add:
+
+```nix
+services.dnsmasq = {
+  enable = true;
+  settings = {
+    # Proxy mode: respond only to PXE DHCP requests, leave normal leases to router
+    dhcp-range = [ "192.168.2.0,proxy" ];
+    # iPXE client detection
+    dhcp-match = [
+      "set:ipxe,175"
+      "set:efi64,option:client-arch,7"
+      "set:efi64,option:client-arch,9"
+    ];
+    dhcp-userclass = "set:ipxe,iPXE";
+    # Boot file selection
+    dhcp-boot = [
+      "tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
+      "tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
+      "tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
+      "undionly.kpxe,,${vars.pxeServerIp}"
+    ];
+  };
+};
+```
+
+### 2b. Rebuild and deploy the pxe-boot CT
+
+```bash
+# On pve1 — build the new tarball
+nix build .#lxc-pxe-boot.config.system.build.tarball
+
+# Verify dnsmasq starts correctly in the CT after deploy
+ssh nixos@192.168.2.247 systemctl status dnsmasq
+```
+
+### 2c. Remove the iPXE config from Pi-hole
+
+In the Pi-hole CT, remove `/etc/dnsmasq.d/99-ipxe-chainload.conf` and
+restart the FTL service:
+
+```bash
+ssh wayne@pve1.sweet.home \
+  "sudo pct exec 100 -- bash -c 'rm /etc/dnsmasq.d/99-ipxe-chainload.conf && systemctl restart pihole-FTL'"
+```
+
+**Verify:** PXE boot a test machine — it should still get an iPXE response and
+reach the boot menu.
+
+**Rollback 2c:** restore the file from the Pi-hole config backup at
+`/etc/pihole/config_backups/` and restart pihole-FTL.
+
+---
+
+## Stage 3 — DHCP migration: Pi-hole → router (brief maintenance window)
+
+**Do this in the evening.** Existing DHCP leases stay valid during the
+switchover so connected devices don't drop — only new lease requests fail
+during the gap, which is under 60 seconds if you follow the steps in order.
+
+The key: configure the router's DHCP DNS option to point at `.253` (Pi-hole's
+current IP). This way, all new leases issued by the router still get the same
+DNS server address — clients never need to change their DNS config. When Pi-hole
+is retired and the DC takes `.253` in Stage 4, `.253` just starts answering
+differently. No client reconfiguration.
+
+### 3a. Pre-configure router DHCP (do not enable yet)
+
+Log into `http://192.168.2.254`, find the DHCP settings and fill in — but
+leave DHCP **disabled** until step 3b:
+
+| Setting | Value |
+|---|---|
+| Start IP | 192.168.2.10 |
+| End IP | 192.168.2.59 |
+| Subnet mask | 255.255.255.0 |
+| Gateway | 192.168.2.254 |
+| Primary DNS | 192.168.2.253 |
+| Secondary DNS | *(leave blank)* |
+| Lease time | 24h |
+
+Save without enabling.
+
+### 3b. Switchover (do steps in quick succession)
+
+1. **Disable Pi-hole DHCP:** Pi-hole admin UI → Settings → DHCP → uncheck
+   "DHCP server enabled" → Save
+2. **Enable router DHCP** immediately after step 1
+
+### 3c. Verify router DHCP is working
+
+On a phone or laptop, disconnect from WiFi and reconnect (or run
+`sudo dhclient -r && sudo dhclient` on a Linux host):
+
+```bash
+ip addr show        # IP should be in 192.168.2.10–59 range
+dig google.com      # should resolve (Pi-hole DNS still running at .253)
+dig pve1.sweet.home # should resolve via FreeIPA at .138 (relayed via Pi-hole)
+```
+
+Wait 10–15 minutes for the most active devices to renew their leases. There's
+no need to wait for all leases to expire before proceeding.
+
+**Rollback 3b:** Re-enable Pi-hole DHCP. Disable router DHCP. Done — existing
+leases remain valid so most devices are unaffected.
+
+---
+
+## Stage 4 — Move domain-controller from .138 to .253
+
+Pi-hole lives at `.253`. The DC must take `.253` the moment Pi-hole stops so
+clients that still have `.253` as their DNS server don't notice the change.
+Script these commands in advance and run them in rapid succession.
+
+**Pre-stage: have this SSH command ready before running step 4a:**
+```bash
+ssh wayne@192.168.2.138 "
+  sudo nmcli connection modify 'System eth0' \
+    ipv4.addresses '192.168.2.253/24' \
+    ipv4.gateway '192.168.2.254' \
+    ipv4.dns '127.0.0.1' \
+    ipv4.method manual && \
+  sudo nmcli connection up 'System eth0'
+"
+```
+
+**Also update the Proxmox VM config to match (run from pve1):**
+```bash
+sudo qm set 108 \
+  --ipconfig0 ip=192.168.2.253/24,gw=192.168.2.254 \
+  --nameserver 192.168.2.253
+```
+
+### 4a. Stop Pi-hole
+
+```bash
+ssh wayne@pve1.sweet.home "sudo pct stop 100"
+```
+
+### 4b. Immediately: change DC's IP to .253
+
+Run the pre-staged SSH command from above. You have ~30 seconds before any
+client notices Pi-hole is gone. If SSH to `.138` refuses (the IP is already
+changing), open a Proxmox console to VM 108 and run the `nmcli` commands
+there.
+
+### 4c. Update Proxmox VM config
+
+Run the pre-staged `qm set 108` command from above.
+
+### 4d. Verify
+
+```bash
+ssh wayne@192.168.2.253   # must connect (new DC IP)
+dig @192.168.2.253 pve1.sweet.home +short   # must return 192.168.2.250
+dig @192.168.2.253 google.com +short        # must return an IP
+```
+
+From a client device that renewed its DHCP lease in Stage 3:
+```bash
+cat /etc/resolv.conf    # should show 192.168.2.253
+dig pve1.sweet.home     # should resolve
+```
+
+**Rollback 4:** `ssh wayne@pve1.sweet.home "sudo pct start 100"`. Change DC IP
+back to .138 via Proxmox console. This restores full Pi-hole DNS/DHCP service.
+Leave Pi-hole CT stopped-but-intact for 48 hours before deleting it.
+
+---
+
+## Stage 5 — Host renumbering (one at a time, any order)
+
+For each host:
+1. Update FreeIPA DNS A record and PTR record to the new IP
+2. Change the static IP on the host itself
+3. Verify SSH to new IP
+4. Update `variables.nix` if that host has an IP variable (pxe-boot, pbs — already done in this PR)
+
+**FreeIPA record update template** (run as admin on domain-controller):
+```bash
+ipa dnsrecord-mod sweet.home <hostname> --a-rec <new-ip>
+ipa dnsrecord-del 2.168.192.in-addr.arpa <old-last-octet> --ptr-rec <hostname>.sweet.home.
+ipa dnsrecord-add 2.168.192.in-addr.arpa <new-last-octet> --ptr-rec <hostname>.sweet.home.
+```
+
+### Renumbering order
+
+| # | Host | Old IP | New IP | How to change IP |
+|---|---|---|---|---|
+| 1 | nixos workstation | .119 | .243 | NetworkManager on guest; or `nmcli connection modify` |
+| 2 | nix-cache | .120 | .224 | `pct set 102 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.224/24,gw=192.168.2.254` then `pct reboot 102` |
+| 3 | tailscale-router | .121 | .222 | Static config on guest; check Tailscale ACLs if IP is referenced there |
+| 4 | tor-relay | .107 | .221 | `pct set 104 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.221/24,gw=192.168.2.254` then `pct reboot 104` |
+| 5 | pdm | .248 | .220 | `pct set 106 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.220/24,gw=192.168.2.254` then `pct reboot 106` |
+| 6 | pxe-boot | .247 | .223 | `pct set 103 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.223/24,gw=192.168.2.254` then rebuild NixOS (already updated in variables.nix) |
+| 7 | server | .252 | .226 | Static config on guest; NFS clients (docker) lose mounts briefly — they remount automatically |
+| 8 | docker | .249 | .225 | Static config on guest; do this after server is at .226 |
+| 9 | pbs | .108 | .244 | Static config on PBS host itself; update in `pbsIp` already done in variables.nix |
+| 10 | pve1 | .250 | .245 | Edit `/etc/network/interfaces` on the Proxmox host — see below |
+
+### pve1 renumber (step 10 — do last)
+
+All guests keep running; only the Proxmox web UI is briefly unreachable.
+
+```bash
+ssh wayne@pve1.sweet.home
+
+# Edit /etc/network/interfaces: change address from .250 to .245
+sudo nano /etc/network/interfaces
+# Change:  address 192.168.2.250/24
+# To:      address 192.168.2.245/24
+
+sudo systemctl restart networking
+# SSH will drop here — reconnect to new IP
+```
+
+```bash
+ssh wayne@192.168.2.245    # verify
+```
+
+Update FreeIPA DNS:
+```bash
+ipa dnsrecord-mod sweet.home pve1 --a-rec 192.168.2.245
+ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
+ipa dnsrecord-add 2.168.192.in-addr.arpa 245 --ptr-rec pve1.sweet.home.
+```
+
+**Rollback any step 5 host:** change the IP back on the guest and update the
+FreeIPA record back to the old IP. The old IP is unoccupied so you can
+temporarily use either.
+
+---
+
+## Stage 6 — Final cleanup
+
+Once all hosts are at their new IPs and verified:
+
+```bash
+# Delete the Pi-hole CT (already stopped since Stage 4)
+ssh wayne@pve1.sweet.home "sudo pct destroy 100"
+
+# Remove stale FreeIPA records for retired addresses
+ipa dnsrecord-del sweet.home pihole --del-all
+ipa dnsrecord-del 2.168.192.in-addr.arpa 253 --ptr-rec pihole.sweet.home.
+
+# Rebuild any NixOS hosts that reference pbsIp or pxeServerIp to pick up
+# the updated variables.nix values (pxe-boot mandatory; others as convenient)
+```
+
+---
+
+## Rollback summary
+
+| What broke | How to roll back |
+|---|---|
+| FreeIPA DNS not resolving | Check `systemctl status named` on DC; restart if failed |
+| FreeIPA DNS unreachable | `pct start 100` on pve1 (restores Pi-hole) |
+| Router DHCP not handing out leases | Re-enable Pi-hole DHCP; disable router DHCP |
+| DC unreachable after IP change | Proxmox console on VM 108 → `nmcli connection up "System eth0"` with old IP |
+| Host unreachable after renumber | Proxmox console → revert IP; or `pct set <id> --net0 ...` old IP and reboot CT |
+| pve1 web UI gone after renumber | SSH to .245 and check `/etc/network/interfaces`; if wrong, fix and restart networking |

hosts/docker/host.nix

@@ -1,8 +1,17 @@
-_:
+{ vars, ... }:
 
 {
-  networking.hostName = "docker";
-  networking.hostId = "007f0200";
+  networking = {
+    hostName = "docker";
+    hostId = "007f0200";
+    useDHCP = false;
+    interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
+      address = vars.dockerIp;
+      prefixLength = vars.lanPrefixLength;
+    }];
+    defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
+    nameservers = [ vars.domainControllerIp ];
+  };
   boot.zfs.forceImportRoot = false;
 
   # Preserved from the pre-refactor `docker` target — stateVersion must never

hosts/ha-server-1/host.nix

@@ -11,12 +11,16 @@
     hostName = vars.haServer1Host;
     hostId = "3a4b5c6d";
     useDHCP = false;
-    interfaces.ens18.ipv4.addresses = [{
+    interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
       address = vars.haServer1Ip;
-      prefixLength = 24;
+      prefixLength = vars.lanPrefixLength;
     }];
-    defaultGateway = "192.168.2.1";
-    nameservers = [ "192.168.2.1" "8.8.8.8" ];
+    interfaces.${vars.vmStorageInterface}.ipv4.addresses = [{
+      address = vars.haServer1StorageIp;
+      prefixLength = vars.haStoragePrefixLength;
+    }];
+    defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
+    nameservers = [ vars.domainControllerIp ];
   };
 
   # Set KEY after pairing this host with the beszel hub; the token is sops-managed.

hosts/ha-server-2/host.nix

@@ -11,12 +11,16 @@
     hostName = vars.haServer2Host;
     hostId = "7e8f9a0b";
     useDHCP = false;
-    interfaces.ens18.ipv4.addresses = [{
+    interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
       address = vars.haServer2Ip;
-      prefixLength = 24;
+      prefixLength = vars.lanPrefixLength;
     }];
-    defaultGateway = "192.168.2.1";
-    nameservers = [ "192.168.2.1" "8.8.8.8" ];
+    interfaces.${vars.vmStorageInterface}.ipv4.addresses = [{
+      address = vars.haServer2StorageIp;
+      prefixLength = vars.haStoragePrefixLength;
+    }];
+    defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
+    nameservers = [ vars.domainControllerIp ];
   };
 
   # Set KEY after pairing this host with the beszel hub; the token is sops-managed.

hosts/nix-cache/host.nix

@@ -8,7 +8,16 @@
     })
   ];
 
-  networking.hostName = vars.nixCacheHost;
+  networking = {
+    hostName = vars.nixCacheHost;
+    useDHCP = false;
+    interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
+      address = vars.nixCacheIp;
+      prefixLength = vars.lanPrefixLength;
+    }];
+    defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
+    nameservers = [ vars.domainControllerIp ];
+  };
 
   services.beszel.agent.environment = {
     #DOCKER_HOST = "tcp://docker-socket-proxy:2375";

hosts/pxe-boot/host.nix

@@ -1,7 +1,16 @@
-_:
+{ vars, ... }:
 
 {
-  networking.hostName = "pxe-boot";
+  networking = {
+    hostName = "pxe-boot";
+    useDHCP = false;
+    interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
+      address = vars.pxeServerIp;
+      prefixLength = vars.lanPrefixLength;
+    }];
+    defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
+    nameservers = [ vars.domainControllerIp ];
+  };
 
   # Preserved from the pre-refactor `pxe-boot` target — stateVersion must
   # never be bumped on an already-installed machine.

hosts/server/host.nix

@@ -8,8 +8,17 @@
     })
   ];
 
-  networking.hostName = vars.nfsServerHost;
-  networking.hostId = "6689f93e";
+  networking = {
+    hostName = vars.nfsServerHost;
+    hostId = "6689f93e";
+    useDHCP = false;
+    interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
+      address = vars.serverIp;
+      prefixLength = vars.lanPrefixLength;
+    }];
+    defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
+    nameservers = [ vars.domainControllerIp ];
+  };
 
   services.beszel.agent.environment = {
     #DOCKER_HOST = "tcp://docker-socket-proxy:2375";

hosts/tailscale-router/host.nix

@@ -1,4 +1,4 @@
-_:
+{ vars, ... }:
 
 {
   imports = [
@@ -8,7 +8,16 @@ _:
     })
   ];
 
-  networking.hostName = "tailscale-router";
+  networking = {
+    hostName = "tailscale-router";
+    useDHCP = false;
+    interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
+      address = vars.tailscaleRouterIp;
+      prefixLength = vars.lanPrefixLength;
+    }];
+    defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
+    nameservers = [ vars.domainControllerIp ];
+  };
 
   services.beszel.agent.environment = {
     KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";

hosts/tor-relay/host.nix

@@ -1,4 +1,4 @@
-{ ... }:
+{ vars, ... }:
 
 {
   imports = [
@@ -8,10 +8,19 @@
     })
   ];
 
-  networking.hostName = "tor-relay";
+  networking = {
+    hostName = "tor-relay";
+    useDHCP = false;
+    interfaces.${vars.lxcLanInterface}.ipv4.addresses = [{
+      address = vars.torRelayIp;
+      prefixLength = vars.lanPrefixLength;
+    }];
+    defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
+    nameservers = [ vars.domainControllerIp ];
+  };
 
-  # No networking.hostId: only ZFS-touching hosts (server, docker) need one
-  # for pool-import safety, and this host does neither.
+  # No networking.hostId: only ZFS-touching hosts need one for pool-import
+  # safety, and this host does neither.
 
   services.beszel.agent.environment = {
     KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";

modules/build-types/pxe-boot.nix

@@ -82,7 +82,7 @@ let
   rockyFreeIpaKs = pkgs.writeText "rocky-freeipa.ks" ''
     #version=RHEL9
     # Unattended Rocky Linux 9 + FreeIPA install
-    # Target: domain-controller.sweet.home  192.168.2.138
+    # Target: domain-controller.${vars.homeDomain}  ${vars.domainControllerIp}
 
     url --url=${rockyMirror}/BaseOS/${rockyArch}/os/
     repo --name=appstream --baseurl=${rockyMirror}/AppStream/${rockyArch}/os/
@@ -136,10 +136,10 @@ let
 
     [ipv4]
     method=manual
-    addresses=192.168.2.138/24
-    gateway=192.168.2.254
-    dns=192.168.2.253;
-    dns-search=sweet.home;
+    addresses=${vars.domainControllerIp}/${toString vars.lanPrefixLength}
+    gateway=${vars.lanGateway}
+    dns=${vars.domainControllerIp};
+    dns-search=${vars.homeDomain};
 
     [ipv6]
     method=auto
@@ -148,7 +148,7 @@ let
 
     # -- /etc/hosts: FQDN must resolve to the real IP (not loopback) for IPA --
     sed -i '/domain-controller/d' /etc/hosts
-    echo '192.168.2.138 domain-controller.sweet.home domain-controller' >> /etc/hosts
+    echo '${vars.domainControllerIp} domain-controller.${vars.homeDomain} domain-controller' >> /etc/hosts
 
     # -- Generate IPA passwords and store securely --
     DM_PASS=$(openssl rand -base64 24 | tr -dc 'A-Za-z0-9' | head -c 24)
@@ -420,6 +420,29 @@ in
     };
   };
 
+  services.dnsmasq = {
+    enable = true;
+    settings = {
+      # Disable DNS listener — only proxy DHCP is needed here.
+      # Without this dnsmasq tries to bind port 53 which systemd-resolved
+      # already owns, causing startup failure.
+      port = 0;
+      dhcp-range = [ "192.168.2.0,proxy" ];
+      dhcp-match = [
+        "set:ipxe,175"
+        "set:efi64,option:client-arch,7"
+        "set:efi64,option:client-arch,9"
+      ];
+      dhcp-userclass = "set:ipxe,iPXE";
+      dhcp-boot = [
+        "tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
+        "tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
+        "tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
+        "undionly.kpxe,,${vars.pxeServerIp}"
+      ];
+    };
+  };
+
   networking.firewall.allowedTCPPorts = [ vars.ports.pxeBootHttp ];
-  networking.firewall.allowedUDPPorts = [ vars.ports.pxeBootTftp ];
+  networking.firewall.allowedUDPPorts = [ vars.ports.pxeBootTftp 67 ];
 }

modules/build-types/server.nix

@@ -44,14 +44,8 @@ in
         exit 0
       fi
 
-      # Pool exists on a device but not yet imported — let the standard
-      # zfs-import-${poolName}.service handle it normally.
-      if zpool import -d /dev/disk/by-id -N "${poolName}" 2>/dev/null; then
-        exit 0
-      fi
-
-      # No pool found at all.  Create it on the Proxmox data disk (scsi1),
-      # which appears as /dev/disk/by-id/scsi-*drive-scsi1 inside the VM.
+      # Locate the data disk first — used for both the fallback import
+      # attempt and, only if the disk is genuinely blank, pool creation.
       DATA_DISK=""
       for candidate in /dev/disk/by-id/scsi-*drive-scsi1; do
         [[ "$candidate" == *-part* ]] && continue
@@ -63,8 +57,31 @@ in
         exit 1
       fi
 
+      # Try importing via the by-id symlink directory first (normal path),
+      # then fall back to scanning the disk directly.  The two-step exists
+      # because of a udev race: systemd-udev-settle.service can clear before
+      # /dev/disk/by-id/ entries are fully populated, causing the first
+      # import to fail even when the pool is intact on the disk.
+      if zpool import -d /dev/disk/by-id -N "${poolName}" 2>/dev/null; then
+        exit 0
+      fi
+      if zpool import -d "$DATA_DISK" -N "${poolName}" 2>/dev/null; then
+        exit 0
+      fi
+
+      # Both import attempts failed.  Before creating a new pool, verify the
+      # disk is genuinely blank — if ZFS label metadata is present the import
+      # failed for some other reason and we must not clobber existing data.
+      if zdb -l "$DATA_DISK" 2>/dev/null | grep -q "name: '${poolName}'"; then
+        echo "zfs-init-${poolName}: $DATA_DISK has ZFS pool '${poolName}' metadata but import failed — refusing to overwrite existing data. Run 'zpool import -d $DATA_DISK ${poolName}' manually to investigate." >&2
+        exit 1
+      fi
+
+      # Disk is genuinely blank: create the pool.  -f is intentionally
+      # omitted so that if we somehow reach this point with an existing pool
+      # on the disk, zpool refuses rather than silently destroying data.
       echo "zfs-init-${poolName}: creating pool on $DATA_DISK"
-      zpool create -f "${poolName}" "$DATA_DISK"
+      zpool create "${poolName}" "$DATA_DISK"
       ${lib.concatMapStrings (ds: ''
         zfs create "${poolName}/${ds}"
       '') poolDatasets}

modules/build-types/tailscale-router.nix

@@ -3,6 +3,7 @@
 {
   imports = [
     ../tailscale/subnet-router.nix
+    ../tailscale/ts-dns-forwarder.nix
     ../beszel/enable-agent.nix
   ];
 

modules/ha/cluster-config.nix

@@ -47,11 +47,11 @@
         }
 
         on ${vars.haServer1Host} {
-          address ${vars.haServer1Ip}:${toString vars.ports.haServerDrbd};
+          address ${vars.haServer1StorageIp}:${toString vars.ports.haServerDrbd};
         }
 
         on ${vars.haServer2Host} {
-          address ${vars.haServer2Ip}:${toString vars.ports.haServerDrbd};
+          address ${vars.haServer2StorageIp}:${toString vars.ports.haServerDrbd};
         }
       }
     '';
@@ -75,8 +75,8 @@
   services.corosync = {
     clusterName = "ha-cluster";
     nodelist = [
-      { nodeid = 1; name = vars.haServer1Host; ring_addrs = [ vars.haServer1Ip ]; }
-      { nodeid = 2; name = vars.haServer2Host; ring_addrs = [ vars.haServer2Ip ]; }
+      { nodeid = 1; name = vars.haServer1Host; ring_addrs = [ vars.haServer1StorageIp ]; }
+      { nodeid = 2; name = vars.haServer2Host; ring_addrs = [ vars.haServer2StorageIp ]; }
     ];
   };
 
@@ -101,6 +101,7 @@
     extraCommands = ''
       iptables -A INPUT -s ${vars.haServer1Ip}/32 -j ACCEPT
       iptables -A INPUT -s ${vars.haServer2Ip}/32 -j ACCEPT
+      iptables -A INPUT -s ${vars.haStorageCidr} -j ACCEPT
     '';
   };
 }

modules/tailscale/ts-dns-forwarder.nix

@@ -0,0 +1,52 @@
+{ vars, ... }:
+
+{
+  # Run dnsmasq on the LAN interface as a forwarding-only resolver for
+  # *.ts.net (Tailscale MagicDNS names). FreeIPA's bind-dyndb-ldap
+  # cannot reach 100.100.100.100 (Tailscale's internal resolver) directly
+  # because the DC is not a Tailscale node. This host IS a Tailscale node
+  # and can reach 100.100.100.100 via its tailscale0 interface, so it
+  # acts as an intermediary: FreeIPA has a conditional forward zone for
+  # ts.net pointing here (vars.tailscaleRouterIp), and this dnsmasq
+  # instance forwards those queries onward to Tailscale's resolver.
+  #
+  # Configure FreeIPA once after deploying this host:
+  #   kinit admin
+  #   ipa dnsforwardzone-add ${vars.tailnetDomain} \
+  #     --forwarder=${vars.tailscaleRouterIp} \
+  #     --forward-policy=only
+  # Note: IPA refuses to shadow ts.net (a real public TLD); use the
+  # tailnet-specific subdomain (vars.tailnetDomain) instead.
+  services.dnsmasq = {
+    enable = true;
+    settings = {
+      # Listen only on the LAN interface — not tailscale0 or loopback.
+      # bind-interfaces prevents dnsmasq from binding to 0.0.0.0 and
+      # then filtering by interface later; combined with `interface` this
+      # ensures it genuinely listens only on eth0.
+      bind-interfaces = true;
+      interface = [ vars.lxcLanInterface ];
+
+      # Forward-only: no local /etc/hosts or /etc/resolv.conf reading,
+      # no negative caching of NXDOMAIN for names this instance doesn't
+      # serve. All ts.net queries come from FreeIPA's conditional forwarder
+      # and must be answered by Tailscale's resolver.
+      no-hosts = true;
+      no-resolv = true;
+
+      # Tailscale's internal "Quad100" resolver — reachable from any
+      # Tailscale node via the tailscale0 interface. Scoped to the
+      # specific tailnet subdomain (vars.tailnetDomain) rather than
+      # all of ts.net: FreeIPA refuses to shadow ts.net (a real public
+      # TLD with DNSimple nameservers) so the conditional forward zone
+      # in FreeIPA must use the tailnet-specific subdomain instead:
+      #   ipa dnsforwardzone-add ${vars.tailnetDomain} \
+      #     --forwarder=${vars.tailscaleRouterIp} \
+      #     --forward-policy=only
+      server = [ "/${vars.tailnetDomain}/100.100.100.100" ];
+    };
+  };
+
+  networking.firewall.allowedUDPPorts = [ 53 ];
+  networking.firewall.allowedTCPPorts = [ 53 ];
+}

variables.nix

@@ -4,9 +4,19 @@
   homeDomain = "sweet.home"; # base LAN domain for service subdomains (pve., docker.)
   tailnetDomain = "tail13f623.ts.net"; # Tailscale MagicDNS suffix
   lanCidr = "192.168.2.0/24"; # LAN subnet
-  pxeServerIp = "192.168.2.223"; # pxe-boot host's LAN IP
-  pbsIp = "192.168.2.244"; # Proxmox Backup Server LAN IP
-  domainControllerIp = "192.168.2.253"; # FreeIPA domain controller / primary DNS
+  lanGateway = "192.168.2.254"; # LAN default gateway (router)
+  lanPrefixLength = 24; # LAN subnet prefix length (/24 = 255.255.255.0)
+  lxcLanInterface = "eth0"; # LAN NIC name in LXC containers (set by Proxmox --net0 name=eth0)
+  vmLanInterface = "ens18"; # LAN NIC name in Proxmox VMs (virtio, first NIC)
+  vmStorageInterface = "ens19"; # storage NIC name in HA server VMs (virtio, second NIC on vmbr1)
+  pxeServerIp = "192.168.2.223"; # pxe-boot LXC container LAN IP
+  nixCacheIp = "192.168.2.224"; # nix-cache LXC container LAN IP
+  tailscaleRouterIp = "192.168.2.222"; # tailscale-router LXC container LAN IP
+  torRelayIp = "192.168.2.221"; # tor-relay LXC container LAN IP
+  serverIp = "192.168.2.226"; # server (NFS/ZFS) Proxmox VM LAN IP
+  dockerIp = "192.168.2.225"; # docker Proxmox VM LAN IP
+  pbsIp = "192.168.2.244"; # Proxmox Backup Server LAN IP (not NixOS-managed)
+  domainControllerIp = "192.168.2.253"; # FreeIPA domain controller / primary DNS (not NixOS-managed)
 
   # Cross-host references (LAN hostnames/users other hosts reach over the network)
   nixCacheHost = "nix-cache"; # substituter/remote-builder hostname
@@ -70,16 +80,20 @@
   primaryUser = "nixos";
 
   # HA file server cluster
-  # haServer1Ip / haServer2Ip: static LAN IPs for both HA nodes (must be
-  # fixed — DRBD and corosync ring addresses are baked into the NixOS config).
+  # LAN IPs (vmbr0 / ens18) — client-facing: iSCSI initiators, NFS, management.
+  # Storage IPs (vmbr1 / ens19) — isolated internal bridge, used for DRBD
+  # replication and Corosync heartbeat only; never leaves pve1.
   # haServerVip: floating virtual IP managed by Pacemaker's IPaddr2 resource;
   # NFS and iSCSI clients connect here regardless of which node is Active.
-  # Set all three to real values in variables.nix before deploying.
   haServer1Host = "ha-server-1";
   haServer2Host = "ha-server-2";
-  haServer1Ip = "192.168.2.200"; # TODO: confirm production IP
-  haServer2Ip = "192.168.2.201"; # TODO: confirm production IP
-  haServerVip = "192.168.2.202"; # TODO: confirm floating VIP
+  haServer1Ip = "192.168.2.228"; # LAN IP, node 1
+  haServer2Ip = "192.168.2.227"; # LAN IP, node 2
+  haServerVip = "192.168.2.229"; # floating VIP (Pacemaker IPaddr2)
+  haServer1StorageIp = "192.168.4.228"; # storage-net IP, node 1 (vmbr1 / ens19)
+  haServer2StorageIp = "192.168.4.227"; # storage-net IP, node 2 (vmbr1 / ens19)
+  haStorageCidr = "192.168.4.0/29"; # storage subnet — internal to pve1 only
+  haStoragePrefixLength = 29; # storage subnet prefix length (/29)
   haStorageRoot = "/srv/ha-data"; # XFS-over-DRBD mount point on the Active node
   haIscsiIqn = "iqn.2026-01.home.sweet:ha-storage";