NixOS's dnsmasq module defaults resolveLocalQueries to true, which adds
127.0.0.1 to networking.nameservers and binds dnsmasq to listen-address=127.0.0.1.
This made the host route all its own DNS through dnsmasq, which had
no-resolv=true and no upstream for anything outside the tailnet domain —
so every non-tailscale DNS query from the host itself (including SSSD
resolving the IPA server FQDN after the IPA client module was added) failed.
Setting resolveLocalQueries=false limits dnsmasq to its intended role: a
forwarding proxy reachable on the LAN interface for IPA's conditional
forwarder. The host uses domainControllerIp directly (already set in
networking.nameservers in host.nix).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ipa-getkeytab runs as root via sudo so the temp file is root-owned;
scp as wayne gets Permission denied. Pipe through `sudo cat` over SSH
instead, which reads as root but writes locally as the invoking user.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single command to enroll a NixOS host in FreeIPA and produce a
sops-encrypted keytab at secrets/<hostname>.keytab:
- Adds the .sops.yaml creation rule automatically (with all registered
platform-variant age keys as recipients)
- SSHes to the domain controller to run ipa host-add + ipa-getkeytab
- Refreshes the admin Kerberos ticket via `ssh -t ... kinit admin` if
missing or expired, so no manual kinit step is needed
- SCPs the keytab and encrypts it in-place with sops (file must be at
secrets/<hostname>.keytab before encryption so the path-based creation
rule matches — the common failure point when doing this manually)
Also adds HOME_DOMAIN and IPA_SERVER to scripts/env.sh, matching
variables.nix's homeDomain/ipaServer (same manual-sync pattern as
NIX_CACHE_HOST/LAN_DOMAIN).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sops matches creation rules against the input file path, so encrypting
/tmp/<host>.keytab directly with stdout redirect fails to find the rule.
Copy to secrets/ first, then use -i to encrypt in-place.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ipa-getkeytab confirmed ipa.sweet.home doesn't respond to LDAP;
domain-controller.sweet.home is the actual IPA server FQDN.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds modules/ipa/client.nix — a parameterized module that joins a NixOS host
to the sweet.home FreeIPA domain without ipa-client-install. It configures
security.ipa (SSSD, Kerberos, PAM, NSSwitch) and places a pre-provisioned host
keytab via sops-nix binary secret so enrollment is fully reproducible from the
flake.
- variables.nix: adds ipaServer (FQDN of the FreeIPA KDC; security.ipa.server
requires a hostname, not an IP, for Kerberos/TLS)
- certs/ipa-ca.crt: placeholder for the IPA CA public certificate (operator
replaces with: curl http://<ipa-server>/ipa/config/ca.crt)
- secrets/nix-cache.keytab: placeholder binary sops file (operator replaces
with the encrypted keytab after ipa host-add + ipa-getkeytab)
- .sops.yaml: adds creation rule for secrets/nix-cache.keytab (same recipients
as secrets/nix-cache.yaml)
- hosts/nix-cache/host.nix: imports the IPA client module; adds
networking.domain so the host's FQDN resolves correctly
Module header documents the three operator steps needed per host before deploy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
zpool create -f was called if `zpool import -d /dev/disk/by-id` failed,
which could happen due to a race with systemd-udev-settle. The disk
would then be visible by the time zpool create ran, silently destroying
all data on an otherwise-intact pool.
Fix: locate the data disk first, retry the import directly against it
as a fallback, then check zdb -l for existing ZFS label metadata before
concluding the disk is blank. Remove -f so zpool create refuses rather
than overwrites if a pool is present.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
IPA refuses to create a forward zone for ts.net because it's a real
public TLD with DNSimple nameservers. The forward zone must use the
tailnet-specific subdomain (vars.tailnetDomain, e.g. tail13f623.ts.net)
instead. Update dnsmasq server selector and comments to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
FreeIPA (the new authoritative DNS) cannot reach 100.100.100.100
(Tailscale's internal MagicDNS resolver) directly because the DC is not
a Tailscale node. The tailscale-router IS a Tailscale node and can
reach 100.100.100.100 via tailscale0, so it now runs a dnsmasq
instance on its LAN interface that forwards all ts.net queries to
Tailscale's resolver.
After deploying this host, configure FreeIPA with:
kinit admin
ipa dnsforwardzone-add ts.net \
--forwarder=192.168.2.222 \
--forward-policy=only
This replaces Pi-hole's conditional forwarder for ts.net and restores
resolution of Tailscale MagicDNS names (e.g. raspberrypi.tail13f623.ts.net)
for all LAN hosts using FreeIPA as their DNS server.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
Adds static IP configuration to every NixOS host in the flake that
has a fixed LAN address, and centralises all network primitives
(IPs, gateway, prefix length, interface names) in variables.nix so
there is one place to update if any of them change.
variables.nix additions:
- lanGateway / lanPrefixLength — LAN gateway and /24 prefix, replacing
every hardcoded 192.168.2.254 / 24 across host files
- lxcLanInterface / vmLanInterface / vmStorageInterface — NIC names for
LXC containers (eth0), Proxmox VMs (ens18), and the HA storage NIC
(ens19), used as attribute keys so changing the name is a one-line edit
- haStoragePrefixLength — /29 for the storage subnet, mirrors haStorageCidr
- Per-host IP variables: nixCacheIp (.224), tailscaleRouterIp (.222),
torRelayIp (.221), serverIp (.226), dockerIp (.225)
host.nix changes:
- tailscale-router, tor-relay, nix-cache, pxe-boot: useDHCP = false,
static address on eth0 (lxcLanInterface), struct-form defaultGateway
(required when using systemd-networkd which LXC containers use)
- server, docker: useDHCP = false, static address on ens18 (vmLanInterface),
struct-form defaultGateway (works for both scripted networking and networkd)
- ha-server-1, ha-server-2: replace hardcoded 192.168.2.254 / 24 / 29
with the new variables; no functional change for these hosts
modules/build-types/pxe-boot.nix:
- Domain-controller kickstart template: replace hardcoded 192.168.2.138
and 192.168.2.254 with vars.domainControllerIp / vars.lanGateway /
vars.lanPrefixLength / vars.homeDomain so the template stays correct
if the DC IP or domain is ever changed again
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
vars.pxeServerIp was already set to the post-renumber target (.223) but the
pxe-boot CT is still at .247, so dnsmasq was advertising .223 as the TFTP server
and the VM couldn't reach it. Update to .247 so PXE boot works now; the comment
reminds us to flip it back to .223 when Stage 5 step 6 renumbers the CT.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the proxy DHCP handshake, the UEFI client queries the PXE boot
server on UDP port 4011 to get the actual boot filename. This port was
not open in the NixOS firewall, so all four PXE discovery attempts from
the test VM silently timed out. dnsmasq was bound to 0.0.0.0:4011 but
never received the packets.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dnsmasq 2.93 does not send proxy DHCP offers from dhcp-boot alone in proxy
mode — pxe-service is required to trigger proxy offers for the initial boot.
Replace the dhcp-boot-based arch detection with pxe-service entries for
arch 0 (BIOS), 7 (EFI BC / OVMF), and 9 (EFI x86-64), then use dhcp-boot
only for the iPXE second-boot HTTP chain (tag:ipxe clients). Add log-dhcp
for post-deploy verification.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dnsmasq binds port 53 by default even when only proxy DHCP is needed.
systemd-resolved already owns port 53 on the pxe-boot container, so
dnsmasq crashes at startup. Setting port=0 disables the DNS listener
entirely — pxe-boot only needs the proxy DHCP path for iPXE chainloading.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dnsmasq config was placed inside systemd.services{} block, making Nix
interpret it as systemd.services.dnsmasq.settings (which doesn't exist).
Move to top-level services.dnsmasq and add UDP 67 to firewall for DHCP
proxy mode.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds proxmox-ha-server-1 and proxmox-ha-server-2 as real mkTarget entries
alongside the existing proxmox-server, backed by a new ha-server build type.
New modules
modules/ha/cluster-config.nix — DRBD resource + corosync nodelist sourced
from vars (haServer1Host/Ip, haServer2Host/Ip); resource-only fencing for
production STONITH; HA port firewall rules for DRBD, iSCSI, Corosync, pcsd
modules/build-types/ha-server.nix — imports pacemaker-stack + iscsi-target
+ cluster-config + beszel; NFS exports from vars.haStorageRoot (XFS-over-DRBD
mount); nfs-server.service.wantedBy force-cleared so Pacemaker controls
start/stop on the Active node only
New hosts
hosts/ha-server-{1,2}/host.nix — static IP from vars, unique hostId; sops
secrets (beszel, corosync authkey) are TODOs pending sync-host-keys.sh
variables.nix
haServer1/2Host, haServer1/2Ip, haServerVip, haStorageRoot, haIscsiIqn
ports.haServerDrbd/Iscsi/Corosync{1,2,Crypto}/PacemakerRemoted/Pcsd
scripts/ha/ (migrated + updated from test-lab/ha/)
cluster-init.sh — generates corosync authkey, initialises DRBD/XFS/iSCSI,
creates NFS dataset dirs, configures Pacemaker with DRBD + XFS + iSCSI
+ nfs-server + VIP; STONITH disabled initially (enable separately)
cluster-enable-stonith.sh — enables fence_pve_ssh STONITH after key deploy
fence-pve-ssh.py — Proxmox SSH fence agent (node names updated to ha-server-1/2)
acceptance-tests.sh — T1–T7 production acceptance tests
test-lab/ha/ removed — all Nix config moved to modules/ha/ and
modules/build-types/; scripts moved to scripts/ha/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
All 7 acceptance tests pass on live NixOS 25.11 VMs (VMIDs 200/201 on
pve1). Failover completes in ~5 s with data integrity verified.
modules/ha/pacemaker-stack.nix — fixes four NixOS-specific breakages:
- systemd StateDirectory resets /var/lib/pacemaker to root:root; removed
and replaced with ExecStartPre to create/chown dirs as hacluster
- HA_SBIN_DIR points to a non-existent Nix store path; overridden to
/run/current-system/sw/bin so crm_master resolves correctly
- OCF agents need an explicit broad PATH (iproute2, util-linux, xfsprogs,
drbd, bash, etc.) — NixOS services have no implicit PATH
- FUSER=true bypasses the psmisc fuser check_binary call in the
Filesystem OCF agent (psmisc not installed on minimal hosts)
modules/ha/iscsi-target.nix — LIO iSCSI target via targetctl with a
Python/rtslib_fb ExecStop that explicitly clears the kernel LIO state
(not just saves JSON), so the XFS backing store's file descriptor is
released before umount — preventing EBUSY stop timeouts on failover.
Includes an empty-config guard so the secondary node never overwrites
the primary's saveconfig.json with an empty one.
test-lab/ha/common.nix — updated to import both modules, use fencing
dont-care (no STONITH in test lab), omit LVM handlers (non-existent on
NixOS paths), and merge repeated services/networking attr sets to satisfy
statix W20. test-lab/ha/acceptance-tests.sh — final v4 with crm_standby
fix (pacemaker 3.x API).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Adds an unattended install option to the PXE boot menu that installs
Rocky Linux 9 and configures FreeIPA on the domain-controller.sweet.home
host without any operator interaction after selecting the menu entry.
How it works:
- fetch-rocky-pxeboot.service downloads the Rocky 9 Anaconda pxeboot
kernel and initrd from the Rocky mirror on first pxe-boot deploy
(idempotent, same pattern as fetch-debian-netboot)
- rocky-freeipa.ipxe boots Anaconda with inst.ks pointing at the
hosted Kickstart and net.ifnames=0 biosdevname=0 for stable eth0
- rocky-freeipa.ks (generated, includes vars.adminSshKey) performs:
- Minimal Rocky 9 install with ipa-server + ipa-server-dns
- Static IP 192.168.2.138 via NM connection file written in %post
- /etc/hosts fixed for FreeIPA FQDN requirement
- Random DM + admin passwords generated and saved to
/root/ipa-credentials.txt (chmod 600, never hardcoded)
- freeipa-first-boot.service oneshot enabled to run
ipa-server-install on the first real boot (~20 min)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Jbvxx4xbHVcx1NkK3vtmK
beatzaplenty
