Commit Graph
152 Commits
Author SHA1 Message Date
beatzaplentyandClaude Sonnet 4.6 7b4ce0ab3d fix(server): prevent zfs-init-tank from wiping pool on udev race
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m46s
zpool create -f was called if `zpool import -d /dev/disk/by-id` failed,
which could happen due to a race with systemd-udev-settle. The disk
would then be visible by the time zpool create ran, silently destroying
all data on an otherwise-intact pool.

Fix: locate the data disk first, retry the import directly against it
as a fallback, then check zdb -l for existing ZFS label metadata before
concluding the disk is blank. Remove -f so zpool create refuses rather
than overwrites if a pool is present.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 21:52:39 +10:00
beatzaplentyandClaude Sonnet 4.6 3123565011 fix(tailscale-router): scope ts.net forwarder to tailnet subdomain
IPA refuses to create a forward zone for ts.net because it's a real
public TLD with DNSimple nameservers. The forward zone must use the
tailnet-specific subdomain (vars.tailnetDomain, e.g. tail13f623.ts.net)
instead. Update dnsmasq server selector and comments to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 19:23:46 +10:00
beatzaplentyandClaude Sonnet 4.6 36f5ebdf86 feat(tailscale-router): serve ts.net DNS forward zone for LAN hosts
FreeIPA (the new authoritative DNS) cannot reach 100.100.100.100
(Tailscale's internal MagicDNS resolver) directly because the DC is not
a Tailscale node. The tailscale-router IS a Tailscale node and can
reach 100.100.100.100 via tailscale0, so it now runs a dnsmasq
instance on its LAN interface that forwards all ts.net queries to
Tailscale's resolver.

After deploying this host, configure FreeIPA with:
  kinit admin
  ipa dnsforwardzone-add ts.net \
    --forwarder=192.168.2.222 \
    --forward-policy=only

This replaces Pi-hole's conditional forwarder for ts.net and restores
resolution of Tailscale MagicDNS names (e.g. raspberrypi.tail13f623.ts.net)
for all LAN hosts using FreeIPA as their DNS server.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 19:20:12 +10:00
beatzaplentyandClaude Sonnet 4.6 ee96713a50 feat(networking): declare static IPs for all NixOS-managed hosts in flake
Adds static IP configuration to every NixOS host in the flake that
has a fixed LAN address, and centralises all network primitives
(IPs, gateway, prefix length, interface names) in variables.nix so
there is one place to update if any of them change.

variables.nix additions:
- lanGateway / lanPrefixLength — LAN gateway and /24 prefix, replacing
  every hardcoded 192.168.2.254 / 24 across host files
- lxcLanInterface / vmLanInterface / vmStorageInterface — NIC names for
  LXC containers (eth0), Proxmox VMs (ens18), and the HA storage NIC
  (ens19), used as attribute keys so changing the name is a one-line edit
- haStoragePrefixLength — /29 for the storage subnet, mirrors haStorageCidr
- Per-host IP variables: nixCacheIp (.224), tailscaleRouterIp (.222),
  torRelayIp (.221), serverIp (.226), dockerIp (.225)

host.nix changes:
- tailscale-router, tor-relay, nix-cache, pxe-boot: useDHCP = false,
  static address on eth0 (lxcLanInterface), struct-form defaultGateway
  (required when using systemd-networkd which LXC containers use)
- server, docker: useDHCP = false, static address on ens18 (vmLanInterface),
  struct-form defaultGateway (works for both scripted networking and networkd)
- ha-server-1, ha-server-2: replace hardcoded 192.168.2.254 / 24 / 29
  with the new variables; no functional change for these hosts

modules/build-types/pxe-boot.nix:
- Domain-controller kickstart template: replace hardcoded 192.168.2.138
  and 192.168.2.254 with vars.domainControllerIp / vars.lanGateway /
  vars.lanPrefixLength / vars.homeDomain so the template stays correct
  if the DC IP or domain is ever changed again

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
2026-07-27 18:14:31 +10:00
beatzaplentyandClaude Sonnet 4.6 c770feebc9 fix(pxe-boot): disable dnsmasq DNS listener to avoid port 53 conflict
dnsmasq binds port 53 by default even when only proxy DHCP is needed.
systemd-resolved already owns port 53 on the pxe-boot container, so
dnsmasq crashes at startup. Setting port=0 disables the DNS listener
entirely — pxe-boot only needs the proxy DHCP path for iPXE chainloading.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 14:29:30 +10:00
beatzaplentyandClaude Sonnet 4.6 58c40292e2 fix(pxe-boot): move dnsmasq out of systemd.services into services.dnsmasq
dnsmasq config was placed inside systemd.services{} block, making Nix
interpret it as systemd.services.dnsmasq.settings (which doesn't exist).
Move to top-level services.dnsmasq and add UDP 67 to firewall for DHCP
proxy mode.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 14:02:41 +10:00
beatzaplenty 94842875d0 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos into worktree-network-cutover-plan 2026-07-27 14:01:24 +10:00
beatzaplenty cfa36b97fc added dnsmasq to pxe-boot 2026-07-27 13:44:29 +10:00
beatzaplentyandClaude Sonnet 4.6 4444398cac fix(network): wire correct IPs throughout and add cutover plan
variables.nix:
- HA server LAN IPs: .200/.201/.202 → .228/.227/.229 (from ip-addressing.md)
- Add haServer1StorageIp (.228), haServer2StorageIp (.227) for 192.168.4.0/29
- Add haStorageCidr for firewall rules

ha-server host.nix (both nodes):
- Add ens19 interface on storage subnet (/29)
- Fix defaultGateway: 192.168.2.1 → 192.168.2.254
- Fix nameservers: 192.168.2.1/8.8.8.8 → domainControllerIp (.253)

cluster-config.nix:
- DRBD replication addresses: LAN IPs → storage IPs (keep replication off LAN)
- Corosync ring_addrs: LAN IPs → storage IPs
- Firewall: add haStorageCidr to allowed sources

docs/network-cutover.md: step-by-step cutover plan with rollback at every stage

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 12:32:54 +10:00
beatzaplenty 6ce4784376 Merge pull request 'Worktree ha file server test' (#81) from worktree-ha-file-server-test into main
Reviewed-on: #81
2026-07-27 02:18:06 +00:00
beatzaplentyandClaude Sonnet 4.6 5856d45575 feat(ha): wire sops secrets and disable NetworkManager for HA servers
- cluster-config.nix: add corosync_authkey sops binary secret
  (/etc/corosync/authkey, mode 0400) and force-disable NetworkManager
  (common config enables it; HA nodes need stable static IP networking)
- hosts/ha-server-{1,2}/host.nix: add host-token.nix import for
  sops-managed beszel-token; add KEY placeholder for beszel hub pairing
- .sops.yaml: add creation rules for secrets/ha-server-{1,2}.yaml and
  secrets/ha-corosync-authkey (admin-only until sync-host-keys.sh runs)
- secrets/ha-server-{1,2}.yaml, secrets/ha-corosync-authkey: stub files
  so eval passes before real secrets are provisioned

Bootstrap order (post-merge):
  1. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
  2. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
  3. sops updatekeys secrets/common.yaml  (grants HA nodes common secrets)
  4. sops secrets/ha-server-{1,2}.yaml   (set beszel-token values)
  5. On node1: corosync-keygen; sops -e --input-type binary
     /etc/corosync/authkey > secrets/ha-corosync-authkey; git add/commit

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 11:45:36 +10:00
beatzaplentyandClaude Sonnet 4.6 e3498b1087 feat(ha): promote HA file server to production flake targets
Adds proxmox-ha-server-1 and proxmox-ha-server-2 as real mkTarget entries
alongside the existing proxmox-server, backed by a new ha-server build type.

New modules
  modules/ha/cluster-config.nix — DRBD resource + corosync nodelist sourced
    from vars (haServer1Host/Ip, haServer2Host/Ip); resource-only fencing for
    production STONITH; HA port firewall rules for DRBD, iSCSI, Corosync, pcsd
  modules/build-types/ha-server.nix — imports pacemaker-stack + iscsi-target
    + cluster-config + beszel; NFS exports from vars.haStorageRoot (XFS-over-DRBD
    mount); nfs-server.service.wantedBy force-cleared so Pacemaker controls
    start/stop on the Active node only

New hosts
  hosts/ha-server-{1,2}/host.nix — static IP from vars, unique hostId; sops
    secrets (beszel, corosync authkey) are TODOs pending sync-host-keys.sh

variables.nix
  haServer1/2Host, haServer1/2Ip, haServerVip, haStorageRoot, haIscsiIqn
  ports.haServerDrbd/Iscsi/Corosync{1,2,Crypto}/PacemakerRemoted/Pcsd

scripts/ha/ (migrated + updated from test-lab/ha/)
  cluster-init.sh — generates corosync authkey, initialises DRBD/XFS/iSCSI,
    creates NFS dataset dirs, configures Pacemaker with DRBD + XFS + iSCSI
    + nfs-server + VIP; STONITH disabled initially (enable separately)
  cluster-enable-stonith.sh — enables fence_pve_ssh STONITH after key deploy
  fence-pve-ssh.py — Proxmox SSH fence agent (node names updated to ha-server-1/2)
  acceptance-tests.sh — T1–T7 production acceptance tests

test-lab/ha/ removed — all Nix config moved to modules/ha/ and
  modules/build-types/; scripts moved to scripts/ha/

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 11:26:37 +10:00
beatzaplentyandClaude Sonnet 4.6 724d9a45af feat(ha): add NixOS modules for DRBD+XFS+LIO+Corosync+Pacemaker HA stack
All 7 acceptance tests pass on live NixOS 25.11 VMs (VMIDs 200/201 on
pve1).  Failover completes in ~5 s with data integrity verified.

modules/ha/pacemaker-stack.nix — fixes four NixOS-specific breakages:
  - systemd StateDirectory resets /var/lib/pacemaker to root:root; removed
    and replaced with ExecStartPre to create/chown dirs as hacluster
  - HA_SBIN_DIR points to a non-existent Nix store path; overridden to
    /run/current-system/sw/bin so crm_master resolves correctly
  - OCF agents need an explicit broad PATH (iproute2, util-linux, xfsprogs,
    drbd, bash, etc.) — NixOS services have no implicit PATH
  - FUSER=true bypasses the psmisc fuser check_binary call in the
    Filesystem OCF agent (psmisc not installed on minimal hosts)

modules/ha/iscsi-target.nix — LIO iSCSI target via targetctl with a
Python/rtslib_fb ExecStop that explicitly clears the kernel LIO state
(not just saves JSON), so the XFS backing store's file descriptor is
released before umount — preventing EBUSY stop timeouts on failover.
Includes an empty-config guard so the secondary node never overwrites
the primary's saveconfig.json with an empty one.

test-lab/ha/common.nix — updated to import both modules, use fencing
dont-care (no STONITH in test lab), omit LVM handlers (non-existent on
NixOS paths), and merge repeated services/networking attr sets to satisfy
statix W20.  test-lab/ha/acceptance-tests.sh — final v4 with crm_standby
fix (pacemaker 3.x API).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-27 09:13:55 +10:00
beatzaplentyandClaude Sonnet 4.6 40856b2e5e feat(pxe-boot): add FreeIPA Server (Rocky Linux 9) iPXE menu entry
Adds an unattended install option to the PXE boot menu that installs
Rocky Linux 9 and configures FreeIPA on the domain-controller.sweet.home
host without any operator interaction after selecting the menu entry.

How it works:
- fetch-rocky-pxeboot.service downloads the Rocky 9 Anaconda pxeboot
  kernel and initrd from the Rocky mirror on first pxe-boot deploy
  (idempotent, same pattern as fetch-debian-netboot)
- rocky-freeipa.ipxe boots Anaconda with inst.ks pointing at the
  hosted Kickstart and net.ifnames=0 biosdevname=0 for stable eth0
- rocky-freeipa.ks (generated, includes vars.adminSshKey) performs:
  - Minimal Rocky 9 install with ipa-server + ipa-server-dns
  - Static IP 192.168.2.138 via NM connection file written in %post
  - /etc/hosts fixed for FreeIPA FQDN requirement
  - Random DM + admin passwords generated and saved to
    /root/ipa-credentials.txt (chmod 600, never hardcoded)
  - freeipa-first-boot.service oneshot enabled to run
    ipa-server-install on the first real boot (~20 min)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Jbvxx4xbHVcx1NkK3vtmK
2026-07-27 07:35:13 +10:00
beatzaplenty 2514d3bc89 Merge pull request 'fix(lxc-pxe-boot): make privileged so NFS mounts work' (#78) from worktree-debian-pxe into main
Reviewed-on: #78
2026-07-26 19:29:25 +00:00
beatzaplentyandClaude Sonnet 4.6 48d6d6f7a2 fix(lxc): auto-derive privileged from NFS fileSystems, not hostname list
Replace the hardcoded hostname check (docker, pxe-boot) with a check
on config.fileSystems: any lxc-* host whose NixOS config declares an
NFS fileSystem entry is automatically made privileged. The script
already reads proxmoxLXC.privileged dynamically via
flake_target_lxc_privileged, so no logic change is needed there —
only the comment is updated to describe the new derivation.

Result: lxc-docker and lxc-pxe-boot (the two with NFS mounts) evaluate
as privileged=true; lxc-nix-cache, lxc-minimal, lxc-server,
lxc-tailscale-router, lxc-tor-relay evaluate as privileged=false.
Any future lxc-* host that declares an NFS mount gets the correct
privilege level for free without a separate manual edit.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:27:56 +10:00
beatzaplentyandClaude Sonnet 4.6 cf0d62696f fix(lxc-pxe-boot): make privileged so NFS mounts work
The kernel's NFS client (FS_USERNS_MOUNT not set) rejects NFS mounts
from inside any unprivileged container's user namespace with EPERM —
AppArmor's mount=nfs feature only whitelists the AppArmor layer; the
VFS-level rejection happens before AppArmor is consulted.

lxc-pxe-boot mounts server.sweet.home:/tank/pxe-boot/images at
/mnt/pxe-images so nginx can serve large ISOs without filling the
container's root disk. Same pattern as lxc-docker (already privileged
for the same reason since it also mounts several NFS shares).

Operator action required: VMID 103 must be recreated as a privileged
container (the UID mapping on disk differs between privileged and
unprivileged; changing it in-place with pct set is unsafe). Rebuild
the tarball and use create-proxmox-resource.sh to replace it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:23:22 +10:00
beatzaplentyandClaude Sonnet 4.6 b8d21d78d9 fix: wire vars.nfsShares.options and correct stale share key names
- Filter non-attrset values from nfsShares in server.nix so the
  poolDatasets loop skips the new `options` string entry
- Fix typo proxomoxLxcImages → proxmoxLxcImages in server.nix exports
- Rename proxmoxPxeImages → pxebootImages in mount-pxe-images.nix and
  pxe-boot.nix to match the actual key in variables.nix

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 04:45:16 +10:00
beatzaplenty dcce023b14 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos 2026-07-27 04:34:53 +10:00
beatzaplenty 4c5ade5605 updated NFS shares and added vars.nfsShares.options 2026-07-27 04:34:32 +10:00
beatzaplentyandClaude Sonnet 4.6 b65736c0dc fix(pxe-boot): fix NFS mount in LXC — NFSv3+nolock and skip rpc_pipefs
Proxmox LXC containers block the sunrpc filesystem (rpc_pipefs) via AppArmor
unless the container has `features: mount=nfs` set. NFSv4 requires rpc_pipefs
for client state management, so the mount fails outright in a default LXC.

Two fixes for the LXC case (config.boot.isContainer):
- Switch from nfsvers=4.2 to nfsvers=3,proto=tcp,nolock,nofail: NFSv3 doesn't
  need rpc_pipefs at the protocol level, and nofail keeps boot clean if the
  NFS server is unreachable.
- Add ConditionVirtualization=!container to var-lib-nfs-rpc_pipefs.mount via
  systemd drop-in: NixOS pulls this unit into nfs-client.target for any NFS
  fileSystems entry. With the condition, systemd skips (not fails) the unit in
  containers, keeping nfs-client.target green and activation reporting clean.

Proxmox VM hosts (not isContainer) continue to use nfsvers=4.2 with
x-systemd.automount unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 04:14:18 +10:00
beatzaplenty 34bb14d9f6 Merge pull request 'fix(server): open mountd port 20048 so NFS clients can scan and mount' (#74) from worktree-fix-nfs-mountd-firewall into main
Reviewed-on: #74
2026-07-26 17:57:40 +00:00
beatzaplentyandClaude Sonnet 4.6 2e9d3da301 fix(server): open mountd port 20048 so NFS clients can scan and mount
showmount and the NFSv3 mount protocol need mountd reachable after querying
portmapper on 111; the server was only opening TCP 111 and 2049, causing
clients (e.g. Proxmox GUI NFS storage scan) to time out connecting to
mountd on 20048. Also adds UDP for all three ports — portmapper, nfsd, and
mountd all use both protocols.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 03:56:39 +10:00
beatzaplentyandClaude Sonnet 4.6 321048626e feat(pxe-boot): mount pxe-images NFS share and symlink to /srv/pxe/http/images
Adds modules/pxe-boot/mount-pxe-images.nix, which mounts
server.sweet.home:/tank/proxmox/pxe-images at /mnt/pxe-images via NFSv4.2
(x-systemd.automount on Proxmox VMs, nofail on LXC containers — same pattern
as docker/mount-data.nix). The pxe-boot build-type now imports this module and
replaces the previous local /srv/pxe/http/images directory rule with an L+
symlink pointing to /mnt/pxe-images, so large images (ISOs, disk images) live
on the NFS share rather than the host's own root disk.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 03:54:37 +10:00
beatzaplenty 16d6baea5f Merge remote-tracking branch 'origin/main' into worktree-debian-pxe 2026-07-27 03:50:56 +10:00
beatzaplenty d082c6a084 updated nfs shares on server build type 2026-07-27 03:40:19 +10:00
beatzaplenty d1ce8d3e71 add proxmox iso share 2026-07-27 03:26:31 +10:00
beatzaplentyandClaude Sonnet 4.6 f7c32aff12 feat(pxe-boot): add Debian bookworm minimal netboot menu entry
Adds a `fetch-debian-netboot.service` oneshot that downloads the Debian
bookworm netboot kernel and initrd from deb.debian.org on first boot,
stages them under /srv/pxe/http/debian/, and serves them via a generated
debian.ipxe chain script. The service is idempotent — it skips the
download if both files are already present.

Also merges the previously split systemd.tmpfiles.rules and
systemd.services blocks into a single systemd = { ... } attrset to
satisfy statix's repeated-keys lint.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 03:23:14 +10:00
beatzaplenty de508141a8 removed tailscale from docker updated secrets
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
2026-07-26 11:12:46 +10:00
beatzaplenty 18cd9c2342 Merge pull request 'feat(beszel): add beszel agent to tailscale-router' (#72) from worktree-beszel-tailscale-router into main
Check NixOS configurations / eval-hosts (push) Failing after 11m27s
Reviewed-on: #72
2026-07-26 01:08:29 +00:00
beatzaplentyandClaude Sonnet 4.6 997918e2f7 feat(beszel): add beszel agent to tailscale-router
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m33s
Wires beszel-agent into all tailscale-router variants (lxc/linode/proxmox)
by importing enable-agent.nix in the build type and host-token.nix in the
host file. Adds the sops creation rule for secrets/tailscale-router.yaml
(all three platform variants as recipients). The secrets file must be
created manually before deploying — see instructions in PR.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 11:07:51 +10:00
beatzaplentyandClaude Sonnet 4.6 e9b225d2d6 fix(tailscale-router): add MASQUERADE to POSTROUTING, not nixos-nat-post
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
extraCommands runs after nixos-nat-post is deleted but before it is
re-created, so -A nixos-nat-post silently fails every time.  POSTROUTING
is a built-in chain that always exists; target it directly instead.

The -C idempotency check prevents duplicate rules on firewall reloads.
Drop networking.nat.enable -- it was only needed for the sub-chain that
turned out to be the wrong target.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 10:00:47 +10:00
beatzaplentyandClaude Sonnet 4.6 781b1d324e fix(tailscale-router): actually insert MASQUERADE rule via extraCommands
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
networking.nat.externalInterface without internalInterfaces creates the
nixos-nat-post chain but inserts no MASQUERADE rule into it — confirmed
by inspecting the live firewall-start script on the deployed host.

Add the rule explicitly via firewall.extraCommands targeting nixos-nat-post,
scoped to LAN source traffic (vars.lanCidr) going out tailscale0.
extraStopCommands removes it on firewall stop.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 09:54:33 +10:00
beatzaplentyandClaude Sonnet 4.6 cda2132d6a fix(tailscale-router): masquerade LAN traffic into Tailscale
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
Without SNAT on tailscale0, Tailscale drops forwarded packets from LAN
source IPs (192.168.2.x) because they are not recognised Tailscale
addresses.  With networking.nat.externalInterface = "tailscale0", all
traffic leaving through the Tailscale tunnel is masqueraded to the
router's own Tailscale IP (100.x.x.x), making it indistinguishable from
locally-originated traffic.  Conntrack handles the return path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 09:45:07 +10:00
beatzaplentyandClaude Sonnet 4.6 4be064572d feat(tailscale): enable UDP GRO forwarding on subnet router uplink
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m42s
Adds a oneshot systemd service that sets ethtool rx-udp-gro-forwarding on
and rx-gro-list off on the default-route interface at boot, silencing
Tailscale's warning about suboptimal UDP GRO forwarding on subnet routers.
Interface is discovered dynamically via `ip route get` so it works on all
platforms regardless of NIC naming.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-26 09:13:28 +10:00
beatzaplentyandClaude Sonnet 4.6 6e1e992652 fix(proxmox): embed SSH host key via NIXOS_HOST_KEYS_DIR so sops can decrypt on first boot
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m30s
--pre-format-files placed the key on the QEMU builder VM's rootfs, not the
target disk. nixos-install chroots into the target and runs sshd-keygen, which
found no key in the chroot and generated a fresh (unregistered) one. sops then
could not decrypt on first boot because the key didn't match .sops.yaml, leaving
both root and nixos with '!' in /etc/shadow even after mutableUsers = false was
set (hashedPasswordFile pointed to paths sops never wrote).

Fix modules/platforms/proxmox.nix to embed the clan SSH host key in
environment.etc via NIXOS_HOST_KEYS_DIR at eval time -- the same pattern
lxc.nix uses. nixos-install's own activation places the key on the target disk,
sshd-keygen finds it already present and skips generation, and sops decrypts
correctly on first boot. Includes the same preserveSshHostKey/restoreSshHostKey
activation scripts as lxc.nix so subsequent nixos-rebuild switch calls (without
NIXOS_HOST_KEYS_DIR) don't remove the key as "obsolete" from environment.etc.

Update create-proxmox-resource.sh: switch VM builds from
  ./result-<target> --pre-format-files ... --build-memory 2048
to
  NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build --impure ... diskoImagesScript
  ./result-<target> --build-memory 2048
matching the LXC build path.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
2026-07-26 08:54:39 +10:00
beatzaplentyandClaude Sonnet 4.6 dfa5452af5 fix(common): set mutableUsers = false to fix password setup on disk images
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
When a proxmox-* disk image is built, activation runs during the image
build without a valid sops age key (the SSH host key doesn't exist yet),
so root and nixos land in /etc/shadow with locked '!' entries. With the
default mutableUsers = true, update-users-groups.pl preserves existing
shadow entries for accounts that already exist, so hashedPasswordFile is
silently ignored on every subsequent boot — passwords are never fixed.

Setting mutableUsers = false forces update-users-groups.pl to apply
hashedPasswordFile unconditionally on every activation. On first real
boot the sops-decrypted hash is now written regardless of whether the
account already existed in shadow from the image build.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
2026-07-26 07:36:06 +10:00
beatzaplentyandClaude Sonnet 4.6 1a14b1d4d3 fix(lxc): move sops-reinstall service from sysinit to network.target
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
nixos-lxc-sops-reinstall.service called switch-to-configuration test at
sysinit.target time (DefaultDependencies=false), before D-Bus was up.
D-Bus is required to restart systemd targets after activation scripts
run. The service reported failure on every boot (exit 1: "Failed to open
dbus connection") even though secrets were correctly installed, because
the D-Bus call happens after activation scripts complete.

Move the service to network.target so basic.target (which includes
dbus-broker.service) runs first. Also drop DefaultDependencies=false so
systemd auto-adds After=basic.target. Add SuccessExitStatus=11 to handle
the edge case where switch-to-configuration holds the lock during a
concurrent rebuild (exit 11 = "Could not acquire lock" -- the rebuild's
own activation already installed the secrets, so treating it as success
is correct).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 19:50:35 +10:00
beatzaplentyandClaude Sonnet 4.6 055577ee91 fix(lxc): fix activation ordering and add boot-time sops reinstall
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
Two bugs prevented nixos-rebuild switch from working on lxc-* hosts after
first boot, both confirmed live on a deployed lxc-tor-relay container:

1. Ordering bug: preserveSshHostKey had no explicit deps, so the topological
   sort placed it at position 7 — after etc at position 5. By the time it
   tried to save the SSH key, etc had already removed it as "obsolete"
   (absent from the current generation's environment.etc when built without
   NIXOS_HOST_KEYS_DIR). Consolidate all four system.activationScripts entries
   into one block and add etc = { deps = ["preserveSshHostKey"]; } and
   setupSecrets = { deps = ["restoreSshHostKey"]; } to enforce the correct
   save→etc→restore→sops chain.

2. No boot-time secrets: /run/secrets is a tmpfs cleared on every reboot, and
   sops-nix does NOT generate a boot-time service in this configuration
   (confirmed live: no sops-nix.service in systemctl list-unit-files).
   Add nixos-lxc-sops-reinstall.service, modelled after sops-nix's own service
   placement (wantedBy/before sysinit.target, DefaultDependencies=false), so
   secrets are reinstalled before basic.target on every non-first boot.
   ConditionPathExists skips it on first boot; nixos-lxc-first-boot-activate
   handles that case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 19:07:30 +10:00
beatzaplentyandClaude Sonnet 4.6 a63e1c70c3 feat(provision): Phase 2 — migrate SSH host keys to clan vars
Replaces the gitignored host-keys/ directory with clan vars as the
authoritative storage for SSH host keys. Keys are now generated as
sops-binary-encrypted clan var files (admin-key only) and checked into
vars/per-machine/<target>/openssh/, eliminating the plaintext private
key that previously had to live outside the repo.

Changes:
- modules/clan/ssh-host-key.nix: clan vars generator for the ed25519
  SSH host key pair (neededFor="activation" — not mapped to sops.secrets,
  delivered via tarball baking for LXC or --pre-format-files for VMs)
- flake.nix: add clanCore module + required settings to every mkTarget;
  deduplicate bundled disko/sops-nix via follows; all 27 hosts eval clean
- flake.lock: updated to reflect the new follows constraints
- scripts/lib/clan-vars.sh: new helper library with
  clan_ssh_key_exists / clan_ssh_pubkey_path / clan_decrypt_ssh_key /
  clan_generate_ssh_key for use by the provisioning and sync scripts
- scripts/secrets/sync-host-keys.sh: queue_host_sync() now checks clan
  vars first; generates via clan_generate_ssh_key if no key exists;
  derives age fingerprint from clan pub key for .sops.yaml registration
- scripts/proxmox/create-proxmox-resource.sh: key management simplified
  (sync-host-keys.sh now generates the key if missing, so the inline
  prepare-host-key.sh call is gone); sync_remote_host_keys() decrypts
  the clan key into a temp dir and scps just the two files to the node
  when a clan key exists, falling back to the old host-keys/ scp for
  any remaining legacy entries

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
2026-07-25 18:22:33 +10:00
beatzaplentyandClaude Sonnet 4.6 c844ccc4e3 fix(tailscale): rename tailscale-subnet-router → tailscale-router everywhere
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m28s
Complete the rename so all identifiers match networking.hostName:
- flake.nix: attribute names and buildType strings
  (linode/proxmox/lxc-tailscale-subnet-router → *-tailscale-router)
- modules/build-types/tailscale-subnet-router.nix → tailscale-router.nix
- .sops.yaml: anchor and alias names (age keys unchanged, no re-encrypt needed)
- host-keys/: local gitignored key files renamed (not committed)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 16:33:03 +10:00
beatzaplentyandClaude Sonnet 4.6 f8719437ba fix(nix-cache): use FQDN to fix hostname resolution on clients
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
systemd-resolved only uses LLMNR for single-label hostnames, never DNS —
same issue mount-data.nix already documented and fixed for NFS by switching
to server.sweet.home. Change the substituter URL, SSH knownHosts, and
remote-builder hostName from bare "nix-cache" to "nix-cache.sweet.home",
and update nginx's virtualHost to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-25 16:01:56 +10:00
beatzaplentyandClaude Sonnet 4.6 e4b335be23 refactor(tailscale): rename exit-node to subnet-router; drop --advertise-exit-node
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s
The host was always intended as a LAN subnet router (--advertise-routes),
not a full exit node (--advertise-exit-node). Rename every trace of
"exit-node" to "subnet-router" and remove the --advertise-exit-node flag
from extraSetFlags; the operator supplies --advertise-routes at first
tailscale up and Tailscale persists it in state across reboots.

Routing sysctls (useRoutingFeatures = "server"), openFirewall, and
trustedInterfaces = ["tailscale0"] are still required for subnet routing
to work, so the module is kept — just correctly named.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TF2dsuKZAiyZWQ1D7CuHJm
2026-07-25 14:53:28 +10:00
beatzaplentyandClaude Sonnet 4.6 dce3788499 fix(lxc): prevent SSH host key deletion on every rebuild; add recovery script
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
NixOS's etc activation removes files that were in a previous generation's
environment.etc but absent from the current one -- even real copies, not
only symlinks.  LXC tarballs bake the host key into environment.etc (via
NIXOS_HOST_KEYS_DIR), but every subsequent nixos-rebuild switch lacks that
env var, so the key is removed as "obsolete".  sops-nix derives its age
decryption key from /etc/ssh/ssh_host_ed25519_key, so deletion cascades
into "Error getting data key: 0 successful groups required, got 0" for
every sops secret on the host.

Fix: two activation scripts bracket the etc step.
  preserveSshHostKey (no deps, runs before etc): copies the live key to
    /run (tmpfs) before etc can delete it.
  restoreSshHostKey (deps=[etc], runs after etc): reinstalls via `install`
    if etc removed the key.  The resulting file is not tracked in either
    generation's environment.etc, so subsequent rebuilds leave it alone.

scripts/recover-hosts.sh: restore both private and public key files (not
just the private key), use install(1) for atomic mode setting, and add a
post-rebuild sops-nix verification step to confirm success.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014zT1L6hmsq6i1evAEH7dmi
2026-07-24 09:17:58 +10:00
beatzaplenty 4800aebf43 enable beszel agent
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
2026-07-24 07:03:01 +10:00
beatzaplentyandClaude Sonnet 4.6 6002c5c738 server: auto-create tank ZFS pool on first boot if data disk is blank
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m40s
A fresh proxmox-server deploy has a blank scsi1 disk, so
zfs-import-tank.service spun 60 s then failed with no pool found.
Add zfs-init-tank.service that runs before the import: exits immediately
if the pool already exists, imports it if it exists but isn't imported
yet, or creates it on /dev/disk/by-id/scsi-*drive-scsi1 (Proxmox's
virtio-scsi naming for the second disk) with all required NFS datasets
if the disk is blank.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-24 06:13:58 +10:00
beatzaplentyandClaude Sonnet 4.6 5497a5b0ae feat: add nixos@nixos workstation SSH key to all hosts
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m44s
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rcq7iXrN9eYUTUEWrbWqyX
2026-07-23 11:05:51 +10:00
beatzaplenty bcae177d8e added jq
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
2026-07-23 06:59:37 +10:00
beatzaplenty d35aca3138 added jq and direnv 2026-07-23 06:59:00 +10:00
beatzaplenty 5d7a6327b7 Allow initrd emergency shell access on baremetal-gui
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m45s
The systemd-based initrd (default here, since this host has a ZFS
root) locks the root account by default, so sulogin refuses a shell
if something in the initrd fails and it drops to emergency mode --
confirmed live: it just loops re-entering the target instead of
prompting, making an initrd-level ZFS import failure impossible to
diagnose from the console. Only affects the pre-switch-root initrd
shell, not the installed system's own login.
2026-07-22 03:47:54 +00:00