nixos user is in the docker group so sudo is not needed (and breaks because
sudo requires a TTY for password input). Also grep -q 'active' matched
'inactive' causing ha-docker-2 to be seen as already joined.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
The phase 3.5 branch-switch step was hardcoded to /home/user/nixos, which
assumes the standalone nixos Gitea repo is checked out there. When the
infrastructure mono-repo is used instead, the path differs. Now reads
PROXMOX_REMOTE_REPO_DIR (same variable create-proxmox-resource.sh uses)
with graceful fallback if the branch switch fails.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
Adds sops-encrypted SSH host key pairs for proxmox-ha-docker-1 and
proxmox-ha-docker-2 (clan vars), and re-encrypts secrets/common.yaml
to include both hosts as recipients so they can decrypt on first boot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
Two new NixOS Proxmox VMs (VMIDs 202/203) forming a dual-manager Docker
Swarm on dedicated vmbr3 (192.168.30.0/24, VLAN 30) for gossip and VXLAN,
with NFS via the storage-client network (vmbr2) from the existing HA cluster.
- nixos/variables.nix: add ha-docker IP/interface/port vars and swarm CIDR
- nixos/modules/build-types/ha-docker.nix: new build type — Docker 29,
NFS mounts, beszel-agent, health monitoring, swarm firewall rules with
checkReversePath = "loose" for VXLAN routing mesh
- nixos/hosts/ha-docker-{1,2}/host.nix: per-host identity — three NICs
(LAN, storage, swarm), IPA dyndns pinned to LAN interface
- nixos/flake.nix: add proxmox-ha-docker-{1,2} targets; build-validated
with nix build --dry-run (169 derivations, no errors)
- nixos/docs/ip-addressing.md: document VLAN 30 / swarm.home zone,
ha-docker IP allocations across all three subnets
- nixos/scripts/docker-swarm/deploy.sh: 10-phase lifecycle script
(bridge, keys, IPA, VMs, swarm init, DNS, verify); modelled on
scripts/ha/deploy.sh with --destroy mode
- nixos/docs/internal/docker-swarm-cutover.md: service-by-service
migration guide covering Traefik log rotation, Nextcloud cron sidecar,
docker-health-to-gotify swarm awareness updates, Passbolt/Gitea steps,
DNS cutover, and CT 105 decommission checklist
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
- Document bidirectional LAN ↔ Tailscale routing via tailscale-router
(static route on router for 100.64.0.0/10, subnet advertisement into Tailnet)
- Document DNS proxy chain: FreeIPA conditional forwarder for *.ts.net
→ tailscale-router → MagicDNS; reverse for *.sweet.home from Tailscale clients
- Remove stale pihole.sweet.home from IP table (decommissioned)
- Fix freeipa role: remove Pi-hole IP from ipa_dns_forwarder, document
ts.net conditional forwarder as a post-install step
- Fix inventory: remove Pi-hole forwarder IP, add ts.net forwarder note
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvNjoxTWEDkhXsd1Dq2ETP
Consolidates nixos, docker, raspi, and debian-configuration into a single
infrastructure-as-code repo. Includes:
- ansible/: full inventory + proxmox-hardening, freeipa, and raspberrypi
roles (converted from debian-configuration bash scripts)
- terraform/: Proxmox VMs, Dynu DNS, Pi-hole (decommissioned stub),
Docker container catalog — migrated from docker/infrastructure/terraform/
- stacks/docker/, stacks/raspi/, nixos/: placeholder READMEs pending
git subtree population (see implementation plan)
- docs/: internal MkDocs site with architecture, network topology, runbooks,
and drift-detection guide; external sanitized site
- scripts/: drift-detect.sh, docs-build.sh, install-hooks.sh, check-secrets.sh
- CI: secret-scan (push/PR), drift-detect (daily), docs-build (on change)
- Pi-hole removed throughout — DNS is FreeIPA, DHCP is router
See docs/internal/implementation-plan.md for the phased rollout after
pushing to Gitea.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvNjoxTWEDkhXsd1Dq2ETP
- variables.nix: switch to rec {}, extract giteaDomain/giteaRepoPath,
extraAdminSshKeys, haLanNfsFqdn, tailscaleResolverIp, ports.dhcp,
ports.dns; ipaServer now derives from homeDomain ref; section headers
- modules: use new vars throughout (pxe-boot, ts-dns-forwarder,
cluster-config, configuration.nix, mount-pxe-images) — eval unchanged
- docs: delete ephemeral planning docs (AUDIT_REPORT, ha-network-audit,
network-cutover); add docs/ha.md; drop migration reference table from
ip-addressing.md; remove stale server example from beszel.md
- CLAUDE.md/README.md/AGENTS.md: fix build types (tailscale-router,
ha-server, drop server); document scripts/ha/, scripts/ipa/, and
all previously undocumented top-level and lib scripts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Updates mount-pxe-images.nix to mount pxe-boot/images from
ha-vip-lan.sweet.home (Pacemaker vip-lan, 192.168.2.229) instead of
server.sweet.home. Storage root changes from /tank to haStorageRoot
(/srv/ha-data). NFSv3+nolock options for LXC are unchanged.
Removes nfsServerHost and storageRoot from variables.nix — all NFS
clients now reference the HA cluster directly (docker via haStorageNfsFqdn
on VLAN 20, pxe-boot via ha-vip-lan on VLAN 2). Updates the nfsShares
comment to reflect the new single-source-of-truth role without the
dead server.nix references.
DNS records for server.sweet.home and server.storage.home were already
cleaned up automatically by IPA dynamic DNS when the VM was deleted.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Removes all flake artifacts for the retired NFS/ZFS file server:
- flake.nix: drop linode-server, proxmox-server, lxc-server targets
- hosts/server/: delete host identity
- modules/build-types/server.nix: delete build type (ZFS pool + NFS exports)
- vars/per-machine/{linode,lxc,proxmox}-server/: delete clan vars
- secrets/server.keytab: delete IPA keytab
- .sops.yaml: remove server key anchors, strip from common.yaml recipients,
delete server.yaml and server.keytab creation rules
- variables.nix: remove serverIp, serverStorageIp (nfsServerHost/storageRoot
retained — pxe-boot/mount-pxe-images.nix still references them)
- docs: mark server decommissioned in ip-addressing.md
Docker NFS mounts already pointed at nfs.storage.home (previous commit).
VM destruction and DNS cleanup (server.sweet.home, server.storage.home
A/PTR records) to follow once the VM is confirmed stopped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Updates mount-data.nix to mount all docker shares from
nfs.storage.home (Pacemaker vip-storage, 192.168.20.229) over docker's
eth1/vmbr2 interface instead of server.sweet.home over the LAN.
Adds haStorageNfsFqdn variable to variables.nix for the storage.home
zone FQDN so mounts survive a future VIP renumber without a rebuild.
Storage root changes from /tank (server ZFS pool) to /srv/ha-data
(HA cluster XFS-over-DRBD). Share subpaths are unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Adds a DNS Zones section to ip-addressing.md covering the three FreeIPA
zones (sweet.home / cluster.home / storage.home) with all A records and PTR
records as configured. Updates Stage 6 in network-cutover.md to reference
nfs.storage.home by DNS name rather than the raw VIP IP, and marks DNS as a
completed prerequisite for the HA storage cutover.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
ip-addressing.md:
- Storage-client network purpose updated: NFS primary, iSCSI available
- Protocol separation box updated: NFS on both VIPs (each CIDR-restricted);
iSCSI on VLAN 20 only but not in active use
- docker role updated to NFS client (not iSCSI initiator)
- vip-storage description updated to NFS + iSCSI endpoint
ha-network-audit.md:
- Implementation status header added noting all audit items resolved
- Records decisions that diverged from original recommendations
(VLAN renumbering, dual VIP, NFS-over-iSCSI for docker/swarm)
- Notes deferred items (STONITH, iSCSI ACLs)
network-cutover.md:
- New Stage 6: HA storage cutover — docker NFS remount from
server.sweet.home to vip-storage (192.168.20.229), final rsync
steps, and server.sweet.home decommission procedure
- Former Stage 6 (final cleanup) renumbered to Stage 7
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
vip-storage (192.168.20.229) now serves NFS as well as iSCSI, firewalled
to haClientCidr (192.168.20.0/24) only. This allows docker and future
swarm nodes to NFS-mount shared volumes from the storage network rather
than the LAN, keeping storage traffic off vmbr0.
LAN NFS (vip-lan 192.168.2.229) remains for pxe-boot and other LAN
clients. Protocol and subnet boundaries enforced by firewall on both nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Storage VIP (192.168.20.229) is on vmbr2 — an internal-only bridge with no
physical uplink, unreachable from the LAN. Test it via SSH to the active node
(which has VLAN 20 connectivity) instead of directly. LAN VIP (192.168.2.229)
continues to be tested directly from wherever the script runs.
Also adds VIP_LAN variable to config section.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Renumber internal HA bridges so VLAN ID matches IP third octet:
vmbr1 (cluster): 192.168.4.x → 192.168.10.x (VLAN 10, /29)
vmbr2 (storage): 192.168.5.x → 192.168.20.x (VLAN 20, /24)
Enforce protocol separation at the firewall on both HA nodes:
- iSCSI (3260/tcp): storage-client subnet (192.168.20.0/24) only
- NFS (111,2049,20048 tcp+udp): LAN subnet (192.168.2.0/24) only
- Cluster subnet (192.168.10.224/29) accepted wholesale (DRBD+Corosync)
Moves from allowedTCPPorts/allowedUDPPorts to source-restricted extraCommands
rules on the nixos-fw chain.
NFS exports reduced to LAN-only (drop haClientCidr lines from mkNfsExports).
Storage-client network clients use iSCSI only; no NFS on VLAN 20.
All HA script VIP/storage IP defaults updated to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
mkTarget computed flakeTarget as "${platform}-${buildType}" which gives
"proxmox-ha-server" for both nodes, but the actual nixosConfigurations
keys are "proxmox-ha-server-1" / "proxmox-ha-server-2". This caused
Switch-nix to fail immediately with an "unknown flake attribute" error.
Add an optional nameSuffix parameter to mkTarget and pass "-1" / "-2"
for the ha-server entries so /etc/flake-target matches the real
attribute name on each node.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Both VIPs fail over together in ha-group:
vip-lan: 192.168.2.229/24 on ens18 (vmbr0) — LAN iSCSI + NFS access
vip-storage: 192.168.5.229/24 on ens20 (vmbr2) — storage-client iSCSI + NFS
variables.nix: haServerLanVip = "192.168.2.229"
cluster-init.sh: both VIP primitives in ha-group XML; VIP_LAN var added
ip-addressing.md: both VIP entries documented
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Infrastructure changes already applied to pve1:
- vmbr2 internal bridge created (192.168.5.0/24, no physical uplink)
- VM 200 (ha-server-1): net2 added → vmbr2 (ens20)
- VM 201 (ha-server-2): net2 added → vmbr2 (ens20)
- CT 105 (docker): net1 added → vmbr2 (eth1)
- VM 101 (server): net1 added → vmbr2 (ens19) — needs reboot to activate
NixOS config (deploy to ha nodes to complete; docker/server at cutover):
- ha-server-{1,2}/host.nix: ens20 with 192.168.5.{228,227}/24
- docker/host.nix: eth1 with 192.168.5.225/24
- server/host.nix: ens19 with 192.168.5.226/24
- cluster-config.nix: corosync ring1 on LAN IPs as backup heartbeat path
- cluster-config.nix: allow haClientCidr (192.168.5.0/24) in iptables
- ha-server.nix: NFS exports now allow both lanCidr and haClientCidr
- VIP moves from 192.168.2.229 (vmbr0/LAN) to 192.168.5.229 (vmbr2)
- iSCSI portal to be rebound from [::0] to 192.168.5.229 at cutover
variables.nix: haStorageCidr corrected to 192.168.4.224/29; new vars:
vmStorageClientInterface, lxcStorageInterface, haServer{1,2}ClientIp,
dockerStorageIp, serverStorageIp, haClientCidr/PrefixLength; haServerVip
updated to 192.168.5.229.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Read-only discovery of pve1 bridges, HA cluster VMs (200/201), and
Docker CT (105) — current state, risk flags, recommended vmbr2 storage
bridge layout, gap list, and migration notes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pve1 runs non-NixOS guests (FreeIPA domain-controller, Proxmox Data Manager, etc.)
alongside NixOS ones. Validate each discovered hostname against the set of
hostnames defined in nixosConfigurations before adding it to the gc list.
Brings back the flake eval but uses it correctly: extract all hostname values
(not target-name keys) to build a filter set, then only include pve1 guests
whose name matches a flake-managed NixOS host.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The by-id lookup is read-only so it's safe to run in dry-run mode.
Previously it was gated behind `if ! $DRY_RUN`, which always triggered
the sdb fallback warning in dry-run — making it look like the device
path wasn't reliable when the symlink actually exists on both servers.
Now the lookup always runs and the script errors out with a clear message
if the by-id symlink is genuinely missing, instead of silently falling
back to a guessed /dev/sd* name.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The flake hostname map approach was wrong: create-proxmox-resource.sh names
every guest after its NixOS hostname (pct create --hostname <nixos-hostname>,
qm create --name <nixos-hostname>), so pct/qm list already output the final
hostname directly. Looking those names up in a map keyed by flake target names
(lxc-docker, proxmox-server, ...) found nothing and the dynamic list was
always empty.
Drop the flake eval entirely; use the pct/qm list output as-is, filtering
only nix-cache and already-seen entries.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
qm lives at /usr/sbin/qm, which is not in the default PATH for
non-interactive SSH sessions as a non-root user. Add PVE_SUDO (set to
"sudo" when PVE_SSH_USER != root, matching create-proxmox-resource.sh's
own sudo_prefix pattern) and prepend it to all three qm invocations in
the script.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs:
1. Dynamic discovery used BatchMode=yes for the pct/qm list SSH call, which
silently suppressed any sudo password prompt and returned empty output.
Fix: split SSH_OPTS into SSH_OPTS (BatchMode, for gc) and SSH_QUERY_OPTS
(no BatchMode, for discovery) so sudo can prompt when needed.
2. pve1 gc used a non-login SSH session which doesn't source /etc/profile,
so nix-collect-garbage was never on PATH. The Nix installer registers
itself via /etc/profile.d/nix-daemon.sh which only runs in login shells.
Fix: use "bash -l -c 'nix-collect-garbage -d'" instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
BatchMode SSH sessions don't source /etc/profile on non-NixOS hosts, so
nix-collect-garbage isn't on PATH for the wayne user. Source the nix-daemon
profile script explicitly, matching the pattern in scripts/lib/nix-bootstrap.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the static host list with dynamic discovery: workstation (nixos)
and pve1 are hard-wired first and second; remaining hosts are discovered
on every run by SSHing to pve1, listing running VMs/containers via
pct/qm list, and resolving their NixOS hostnames from a single flake eval.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Runs nix-collect-garbage -d on all deployed NixOS hosts and pve1 in
parallel, skipping nix-cache to avoid evicting shared cache paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Generated /home/wayne/.ssh/id_ed25519 on pve1 (single-user Nix install
owned by wayne, no daemon — builds run as wayne not root). Adds the
public key to remoteBuilderAuthorizedKeys so nix-cache accepts it.
After deploying nix-cache, re-run configure-nix-cache-client.sh as
wayne on pve1 to wire in the builder line and known_hosts entry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nfs-utils changed ConditionPathExists from !/etc/krb5.keytab to
/etc/krb5.keytab, so on IPA-joined hosts the service now starts instead
of skipping. Exports use standard auth (no sec=krb5) so the nfs/ Kerberos
principal is never provisioned and the service fails.
enable = false masks the unit so nfs-server's Wants= can't pull it in.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
KEY (hub's SSH public key) is now stored in variables.nix as beszelHubKey
and set by enable-agent.nix — no host file ever needs to touch it.
TOKEN was already universal; this removes the last per-host beszel config.
Importing enable-agent.nix in a build type is now the only step needed to
add a new host to beszel monitoring.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five files contained only beszel-token and are deleted entirely.
nix-cache.yaml retains cache-priv-key; beszel-token removed from it.
The universal token in secrets/common.yaml replaces all of these.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace per-host host-token.nix imports with a single beszel-token secret
in secrets/common.yaml, wired once in enable-agent.nix. Host files now only
need services.beszel.agent.environment.KEY — no imports block required.
Delete modules/beszel/host-token.nix (no longer referenced anywhere).
Action needed: run `sops secrets/common.yaml` and add `beszel-token: <value>`
from the beszel hub UI before deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents how to wire beszel-agent into any flake build: the two-module
split (enable-agent.nix in build types, host-token.nix in host files),
sops secret creation, hub pairing flow, and optional EXTRA_FILESYSTEMS /
Docker monitoring knobs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three-phase online resize: Proxmox qm resize → guest block device
rescan → drbdadm resize + xfs_growfs. No downtime required.
Detects active node via crm_mon, pre-checks DRBD UpToDate/UpToDate,
auto-discovers VMIDs from qm list, and resolves the block device name
from the stable scsi-0QEMU... by-id symlink with a slot-index fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
crm_mon 2.x formats the Promoted line as " * Promoted: [ node ]" — the
asterisk bullet means the previous grep -E '^\s*(Promoted|Masters):' never
matched, so active-node detection silently returned empty in health.sh,
failover.sh, and acceptance-tests.sh.
Fix: pipe through grep -v Unpromoted first, then grep -E '(Promoted|Masters):'
without anchoring to start-of-line.
Also: remove the SSH_OPTS=-i ~/.ssh/... variable pattern in health.sh and
failover.sh; tilde is not expanded inside double-quoted strings, so $SSH_OPTS
was passing a literal ~ to SSH. Inline the key path in each function
definition instead (same as acceptance-tests.sh already did).
Also: drop the 2>/dev/null embedded in the crm_mon argument string — the
outer 2>/dev/null on the n1/n2 call already suppresses SSH stderr; the
embedded one was harmless but noisy to reason about.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
failover.sh: graceful active→standby resource migration with pre-checks
(DRBD UpToDate/UpToDate, quorum), spinner wait for XFS mount on target,
rollback on timeout, --to/--force/--dry-run/--timeout flags.
health.sh: read-only status panel — node reachability, quorum, DRBD
role/dstate/cs per node, Pacemaker resources, failure history, XFS mount
usage, and service port checks (NFS 2049, iSCSI 3260) via VIP.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Even with both nodes in standby, Pacemaker's monitor operations keep running.
When the monitor sees DRBD is Primary on a standby node (that it didn't start),
it triggers a stop action — killing the initial sync after ~10 s.
Enable maintenance-mode after standby stops the existing resources but before
DRBD is manually brought up for the sync. Maintenance-mode suspends all
start/stop/monitor actions so Pacemaker is completely hands-off during the
sync. Disable it alongside crm_standby -v off once UpToDate/UpToDate is
confirmed.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The metadata creation check used grep -q "UpToDate" which matches when the
resource is DOWN — drbdadm dstate reads metadata directly and returns just
"UpToDate" (no slash) for a down-but-previously-synced resource. This caused
metadata creation to be silently skipped, leaving mismatched UUIDs from an
interrupted sync which cause DRBD to go WFConnection→StandAlone immediately.
Fix: require exact "UpToDate/UpToDate" (with slash, only possible when the
resource is UP and both disks are confirmed synced) before skipping.
Also give the StandAlone detection a 5-iteration (15 s) grace period before
dying, to avoid false-positive on the first few ticks while the peer is still
coming up.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Clearing crm_standby before the initial sync finished caused Pacemaker's OCF
DRBD agent to race with the manual drbdadm up/primary calls. The agent saw
DRBD in WFConnection or SyncSource and tore it down, driving the resource back
to StandAlone and killing the sync in seconds.
Move the crm_standby -v off calls to immediately after the sync-complete
break, so Pacemaker only resumes once DRBD is UpToDate/UpToDate and safe
to hand back.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If the DRBD peer connection drops during the initial sync wait (cs:StandAlone),
the loop would spin forever printing "waiting for sync progress" with no
indication of what's wrong. Now parses cs: from /proc/drbd each tick and
dies with a clear error if StandAlone is detected. Also shows the cs: field
in the non-syncing message so the actual connection state is always visible.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
When the DRBD kernel module invokes fence-peer via the User Mode Helper
mechanism it provides a minimal PATH that omits /run/current-system/sw/bin.
crm-fence-peer.sh calls cibadmin, crm_mon etc.; without those in PATH a
pipeline breaks with SIGPIPE. A signal-killed process has WEXITSTATUS()==0,
so the kernel sees exit code 0, logs "fence-peer helper broken, returned 0",
and retries forever — blocking Pacemaker failover completely.
Fix: use pkgs.writeShellScript to create thin PATH-fixing wrappers in the
Nix store. The wrappers prepend /run/current-system/sw/bin before exec-ing
the real crm-fence-peer.sh / crm-unfence-peer.sh, giving them a working
Pacemaker toolchain regardless of what PATH the kernel provides.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The TTY check ([[ -t 1 ]]) evaluated false under sudo, causing each update
to print on a new line via log(). Since \r worked correctly in the original
code (user confirmed), drop the branching and always use printf \r.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The sync progress grep was silently failing because /proc/drbd formats the
percentage line as "sync'ed: 5.2%" (two spaces after colon), but the
pattern matched only one. Use [[:space:]]+ throughout to tolerate any
whitespace.
Also make the display TTY-aware: interactive sessions get the spinning
\r in-place update; piped/logged sessions get a plain log line printed
once per unique percentage point (avoiding scroll spam).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cluster-init.sh: replace the fixed 300-iteration sync wait with an
indefinite loop that prints an in-place progress line (%done, ETA, speed)
from /proc/drbd every 3 s. Clears the line with printf \r before logging
completion, so the output stays clean alongside the [cluster-init] log
lines.
acceptance-tests.sh: add a pre-flight check that hard-exits if
drbdadm dstate is not UpToDate/UpToDate, with a hint to the watch
command for monitoring progress. Tests cannot give accurate results
while the initial full-sync is in progress.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
acceptance-tests.sh: replace drbdadm-role-based Active node detection with
crm_mon-based detection. DRBD role briefly reads Secondary during Pacemaker
transitions, causing T2/T3/T4 to target the wrong node. crm_mon Promoted:/
Masters: lines are authoritative; wait up to 90 s for Pacemaker to settle
before falling back to NODE1.
cluster-init.sh: add crm_resource --cleanup before the VIP wait loop so
stale migration-threshold failure counters from previous cluster-init runs
are cleared before Pacemaker decides resource placement.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/dev/sda and /dev/sdb are assigned by the OS based on Proxmox disk-add
order, which is not consistent across VMs. Use the SCSI controller path
instead — drive-scsi1 is always the dedicated data disk on all HA nodes
regardless of which sda/sdb it gets assigned to.
variables.nix: replace per-node haServer{1,2}DrbdDisk with a single
haServerDrbdDisk using /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1.
cluster-config.nix: revert to shared volume block (per-node block was
needed for /dev/sd* but the by-id path is identical on both nodes).
cluster-init.sh:
- single DRBD_DISK variable (matching haServerDrbdDisk)
- robust /etc/drbd.conf patch: NixOS manages this file as a symlink to
a read-only Nix store path; cp --remove-destination breaks the symlink
before sed -i so the edit actually takes effect
- scp helper script to NODE2 rather than bash -c over SSH to avoid
quoting complexity
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Proxmox VM disk ordering differs between the two HA nodes:
ha-server-1: sda=OS (50G), sdb=DRBD data (32G)
ha-server-2: sda=DRBD data (32G), sdb=OS (50G)
The DRBD resource config was using a shared disk=/dev/sdb which targeted
the OS disk on ha-server-2, causing drbdmeta and drbdadm up to operate
on the mounted root filesystem (hence "Device or resource busy").
Changes:
- variables.nix: add haServer1DrbdDisk/haServer2DrbdDisk
- cluster-config.nix: move volume block inside per-host on{} sections
so each node uses the correct backing disk
- cluster-init.sh: use NODE1_DRBD_DISK/NODE2_DRBD_DISK variables;
add runtime check that patches /etc/drbd.d/*.res on the running nodes
if the deployed config points to the wrong disk (workaround for VMs
built before this fix; redeploy with --force-rebuild to make permanent)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drbdmeta is in /run/current-system/sw/bin which is already on sudo's
PATH on the HA nodes; the bash -c wrapper with PATH manipulation was
adding complexity without benefit and introduced quoting hazards that
likely caused write-dev-uuid to fail or run in the wrong context.
Use direct n2_ssh calls instead — the same pattern that works when
tested manually on NODE2.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drbdmeta lives in the same Nix store dir as drbdadm but sudo doesn't
inherit the full PATH, so drbdmeta was not found (exit 127) even though
drbdadm was. Resolve drbdmeta's directory from drbdadm's location and
prepend it to PATH.
Replace openssl rand for UUID generation with /proc/sys/kernel/random/uuid
— openssl is not guaranteed to be on PATH in a minimal NixOS root
environment, but /proc/sys/kernel/random/uuid is always present.
Apply the same PATH fix on NODE2 inline in the bash -c invocations that
call drbdmeta over SSH.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cluster-init.sh: replace 'drbdadm create-md --force' with direct drbdmeta
calls using --force on both steps. drbdadm create-md --force passes --force
to the create-md sub-call but NOT to write-dev-uuid, which then fails when
/dev/sdb is busy and stdin is not a TTY ("stdin not a TTY, not waiting for
confirmation" → exit 20). write-dev-uuid failing means DRBD has no UUID,
so the peer can't identify the device → connection fails → no sync → wrong
Active node detected by acceptance tests.
acceptance-tests.sh T4: grep -c returns exit code 1 when the count is 0
(no matches), triggering '|| echo "0"' and producing "0\n0" which breaks
[[ "$IQN_COUNT" -ge 1 ]] with "arithmetic syntax error". Fixed by running
the pipe inside bash -c with '|| true' on the grep to suppress the
non-zero exit code. Same fix applied to T5's IQN_ON_STANDBY check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Root cause of recurring sops failures on new VM boots: disko builds raw
disk images, and create-proxmox-resource.sh only syncs the clan-var SSH
host key to the Proxmox node (for proxmox.nix to bake into the image) when
it actually builds — reusing a cached image skips sync_remote_host_keys, so
destroy+recreate reuses a stale image with the wrong or randomly-generated
key baked in. On first boot the VM gets a different key than what .sops.yaml
was encrypted for, and sops fails permanently.
Fix 1 — deploy.sh Phase 3: always pass --force-rebuild so every VM creation
rebuilds the disko image fresh with the current clan-var key baked in via
NIXOS_HOST_KEYS_DIR (proxmox.nix already reads this under --impure).
Fix 2 — deploy.sh Phase 5.5: after VMs boot, scan their actual ed25519 host
keys and, if they drift from clan vars, update the clan var pub-key files,
rewrite the .sops.yaml age anchors, and re-encrypt all affected sops files.
Defence-in-depth: normally a no-op after Fix 1, but catches any residual
mismatch (e.g. --skip-create-vms reuse of an older image).
Fix 3 — cluster-init.sh: add crm_standby -v on for both nodes before DRBD
metadata init. Without this, Pacemaker's OCF DRBD agent races: it sees
drbdadm down as a failure and immediately calls drbdadm up again, leaving
/dev/sdb busy when create-md / write-dev-uuid runs (drbdmeta exits 20 with
"stdin not a TTY, not waiting for confirmation"). Standby suppresses
resource scheduling during init; crm_standby -v off restores it after DRBD
is up on both nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drbd.service runs drbdadm up all at activation time, but Pacemaker's
OCF drbd agent manages the resource lifecycle (up/down/promote/demote).
When both run simultaneously the device is busy, causing drbdmeta
apply-al to abort with exit 20.
Set wantedBy = [] so the kernel module and config are still present
(via services.drbd.enable = true) but systemd doesn't auto-start the
service. Pacemaker's OCF agent calls drbdadm directly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
At first boot, pacemaker fails with 'Dependency failed' because ipa-activation
fails (IPA not enrolled yet) before corosync/authkey are ready, causing a
systemd boot-ordering race. The service recovers fine when started manually.
Add an explicit 'systemctl start pacemaker' on both nodes immediately after
restarting corosync so cluster-init doesn't time out waiting for it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
The _patch_targetctl runtime dropin was a workaround for the wrong targetctl
binary path in the original iscsi-target.nix (used pkgs.targetcli-fb, but
targetctl is in rtslib-fb's python3 env). The module is now fixed to use
\${python3}/bin/targetctl, so new VM images have the correct ExecStart/ExecStop
baked in and the patch finds nothing — failing with exit 1 and killing the deploy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Adds nextcloud-client, chromium, git, vim, claude-code, fish, sops,
the Nextcloud tray service, SOPS_AGE_KEY_FILE, and Proxmox/PBS desktop
shortcuts to wayne's HM config on GUI hosts. Uses lib.mkDefault on the
EDITOR setting in ipa/client.nix so the GUI layer can override to vim.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both secrets/ha-server-{1,2}.keytab had unresolved conflict markers from
the stash/merge interaction. Keep the upstream version (2 age recipients:
admin + the host's own key) so each host can decrypt its own keytab on boot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
The HA branch merge left conflict markers in the ha-server-{1,2}.keytab
path_regex entries. Keep the upstream side (adds *proxmox-ha-server-{1,2}
as recipients) so each host can decrypt its own IPA keytab on boot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Deploy/init fixes:
- iscsi-target.nix: targetctl binary is in rtslib-fb (python3 env), not
targetcli-fb — fixes ExecStart and ExecStop for the targetctl.service
- deploy.sh: _patch_targetctl() applies runtime dropin to both nodes before
cluster-init so Pacemaker can manage the iSCSI target from first start
- cluster-init.sh: replace crm configure heredoc with cibadmin --replace XML
(pacemaker-4.0 schema: globally-unique in meta_attributes, promoted-max/
promoted-node-max, Promoted role in constraints); force_unmount=true on
xfs-data; DRBD promote timeout 240s
- cluster-config.nix: add crm-fence-peer.sh/crm-unfence-peer.sh handlers;
update fencing comment to reflect resource-only + Pacemaker-aware handler
replacing STONITH during testing phase
- ha-server.nix: add openiscsi to systemPackages for T4 iscsiadm availability
Acceptance test fixes:
- acceptance-tests.sh: fix ((PASS++)) set -e bug → PASS=$((PASS+1));
detect Active/Standby dynamically via drbdadm role (Pacemaker can promote
either node); T4 bash TCP probe instead of iscsiadm; T5 timeout 120s;
T6 echo|sudo tee for root-owned XFS write (bash -c redirect runs as nixos
not sudo — permission denied); use ns cat / ns rm for root-owned reads
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
NixOS's group activation uses plain groupmod, which silently skips the
GID change when SSSD exposes the same GID (50010) via the IPA
docker-access group through NSS. Add an activation script that runs
after the normal 'groups' step and applies groupmod --non-unique so the
local docker group can share GID 50010 with the SSSD-provided IPA group.
If the GID actually changes on a live system the script also restarts
docker.socket + docker.service so the socket is recreated with the new
GID.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cibadmin raw XML fails schema validation under pacemaker-4.0: globally-unique
is not a valid direct <clone> attribute, and master-max/master-node-max are
renamed. Switch to crm configure commands which are schema-version-aware:
- promotable clone with promoted-max/promoted-node-max
- order/colocation constraints using Promoted role (Pacemaker 4.0 naming)
- crm configure handles schema differences automatically
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
clearconfig does not reliably clear kernel LIO configfs state — the kernel
still holds backing-file references, blocking umount. Replace clearconfig
with explicit targetcli delete commands (/iscsi delete, /backstores/fileio
delete) which do release kernel state. Also make DRBD secondary demote
idempotent (skip if already Secondary).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Remove VIP-specific portal binding — the VIP doesn't exist until Pacemaker
assigns it; the default all-IPs portal (::0:3260) is correct for Pacemaker
to manage
- Clear existing LIO targets before re-running targetcli (idempotent on
partial failures)
- Tear down LIO kernel objects after saveconfig so umount succeeds (LIO holds
the backing file open otherwise)
- Guard mount with mountpoint check so re-runs don't fail when already mounted
- Use --replace for cibadmin constraints (idempotent vs --create)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- ha-server.nix: add xfsprogs to systemPackages so mkfs.xfs is on PATH for
root (needed by cluster-init.sh during initial setup)
- cluster-config.nix: create /var/lib/drbd via tmpfiles to silence
lk_bdev_save warnings from drbd-utils
- cluster-init.sh: dynamically find xfsprogs in /nix/store if not on PATH
(fallback for running VMs before xfsprogs is in the system profile)
- cluster-init.sh: fix DRBD metadata check on node2 — broken regex now uses
grep -E for ERE alternation to correctly skip create-md when DRBD is already
set up (previous regex would have triggered create-md on a live secondary)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Root SSH was failing because only the RSA admin key was authorized but the
local dev box only has an ed25519 key. Fix:
- cluster-config.nix: add ed25519 keys to root (same set as nixos user) so
future deployments work without the temp-key workaround
- deploy.sh/acceptance-tests.sh: SSH as nixos user with sudo instead of root@
- cluster-init.sh: HA_USER/HA_KEY env vars + n2_ssh()/n2_scp() helpers so
inter-node SSH works regardless of whether root-to-root is available
- deploy.sh Phase 6: generate temp keypair, authorize on node2, place on node1
for root to use during cluster-init, clean up afterward
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Two independent fixes:
1. hashedPassword = "!" on the IPA user stub
Without any shadow entry, pam_unix returns PAM_AUTHINFO_UNAVAIL
before prompting, so PAM_AUTHTOK is never set. The "!" marker
(account locked, not a real hash) ensures a shadow entry exists so
pam_unix prompts and sets PAM_AUTHTOK — which the subsequent
pam_sss module can then use.
2. pam_sss try_first_pass instead of use_first_pass (login + su)
use_first_pass silently fails when PAM_AUTHTOK is unset.
try_first_pass prompts independently in that case, making IPA
password auth work even if pam_unix returns early for any reason.
Root cause found during incident: wayne (uid 50002) was outside IPA's
auto-SID range (163800000+), so sidgen never assigned him an
ipaNTSecurityIdentifier. Without it, ipadb's handle_authdata step fails
with "Generic error" after SPAKE pre-auth succeeds. Fixed by manually
adding objectClass: ipaNTUserAttrs + ipaNTSecurityIdentifier RID 550002
to wayne's LDAP entry on domain-controller. Any future IPA user with a
manually-assigned uid outside the auto-range needs the same treatment.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pve_check() always uses sudo, so "sudo test -w /nix" passes as root
regardless of whether the SSH user can actually write there. Use a
direct non-sudo SSH command for the writability probe so the check
reflects wayne's own access, not root's.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
When create-proxmox-resource.sh clones the repo to pve1, it stays on
whatever branch was checked out. Add a pre-build phase that detects
branch mismatch and switches the remote clone to the current local
branch before building, so the Proxmox node always builds from the
same commits we're deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
pve1 has a pre-existing /nix store owned by a different UID; wayne's IPA
UID (50002) can't write to it. Add a pre-phase-3 check that uses wayne's
passwordless sudo to chown -R the store before create-proxmox-resource.sh
runs codex-setup.sh on the node.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Add root SSH key + passwordless sudo to ha-server nodes (needed for
deploy script to run cluster-init.sh via SSH as root on node1)
- Fix cluster-init.sh: correct default IPs (228/227/229 per variables.nix),
use \${VAR:-default} for all config so deploy.sh can override via env
- Fix acceptance-tests.sh: same IP corrections, add -i flag to SSH calls,
use \${VAR:-default} pattern
- Fix deploy.sh dry-run bugs: pve_check() always runs SSH for read-only
probes so bridge existence check is accurate; wait_for_ssh skips in
dry-run instead of timing out
- Fix cluster-init invocation: upload script via scp and run via SSH as
root on node1 (was incorrectly trying to run the script locally)
- Fix acceptance-tests invocation: pass IP env vars from deploy.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Run sync-host-keys.sh for proxmox-ha-server-{1,2}: generates SSH host
key pairs in vars/per-machine/, registers age anchors in .sops.yaml,
adds both hosts as recipients for common.yaml, ha-corosync-authkey,
and per-host secrets/keytab files
- Re-encrypt secrets/common.yaml with both new host keys
- Convert all stub secrets to real sops-encrypted files:
secrets/ha-server-{1,2}.yaml (YAML, beszel-token = PLACEHOLDER)
secrets/ha-server-{1,2}.keytab (binary, stub text encrypted)
secrets/ha-corosync-authkey (binary, stub text encrypted)
- Add scripts/ha/deploy.sh: full lifecycle script (bridge setup, VM
creation, DRBD disk + storage NIC attachment, boot wait, cluster-init,
acceptance tests, --destroy)
Bootstrap order (operator runs these before first deploy):
1. bash scripts/ha/deploy.sh # deploys, tests
# Post-deploy secret replacement:
2. sops secrets/ha-server-{1,2}.yaml (set real beszel-token)
3. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
5. Set services.beszel.agent.environment.KEY in host.nix after hub pairing
6. nixos-rebuild switch on both nodes to pick up real secrets
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Add stub keytab files (secrets/ha-server-{1,2}.keytab) so modules/ipa/client.nix
activates for both nodes — IPA security.ipa.enable = true confirmed by eval
- Add .sops.yaml creation rules for both keytabs (admin-only until
sync-host-keys.sh registers the host age keys)
Eval: both proxmox-ha-server-1 and proxmox-ha-server-2 build cleanly
with IPA enabled, beszel agent enabled, corosync authkey + beszel-token
+ ipa-host-keytab all wired via sops-nix.
Bootstrap order before first deploy:
1. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
2. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
3. sops updatekeys secrets/common.yaml
4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
5. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
6. sops secrets/ha-server-1.yaml (set beszel-token)
7. sops secrets/ha-server-2.yaml (set beszel-token)
8. On node1 after first boot: corosync-keygen, then
sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey
9. Set services.beszel.agent.environment.KEY in each host.nix after hub pairing
10. git add/commit the real secrets, then nixos-rebuild switch on both nodes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Documentation fixes:
- README/AGENTS: rename tailscale-exit-node → tailscale-router, add ha-server
build type and proxmox-ha-server-{1,2} host table rows, add baremetal to
platform list, remove references to non-existent flake-target-refactor-spec.md
and remove-sensetive-info-refactor.md
- docs/auto-installer.md: fix lxc-tailscale-exit-node → lxc-tailscale-router,
add pxe-minimal to the flake outputs list
- variables.nix: fix domainControllerIp comment — IPA is the authoritative DNS
at .253 (Pi-hole is gone), not a forwarding intermediary
Code deduplication:
- Extract duplicate SSH host-key preservation activation scripts from
modules/platforms/lxc.nix and modules/platforms/proxmox.nix into a shared
modules/common/preserve-ssh-host-key.nix; both platforms now import it
- Replace 8-line hand-enumerated NFS export lists in server.nix and ha-server.nix
with a mkNfsExports helper that generates exports from vars.nfsShares — adding
a share to variables.nix now propagates to both exporters automatically
Dead code removal:
- modules/common/configuration.nix: remove leftover NixOS skeleton comments
(hardware-configuration import, grub lines) that were never used
- modules/docker/enable-service.nix: remove commented-out listenOptions and
daemon.settings blocks
- hosts/server/host.nix, hosts/nix-cache/host.nix: remove #DOCKER_HOST comments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Create an IPA group "docker-access" (GID 50010) and pin the local
"docker" group to that GID on all Docker hosts. Any IPA user in the
docker-access group automatically gains docker socket access through
SSSD supplementary-group resolution — no per-host docker.members
entry needed.
Specific changes:
- variables.nix: add dockerAccessGid = 50010
- modules/docker/enable-service.nix: lib.mkForce docker GID to
dockerAccessGid, removing the need to name individual IPA users
- modules/build-types/docker.nix: remove direct wayne docker.members
entry (access now comes from IPA group)
- modules/ipa/client.nix: refactor repeated security.* / systemd.*
top-level keys into merged attribute sets (fixes statix W20); add
security.pam.services.lightdm.makeHomeDir so the GUI login path
also creates the home dir on first login
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Grants the domain wayne account docker socket access on docker.sweet.home
without needing sudo, alongside the existing nixos local user membership.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SSSD's dyndns_iface defaults to "*" which registers every interface's IP in
IPA DNS, including all Docker bridge networks (172.x.x.x). This caused
docker.sweet.home to resolve to Docker bridge IPs instead of 192.168.2.225.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On first enrollment /home/wayne doesn't exist until the IPA user's first
login (pam_mkhomedir creates it). home-manager-<user>.service cd's into
the home dir immediately and fails with ENOENT, causing the whole rebuild
activation to return exit code 4.
Add ConditionPathExists so systemd skips the service (condition not met,
no failure) instead. After first login the dir exists and subsequent
rebuilds activate Home Manager normally.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Passing a FQDN like "nixos.sweet.home" instead of the short hostname
"nixos" caused the script to create a double-FQDN IPA host account
(nixos.sweet.home.sweet.home). Add an early check that rejects any
TARGET containing a dot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HM's useUserPackages creates a users.users stub for every configured HM
user, which lands wayne in /etc/passwd. NixOS adds pam_sss.so with the
"localusers" flag to the sudo PAM stack when SSSD is enabled; that flag
causes pam_sss to skip SSSD for any user found in local /etc/passwd,
falling through to pam_unix which has no shadow password for the stub.
Result: sudo auth always fails for the IPA user despite being in wheel.
Use NOPASSWD for the IPA user in sudoers instead. The IPA user already
authenticated to reach a shell (SSH key from IPA or Kerberos), so
re-prompting via a broken PAM path is security theater on a homelab.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
initgroups() uses NSS (groups: files sss) to build the supplemental
group list at login. Adding wheel to the local users.users stub means
the IPA user gets wheel membership from /etc/group, satisfying sudo's
group check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Any enrolled host now automatically gets a Home Manager profile for the
IPA primary user (vars.ipaUser = "wayne"), covering what IPA doesn't:
dotfiles, user-scoped packages (tmux, sshfs), and EDITOR variable.
The home directory is pre-created by systemd-tmpfiles so HM activation
succeeds on steady-state systems before first login; pam_mkhomedir remains
as a fallback for fresh deploys where SSSD hasn't cached the user yet.
A minimal users.users stub satisfies NixOS's assertion requirements
(isNormalUser + group) that arise because home-manager.useUserPackages
creates a users.users entry to install packages to /etc/profiles/per-user/.
The stub is shadowed by SSSD at runtime (security.ipa sets passwd: sss files).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
IPA users (e.g. wayne@) don't get Home Manager so the aliases defined in
aliases.nix were invisible to them. Move Switch-nix, Test-nix, and
buildImage into programs.bash in configuration.nix so every user on every
host gets them via /etc/bashrc. Stub out aliases.nix for future per-user
HM-only additions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The NixOS IPA module adds ConditionPathExists=/etc/krb5.keytab drop-ins
for auth-rpcgss-module.service and rpc-gssd.service via systemd.units.
In LXC containers with keytabs those conditions pass, the services start,
and then fail because auth_rpcgss can't be loaded and rpc_pipefs doesn't
exist in the container namespace.
Use lib.mkForce on our systemd.units text to win the conflict with NixOS's
existing definitions, and include ConditionVirtualization=!container
alongside the ConditionPathExists conditions so the services are skipped
(inactive, not failed) in containers that have a keytab.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
modules/ipa/client.nix is now a self-contained NixOS module (no longer a
parameterized function): it checks builtins.pathExists for
secrets/<hostname>.keytab at eval time and enables itself automatically if
found, making it a no-op for hosts without a keytab.
modules/common/configuration.nix imports it so every host in the flake is
a candidate for IPA enrollment — no per-host wiring needed. Adding a
keytab (via scripts/ipa/create-nixos-ipa-host-account.sh) is now the
only step required to enroll a host.
The module also sets networking.domain and networking.nameservers via
mkDefault when active, so new hosts don't need those set explicitly.
Also:
- Remove explicit IPA imports from hosts/nix-cache and hosts/tailscale-router
- Add secrets/pxe-boot.keytab + creation rule; remove incorrect
secrets/nixos.sweet.home.keytab and its creation rule
- Add .sops.yaml creation rules for all remaining host keytabs
(server, docker, tor-relay, nix-minimal, nixos) so the creation script
can target them without manual .sops.yaml edits
- Fix duplicate tailscale-router.keytab rule and corrupted gui.yaml comment
block in .sops.yaml
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Script fixes:
- Rename HOSTNAME variable to TARGET (shadowed the bash builtin)
- Fix ipa-getkeytab -s to always use IPA_SERVER, not DC_HOST (diverge if
--dc is overridden to a jump host)
- Remove dead REALM variable
- Add EXIT trap to delete the plaintext keytab if the script aborts before
sops encryption completes; cleared after successful encrypt
- Distinguish real ipa host-add failures from "already exists" instead of
swallowing all errors with || true
- Warn explicitly when no platform age keys exist for the target (keytab
would be admin-only and the host couldn't decrypt it at boot)
- Fix sops fallback from pinned nixos-25.11 channel to nixpkgs (uses the
repo's own flake.lock)
- Expand "next steps" output to include networking.domain and nameservers
lines that host.nix requires for IPA membership
Module docs:
- Point to the script as the primary setup path; move manual steps to a
fallback section
- Note that certs/ipa-ca.crt is already committed (no need to re-fetch)
- Document the networking.domain and nameservers requirements in the header
- Add sync-host-keys.sh as explicit step 0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OpenSSH 10.0 tightened AuthorizedKeysCommand security by checking every
path component of the command binary for group/world-write permission.
/nix/store is 1775 (group-writable by nixbld), so sshd silently skips
the command for any binary in the Nix store — causing IPA pubkey auth to
silently fail with no diagnostic.
Fix: copy sss_ssh_authorizedkeys to /usr/local/bin via systemd tmpfiles
(C+ copies the file rather than symlinking, so the path at runtime is
root-owned/755 throughout), and point AuthorizedKeysCommand at the copy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Imports docker/enable-service.nix and pins virtualisation.docker to
docker_29 via a nixpkgs overlay. Fixes indentation on `imports` block
and `programs.direnv.enable` line so nixpkgs-fmt passes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- security.pam.services.sshd.makeHomeDir: IPA users have no pre-created
home directory on the host; without this, sshd opens a session to a
missing directory and resets the connection immediately after auth
- AuthorizedKeysCommand was already added in previous commit
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- modules/ipa/client.nix: add AuthorizedKeysCommand so sshd fetches SSH
public keys from IPA via sss_ssh_authorizedkeys, enabling pubkey login
without per-host authorized_keys files
- hosts/tailscale-router/host.nix: add IPA client module + networking.domain
so SSSD runs and wayne can authenticate on this host
- secrets/tailscale-router.keytab: sops-encrypted keytab for
tailscale-router.sweet.home (generated by create-nixos-ipa-host-account.sh)
- .sops.yaml: creation rule for secrets/tailscale-router.keytab
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NixOS's dnsmasq module defaults resolveLocalQueries to true, which adds
127.0.0.1 to networking.nameservers and binds dnsmasq to listen-address=127.0.0.1.
This made the host route all its own DNS through dnsmasq, which had
no-resolv=true and no upstream for anything outside the tailnet domain —
so every non-tailscale DNS query from the host itself (including SSSD
resolving the IPA server FQDN after the IPA client module was added) failed.
Setting resolveLocalQueries=false limits dnsmasq to its intended role: a
forwarding proxy reachable on the LAN interface for IPA's conditional
forwarder. The host uses domainControllerIp directly (already set in
networking.nameservers in host.nix).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ipa-getkeytab runs as root via sudo so the temp file is root-owned;
scp as wayne gets Permission denied. Pipe through `sudo cat` over SSH
instead, which reads as root but writes locally as the invoking user.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single command to enroll a NixOS host in FreeIPA and produce a
sops-encrypted keytab at secrets/<hostname>.keytab:
- Adds the .sops.yaml creation rule automatically (with all registered
platform-variant age keys as recipients)
- SSHes to the domain controller to run ipa host-add + ipa-getkeytab
- Refreshes the admin Kerberos ticket via `ssh -t ... kinit admin` if
missing or expired, so no manual kinit step is needed
- SCPs the keytab and encrypts it in-place with sops (file must be at
secrets/<hostname>.keytab before encryption so the path-based creation
rule matches — the common failure point when doing this manually)
Also adds HOME_DOMAIN and IPA_SERVER to scripts/env.sh, matching
variables.nix's homeDomain/ipaServer (same manual-sync pattern as
NIX_CACHE_HOST/LAN_DOMAIN).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sops matches creation rules against the input file path, so encrypting
/tmp/<host>.keytab directly with stdout redirect fails to find the rule.
Copy to secrets/ first, then use -i to encrypt in-place.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ipa-getkeytab confirmed ipa.sweet.home doesn't respond to LDAP;
domain-controller.sweet.home is the actual IPA server FQDN.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds modules/ipa/client.nix — a parameterized module that joins a NixOS host
to the sweet.home FreeIPA domain without ipa-client-install. It configures
security.ipa (SSSD, Kerberos, PAM, NSSwitch) and places a pre-provisioned host
keytab via sops-nix binary secret so enrollment is fully reproducible from the
flake.
- variables.nix: adds ipaServer (FQDN of the FreeIPA KDC; security.ipa.server
requires a hostname, not an IP, for Kerberos/TLS)
- certs/ipa-ca.crt: placeholder for the IPA CA public certificate (operator
replaces with: curl http://<ipa-server>/ipa/config/ca.crt)
- secrets/nix-cache.keytab: placeholder binary sops file (operator replaces
with the encrypted keytab after ipa host-add + ipa-getkeytab)
- .sops.yaml: adds creation rule for secrets/nix-cache.keytab (same recipients
as secrets/nix-cache.yaml)
- hosts/nix-cache/host.nix: imports the IPA client module; adds
networking.domain so the host's FQDN resolves correctly
Module header documents the three operator steps needed per host before deploy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
zpool create -f was called if `zpool import -d /dev/disk/by-id` failed,
which could happen due to a race with systemd-udev-settle. The disk
would then be visible by the time zpool create ran, silently destroying
all data on an otherwise-intact pool.
Fix: locate the data disk first, retry the import directly against it
as a fallback, then check zdb -l for existing ZFS label metadata before
concluding the disk is blank. Remove -f so zpool create refuses rather
than overwrites if a pool is present.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
IPA refuses to create a forward zone for ts.net because it's a real
public TLD with DNSimple nameservers. The forward zone must use the
tailnet-specific subdomain (vars.tailnetDomain, e.g. tail13f623.ts.net)
instead. Update dnsmasq server selector and comments to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
FreeIPA (the new authoritative DNS) cannot reach 100.100.100.100
(Tailscale's internal MagicDNS resolver) directly because the DC is not
a Tailscale node. The tailscale-router IS a Tailscale node and can
reach 100.100.100.100 via tailscale0, so it now runs a dnsmasq
instance on its LAN interface that forwards all ts.net queries to
Tailscale's resolver.
After deploying this host, configure FreeIPA with:
kinit admin
ipa dnsforwardzone-add ts.net \
--forwarder=192.168.2.222 \
--forward-policy=only
This replaces Pi-hole's conditional forwarder for ts.net and restores
resolution of Tailscale MagicDNS names (e.g. raspberrypi.tail13f623.ts.net)
for all LAN hosts using FreeIPA as their DNS server.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
Adds static IP configuration to every NixOS host in the flake that
has a fixed LAN address, and centralises all network primitives
(IPs, gateway, prefix length, interface names) in variables.nix so
there is one place to update if any of them change.
variables.nix additions:
- lanGateway / lanPrefixLength — LAN gateway and /24 prefix, replacing
every hardcoded 192.168.2.254 / 24 across host files
- lxcLanInterface / vmLanInterface / vmStorageInterface — NIC names for
LXC containers (eth0), Proxmox VMs (ens18), and the HA storage NIC
(ens19), used as attribute keys so changing the name is a one-line edit
- haStoragePrefixLength — /29 for the storage subnet, mirrors haStorageCidr
- Per-host IP variables: nixCacheIp (.224), tailscaleRouterIp (.222),
torRelayIp (.221), serverIp (.226), dockerIp (.225)
host.nix changes:
- tailscale-router, tor-relay, nix-cache, pxe-boot: useDHCP = false,
static address on eth0 (lxcLanInterface), struct-form defaultGateway
(required when using systemd-networkd which LXC containers use)
- server, docker: useDHCP = false, static address on ens18 (vmLanInterface),
struct-form defaultGateway (works for both scripted networking and networkd)
- ha-server-1, ha-server-2: replace hardcoded 192.168.2.254 / 24 / 29
with the new variables; no functional change for these hosts
modules/build-types/pxe-boot.nix:
- Domain-controller kickstart template: replace hardcoded 192.168.2.138
and 192.168.2.254 with vars.domainControllerIp / vars.lanGateway /
vars.lanPrefixLength / vars.homeDomain so the template stays correct
if the DC IP or domain is ever changed again
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
vars.pxeServerIp was already set to the post-renumber target (.223) but the
pxe-boot CT is still at .247, so dnsmasq was advertising .223 as the TFTP server
and the VM couldn't reach it. Update to .247 so PXE boot works now; the comment
reminds us to flip it back to .223 when Stage 5 step 6 renumbers the CT.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the proxy DHCP handshake, the UEFI client queries the PXE boot
server on UDP port 4011 to get the actual boot filename. This port was
not open in the NixOS firewall, so all four PXE discovery attempts from
the test VM silently timed out. dnsmasq was bound to 0.0.0.0:4011 but
never received the packets.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dnsmasq 2.93 does not send proxy DHCP offers from dhcp-boot alone in proxy
mode — pxe-service is required to trigger proxy offers for the initial boot.
Replace the dhcp-boot-based arch detection with pxe-service entries for
arch 0 (BIOS), 7 (EFI BC / OVMF), and 9 (EFI x86-64), then use dhcp-boot
only for the iPXE second-boot HTTP chain (tag:ipxe clients). Add log-dhcp
for post-deploy verification.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dnsmasq binds port 53 by default even when only proxy DHCP is needed.
systemd-resolved already owns port 53 on the pxe-boot container, so
dnsmasq crashes at startup. Setting port=0 disables the DNS listener
entirely — pxe-boot only needs the proxy DHCP path for iPXE chainloading.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dnsmasq config was placed inside systemd.services{} block, making Nix
interpret it as systemd.services.dnsmasq.settings (which doesn't exist).
Move to top-level services.dnsmasq and add UDP 67 to firewall for DHCP
proxy mode.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds proxmox-ha-server-1 and proxmox-ha-server-2 as real mkTarget entries
alongside the existing proxmox-server, backed by a new ha-server build type.
New modules
modules/ha/cluster-config.nix — DRBD resource + corosync nodelist sourced
from vars (haServer1Host/Ip, haServer2Host/Ip); resource-only fencing for
production STONITH; HA port firewall rules for DRBD, iSCSI, Corosync, pcsd
modules/build-types/ha-server.nix — imports pacemaker-stack + iscsi-target
+ cluster-config + beszel; NFS exports from vars.haStorageRoot (XFS-over-DRBD
mount); nfs-server.service.wantedBy force-cleared so Pacemaker controls
start/stop on the Active node only
New hosts
hosts/ha-server-{1,2}/host.nix — static IP from vars, unique hostId; sops
secrets (beszel, corosync authkey) are TODOs pending sync-host-keys.sh
variables.nix
haServer1/2Host, haServer1/2Ip, haServerVip, haStorageRoot, haIscsiIqn
ports.haServerDrbd/Iscsi/Corosync{1,2,Crypto}/PacemakerRemoted/Pcsd
scripts/ha/ (migrated + updated from test-lab/ha/)
cluster-init.sh — generates corosync authkey, initialises DRBD/XFS/iSCSI,
creates NFS dataset dirs, configures Pacemaker with DRBD + XFS + iSCSI
+ nfs-server + VIP; STONITH disabled initially (enable separately)
cluster-enable-stonith.sh — enables fence_pve_ssh STONITH after key deploy
fence-pve-ssh.py — Proxmox SSH fence agent (node names updated to ha-server-1/2)
acceptance-tests.sh — T1–T7 production acceptance tests
test-lab/ha/ removed — all Nix config moved to modules/ha/ and
modules/build-types/; scripts moved to scripts/ha/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
All 7 acceptance tests pass on live NixOS 25.11 VMs (VMIDs 200/201 on
pve1). Failover completes in ~5 s with data integrity verified.
modules/ha/pacemaker-stack.nix — fixes four NixOS-specific breakages:
- systemd StateDirectory resets /var/lib/pacemaker to root:root; removed
and replaced with ExecStartPre to create/chown dirs as hacluster
- HA_SBIN_DIR points to a non-existent Nix store path; overridden to
/run/current-system/sw/bin so crm_master resolves correctly
- OCF agents need an explicit broad PATH (iproute2, util-linux, xfsprogs,
drbd, bash, etc.) — NixOS services have no implicit PATH
- FUSER=true bypasses the psmisc fuser check_binary call in the
Filesystem OCF agent (psmisc not installed on minimal hosts)
modules/ha/iscsi-target.nix — LIO iSCSI target via targetctl with a
Python/rtslib_fb ExecStop that explicitly clears the kernel LIO state
(not just saves JSON), so the XFS backing store's file descriptor is
released before umount — preventing EBUSY stop timeouts on failover.
Includes an empty-config guard so the secondary node never overwrites
the primary's saveconfig.json with an empty one.
test-lab/ha/common.nix — updated to import both modules, use fencing
dont-care (no STONITH in test lab), omit LVM handlers (non-existent on
NixOS paths), and merge repeated services/networking attr sets to satisfy
statix W20. test-lab/ha/acceptance-tests.sh — final v4 with crm_standby
fix (pacemaker 3.x API).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Adds an unattended install option to the PXE boot menu that installs
Rocky Linux 9 and configures FreeIPA on the domain-controller.sweet.home
host without any operator interaction after selecting the menu entry.
How it works:
- fetch-rocky-pxeboot.service downloads the Rocky 9 Anaconda pxeboot
kernel and initrd from the Rocky mirror on first pxe-boot deploy
(idempotent, same pattern as fetch-debian-netboot)
- rocky-freeipa.ipxe boots Anaconda with inst.ks pointing at the
hosted Kickstart and net.ifnames=0 biosdevname=0 for stable eth0
- rocky-freeipa.ks (generated, includes vars.adminSshKey) performs:
- Minimal Rocky 9 install with ipa-server + ipa-server-dns
- Static IP 192.168.2.138 via NM connection file written in %post
- /etc/hosts fixed for FreeIPA FQDN requirement
- Random DM + admin passwords generated and saved to
/root/ipa-credentials.txt (chmod 600, never hardcoded)
- freeipa-first-boot.service oneshot enabled to run
ipa-server-install on the first real boot (~20 min)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Jbvxx4xbHVcx1NkK3vtmK
Add nixos@nixos session key so the Claude Code session can SSH into
test VMs directly. Also enable services.qemuGuest.enable so
qm guest exec works as a fallback for key injection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SOPS_AGE_KEY_FILE was set in hosts/nixos/home.nix sessionVariables with a
literal ~ that Home Manager injects as-is into the environment. In bash,
tilde expansion does not happen inside double-quoted variable references, so
DEFAULT_SOPS_AGE_KEY_FILE resolved to ~/... literally and the -s file-existence
check in ensure_admin_decrypt_key silently failed. The script then generated
a brand-new age key (to ~/... relative to the repo root) while the real admin
key at ~/.config/sops/age/keys.txt went untouched -- making it appear the key
was lost when it was actually still intact.
Fix the home.nix root cause by using config.home.homeDirectory so the path
is fully resolved. Add tilde expansion in ensure_admin_decrypt_key as a
belt-and-suspenders guard for any caller whose environment has the same issue.
Also replace the auto-generate-a-new-key fallback with a hard failure: auto-
generating a new admin key is never useful (it cannot decrypt existing secrets)
and created serious confusion about whether the original key was lost.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the hardcoded hostname check (docker, pxe-boot) with a check
on config.fileSystems: any lxc-* host whose NixOS config declares an
NFS fileSystem entry is automatically made privileged. The script
already reads proxmoxLXC.privileged dynamically via
flake_target_lxc_privileged, so no logic change is needed there —
only the comment is updated to describe the new derivation.
Result: lxc-docker and lxc-pxe-boot (the two with NFS mounts) evaluate
as privileged=true; lxc-nix-cache, lxc-minimal, lxc-server,
lxc-tailscale-router, lxc-tor-relay evaluate as privileged=false.
Any future lxc-* host that declares an NFS mount gets the correct
privilege level for free without a separate manual edit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The kernel's NFS client (FS_USERNS_MOUNT not set) rejects NFS mounts
from inside any unprivileged container's user namespace with EPERM —
AppArmor's mount=nfs feature only whitelists the AppArmor layer; the
VFS-level rejection happens before AppArmor is consulted.
lxc-pxe-boot mounts server.sweet.home:/tank/pxe-boot/images at
/mnt/pxe-images so nginx can serve large ISOs without filling the
container's root disk. Same pattern as lxc-docker (already privileged
for the same reason since it also mounts several NFS shares).
Operator action required: VMID 103 must be recreated as a privileged
container (the UID mapping on disk differs between privileged and
unprivileged; changing it in-place with pct set is unsafe). Rebuild
the tarball and use create-proxmox-resource.sh to replace it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Correct the NFS path from /tank/proxmox/pxe-images to /tank/pxe-boot/images
(matches variables.nix's proxmoxPxeImages.subpath)
- Clarify that LXC uses NFSv3+nolock while VM uses NFSv4.2+automount
- Add explicit note that lxc-pxe-boot needs features: nesting=1,mount=nfs and
why: nesting=1 is required by systemd 260+ for userns/credential isolation
(AppArmor denies userns_create without it), mount=nfs for NFSv3 access.
pct set replaces the whole features string — include both or the container
will fail to boot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Filter non-attrset values from nfsShares in server.nix so the
poolDatasets loop skips the new `options` string entry
- Fix typo proxomoxLxcImages → proxmoxLxcImages in server.nix exports
- Rename proxmoxPxeImages → pxebootImages in mount-pxe-images.nix and
pxe-boot.nix to match the actual key in variables.nix
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Proxmox LXC containers block the sunrpc filesystem (rpc_pipefs) via AppArmor
unless the container has `features: mount=nfs` set. NFSv4 requires rpc_pipefs
for client state management, so the mount fails outright in a default LXC.
Two fixes for the LXC case (config.boot.isContainer):
- Switch from nfsvers=4.2 to nfsvers=3,proto=tcp,nolock,nofail: NFSv3 doesn't
need rpc_pipefs at the protocol level, and nofail keeps boot clean if the
NFS server is unreachable.
- Add ConditionVirtualization=!container to var-lib-nfs-rpc_pipefs.mount via
systemd drop-in: NixOS pulls this unit into nfs-client.target for any NFS
fileSystems entry. With the condition, systemd skips (not fails) the unit in
containers, keeping nfs-client.target green and activation reporting clean.
Proxmox VM hosts (not isContainer) continue to use nfsvers=4.2 with
x-systemd.automount unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
showmount and the NFSv3 mount protocol need mountd reachable after querying
portmapper on 111; the server was only opening TCP 111 and 2049, causing
clients (e.g. Proxmox GUI NFS storage scan) to time out connecting to
mountd on 20048. Also adds UDP for all three ports — portmapper, nfsd, and
mountd all use both protocols.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds modules/pxe-boot/mount-pxe-images.nix, which mounts
server.sweet.home:/tank/proxmox/pxe-images at /mnt/pxe-images via NFSv4.2
(x-systemd.automount on Proxmox VMs, nofail on LXC containers — same pattern
as docker/mount-data.nix). The pxe-boot build-type now imports this module and
replaces the previous local /srv/pxe/http/images directory rule with an L+
symlink pointing to /mnt/pxe-images, so large images (ISOs, disk images) live
on the NFS share rather than the host's own root disk.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a `fetch-debian-netboot.service` oneshot that downloads the Debian
bookworm netboot kernel and initrd from deb.debian.org on first boot,
stages them under /srv/pxe/http/debian/, and serves them via a generated
debian.ipxe chain script. The service is idempotent — it skips the
download if both files are already present.
Also merges the previously split systemd.tmpfiles.rules and
systemd.services blocks into a single systemd = { ... } attrset to
satisfy statix's repeated-keys lint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wires beszel-agent into all tailscale-router variants (lxc/linode/proxmox)
by importing enable-agent.nix in the build type and host-token.nix in the
host file. Adds the sops creation rule for secrets/tailscale-router.yaml
(all three platform variants as recipients). The secrets file must be
created manually before deploying — see instructions in PR.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
extraCommands runs after nixos-nat-post is deleted but before it is
re-created, so -A nixos-nat-post silently fails every time. POSTROUTING
is a built-in chain that always exists; target it directly instead.
The -C idempotency check prevents duplicate rules on firewall reloads.
Drop networking.nat.enable -- it was only needed for the sub-chain that
turned out to be the wrong target.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
networking.nat.externalInterface without internalInterfaces creates the
nixos-nat-post chain but inserts no MASQUERADE rule into it — confirmed
by inspecting the live firewall-start script on the deployed host.
Add the rule explicitly via firewall.extraCommands targeting nixos-nat-post,
scoped to LAN source traffic (vars.lanCidr) going out tailscale0.
extraStopCommands removes it on firewall stop.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without SNAT on tailscale0, Tailscale drops forwarded packets from LAN
source IPs (192.168.2.x) because they are not recognised Tailscale
addresses. With networking.nat.externalInterface = "tailscale0", all
traffic leaving through the Tailscale tunnel is masqueraded to the
router's own Tailscale IP (100.x.x.x), making it indistinguishable from
locally-originated traffic. Conntrack handles the return path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a oneshot systemd service that sets ethtool rx-udp-gro-forwarding on
and rx-gro-list off on the default-route interface at boot, silencing
Tailscale's warning about suboptimal UDP GRO forwarding on subnet routers.
Interface is discovered dynamically via `ip route get` so it works on all
platforms regardless of NIC naming.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--pre-format-files placed the key on the QEMU builder VM's rootfs, not the
target disk. nixos-install chroots into the target and runs sshd-keygen, which
found no key in the chroot and generated a fresh (unregistered) one. sops then
could not decrypt on first boot because the key didn't match .sops.yaml, leaving
both root and nixos with '!' in /etc/shadow even after mutableUsers = false was
set (hashedPasswordFile pointed to paths sops never wrote).
Fix modules/platforms/proxmox.nix to embed the clan SSH host key in
environment.etc via NIXOS_HOST_KEYS_DIR at eval time -- the same pattern
lxc.nix uses. nixos-install's own activation places the key on the target disk,
sshd-keygen finds it already present and skips generation, and sops decrypts
correctly on first boot. Includes the same preserveSshHostKey/restoreSshHostKey
activation scripts as lxc.nix so subsequent nixos-rebuild switch calls (without
NIXOS_HOST_KEYS_DIR) don't remove the key as "obsolete" from environment.etc.
Update create-proxmox-resource.sh: switch VM builds from
./result-<target> --pre-format-files ... --build-memory 2048
to
NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build --impure ... diskoImagesScript
./result-<target> --build-memory 2048
matching the LXC build path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
Three bugs combined to leave every VM build with a shell but no boot disk:
1. The remote build script moved the raw image to /var/lib/vz/import/ before
qm importdisk could use it. If the mv failed (cross-filesystem copy, sudo
path, or any other reason) the remote script exited non-zero -- but the
local script's set -e handling of the SSH heredoc was inconsistent, so
qm create sometimes ran anyway, leaving a diskless VM shell.
Fix: skip the mv entirely. The diskoImagesScript writes <hostname>.raw into
its CWD (the remote repo dir, $out = $PWD at invocation). Import directly
from that path; clean it up after a successful import.
2. The qm importdisk output regex expected "Successfully imported disk as '...'"
but current Proxmox emits "unusedN: successfully imported disk '...'"
(lowercase, no "as"). The grep returned no match and exited 1.
3. The disk_id assignment used $(... | grep ...) without || true inside the
substitution. With set -euo pipefail, a non-zero grep exit aborts the
script before the fallback could run -- so the VM was always left with an
unattached unused0 disk.
Fix: update the primary regex to match the actual PVE format; add || true
inside the substitution so set -e never fires on a grep miss; add a qm
config fallback (scan for unusedN: lines) that works regardless of PVE
output format changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
When a proxmox-* disk image is built, activation runs during the image
build without a valid sops age key (the SSH host key doesn't exist yet),
so root and nixos land in /etc/shadow with locked '!' entries. With the
default mutableUsers = true, update-users-groups.pl preserves existing
shadow entries for accounts that already exist, so hashedPasswordFile is
silently ignored on every subsequent boot — passwords are never fixed.
Setting mutableUsers = false forces update-users-groups.pl to apply
hashedPasswordFile unconditionally on every activation. On first real
boot the sops-decrypted hash is now written regardless of whether the
account already existed in shadow from the image build.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
qm importdisk in QEMU 11.x outputs lowercase "successfully imported disk
as '...'" rather than the capitalised form the original grep expected.
The case mismatch made disk_id always empty, which caused the script to
exit 1 after qm create had already run -- leaving the VM with only an
EFI disk, no scsi0, and boot order still set to net0.
Fix by adding -i (case-insensitive) to the grep. Both the old capitalised
format (where the disk id had an "unused0:" prefix inside the quotes) and
the new lowercase format are handled correctly: the sed strip of unused0:
is preserved for backward compatibility, and the regex result is identical
either way.
Also add an early warning when --disk-size is passed for --type vm: the
flag is LXC-only for create mode and was silently ignored, leaving users
expecting a different size than the proxmoxImageSize in variables.nix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the clan vars migration all keys are in vars/per-machine/, not
host-keys/. Update:
- File header: "existing clan var is never overwritten" (not host-keys/ file)
- Header --remove/--regenerate description: mention clan vars as primary
- usage() --remove, --regenerate-all-keys, --dry-run text
- cmd_remove/cmd_regenerate_all empty-guard messages
- README.md vars/per-machine/ row: "all deployed hosts" (not "LXC hosts")
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
locally_managed_hosts() only scanned host-keys/ (now empty for all
current targets), so --remove and --regenerate-all-keys silently did
nothing. Fix:
- locally_managed_hosts(): also yields targets from
vars/per-machine/*/openssh/ssh_host_ed25519_key/secret, deduped
- cmd_remove: shows [clan-vars] or [host-keys/] label per entry;
deletes vars/per-machine/<target>/openssh/ in addition to host-keys/
- cmd_regenerate_all: same -- removes clan vars dirs before regenerating
Also update CLAUDE.md and README.md to reflect that all flake targets
now use clan vars (not just lxc-*); host-keys/ is only for the
auto-installer's own pre-seeding path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Adds vars/per-machine/<target>/openssh/ for every flake target except
lxc-tor-relay and lxc-nix-cache (already committed). 18 targets recovered
from pve1 host-keys/ backup; lxc-gui and proxmox-minimal have no prior
live deployment and no backup key, so fresh ed25519 keys were generated —
their .sops.yaml anchors were updated to match.
All secrets are admin-only encrypted (matching clan_generate_ssh_key
convention). Age fingerprints verified against .sops.yaml anchors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
The previous Phase 4 commit had sed-mangled fingerprints for these two
targets (old and new fingerprints concatenated into one line). The correct
new fingerprints are:
- proxmox-minimal: age19m0m7vdfg... (freshly generated, no prior key on pve1)
- lxc-gui: age1rrxqea6q6... (freshly generated, no prior key on pve1)
Re-run sops updatekeys on common.yaml and gui.yaml to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Remove four stale age key anchors from .sops.yaml that correspond to
non-lxc build-type variants that were never deployed (or are now
superseded by their lxc-* counterparts):
&docker → superseded by &lxc-docker (active running host)
&server → superseded by &lxc-server (active running host)
&nix-cache → superseded by &lxc-nix-cache (active running host)
&nix-minimal → superseded by &lxc-minimal (active running host)
Also remove the secrets/docker.yaml creation_rules block entirely since
that file does not exist.
Re-encrypt secrets/common.yaml, secrets/nix-cache.yaml, and
secrets/server.yaml with sops updatekeys to drop the stale recipients.
The four removed keys can no longer decrypt these files.
Update README.md and CLAUDE.md to clarify that deployed lxc-* hosts
now use clan vars (vars/per-machine/<target>/openssh/) rather than the
gitignored host-keys/ directory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
All actively deployed lxc-* hosts now have clan vars. Remove the legacy
scp -pr host-keys/ fallback in sync_remote_host_keys(): instead of
silently copying the gitignored directory, error clearly if no clan var
exists for the target and tell the operator how to generate one.
Also extend the uncommitted-changes check to cover vars/per-machine/ in
addition to .sops.yaml and secrets/, since clan vars must be committed
before the remote build git-pulls them.
Update the script header and sync_remote_host_keys comment to reflect
the new clan-only key flow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Encrypted private key matches the running container's key and the
vars.nixCacheHostKey in variables.nix (no rotation). Age fingerprint
age1ufg390... matches the &lxc-nix-cache anchor in .sops.yaml.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
nixos-lxc-sops-reinstall.service called switch-to-configuration test at
sysinit.target time (DefaultDependencies=false), before D-Bus was up.
D-Bus is required to restart systemd targets after activation scripts
run. The service reported failure on every boot (exit 1: "Failed to open
dbus connection") even though secrets were correctly installed, because
the D-Bus call happens after activation scripts complete.
Move the service to network.target so basic.target (which includes
dbus-broker.service) runs first. Also drop DefaultDependencies=false so
systemd auto-adds After=basic.target. Add SuccessExitStatus=11 to handle
the edge case where switch-to-configuration holds the lock during a
concurrent rebuild (exit 11 = "Could not acquire lock" -- the rebuild's
own activation already installed the secrets, so treating it as success
is correct).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
The key was generated in a prior session but not committed — the clan vars
files existed only in that session's working tree. Recovered the original
private key from pve1's host-keys/ backup (fingerprint age16kqf... matches
the &lxc-tor-relay anchor already in .sops.yaml), re-encrypted for admin
age key only, and stored in the canonical clan vars layout.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Two bugs prevented nixos-rebuild switch from working on lxc-* hosts after
first boot, both confirmed live on a deployed lxc-tor-relay container:
1. Ordering bug: preserveSshHostKey had no explicit deps, so the topological
sort placed it at position 7 — after etc at position 5. By the time it
tried to save the SSH key, etc had already removed it as "obsolete"
(absent from the current generation's environment.etc when built without
NIXOS_HOST_KEYS_DIR). Consolidate all four system.activationScripts entries
into one block and add etc = { deps = ["preserveSshHostKey"]; } and
setupSecrets = { deps = ["restoreSshHostKey"]; } to enforce the correct
save→etc→restore→sops chain.
2. No boot-time secrets: /run/secrets is a tmpfs cleared on every reboot, and
sops-nix does NOT generate a boot-time service in this configuration
(confirmed live: no sops-nix.service in systemctl list-unit-files).
Add nixos-lxc-sops-reinstall.service, modelled after sops-nix's own service
placement (wantedBy/before sysinit.target, DefaultDependencies=false), so
secrets are reinstalled before basic.target on every non-first boot.
ConditionPathExists skips it on first boot; nixos-lxc-first-boot-activate
handles that case.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Replaces the gitignored host-keys/ directory with clan vars as the
authoritative storage for SSH host keys. Keys are now generated as
sops-binary-encrypted clan var files (admin-key only) and checked into
vars/per-machine/<target>/openssh/, eliminating the plaintext private
key that previously had to live outside the repo.
Changes:
- modules/clan/ssh-host-key.nix: clan vars generator for the ed25519
SSH host key pair (neededFor="activation" — not mapped to sops.secrets,
delivered via tarball baking for LXC or --pre-format-files for VMs)
- flake.nix: add clanCore module + required settings to every mkTarget;
deduplicate bundled disko/sops-nix via follows; all 27 hosts eval clean
- flake.lock: updated to reflect the new follows constraints
- scripts/lib/clan-vars.sh: new helper library with
clan_ssh_key_exists / clan_ssh_pubkey_path / clan_decrypt_ssh_key /
clan_generate_ssh_key for use by the provisioning and sync scripts
- scripts/secrets/sync-host-keys.sh: queue_host_sync() now checks clan
vars first; generates via clan_generate_ssh_key if no key exists;
derives age fingerprint from clan pub key for .sops.yaml registration
- scripts/proxmox/create-proxmox-resource.sh: key management simplified
(sync-host-keys.sh now generates the key if missing, so the inline
prepare-host-key.sh call is gone); sync_remote_host_keys() decrypts
the clan key into a temp dir and scps just the two files to the node
when a clan key exists, falling back to the old host-keys/ scp for
any remaining legacy entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Introduces clan-core pinned to its 26.05 release alongside the existing nixpkgs
26.05 input. No host configuration is changed — this is a pure dependency
addition so Phase 2 (per-host vars/secret management migration) has the input
available without a separate flake.lock bump.
clan-core.inputs.nixpkgs.follows = "nixpkgs" keeps a single nixpkgs closure.
sops-nix remains as a flake input; vars layers on top of it rather than
replacing it (clan's sops storage backend still needs sops-nix).
All hosts evaluate cleanly (codex-maintenance.sh --full-check equivalent
triggered by the flake.nix change).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three ordering-related fixes to the Proxmox provisioning flow:
1. prepare-host-key.sh: make idempotent -- if the key already exists, print
a note and exit 0 instead of erroring. The caller (create-proxmox-resource.sh)
already guards standalone calls, but the script itself should be safe to
run directly on a host that was already keyed.
2. create-proxmox-resource.sh: after sync-host-keys.sh updates .sops.yaml /
secrets/, detect uncommitted changes and block with a prompt until the
operator confirms they've committed and pushed. The PVE node's git pull
only picks up committed+pushed state; without this gate, a new host's sops
recipient is missing from the secrets files the image build uses, so the
host can't decrypt secrets on first boot.
3. create-proxmox-resource.sh: add an explicit existence check for the host
key in both the LXC and VM remote build heredocs, before it's passed as
--pre-format-files / NIXOS_HOST_KEYS_DIR input. Gives a clear error
pointing at sync-host-keys.sh instead of a raw `cp: cannot stat` from
disko deep in the build.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The disko images script does `cd "$TMPDIR"` before parsing its arguments,
so relative paths passed to --pre-format-files resolve against the temp
dir instead of the repo root. Use $(pwd) to capture the absolute repo
path before the disko script changes directory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Complete the rename so all identifiers match networking.hostName:
- flake.nix: attribute names and buildType strings
(linode/proxmox/lxc-tailscale-subnet-router → *-tailscale-router)
- modules/build-types/tailscale-subnet-router.nix → tailscale-router.nix
- .sops.yaml: anchor and alias names (age keys unchanged, no re-encrypt needed)
- host-keys/: local gitignored key files renamed (not committed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If host-keys/<target>_ssh_host_ed25519_key doesn't exist, run
prepare-host-key.sh to generate it before sync-host-keys.sh runs.
Prevents sync-host-keys.sh from hitting its SKIP/exit-1 path (anchor
in .sops.yaml but no local key) and the downstream disko build failure
when --pre-format-files can't find the key file.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The directory name was tailscale-subnet-router but networking.hostName
was already tailscale-router, causing a mismatch that confused scripts
comparing directory paths against actual hostnames. Flake attribute
names (linode/proxmox/lxc-tailscale-subnet-router) and .sops.yaml
anchors are unchanged — they describe the build type, not the hostname.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
systemd-resolved only uses LLMNR for single-label hostnames, never DNS —
same issue mount-data.nix already documented and fixed for NFS by switching
to server.sweet.home. Change the substituter URL, SSH knownHosts, and
remote-builder hostName from bare "nix-cache" to "nix-cache.sweet.home",
and update nginx's virtualHost to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
read exits non-zero when stdin is not a terminal (set -e killed the
script silently). Catch that and emit a clear error pointing to the
SUDO_PASS environment variable rather than crashing with no output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/dev/tty exists as a device node even without a controlling terminal,
so -r/-w tests pass but opening it fails. Plain 'read -r -s' from stdin
is enough: works interactively from a real terminal, and from a non-tty
context the caller should set SUDO_PASS in the environment instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Environments without a controlling terminal (containers, CI agents)
don't have /dev/tty. Try it first for the sudo password prompt, fall
back to plain stdin so the script works in both contexts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Instead of ssh -t (requires PTY on both sides), prompt for the sudo
password once at startup and pipe it to each remote invocation via
sudo -S. This works from any context -- interactive terminal, background
agent, or script -- with no PTY needed on either end.
Also accepts SUDO_PASS from the environment for non-interactive callers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ssh -t won't allocate a PTY when its own stdin is redirected (by a
heredoc). Replaced the heredoc-fed 'sudo bash -s' with commands passed
as an argument string so stdin stays free and -t can properly allocate
a PTY for the sudo password prompt.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New script: scripts/secrets/push-host-keys.sh
- Pushes newly-generated SSH host keys from host-keys/ to already-running
NixOS hosts after sync-host-keys.sh --regenerate-all-keys.
- Before pushing any key, checks that .sops.yaml and secrets/*.yaml are
committed and pushed to the remote Gitea flake (hosts rebuild from there,
so recipient changes must land first); offers to auto-commit/push if not.
- Reads /etc/flake-target from each host to confirm which key to install,
handling the case where multiple flake targets share a hostname.
- Deduplicates by hostname in --all mode; skips hand-registered targets
that have no host-keys/ entry.
- --dry-run, --skip-git-check, SSH_USER override (default: nixos).
sync-host-keys.sh --regenerate-all-keys:
- Updated pre-confirmation warning to distinguish already-running hosts
(need push-host-keys.sh) from not-yet-deployed hosts (need installer
image rebuild).
- Added next-steps block after regeneration completes pointing to
push-host-keys.sh --all.
recover-hosts.sh:
- Header and SSH host key mismatch warn now cross-reference
push-host-keys.sh as the proactive (pre-drift) alternative.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The host was always intended as a LAN subnet router (--advertise-routes),
not a full exit node (--advertise-exit-node). Rename every trace of
"exit-node" to "subnet-router" and remove the --advertise-exit-node flag
from extraSetFlags; the operator supplies --advertise-routes at first
tailscale up and Tailscale persists it in state across reboots.
Routing sysctls (useRoutingFeatures = "server"), openFirewall, and
trustedInterfaces = ["tailscale0"] are still required for subnet routing
to work, so the module is kept — just correctly named.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TF2dsuKZAiyZWQ1D7CuHJm
- Add hosts/tor-relay/host.nix import of host-token.nix so the agent
gets its TOKEN from a sops-managed environment file
- Add secrets/tor-relay.yaml (encrypted beszel token for this host)
- Add creation_rules entry for secrets/tor-relay.yaml in .sops.yaml
- Update &lxc-tor-relay age key to the host's actual current key
(old key was from a prior LXC incarnation; new key extracted from
Switch-nix output: age1gl5ujmhd2pe37...)
- Re-encrypt secrets/common.yaml via sops updatekeys to swap in the
new key, so the host can decrypt its password hash on next boot
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NixOS's etc activation removes files that were in a previous generation's
environment.etc but absent from the current one -- even real copies, not
only symlinks. LXC tarballs bake the host key into environment.etc (via
NIXOS_HOST_KEYS_DIR), but every subsequent nixos-rebuild switch lacks that
env var, so the key is removed as "obsolete". sops-nix derives its age
decryption key from /etc/ssh/ssh_host_ed25519_key, so deletion cascades
into "Error getting data key: 0 successful groups required, got 0" for
every sops secret on the host.
Fix: two activation scripts bracket the etc step.
preserveSshHostKey (no deps, runs before etc): copies the live key to
/run (tmpfs) before etc can delete it.
restoreSshHostKey (deps=[etc], runs after etc): reinstalls via `install`
if etc removed the key. The resulting file is not tracked in either
generation's environment.etc, so subsequent rebuilds leave it alone.
scripts/recover-hosts.sh: restore both private and public key files (not
just the private key), use install(1) for atomic mode setting, and add a
post-rebuild sops-nix verification step to confirm success.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014zT1L6hmsq6i1evAEH7dmi
A fresh proxmox-server deploy has a blank scsi1 disk, so
zfs-import-tank.service spun 60 s then failed with no pool found.
Add zfs-init-tank.service that runs before the import: exits immediately
if the pool already exists, imports it if it exists but isn't imported
yet, or creates it on /dev/disk/by-id/scsi-*drive-scsi1 (Proxmox's
virtio-scsi naming for the second disk) with all required NFS datasets
if the disk is blank.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The post-build cp/mv into /var/lib/vz/template/cache (LXC) and
/var/lib/vz/import (VM) are Proxmox-owned root directories -- they need
sudo_pfx just like pct/qm/pvesh do. nix build writes to the nix store
as the SSH user, but staging into /var/lib/vz/ requires root.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The script was root-only and hard-coded /etc/nix/nix.conf and
/etc/ssh/ssh_known_hosts, making it always fail (non-fatally) when
called as a non-root SSH user from create-proxmox-resource.sh.
Add dual-mode detection based on EUID:
- root (multi-user/daemon): existing behavior unchanged -- writes
/etc/nix/nix.conf, /etc/ssh/ssh_known_hosts, restarts nix-daemon
- non-root (single-user): writes ~/.config/nix/nix.conf and
~/.ssh/known_hosts, creates the config file if missing, skips the
daemon restart (single-user has no daemon), defaults REMOTE_BUILDER_KEY
to ~/.ssh/id_ed25519 instead of /root/.ssh/id_ed25519
create-proxmox-resource.sh already calls the script without sudo (as the
SSH user), so no change is needed there -- the script now handles both
cases on its own.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nix was installed as root on pve1 (the codex-setup.sh root path, meant
for container/Codex environments), making nix build require sudo there.
After cleaning up the root install and reinstalling as the SSH user
(wayne), nix is owned by that user and runs directly without sudo.
create-proxmox-resource.sh: drop sudo_pfx from nix build in both
remote scripts. The SSH user owns the store after reinstall; nix build
goes through the nix daemon-or-store directly. sudo stays on pct/qm/pvesh
(cluster IPC) and the disko image-writer script (writes to disk).
codex-setup.sh: add build-users-group = (empty) to the user nix.conf
written by the non-root install path. Guards against a stale
/etc/nix/nix.conf from a prior root install (which sets
build-users-group = nixbld) silently breaking single-user builds.
Manual cleanup required once on each Proxmox node that had root's nix:
sudo rm -rf /nix /etc/nix
sudo rm -f /etc/profile.d/nix.sh /etc/profile.d/nix-daemon.sh
for i in $(seq 1 10); do sudo userdel nixbld$i 2>/dev/null||true; done
sudo groupdel nixbld 2>/dev/null || true
After that, the next create-proxmox-resource.sh run auto-reinstalls
nix as the SSH user via codex-setup.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nix on pve1 was installed as root (single-user), so wayne can't access
/nix/var/nix/db/big-lock without root -- nix build genuinely needs sudo
there. The previous fix to drop sudo_pfx was wrong.
The real fix is node config: add nix to wayne's NOPASSWD rules in
sudoers on pve1 (see below). pct/qm/pvesh already have NOPASSWD and
work fine in non-interactive SSH heredocs; nix was just missing from
that list.
On pve1 as root:
echo 'wayne ALL=(root) NOPASSWD: ALL' | tee /etc/sudoers.d/wayne-nopasswd
chmod 440 /etc/sudoers.d/wayne-nopasswd
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nix build runs through the nix daemon and doesn't need root; the
tooling-check step already confirms the SSH user can run nix directly
(ensure_nix_profile + command -v nix). sudo without a TTY blocks
non-interactive SSH heredoc sessions with "a terminal is required".
Keep sudo on pct/qm/pvesh (cluster IPC) and the disko image-writer
script (writes to block devices) -- those actually require root.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single-user Nix installations are owned by root. When PROXMOX_SSH_USER
is not root, prefix the remote nix build command with sudo_prefix, same
as the Proxmox tool invocations. Passes sudo_prefix as an extra arg to
both the LXC tarball and VM disko image build heredocs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/root/nixos was only correct when PROXMOX_SSH_USER=root. Now that it
defaults to wayne, use /home/${PROXMOX_SSH_USER}/nixos so git clone
goes somewhere the SSH user can actually write to.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PROXMOX_SSH_USER was changed from root to wayne, but all remote pvesh/qm/pct
invocations assumed root. When run as a non-root user these commands fail with
ipcc_send_rec errors because they can't reach the pve-cluster IPC socket.
Adds a global sudo_prefix (empty when PROXMOX_SSH_USER=root, "sudo" otherwise)
and applies it to every remote Proxmox command in the script, including the
duplicate-host heredoc check, pvesh nextid, vmid existence checks, resource
destruction, and all create/start commands. Removes the now-redundant local
sudo_prefix definition that was previously only in the VM image build branch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous fix (#42) only unmounted /mnt/{dev,proc,sys,run}, but disko
also mounts the ESP at /mnt/boot (modules/disko/baremetal.nix) -- another
nested mount blocking ZFS from unmounting its own root dataset at /mnt
the same way. Confirmed live: zpool export still failed with "cannot
unmount '/mnt': pool or dataset busy" after the chroot-only fix.
Replace the manual dev/proc/sys/run list with a single recursive
`umount -R /mnt`, which clears everything nested under /mnt -- current
and future mountpoints alike -- rather than needing to keep enumerating
whatever nixos-install/disko happen to leave mounted.
nixos-install bind-mounts /dev, /proc, /sys (and usually /run) into
/mnt to run the target's activation script in a chroot, and doesn't
unmount them again afterward. Left in place, those nested mounts made
ZFS refuse to unmount its own root dataset at /mnt: zpool export
failed with "cannot unmount '/mnt': pool or dataset busy", and because
of this script's set -e, that killed the script before it ever reached
reboot -- silently defeating the export-before-reboot fix from #40 on
every real run, which is why the ZFS-import stall kept recurring.
The systemd-based initrd (default here, since this host has a ZFS
root) locks the root account by default, so sulogin refuses a shell
if something in the initrd fails and it drops to emergency mode --
confirmed live: it just loops re-entering the target instead of
prompting, making an initrd-level ZFS import failure impossible to
diagnose from the console. Only affects the pre-switch-root initrd
shell, not the installed system's own login.
disko's --mode ...,mount leaves the pool imported (needed for
nixos-install to write into /mnt), and the script rebooted straight
into the newly-installed system without exporting it. That pool is
still stamped with the live installer's own hostid, which never
matches the target host's declared networking.hostId, and since
boot.zfs.forceImportRoot is false (the recommended setting, not a bug),
the first real boot refuses to force-import an unexported pool from a
different hostid -- which is exactly the ZFS-import stall baremetal-gui
was hitting after install. Exporting all pools right before reboot (a
no-op for non-ZFS hosts) clears the in-use state so import succeeds
regardless of hostid.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
If neither /etc/host-keys nor /root/host-keys has the target's SSH
host key, auto-install.sh previously went straight to "continue
without one anyway?". Added a third option in between, gated on
[[ -t 0 ]] (only offered when there's an actual operator at stdin, never
in an unattended/non-interactive run): prompt for an arbitrary
directory (USB stick, other mount, etc.), and if the key pair is
there, copy it into /root/host-keys and install it to /mnt same as the
existing pre-seeded-key path. Falls through to the original
warning+confirm if the prompt is skipped, the path doesn't have the
key, or the run isn't interactive at all.
docs/auto-installer.md updated to mention the new fallback. Quick
bash -n + shellcheck pass only, per request.
Running the script standalone (its whole point per the last commit)
failed with "disko: command not found" -- jq/disko/nixos-install are
only guaranteed present via the built installer image's
environment.systemPackages, not on a plain checkout.
Added a #!/usr/bin/env nix-shell / #!nix-shell -i bash -p jq disko
nixos-install-tools shebang instead of a per-tool fallback: disko's own
generated scripts already hardcode absolute Nix store paths for
everything they shell out to internally (parted/sgdisk/mkfs.*/zfs/...
confirmed by inspecting a generated system.build.formatScript earlier),
so these three are the only genuinely external dependencies the script
itself has. This is a fast no-op on the built installer image (already
has all three) and what makes it also work standalone.
Quick syntax + shellcheck pass only this round (bash -n, shellcheck
with a `shellcheck shell=bash` directive since it doesn't recognize
nix-shell shebangs natively) -- skipping the full codex-maintenance.sh
sweep per request, to get this out for a real hardware test.
Two real bugs, both hit live:
1. Shebang: #!/run/current-system/sw/bin/bash only resolves on an
already-activated NixOS system -- running the checked-out script
directly (e.g. from a stock ISO, cloned repo) failed with "cannot
execute: required file not found" on a non-NixOS box. Switched to
#!/usr/bin/env bash, which resolves identically on NixOS
(environment.usrbinenv's own default) and any normal Linux distro.
Also fixed the file's missing executable bit.
2. FLAKE_BASE_URL: previously depended on pkgs.replaceVars substituting
a Nix-templated @lanDomain@ placeholder at build time -- meaning it
only ever worked when baked into the built installer image, not when
run straight from a checkout (the literal, unexpanded "@lanDomain@"
string reached git as a bogus hostname). Replaced with LAN_DOMAIN in
scripts/env.sh (manually kept in sync with variables.nix's lanDomain,
same pattern as NIX_CACHE_HOST/nixCacheHost already), sourced by the
script itself like every other script in scripts/. Dropped
pkgs.replaceVars from modules/installer/common.nix entirely --
scripts/env.sh is now baked into the image alongside auto-install.sh
at a matching relative path (/etc/nixos-installer/env.sh next to
/etc/nixos-installer/installer/auto-install.sh) so the script's own
relative `source` line resolves the same way in both contexts.
loginShellInit's invocation path and docs/auto-installer.md updated to
match. Verified: shellcheck clean on both scripts, the baked files are
byte-identical to their checked-in sources (no templating left to
verify), and codex-maintenance.sh (secret grep, fmt, statix, full eval
of every host/package including the installer/pxe artifacts) passes
clean.
These were dropped from variables.nix by a stash/merge conflict
resolution on main (commit fb6ee27) that kept the new wifiSsid value
but discarded the two disk-path variables entirely, leaving unresolved
`<<<<<<< Updated upstream` markers in an intermediate commit before
being cleaned up. modules/disko/baremetal.nix references both directly
with no fallback, so baremetal-gui has been failing to evaluate on main
since that commit ("attribute 'guiRootDisk1' missing") -- confirmed by
cloning main fresh and evaluating config.disko.devices.disk.disk1.device
directly.
This commit is rebased onto latest main (through "updated secrets",
which registered baremetal-gui's real sops recipient) rather than the
older base this branch started from.
Moves the auto-installer's shell script from an inline Nix string in
modules/installer/common.nix to scripts/installer/auto-install.sh, a
real, version-controlled, directly-editable/shellcheck-able file.
common.nix now wires it in with pkgs.replaceVars, substituting the one
value that actually needs to come from variables.nix (lanDomain) --
every other `${...}` in the script is a literal bash reference, left
untouched. replaceVars fails the build if any @name@-shaped placeholder
is left unsubstituted, so a typo'd or renamed variable is caught at
eval time rather than silently shipping broken.
Verified: built the substituted derivation and diffed it against the
source template -- identical except for the one substituted line, no
leftover unsubstituted placeholders. Full codex-maintenance.sh (secret
grep, fmt, statix, full eval of every host/package including the
installer/pxe artifacts that consume this) passes clean.
codex-maintenance.sh evaluated each affected host/package one at a time,
even though those calls are independent. Added scripts/lib/nix-parallel.sh
(run_nix_parallel) and wired it into the host-eval, package-eval, and
dry-run-build loops.
Concurrency defaults to core count capped by available memory (~1GB/job)
rather than plain nproc: empirically, nproc-many concurrent full-flake
evals OOM-killed each other on a 4GB/6-core box, while 3-4 ran clean and
were still ~2x faster than serial. Override via NIX_PARALLEL_JOBS.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Wires everything staged so far into a real flake target:
- modules/platforms/baremetal.nix (new): the bare-metal platform module,
composed from a real nixos-generate-config run on the actual gui-host
hardware (AMD CPU, ahci/xhci/usb storage -- modules/hardware-configuration/baremetal.nix).
Enables hardware.enableRedistributableFirmware (real wifi/GPU/microcode
firmware VMs never needed), amdgpu as the Xorg video driver plus
hardware.graphics for Mesa OpenGL/Vulkan, and imports the ZFS RAID0 disko
layout + modules/services/zfs/enable-service.nix for root-on-ZFS boot
support.
- flake.nix: new baremetal-gui target, reusing hosts/nixos/host.nix (same
identity already shared across linode/proxmox/lxc-gui).
- hosts/nixos/host.nix: added networking.hostId, required now that a ZFS
root pool is in the picture.
- variables.nix: guiRootDisk1/guiRootDisk2 filled in (/dev/sda, /dev/sdb --
only used transiently at disko-format time, same as modules/disko/proxmox.nix's
own plain device path). wifiPassword removed.
- modules/networking/wifi.nix: reworked to pull the wifi password from a
new sops secret (secrets/gui.yaml, wifi-password) instead of a plaintext
variable -- NetworkManager's ensureProfiles renders `psk = "$WIFI_PASSWORD"`
literally (nixpkgs' own documented pattern for this) and envsubst-expands
it from a sops-rendered EnvironmentFile at activation, so the real value
never touches the Nix store, only /run.
- .sops.yaml: new secrets/gui\.yaml rule, admin + the currently-registered
lxc-gui recipient (the only gui variant with a provisioned host key so
far -- whichever variant is actually deployed next still needs
scripts/secrets/sync-host-keys.sh run for its own recipient).
- README.md/CLAUDE.md: documented the new platform/target and its module
layout, per this repo's own drift-prevention note.
Verified end-to-end: nix eval of every existing target (nothing broke),
a temporary real nixosSystem build against the actual disko.nixosModules.disko
confirming the generated zpool create has no mirror/raidz keyword (genuine
stripe), and a temporary test SSID confirming the sops secret/template/
ensureProfiles chain renders correctly before reverting to blank/real values.
Full scripts/codex-maintenance.sh (secret-grep, fmt, statix, full-fallback
eval of every host/package) passes clean.
Adds modules/disko/baremetal.nix: two disks, each its own top-level
zpool vdev with no mirror/raidz between them (disko's zpool `mode`
defaults to "" for a plain stripe), ESP + systemd-boot on disk1. Device
paths are placeholders in variables.nix (guiRootDisk1/guiRootDisk2)
until the real hardware profile arrives.
Verified structurally by building a throwaway nixosSystem with the
actual disko.nixosModules.disko and reading the generated
system.build.formatScript: it emits `zpool create rpool ... disk1
disk2` with no mirror/raidz keyword, confirming a genuine stripe.
Not yet wired into any flake target -- that happens once the hardware
config lands and a new bare-metal platform module is added, per the
agreed sequencing.
Adds modules/networking/wifi.nix using NetworkManager's ensureProfiles
mechanism so the gui host associates to a known SSID on first boot with
no manual nmtui step. Credentials are placeholders in variables.nix
(wifiSsid/wifiPassword, both empty) to be filled in once the bare-metal
hardware profile is wired up — the module is a no-op until then.
The iPXE menu's "nixos" entry actually chain-loads this flake's own
custom auto-installer image, not a stock NixOS image — rename it to
"auto-installer" (label "NixOS Auto-Installer") so the menu says what it
boots, and set networking.hostName on netbootSystem to match, so the
generated system name (nixos-system-auto-installer-*) and staged
directory (/srv/pxe/http/auto-installer) agree with the menu entry too.
Add a second, genuinely vanilla NixOS minimal netboot image
(netbootMinimalSystem in flake.nix — nixpkgs' netboot-minimal.nix on its
own, none of modules/installer/common.nix's auto-installer wiring),
built from source the same way as the auto-installer image and exposed
as packages.x86_64-linux.pxe-minimal. Staged and menu-wired the same
way, as "nixos-minimal" (item, hostname, and directory all matching).
modules/pxe-boot/stage-installer-artifacts.nix is generalized to stage
both images via a shared rule-builder instead of one hardcoded set of
paths.
Verified: nix eval confirms both images' config.system.name matches
their menu entry/directory names, the pxe-boot host itself builds
clean with the new menu.ipxe, and the new pxe-minimal image was booted
directly under QEMU (kernel+initrd, no KVM) to a working login shell
with hostname nixos-minimal, no hang.
The netboot build composed ./modules/installer/iso.nix (which pulls in
nixpkgs' installation-cd-minimal.nix) together with nixpkgs'
netboot-minimal.nix. Both installation-cd-base.nix and netboot.nix set
fileSystems."/" via the identical lib.mkImageMediaOverride (mkOverride
60) priority - genuinely conflicting root-filesystem strategies
(ISO-by-label vs. netboot-tmpfs) at the same priority, and the ISO one
was winning. Every netboot boot hung waiting for a device that can
never exist outside a real CD/USB:
A start job is running for /dev/disk/by-label/nixos-minimal-...
Reproduced live: deployed a scratch lxc-pxe-boot on pve-test, pulled its
built kernel/initrd, and booted them directly with QEMU to confirm the
hang and capture full console output. netboot-minimal.nix's own chain
(netboot-base.nix) already imports profiles/installation-device.nix
independently, so common.nix's initialHashedPassword override still
applies correctly with iso.nix removed from this composition. Rebuilt
and re-booted the same way after the fix - full boot to a working shell
with SSH up, no hang.
Backs up a VM/CT on pve1 (snapshot mode by default, so the source stays
online), relays the archive to pve-test, restores it there with fresh
MAC addresses (--unique), and deletes both the source and relayed
backup copies afterward -- no ad hoc backup files left behind on
either node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pve1.sweet.home is production; scripts/env.sh now also defines
PVE_TEST_HOST for a separate sandbox node, individually targetable via
--node/PROXMOX_HOST. Tooling defaults are unchanged (still pve1) -- the
new restriction (Claude defaults to pve-test unless explicitly told to
use pve1) is documented as policy in CLAUDE.md, not enforced in the
scripts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The kernel's NFS client filesystem doesn't set FS_USERNS_MOUNT, so mounting
NFS from inside any non-init user namespace -- exactly what an unprivileged
LXC container's UID-mapped root runs in -- is rejected at the VFS layer
with EPERM, regardless of Proxmox's mount=nfs;nfs4 container feature (which
only patches the AppArmor layer). Confirmed live on the redeployed lxc-docker
container: TCP to the NFS server's port 2049 succeeds, the server's export
table matches the container's IP, and mount.nfs: Operation not permitted
still fires immediately with no corresponding denial anywhere in the
server's own logs -- a kernel-level rejection that no amount of DNS/
automount/export tweaking (this branch's earlier commits) could ever fix.
modules/platforms/lxc.nix now keys proxmoxLXC.privileged off hostName
("docker" -> true) rather than a blanket false, since build-types/docker.nix
is also composed for linode-docker/proxmox-docker, which don't import
proxmox-lxc.nix at all -- setting this option there would break their eval.
create-proxmox-resource.sh reads the value back via a new
flake_target_lxc_privileged helper instead of hardcoding --unprivileged 1,
so the two stay in sync automatically for every lxc-* target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
Codifies read-only access to existing Proxmox config/VMs/containers,
allows scratch test VMs/containers as long as they're torn down again,
and forbids any change to production on the node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
create-proxmox-resource.sh's `qm create` never passed --agent, so despite
services.qemuGuest.enable = true being set on every host, Proxmox never
created the virtio-serial channel the guest agent needs -- qm guest exec
and the UI's IP-address display silently never worked for any VM this
script created. Found while live-testing every lxc-*/proxmox-* build type
against pve.sweet.home for an end-to-end flake audit.
Also registers a fresh sops age key for lxc-gui (no prior registration
existed), generated while testing that target live -- needed before
lxc-gui can be deployed with working secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZHwwAM7cacQKkqKfQ5e8a
The previous commit's networking.search fix was wrong. Confirmed live
on lxc-docker (vmid 105) after redeploying with it: `resolvectl query
server.sweet.home` started failing again, even though
`resolvectl query --interface=eth0 server.sweet.home` still resolved
correctly to the right IP via the LAN's real DNS server. The debug log
showed why -- adding a *global* search domain via networking.search
gave systemd-resolved a domain-matched but server-less "global" scope,
which it now prioritizes over eth0's correctly-configured scope for
every "*.sweet.home" query, silently sending them to public fallback
DNS (1.1.1.1 et al) instead, which of course returns NXDOMAIN for an
internal-only name. Bare single-label names (e.g. "server") were never
going to work either way -- systemd-resolved only ever tries LLMNR for
those, never DNS search-suffixing, regardless of configuration.
Reverts the networking.search addition and instead has
modules/docker/mount-data.nix build each NFS device string from
"${vars.nfsServerHost}.${vars.homeDomain}" (a plain FQDN, no dependency
on search-domain behavior at all) -- the same pattern
modules/raspi/mount-data.nix already uses for the Raspberry Pi's share
and for the identical reason.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
Cross-host references throughout this repo (vars.nfsServerHost,
vars.nixCacheHost, vars.dockerHost) are bare short names, not FQDNs.
Resolving them has always depended on whatever network stack happens
to be in play picking up the DHCP-advertised domain as a search
suffix -- NetworkManager does this by default, which is why it went
unnoticed everywhere else, but LXC containers force-disable
NetworkManager and get their systemd-networkd config written directly
by Proxmox instead, which never sets one.
Confirmed live on lxc-docker (vmid 105) after today's earlier fix for
the automount/mount=nfs bugs: systemd-resolved had no search domain
for eth0, so "server" failed to resolve ("Name or service not known")
while "server.sweet.home" resolved fine via the same DNS server --
every NFS mount in modules/docker/mount-data.nix was still failing.
networking.search sets systemd-resolved's own Domains= globally
rather than depending on the per-link DHCP path, so it isn't at the
mercy of whichever component owns a given host's interface file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
CI was running a full eval of every host + package on every push/PR,
which was slow enough to routinely time out the Gitea runner. Default
mode now diffs against a base ref and scopes nixpkgs-fmt/statix/eval to
the files that changed and the hosts/packages they can affect; a change
to flake.nix/flake.lock/variables.nix/modules/common/* (or any other
modules/*.nix outside platforms//build-types, whose blast radius isn't
inferable from the path) falls back to evaluating everything. The old
full sweep moves behind --full-check, which CI never passes; --dry-run
adds build-planning on top of whichever scope is active.
Also trims codex-setup.sh's redundant full host eval loop -- that's
what codex-maintenance.sh is for; setup should just install tooling.
variables.nix's nixCacheHostKey no longer matched nix-cache's actual SSH
host key (confirmed via ssh-keyscan against the live container), so every
declaratively-configured client's programs.ssh.knownHosts trusted the
wrong key -- distributed builds would fail host-key verification. Also,
modules/nix-cache/remote-builder-client.nix hardcoded sshKey to
/root/.ssh/nixremote, but the `server` host only has its own default
/root/.ssh/id_ed25519 installed (confirmed live via qm guest-agent) --
that file was never even present, so the build machine config pointed at
nothing. Standardize on each client's own default identity, matching the
per-host-key pattern vars.remoteBuilderAuthorizedKeys already uses instead
of a shared/differently-named keypair, and add
scripts/secrets/sync-nix-cache-host-key.sh (wired into
codex-maintenance.sh's --check) so the host-key drift doesn't silently
recur next time nix-cache is rebuilt or recreated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V7yVH71vGrDVzovh9UaMu8
Following the existing per-client-key pattern in
vars.remoteBuilderAuthorizedKeys (one entry per host's own default root
key, not a shared dedicated keypair -- docs/nix-cache.md's "./nixremote"
filename is just a generic placeholder in the instructions). This box
had no entry yet, which is why its earlier `ssh-copy-id nixremote@nix-cache`
never stuck: nix-cache's nixremote authorized_keys is NixOS-declarative
and gets overwritten on every rebuild.
Public key only; requires a nix-cache rebuild to take effect.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scripts/ had grown to 10 top-level scripts covering three distinct
concerns (sops/age + SSH host-key management, Proxmox deployment, and
repo-wide bootstrap/CI) with no grouping. Move the key-management scripts
(backup-admin-key.sh, rotate-admin-key.sh, prepare-host-key.sh,
sync-host-keys.sh) into scripts/secrets/, and the Proxmox scripts
(create-proxmox-resource.sh, configure-nix-cache-client.sh) into
scripts/proxmox/; leave env.sh, codex-setup.sh, codex-maintenance.sh, and
bump-nixpkgs-release.sh at the top level (frequently hand-typed or pure
shared config) and scripts/lib/ as-is.
Updates every cross-reference: each moved script's repo_root computation
(now one directory deeper), shellcheck source= directives, inter-script
paths (create-proxmox-resource.sh's call into sync-host-keys.sh and its
remote bootstrap of configure-nix-cache-client.sh on the Proxmox node),
and every doc/module mention (CLAUDE.md's Scripts section reorganized to
match, README.md, docs/auto-installer.md, docs/proxmox-images.md,
modules/installer/common.nix, modules/platforms/lxc.nix). CI workflows
need no change -- they only invoke codex-maintenance.sh, which didn't
move. Verified via bash -n, shellcheck (no new warnings beyond the
pre-existing SC1091/SC2029/SC2095 baseline), and live dry-runs of
sync-host-keys.sh --all and create-proxmox-resource.sh --list from their
new paths.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scripts/backup-admin-key.sh, rotate-admin-key.sh, and sync-host-keys.sh
each independently resolved sops/age's default key-file path, derived an
age pubkey from an identity file, and (two of them) ran `sops updatekeys`
the same way -- now shared via scripts/lib/sops-age.sh. Also extracted the
"type X to confirm" prompt duplicated across create-proxmox-resource.sh
and sync-host-keys.sh into scripts/lib/confirm.sh. Pure extraction, no
behavior change -- each call site produces identical commands/output to
before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
.sops.yaml's &lxc-nix-cache anchor was rotated (commit b3c8145) after
today's LXC rebuild, but secrets/nix-cache.yaml was never re-run through
sops updatekeys -- it stayed encrypted for the pre-rotation key. Per
docs/nix-cache.md and docs/auto-installer.md, sops-nix decrypt failures
are silent: cache-priv-key never materialized, nix-serve had nothing to
bind its secretKeyFile to and failed to start, and nginx proxy_pass had
no upstream -- the 502 Bad Gateway seen from clients.
Confirmed the new key matches the currently-deployed host by deriving
the age identity from host-keys/lxc-nix-cache_ssh_host_ed25519_key.pub
via ssh-to-age and comparing to .sops.yaml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Run it once, right after a node's first-time Nix bootstrap (not on every
invocation, and not inside codex-setup.sh/codex-maintenance.sh themselves),
so a freshly-bootstrapped Proxmox node substitutes from and can offload
builds to nix-cache on every subsequent run. Non-fatal on failure -- the
build still proceeds, just without nix-cache.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Non-NixOS machines with just the Nix package manager installed have no
module system to pick up modules/nix-cache/client.nix, so this edits
/etc/nix/nix.conf directly (extra-substituters/extra-trusted-public-keys,
plus the SSH remote-builder config once the nixremote key is installed),
falling back to cache.nixos.org when nix-cache is unreachable.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-run tooling verification (added in the previous commit) was hardcoded
to checking just `nix`. Turn it into a small array instead, so a future
remote build step needing another tool extends that list rather than
growing a second parallel check.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two bugs found running scripts/create-proxmox-resource.sh against a real
Proxmox node for the first time:
- The Nix installer's single-user root path still shells out to `sudo`
to create /nix even though it already detected it's running as root,
which fails outright on a minimal, sudo-less Debian/Proxmox node.
codex-setup.sh now pre-creates /nix itself so that branch of the
installer is skipped.
- ensure_remote_repo() only ran scripts/codex-setup.sh right after a
fresh git clone, so a bootstrap that cloned the repo but then failed
installing Nix (exactly the failure above) left every subsequent run
silently building with a `nix` that was never actually installed,
since the repo already existing skipped tooling setup entirely. It
now checks `command -v nix` (via the same ensure_nix_profile used
elsewhere, since a non-interactive ssh session won't otherwise have a
single-user install on PATH) on every run and re-bootstraps if it's
missing. Both remote build heredocs also now source
scripts/lib/nix-bootstrap.sh themselves for the same PATH reason.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New tor-relay build type (currently lxc-only) running a plain Tor
middle relay via modules/tor/enable-relay.nix, plus nyx for
interactive monitoring over the relay's control socket.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
create-proxmox-resource.sh no longer builds locally and scp's a
multi-gigabyte image over -- it now clones/pulls this repo onto the
Proxmox node itself (bootstrapping build tooling via the existing
codex-setup.sh on first use) and runs the nix build / disko image
script there, staging the result straight into the node's own import
directory. host-keys/ (gitignored) is copied over separately since a
git pull doesn't carry it. --image still uploads an explicit local
file for the case where you don't want a build at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
--allow-duplicate-host previously just skipped the existing-resource
check entirely, so re-running e.g. --type lxc --host docker while an
lxc-docker container already existed created a second container
sharing the same hostname/identity instead of replacing it -- both
then fight over DNS/DHCP for that hostname, and it's easy to end up
testing the stale one without realizing.
Now splits matches into "exact" (same --type as the one being
created, e.g. another lxc-docker) and "cross-type" (a different
platform sharing this host identity, e.g. a proxmox-docker VM
alongside an lxc-docker container -- a deliberate, valid coexistence
this script has never managed and still won't). Only an exact match is
destroyed and replaced, after typing the hostname back to confirm; a
cross-type match is always left untouched. Without
--allow-duplicate-host, both cases still refuse to run exactly as
before.
Verified live against pve.sweet.home: correctly split VMID 103 (a
stopped proxmox-docker VM, cross-type -- left untouched) from VMID 105
(the running lxc-docker container, exact-type -- flagged for
destroy+replace), and confirmed the destroy prompt safely aborts on a
non-matching confirmation, leaving both resources untouched.
Two compounding bugs, confirmed live on the running lxc-docker
container (vmid 102 on pve.sweet.home):
1. x-systemd.automount never works inside any Linux container --
systemd logs "Starting of <unit>.automount unsupported" for every
share and never mounts them. modules/docker/mount-data.nix and
modules/raspi/mount-data.nix now key off config.boot.isContainer
(set true by nixpkgs' proxmox-lxc.nix) to mount eagerly with
`nofail` there instead, while VM-based docker targets keep automount
unchanged.
2. The container's Proxmox `features` never included `mount=nfs`, so
AppArmor blanket-denies the nfs/rpc_pipefs mount syscalls NFS
needs ("permission denied"). scripts/env.sh's
PROXMOX_DEFAULT_LXC_FEATURES now includes mount=nfs;nfs4 for future
lxc-* containers -- the semicolon required quoting the --features
value in create-proxmox-resource.sh's remote pct-create command,
since it's sent as a raw string for the remote shell to parse and an
unquoted `;` would be read as a command separator.
The already-running container needs a matching `pct set --features`
plus a restart to pick this up -- that's an operator step outside this
repo.
Companion to rotate-admin-key.sh: copies whatever age identity sops/age
itself would resolve (or an explicit --key-file) to a given destination
path with 0600 perms, validating it's a real identity and round-tripping
the derived public key before/after the write so a corrupted copy is
caught immediately rather than discovered later during a restore.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Automates the manual steps sync-host-keys.sh/create-proxmox-resource.sh
print when they bootstrap a fresh, not-yet-trusted age key: verifies a
backed-up key matches the current &admin entry, swaps in a new key, and
re-encrypts every secrets/*.yaml. Explicitly cds into repo_root before any
sops call, since sops resolves .sops.yaml by walking up from cwd rather
than from the target file's path -- confirmed via a scratch-repo test that
running from elsewhere would otherwise silently rotate against the wrong
config.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same class of problem as the deployedTargets/README fixes: hand-maintained
prose that drifted from reality and nobody was obligated to update.
- CLAUDE.md: "18 hosts" was a stale hardcoded count (actually 20); reworded
to not need updating as hosts are added. Also added the missing
tailscale-exit-node build type to a list that had it everywhere else in
the file except one bullet.
- AGENTS.md: same missing tailscale-exit-node build type.
- docs/auto-installer.md: the hand-enumerated lxc-* list was missing
lxc-tailscale-exit-node.
- flake-target-refactor-spec.md: added a "Status: implemented" note so this
completed historical spec (referenced elsewhere purely for rationale)
can't be mistaken for an open plan with unresolved Open Questions.
- remove-sensetive-info-refactor.md: the "Definition of done" checklist was
entirely unchecked despite most of the work being done. Checked off what's
actually done (sops-nix migration, history scrub just performed, the
pre-commit gitleaks hook), and left rotation of the GitHub PAT found in
history explicitly flagged as the one still-open item -- an operator
action against GitHub, not something this repo can attest to itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The remote bash script run over SSH ended with a for-loop whose last
statement was `[[ "$n" == "$target" ]] && echo ...`. When the last
VM/CT checked on the node didn't match --host, that test evaluated
false and became the exit status of the whole remote script (1) --
which the wrapper then misreported as "couldn't reach the node",
even though SSH connectivity and the check itself were both fine.
The actual signal is the script's stdout, not its exit code, so end
it with an explicit exit 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three chunks of copy-pasted logic were drifting across scripts/*.sh:
- codex-setup.sh and codex-maintenance.sh each carried an identical
NIX_CONFIG bootstrap + ensure_nix_profile() -> scripts/lib/nix-bootstrap.sh
- sync-host-keys.sh and prepare-host-key.sh each ran the same
ssh-keygen/ssh-to-age nix-shell invocations -> scripts/lib/ssh-host-keys.sh
(prepare-host-key.sh now also calls env.sh's nix_extra_opts before using
them, closing a gap where it alone skipped the nix-cache reachability
check env.sh exists for)
- the "list nixosConfigurations attrNames" / "get one target's hostName"
nix eval pattern was repeated across codex-setup.sh, codex-maintenance.sh,
sync-host-keys.sh and create-proxmox-resource.sh (the latter twice, in
its own --list and --host lookup) -> scripts/lib/nix-eval.sh, which also
centralizes the --no-use-registries --no-accept-flake-config flag pair
used on every such call
Verified against the real flake/node config (nix is available here):
create-proxmox-resource.sh --list for both --type lxc/vm, a full
--dry-run create, and prepare-host-key.sh generating and cleaning up a
real key/age-pubkey pair.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same problem as the deployedTargets removal, just in markdown instead of
Nix: which variant of a buildtype is actually deployed is live
infrastructure state, and a committed table can't stay accurate as that
changes -- it already required a manual edit on every migration and had
drifted before. Keep only what doesn't rot: what each target is for, and
stable naming history. Point at the live node / /etc/flake-target instead
for actual deployment status.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
variables.nix's deployedTargets was a manually-maintained list with no
enforcement keeping it in sync with reality -- it caused two separate
false refusals in a row (naming a VM as deployed well after it had been
destroyed, then matching a target against itself once the list was
"corrected"). Static files can't track whether a resource still actually
exists.
create-proxmox-resource.sh's duplicate-host guard now queries the
Proxmox node directly (qm/pct's own name/hostname config, matched
against --host) instead. Also fixes a gap in that live check: it
originally swallowed ssh failures and would have silently treated "can't
reach the node" the same as "checked, nothing there" -- it now refuses
instead of guessing when the node can't be reached.
deployedTargets is removed entirely from variables.nix since nothing
else in the repo consumed it once this script no longer does; README.md's
Hosts table remains the sole source of truth for "(real, deployed)"
status. CLAUDE.md and the script's own --help/comments updated to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The duplicate-host check in create-proxmox-resource.sh compared by
hostName only, so it fired even when the target being created was
exactly the one variables.nix's deployedTargets already names (e.g.
rebuilding lxc-nix-cache after destroying its old container to pick up
new sops secrets) -- there's no other machine at risk of an identity
collision in that case, just the normal redeploy workflow. Skip the
check when dt == flake_target; the later VMID-existence check still
guards against clobbering a resource that's actually live on the node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The old proxmox-nix-cache VM was destroyed and nix-cache is being
redeployed as an LXC container going forward. Without this update,
create-proxmox-resource.sh's duplicate-host check (which only reads this
static list, not live Proxmox state) kept refusing to create
lxc-nix-cache even though nothing named nix-cache actually exists on the
node anymore.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Distributed builds failed with "Host key verification failed" on any
client that had never manually SSH'd to nix-cache before, since
nothing populated root's known_hosts for it. Wire nix-cache's host
public key into programs.ssh.knownHosts via a new vars.nixCacheHostKey
so every client picks it up automatically on rebuild.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
lxc-* hosts need NIXOS_HOST_KEYS_DIR + --impure to bake in a pre-seeded
SSH host key, otherwise sops-nix's .sops.yaml recipient never matches
and every secret permanently fails to decrypt on first boot. That
invocation is easy to forget, so wrap it as `buildImage <flake-target>`
alongside the existing Switch-nix/Test-nix helpers.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The reachability check used `cat < /dev/tcp/${NIX_CACHE_HOST}/22`, which
blocks forever reading for EOF that never comes -- sshd sends its banner
and then holds the connection open waiting for the client to speak next.
Every single check hit the 3s timeout and reported "unreachable"
unconditionally, regardless of whether the remote builder was actually up.
Confirmed live: a plain TCP connect (`exec 3<>/dev/tcp/...`, no read)
returns in ~60ms against a healthy nix-cache instead of always timing out.
Fixing that exposed a second, previously-dormant bug: `printf -v
NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"` on a genuinely empty NIX_OPTS array
still runs one format pass and yields the literal `'' ` rather than an
empty string. A subprocess (e.g. sync-host-keys.sh) reusing this
process's decision via `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` then rebuilt
a 1-element array holding an empty string instead of a 0-element array,
which broke `nix-shell "${NIX_OPTS[@]}" -p <pkg>` with a bogus positional
argument the moment NIX_OPTS was legitimately empty (nix-cache reachable)
-- something the first bug had made impossible to ever hit before.
Also adds a couple of retries (1s apart) to both checks as a secondary
safety net against genuine multi-second blips, on top of fixing the
checks themselves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The tarball/disko-image build previews were hardcoded strings that never
included ${NIX_OPTS[@]}, so --dry-run always showed the same "would build"
command whether nix-cache's substituter/remote-builder got disabled by
nix_extra_opts's reachability probe or not -- the actual (non-dry-run)
build commands already applied it correctly, only the preview lied.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
services.beszel.agent runs under DynamicUser=true with ProtectSystem =
"strict" and no StateDirectory, so /var/lib/beszel-agent -- where the
agent persists the fingerprint that locks its hub pairing to this
machine (github.com/henrygd/beszel/discussions/1542) -- was never
actually writable. Every restart silently failed to persist it and
regenerated a fresh one in memory, permanently desyncing from whatever
the hub had on record after the very first successful pairing. Affects
every host importing modules/beszel/enable-agent.nix (nix-cache, server),
not just full container rebuilds.
Found via nix-cache showing "fingerprint mismatch" after being rebuilt
post-outage; confirmed server was silently exposed to the same bug, just
hadn't restarted since its first pairing. Fixed by declaring
StateDirectory so systemd gives the dynamic user real persistent storage.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
--name (used as pct/qm create's --hostname/--name) defaulted to
$flake_target (e.g. "lxc-nix-cache"), not $host (e.g. "nix-cache"). Since
proxmoxLXC.manageHostName pulls the guest's real networking.hostName
straight from Proxmox's own container config, this silently overrode
host.nix's hostName with a build-type-specific name. Default --name to
--host instead, so the guest's identity matches host.nix regardless of
which platform variant built it.
Found by spinning up a fresh lxc-nix-cache test container and noticing its
hostname was "lxc-nix-cache" instead of "nix-cache".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New build type dedicated to Tailscale exit-node capability, wired up for
linode/proxmox/lxc like every other build type (the lxc variant is the one
actually intended for deployment). Kept separate from the "server" host
rather than bundling exit-node capability onto it.
Trimmed modules/tailscale/exit-node.nix down to pure exit-node behavior:
dropped the old --advertise-routes=${vars.lanCidr} bundling (meaningless
for a Linode-hosted VPS with no path to the LAN), and switched
extraUpFlags -> extraSetFlags. Confirmed against nixpkgs' tailscale.nix
that extraUpFlags is only applied by tailscaled-autoconnect, which itself
only runs when services.tailscale.authKeyFile is set -- nothing in this
repo sets one, so the old flags would never have actually been applied.
extraSetFlags runs unconditionally via tailscaled-set on every boot, so
--advertise-exit-node self-reapplies once the operator has done the
one-time manual `tailscale up` auth.
Verified: all three new targets eval cleanly, nixpkgs-fmt/statix clean,
and a dry-run build of lxc-tailscale-exit-node's tarball resolves its full
closure including tailscaled-set.service.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
virtualisation/proxmox-lxc.nix registers the Nix store DB via a systemd
service, never an activation script -- so neededForUsers sops secrets
(password hashes) and the user-creation step that consumes them never ran
on a real first boot, leaving /etc/shadow stuck with build-time placeholder
entries. boot.postBootCommands looked like the right hook (stage-2-init.sh
does invoke it) but switch-to-configuration behaves unreliably that early,
before systemd itself is up. Fixed with a genuine oneshot systemd service,
gated by ConditionPathExists so it only ever runs once.
Confirmed live via a from-scratch destroy+rebuild+redeploy of the
lxc-nix-cache test container: real password hashes applied automatically,
systemctl is-system-running -> running, zero failed units.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
The real root cause behind the original nix-cache 502, traced all the way
through: modules/installer/host-keys.nix (which NIXOS_HOST_KEYS_DIR=...
--impure actually wires up) is only ever imported by the installer's own
modules/installer/common.nix -- modules/platforms/lxc.nix, which every
real lxc-* host build actually uses, never imported anything like it.
docs/auto-installer.md previously claimed NIXOS_HOST_KEYS_DIR bakes a key
into lxc-* tarballs "the same way it does for the ISO/PXE installer
images" -- that was never actually true; I wrote it without verifying the
mechanism existed for lxc.nix specifically.
In practice this meant every lxc-* container booted with a freshly
self-generated SSH host key that could never match whatever .sops.yaml
actually trusts for that target, so *every* secret -- not just
cache-priv-key -- silently failed to decrypt. No error surfaces in the
boot log for this: the activation step that installs secrets only runs
on a genuinely fresh first activation and silently no-ops once
/run/current-system already exists, so by the time anyone looks the
window has closed. Found by manually invoking sops-install-secrets
directly: "Error getting data key: 0 successful groups required, got 0".
Fixed by giving modules/platforms/lxc.nix the same key-baking mechanism
the installer has, but keyed to its own exact flake target and placing
the key directly at /etc/ssh/ssh_host_ed25519_key (no copy step to stage
for, unlike the installer's /etc/host-keys/ staging area -- an lxc-*
tarball has no install step). The target name comes in via
specialArgs.flakeTarget (new, set by flake.nix's mkTarget) rather than
being read back from config.environment.etc."flake-target" -- reading
that back from within a module that also contributes to
environment.etc is circular (confirmed: "infinite recursion
encountered").
Verified live end-to-end against the real test container (lxc-nix-cache,
VMID 100 on pve.sweet.home): destroyed it, rebuilt the tarball fresh with
the fix, recreated it, and confirmed /run/secrets/ now has all three
secrets this host needs (beszel-token, cache-priv-key, nix-github-token),
nix-serve is active (running), and curl http://localhost/nix-cache-info
succeeds both directly and through nginx.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
exit-node.nix advertised --advertise-exit-node/--advertise-routes but
never enabled IP forwarding, so it wouldn't actually route traffic.
services.tailscale.useRoutingFeatures = "server" is the built-in way to
get the sysctls right without duplicating modules/networking/enable-ip-forwarding.nix.
Also opens the firewall for direct (non-DERP-relayed) peer connections,
and now imports enable-service.nix instead of duplicating `enable = true`,
so "plain tailscale" and "tailscale exit node" are two composable modules.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found and fixed live against a real test container (VMID 100, lxc-nix-cache
on pve.sweet.home) after the previous pct-restore-to-pct-create fix still
produced a container that booted into garbled console output:
1. pct create's own CLI default for --unprivileged is privileged (unlike
the web UI, whose checkbox defaults the other way), but
modules/platforms/lxc.nix sets proxmoxLXC.privileged = false, so the
image assumes it's running unprivileged. Real mismatch -- now passes
--unprivileged 1 explicitly.
2. The actual root cause of the garbled console: modern (v247+) systemd
routinely uses nested user namespaces and credential mounts (even
plain getty units, via LoadCredential=-style mechanisms), which
AppArmor's default LXC confinement denies without --features
nesting=1,keyctl=1. Confirmed via the host's kernel audit log: every
getty unit was crash-looping on a denied /run/credentials/* mount
every ~3s, and core services like nsncd failed userns_create the same
way -- the system never finished activating. Fixed live (pct set +
restart on the running test container) before committing the script
change: systemctl is-system-running went from never completing to
"running" with zero failed units.
3. --memory doesn't touch swap -- confirmed live it silently stayed at
Proxmox's own 512M default with --memory 2048. Now defaults --swap to
whatever --memory resolves to.
docs/auto-installer.md's manual pct create walkthrough gets the same
fixes, with the "why" for each flag, since a human following it by hand
would hit the identical bugs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Two independent problems found while actually running the script:
1. nix build/nix-shell retry each unreachable substituter/builder up to
5x with backoff, per store path -- with nix-cache down this compounds
into minutes of noise. scripts/env.sh gains nix_extra_opts(), which
probes http://nix-cache and nixremote@nix-cache:22 once via plain
curl/TCP (bypassing Nix's own retry logic entirely -- confirmed
nix store ping still retries 5x even with a short connect-timeout)
and exports the decision so create-proxmox-resource.sh and the
sync-host-keys.sh subprocess it shells out to both reuse it instead
of probing independently.
2. The actual failure: "archive contains no configuration file". pct
restore expects a vzdump backup archive with embedded config;
config.system.build.tarball is a plain CT template tarball -- wrong
Proxmox mechanism entirely. Fixed to pct create against it as a vztmpl
template instead, uploaded to /var/lib/vz/template/cache/ rather than
/var/lib/vz/dump/. This same wrong claim had propagated into
docs/auto-installer.md, README.md, and CLAUDE.md from when the script
was first written -- corrected everywhere.
Also: checks for an already-uploaded image on the node (fixed
<flake_target>.tar.xz/.raw naming) before building, skipping build+upload
entirely if found (--force-rebuild to always rebuild).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
sync-host-keys.sh: generates/registers SSH host keys and their
.sops.yaml/secrets/*.yaml recipients for flake targets, idempotently.
--all, <target>, --remove, --regenerate-all-keys, all with --dry-run
(verified zero-side-effect via a sandboxed git-status check across every
mode). Only ever touches anchors with a corresponding host-keys/ file --
&admin and any hand-registered real-host anchor are never listed,
removed, or regenerated. Supersedes running prepare-host-key.sh one host
at a time for any target that already has a flake entry.
create-proxmox-resource.sh: builds a lxc-*/proxmox-* target's
tarball/disk image and creates it on a real Proxmox node, or reconfigures
an existing resource's cores/memory/disk (--modify, always requires
typing the VMID back to confirm). Refuses to create a new resource for a
VMID that already exists, and refuses to duplicate a host identity that
already has a real deployment elsewhere (variables.nix's new
deployedTargets, checked by hostName so it also catches cross-platform
duplicates) unless --allow-duplicate-host is passed. --dry-run throughout.
scripts/env.sh centralizes the Proxmox connection config both scripts
(and future ones) share. Also fixes an unrelated gap found along the way:
proxmox-* Disko image builds write their .raw file straight into the
repo root, and .gitignore never covered it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
nix-serve's secretKeyFile was a manual, undocumented-outside-a-comment
`nix-store --generate-binary-cache-key` step per host -- easy to miss on
a fresh nix-cache instance (as lxc-nix-cache testing just found: systemd
fails the unit with EXIT_CREDENTIALS when LoadCredential can't find the
source file, which nginx then reports as a 502 from clients). It also
can't be regenerated per-host safely: modules/nix-cache/client.nix
hardcodes every client's trust in one specific public key, so every
nix-cache instance has to share the exact same keypair.
Sourced from secrets/nix-cache.yaml's new cache-priv-key entry instead,
via the same sops-nix pattern every other secret in this repo already
uses. Verified the added value derives to the exact public key
modules/nix-cache/client.nix already trusts before committing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Adds nested vars.nfsShares (subpath + mountpoint per dataset, previously
duplicated independently across server.nix's NFS exports, mount-data.nix's
client mounts, docker.nix's tmpfiles rules, traefik's log rotation path,
and hosts/server/host.nix's beszel config), vars.ports (every literal port
in modules/ and hosts/, kept as separate entries per service even where
numbers coincide so changing one can't silently change another), plus
vars.proxmoxImageSize, vars.nixCacheGcMaxAge, vars.traefikLogRotate, and
raspberryPiHost/raspiNfsPath/raspiMountpoint for the Pi's own NFS export.
Also fixes docker.nix/minimal.nix/gui.nix hardcoding the literal "nixos"
username instead of the existing vars.primaryUser, found during the sweep.
system.stateVersion is deliberately left untouched everywhere -- per
NixOS's own docs that value must stay fixed from first install, not
follow any shared variable.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
flake.nix's nixpkgs.url/home-manager.url can't source a version string
from variables.nix -- flake input resolution requires a plain string
literal, confirmed empirically (nix flake metadata errors with
"expected a string or a path but got a thunk" otherwise). This script
is the one-command alternative: bump both release branches in flake.nix
via targeted substitution (never a blind repo-wide replace, so it can't
collide with stateVersion strings elsewhere), with an upstream branch
existence check before writing, plus an optional --tooling flag for
codex-maintenance.sh's separately-pinned nixpkgs-fmt/statix fetch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Linode provisions and sizes /dev/sda (root) and /dev/sdb (swap) itself
as whole, unpartitioned block devices before the OS ever boots.
modules/disko/linode.nix declares them with destroy = false (skips
Disko's wipe stage for these disks entirely) and a bare filesystem/swap
content type matching that existing layout, so re-running it against an
already-provisioned disk only mkfs/mkswaps if blkid shows it isn't
formatted yet -- never repartitions or destroys data.
With every host reachable through the installer menu now carrying a
Disko config, auto-install.sh no longer needs to probe the flake and
branch between `disko --mode destroy,format,mount` and a bind-mount
fallback -- it just always runs Disko.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Both existed only so the installer could boot as an LXC container and
nixos-install some other host from within it, but lxc-* targets are
already excluded from the install menu (nixos-install can't touch its
own running root filesystem), and now have their own direct tarball
path anyway. That left the installer's own LXC form with no real use
case, and packages.all with only two members worth bundling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Both check-nixos.yml workflows (GitHub + Gitea) now call
scripts/codex-maintenance.sh instead of a hand-rolled eval-only loop,
closing a real gap: CI previously enforced none of the secret grep,
nixpkgs-fmt, or statix checks that codex-maintenance.sh already runs
locally — nothing was stopping that from regressing. One script now
backs both, instead of two copies that can drift from each other.
codex-maintenance.sh itself is extended to cover buildable surface
that wasn't validated anywhere before: packages.x86_64-linux.*, plus
config.system.build.tarball (lxc-* hosts) and
config.system.build.diskoImagesScript (proxmox-*, excluding the
installer's own proxmox-lxc target, which has no disko config).
Also:
- scripts/prepare-host-key.sh: dropped the redundant
[path-to-nixos-repo] parameter — it always defaults to the repo the
script itself lives in now, so a second argument never made sense
after the nix-auto-installer migration.
- Removed prepare.sh (dead pre-disko manual parted/mkfs/mkswap
partitioning, fully superseded) and
scripts/create-linode-installer-disk.sh (incomplete draft for an
abandoned dd-via-rescue-mode approach; Linode hosts already deploy
fine through the normal auto-installer flow).
- docs/pxe-boot.md: fixed a stale `nixosConfigurations.pxe-boot` eval
command (pre-refactor flat name, not a real flake attribute
anymore) and added a cross-reference to docs/auto-installer.md.
- CLAUDE.md/README.md: full documentation pass reconciling this
session's changes — modules/installer/, modules/pxe-boot/, the
LXC/Proxmox image-building deployment paths, corrected the
password-hash/SSH-key locations in the safety-rules section (both
had drifted to reference files/paths that no longer exist), and
added session-workflow guidance to prefer targeted host evals over
full-repo sweeps for incremental changes (explicitly scoped to
interactive sessions, not CI).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Adds the proxmox-minimal host's age key (derived from its SSH host
key via scripts/prepare-host-key.sh) as a recipient in .sops.yaml and
re-encrypts secrets/common.yaml for it via sops updatekeys.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
modules/disko/proxmox.nix gains imageSize (20G default) and a
per-host imageName (networking.hostName, so every proxmox-* host
produces a distinctly named image instead of an identical main.raw).
This is the same disko.devices config already used to format a real
disk on install, so it's available for every proxmox-* target with no
per-host changes needed:
nix build .#nixosConfigurations.<host>.config.system.build.diskoImagesScript
sudo ./result --build-memory 2048
docs/proxmox-images.md covers building, host-key pre-seeding via
disko's --pre-format-files (same host-keys/ workflow as the installer
and LXC tarball paths), and the qm import/attach sequence for
deploying the result to Proxmox.
Also fixes a real bug in auto-install.sh found while testing: the
disko confirmation bypass used --yes, which disko's CLI doesn't
recognize at all (the actual flag is --yes-wipe-all-disks) — so the
"skip confirmation" flag was silently a no-op and the interactive
prompt kept appearing regardless.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
LXC hosts (device busy fix):
modules/platforms/lxc.nix now imports nixpkgs' own
virtualisation/proxmox-lxc.nix, giving every lxc-* host a real
config.system.build.tarball output — a directly `pct restore`-able
Proxmox container image. This is the actual bug fix behind the
"cannot remove real root directory: device busy or in use" error:
lxc-* targets were only reachable through nixos-install, which
bind-mounts / onto /mnt for containers (no raw disk to partition)
and then correctly refuses to modify the filesystem it's currently
running on. auto-install.sh's menu now excludes lxc-* targets
entirely (they deploy via nix build + pct restore instead, see
docs/auto-installer.md) — and, on the same reasoning, also excludes
`installer`/`proxmox-lxc`, which are the installer image's own flake
targets, not deployable hosts.
manageHostName = true keeps host.nix's declared hostnames (upstream's
default would let Proxmox's ambient container config win instead);
privileged = false matches how these containers are actually created.
Eval warnings, now zero across all 19 nixosConfigurations + 4 packages:
- Multiple password options (root/nixos in the installer): nixpkgs'
own installer profile sets initialHashedPassword = "" for
passwordless login, conflicting with our explicit hashedPassword.
Force-nulled the upstream option rather than adopting passwordless
login, since this image now also boots over LAN PXE with
PasswordAuthentication enabled.
- boot.zfs.forceImportRoot default value: set explicitly to false
(matching the two places that already did) in
modules/common/configuration.nix and modules/installer/common.nix,
covering every host and the installer alike.
- Deprecated pkgs.system in modules/build-types/gui.nix: switched to
pkgs.stdenv.hostPlatform.system.
All confirmed non-behavioral where it matters: unrelated hosts'
drvPaths are byte-identical to their pre-existing baselines throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Finishes the nix-auto-installer migration: scripts/prepare-host-key.sh
and the local host-keys/ directory (gitignored, private key material,
never committed — moved as plain files, not through git history)
weren't carried over in the initial migration.
Also implements automatic key staging, replacing the manual
scp-after-boot step:
- modules/installer/host-keys.nix reads host-keys/ via
builtins.getEnv, which Nix silently returns as "" under normal
(non---impure) evaluation — the module is a no-op by default, safe
for CI, until explicitly opted into:
NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build .#iso --impure
When built this way every key present gets baked into the image at
/etc/host-keys/, and auto-install.sh installs whichever one matches
the flake target selected at install time — no manual per-host scp.
- This deliberately includes the PXE netboot variant, even though
pxe-boot serves it unauthenticated over LAN HTTP: accepted
explicitly as a reasonable trade-off for a network that sits behind
LAN-only infrastructure, not the open internet. auto-install.sh
still falls back to /root/host-keys (manual scp) if a key isn't
baked in, so images built without --impure keep working exactly as
before.
- docs/auto-installer.md replaces nix-auto-installer's README,
updated for in-repo paths and the new build flow.
Verified: normal `nix eval` (no --impure) evaluates identically across
all 19 nixosConfigurations + 4 packages with zero host-keys/* entries
(CI-unaffected); with --impure + the env var set, all three installer
variants (installer/ISO, proxmox-lxc, pxe) correctly embed every key
in host-keys/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Adds debian@surface's existing SSH key (already used as its admin
key elsewhere in this repo) to vars.remoteBuilderAuthorizedKeys so
nix-cache will accept it as a distributed-build client once deployed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Zero W20 (repeated attribute keys), W10 (empty { ... }: variadic
pattern, use _: instead), and W04 (a = x.a instead of inherit)
warnings remain anywhere in the tree, and nixpkgs-fmt --check is
clean on all 46 .nix files.
Repeated-key merges go as deep as statix actually flags per file
(e.g. boot.loader.* nested under boot.loader = { ... } once the
outer boot.* merge exposed it as its own repeat) — every merge is a
pure attribute-path restructuring with no value changes, verified by
comparing config.system.build.toplevel.drvPath before/after for a
representative host per changed module plus a full 19-host + 4-package
eval sweep.
One indentation slip caught and fixed during this pass: nesting
modules/installer/common.nix's environment.etc."auto-install.sh".text
under an environment = { ... } block initially normalized the
script's shebang/set line indentation, which actually changes the
rendered file (Nix's '' string dedent treats it as real content, not
cosmetic whitespace) — reproduced the original's exact indentation
and reverified the rendered script is byte-identical to before.
modules/services/zfs/auto-mount-volumes.nix picked up formatting too;
worth noting it isn't imported by anything in this flake at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Folds the separate nix-auto-installer repo's build outputs into this
flake so it can build every auto-installer artifact it used to,
negating the need for that repo:
- modules/installer/{common,iso,proxmox-lxc}.nix — migrated from
nix-auto-installer's common.nix/installer.nix/proxmox-lxc.nix.
- flake.nix gains nixosConfigurations.{installer,proxmox-lxc} and
packages.x86_64-linux.{iso,lxc,pxe,all}, matching the original
repo's interface (nix build .#iso / .#lxc / .#pxe / .#all).
- Dropped the live Gitea PAT baked into every installer image via
environment.etc."git-credentials" — gitea.lan.ddnsgeek.com/beatzaplenty/nixos.git
allows anonymous read, so the installer (which only ever reads the
flake, never pushes) doesn't need a credential at all.
- installer_old.nix wasn't migrated — dead code, unreferenced by the
source repo's flake.nix, and carried a second stale leaked token.
- The installer environment's own hardcoded login password hash is
preserved as-is: sops-nix has no stable per-boot host key to derive
an age key from on ephemeral installer media, so it can't reuse the
same per-host secret mechanism the rest of this repo uses without
separate design work.
- vars.adminSshKey / vars.dockerHost-style dedup: the installer's SSH
authorized key and its FLAKE_BASE_URL domain were exact duplicates
of values already in variables.nix / modules/common/configuration.nix,
so both now reference the single source of truth instead.
Verified eval-equivalent for every existing host (drvPath-identical)
and confirmed the migrated auto-install.sh script renders byte-for-byte
identical to the source repo's output.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
CLAUDE.md's "Composition pattern" section still described the
pre-refactor layout (hosts/<host>/configuration.nix as a thin imports
list, hardware-configuration wired in from flake.nix) from before the
platform x build-type matrix landed. Rewrite it to match the current
mkTarget/host.nix architecture and the module moves from the prior
commit. Also fixes docs/nix-cache.md, which referenced a
modules/nix/ path that never existed in this repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Several single-purpose modules sat at modules/ root or in the
services/ catch-all despite the repo's established pattern of one
directory per concern (tailscale/, beszel/, docker/, nix-cache/):
- remote-builder-client.nix -> nix-cache/ (always co-included with
nix-cache/client.nix in flake.nix's mkTarget, same buildType guard)
- set-locale.nix -> common/ (unconditionally imported by
common/configuration.nix already)
- enable-ip-forwarding.nix -> networking/
- rotate-traefik-logs.nix -> traefik/rotate-logs.nix
- services/docker-health-to-gotify.nix and services/nextcloud-cron-job.nix
-> docker/ (both only ever imported by the docker build type, same
as the rest of modules/docker/*)
Pure path moves plus import-path updates in flake.nix,
common/configuration.nix, and build-types/docker.nix — verified
eval-equivalent (drvPath-identical) across representative hosts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
hosts/server/host.nix and hosts/nix-cache/host.nix each hand-rolled the
same sops secret/template/environmentFile wiring for the beszel agent
token, differing only in the sops file path and template name. Factor
it into modules/beszel/host-token.nix ({ name, sopsFile }) so a third
host can adopt it without copy-pasting the boilerplate again. Also
drops two dead, stale commented-out HUB_URL lines left over from
before variables.nix grew a homeDomain var.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Pulls the beszel hub / PVE / PBS ports, the docker-compose host's LAN
name, and the remote-builder client SSH keys out of scattered inline
literals across modules/hosts and into variables.nix as the single
source of truth, matching the existing pattern for other cross-host
references (nixCacheHost, nfsServerHost).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
boot.isContainer disables services.udev, which NetworkManager depends on
to enumerate devices — this left NM unable to reliably manage the
container veth, breaking DHCP-hostname registration in Pi-hole. It also
defaulted networking.useHostResolvConf to true, which assumes a
systemd-nspawn-style resolv.conf bind-mount that real Proxmox LXC doesn't
provide (nixpkgs' own proxmox-lxc.nix module forces this false for the
same reason). Also guard the networkmanager extraGroups membership in the
minimal/gui build-types, since that group only exists when NM is enabled.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
linode-minimal, proxmox-minimal, and lxc-minimal now all share
hosts/nix-minimal/host.nix instead of three separate per-platform
host files with different hostnames — every minimal-buildtype host is
named nix-minimal regardless of which platform it runs on.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
LXC containers share the host kernel — Proxmox starts them by exec'ing
/sbin/init directly in the container's rootfs, no bootloader or initrd
involved at all. Without boot.loader.initScript.enable, that file
isn't wired to launch the current generation, so even a correctly
installed system (see the nix-auto-installer bind-mount fix, same
underlying issue) could still fail to come up after reboot. This is
exactly what nixpkgs' own virtualisation/proxmox-lxc.nix module sets
for the same reason.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
One file (variables.nix) holding every value that was previously
hardcoded and repeated across modules: LAN domain/CIDR, home/tailnet
domains, cross-host references (nix-cache substituter hostname, NFS
server hostname, remote-builder user), PXE/PBS IPs, timezone, and the
primary username.
Wired in via flake.nix's specialArgs (and home-manager's
extraSpecialArgs for the two home.nix files), so any module picks it
up by just adding `vars` to its function arguments — no explicit
import needed. Two hosts (nix-cache, server) now derive their own
networking.hostName from the same variable other hosts use to reach
them, so there's exactly one place to change either identifier.
Purely mechanical: every substituted value matches what was already
there, confirmed by identical toplevel .drv paths for all 17 targets
before and after.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-19 14:51:37 +10:00
193 changed files with 14478 additions and 1265 deletions
Not every combination exists — `pxe-boot` has no `linode` variant, since
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have. The full
list:
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have,
`tor-relay` and `ha-server` currently only exist on `lxc`/`proxmox`, and
`baremetal` currently only exists as `baremetal-gui` (the real gui-host
hardware). The full list:
| Target | Purpose |
| --- | --- |
| `linode-minimal` | Minimal NixOS host profile on a Linode VPS (real, deployed) |
| `proxmox-minimal` | Minimal NixOS host profile on Proxmox (real, deployed — previously the flat `nix-minimal` target) |
| `linode-minimal` | Minimal NixOS host profile on a Linode VPS |
| `proxmox-minimal` | Minimal NixOS host profile on Proxmox — previously the flat `nix-minimal` target |
| `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container |
| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder (`proxmox-nix-cache` is the real, deployed one — previously the flat `nix-cache` target) |
| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host (`proxmox-server` is the real, deployed one — previously the flat `server` target) |
| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack (`proxmox-docker` is the real, deployed one — previously the flat `docker` target) |
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation (`proxmox-gui` is the real, deployed one — previously the flat `nixos` target) |
| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host (`proxmox-pxe-boot` is the real, deployed one — previously the flat `pxe-boot` target) |
| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder — previously the flat `nix-cache` target |
| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack — previously the flat `docker` target |
| `flake.nix` | Flake inputs, the `mkTarget` platform × build-type generator, and `nixosConfigurations` outputs |
| `variables.nix` | Single source of truth for shared values (LAN domain/CIDR, hostnames, timezone, primary username, storage root, NFS share subpaths/mountpoints, service ports, ...) — passed to every module and Home Manager config as the `vars` argument via `specialArgs`/`extraSpecialArgs` |
| `docs/` | Operational notes for cache, builders, lock updates, and boot services |
| `scripts/` | Codex setup and validation helpers |
| `modules/installer/` | Auto-installer environment (ISO, also served as PXE netboot) — see `docs/auto-installer.md` |
| `host-keys/` | Gitignored; only used by the auto-installer environment for pre-seeding SSH host keys before first boot — see `docs/auto-installer.md`. All deployed hosts use clan vars (`vars/per-machine/<target>/openssh/`) instead |
| `vars/per-machine/` | Clan vars: committed, sops-encrypted SSH host keys for all deployed hosts; read by `create-proxmox-resource.sh` at deploy time |
| `docs/` | Operational notes for cache, builders, lock updates, boot services, the auto-installer, and Proxmox image builds |
`scripts/env.sh` in alongside it at `/etc/nixos-installer/env.sh`,
preserving the same relative layout (`installer/auto-install.sh` ->
`../env.sh`) the checked-out repo has, so the script's own
`source ".../env.sh"` line resolves correctly in both places without any
Nix-level templating.
Once running, it:
1. Queries `nixosConfigurations` from this flake over the network (`git+https://<lanDomain>/beatzaplenty/nixos.git`) — this happens at *install* time, not build time, so a generic installer image always sees whatever hosts are currently committed, without needing a rebuild.
2. Presents them as a menu; confirms the choice.
3. Skips the `nix-cache` substituter when installing a `nix-cache` host itself (consistent with that host's own runtime config).
4. Runs `disko --mode destroy,format,mount` (see "Storage" above — every host reachable through this menu has a Disko config, so this is unconditional).
5. Installs the target's SSH host key from `/etc/host-keys` or `/root/host-keys` (see "Host keys" above).
| LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
| Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
| Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink |
| `nix-minimal` | likely `proxmox-minimal` or a platform-less base module | Ambiguous — see Open Questions |
| `nix-cache` | base module consumed by `linode-nix-cache`, `proxmox-nix-cache`, `lxc-nix-cache` | Currently platform-less; needs to become a build-type module, not a standalone target |
| `server` | base module consumed by `linode-server`, `proxmox-server`, `lxc-server` | Same as above |
| `docker` | base module consumed by `linode-docker`, `proxmox-docker`, `lxc-docker` | Confirm docker actually makes sense as an LXC/VM guest build vs. a standalone container image — see Open Questions |
| `pxe-boot` | TBD — may stay a single target rather than a per-platform one | See Open Questions |
| `nixos` | TBD — unclear what this maps to in the new scheme | See Open Questions |
## Open Questions (Claude Code: raise these with me before implementing, don't guess)
1.**`nixos` target** — what is this currently used for (bare metal install, dev
shell, template)? It doesn't obviously map to any of the six build types.
2.**`nix-minimal` vs `linode-minimal`** — are these two different things, or is
`nix-minimal` a leftover/duplicate?
3.**`pxe-boot` and `gui` across all three platforms** — does PXE boot make sense
for an LXC container or a cloud VM (Linode), or is it inherently bare-metal/
network-boot only and should remain a single non-platform target? Does `gui`
make sense inside an LXC container?
4.**`docker` as a build type** — is this "a NixOS host configured to run Docker"
(which would sensibly have linode/proxmox/lxc variants), or "a Docker container
image built by the flake" (which wouldn't take a platform prefix at all, since
it doesn't run on Linode/Proxmox/LXC as a guest OS)? These are structurally
different and change how it should be wired in.
5. Confirm whether all 18 combinations should actually exist, or whether this is
meant to produce only the combinations that are genuinely useful (e.g. maybe no
one needs `lxc-pxe-boot`).
## Implementation approach
1.**Inventory first.** Read the current `flake.nix` and any `nixosConfigurations`/
`modules` structure. Map every existing target to what module(s) it actually
pulls in. Don't assume — confirm against the real file contents.
2.**Separate build-type and platform into their own module directories**, e.g.:
```
modules/build-types/minimal.nix
modules/build-types/nix-cache.nix
modules/build-types/server.nix
modules/build-types/docker.nix
modules/build-types/pxe-boot.nix
modules/build-types/gui.nix
modules/platforms/linode.nix
modules/platforms/proxmox.nix
modules/platforms/lxc.nix
```
Build-type modules should contain only what makes a system "minimal" vs
"server" vs "gui", etc. Platform modules should contain only what's specific
to running as a Linode VM vs Proxmox VM vs LXC container (virtualisation
guest tools, boot method, filesystem/image format, LXC-specific constraints
like no kernel modules, etc).
3. **Generate the target matrix programmatically** in `flake.nix` rather than
hand-writing 18 near-identical `nixosConfigurations` entries — e.g. a small
function that takes a platform name and build-type name, composes the two
modules plus any shared base module, and produces the named output. This
keeps future build types/platforms a one-line addition rather than a copy-paste
job.
4. **Only build combinations we've confirmed make sense** (see Open Questions) —
don't emit all 18 by default if some are structurally invalid.
5. **Preserve existing working configs during the transition.** Don't delete the
old target names until their replacements build successfully — rename/alias
at the end, not the start, so there's no window where the flake is broken.
# Spec: Remove Sensitive Information from NixOS Flake
## Goal
Every secret currently readable in plaintext anywhere in this repo (working tree *and* git history) gets removed, replaced with `sops-nix`-managed encrypted references, and rotated. When this is done, the repo should be safe to make public without exposing anything about the systems it configures.
Treat this as three sequential milestones. Do not start git history rewriting (Milestone 3) until Milestones 1 and 2 are fully verified and the flake still builds. This should be its own branch (`refactor/secrets`) until fully verified, then merged.
---
## Milestone 1 — Audit
Before touching anything, produce a complete inventory. Do not guess at scope — grep the whole tree and the whole history.
1. Run a secret scanner across the working tree and full history. Use both, since they catch different things:
-`gitleaks detect --source . -v --log-opts="--all"` (scans history too)
- TLS cert/key pairs committed under e.g. `secrets/`, `certs/`, `pki/`
- Real name, personal email, home address, or anything in comments/hostnames that maps a machine to your physical identity or network layout (e.g. hostnames like `wayne-desktop`, static LAN IPs, ISP-identifying info)
-`.env` files, `secrets.nix`, `secrets.yaml`, or any file that looks like it was meant to be gitignored but wasn't
3. Produce `secrets-inventory.md` (temporary, delete before finishing) listing: file path, line, secret type, and which host/service it belongs to. This becomes the checklist for Milestone 2 — every row must be either migrated to sops or deleted, with nothing left unaccounted for.
---
## Milestone 2 — Migrate to sops-nix
### 2.1 Set up sops-nix
1. Add the flake input:
```nix
sops-nix.url = "github:Mic92/sops-nix";
sops-nix.inputs.nixpkgs.follows = "nixpkgs";
```
2. Import `sops-nix.nixosModules.sops` into each host's module list (or into a shared `common.nix` if all hosts use it).
3. Generate an age keypair **per host** (not one shared key for everything — a compromised host shouldn't decrypt every other host's secrets):
```
nix-shell -p age --run "age-keygen -o /var/lib/sops-nix/key.txt"
```
Print the public key (`age-keygen -y`) for each host — you'll need it for `.sops.yaml`.
4. Also generate one age key for yourself (your admin workstation) so you can edit secrets without needing to SSH into a host: store it at `~/.config/sops/age/keys.txt`, back it up somewhere outside this repo (password manager, offline). **If this key is lost, every secret encrypted with it is unrecoverable — losing the age key is equivalent to losing the secrets.**
5. Create `.sops.yaml` at the repo root defining creation rules: which age public keys can decrypt which secrets files, keyed by path regex, so e.g. `secrets/hostA.yaml` is decryptable by your admin key + hostA's key, `secrets/hostB.yaml` by your admin key + hostB's key.
### 2.2 Migrate each secret category from the inventory
For each row in `secrets-inventory.md`:
- **Password hashes**: generate hash with `mkpasswd -m sha-512` (or `bcrypt` if your setup wants that), store under `sops.secrets."<name>/hashedPassword"`, reference via `users.users.<name>.hashedPasswordFile = config.sops.secrets."<name>/hashedPassword".path;`. Do not put the *plaintext* password anywhere, only the hash, and only the hash goes into the encrypted sops file.
- **API tokens / auth keys**: move the raw value into the per-host sops YAML, reference in the module via `config.sops.secrets."<service>/token".path` — most NixOS service modules that take a token also accept a `*File` variant (e.g. `environmentFile`, `tokenFile`); use that instead of passing the value directly.
- **Private keys / certs**: move the PEM/key content wholesale into a sops secret, output as a file with appropriate `sops.secrets.<name>.path`, `owner`, `mode`, `restartUnits` so the depending service (sshd, wireguard, nginx) reloads when the secret changes.
- **Personal/identifying info**: this doesn't belong in sops (it's not "secret," it's just information you don't want public). Replace real names/emails with placeholders or move to a small untracked `local.nix` that's `.gitignore`'d and imported conditionally, with a documented template (`local.nix.example`) committed instead.
### 2.3 Verify before moving on
- `nixos-rebuild dry-build --flake .#<host>` succeeds for every host.
- `sudo nixos-rebuild switch --flake .#<host>` on at least one real machine (or a VM) confirms secrets decrypt and services start.
- Confirm decrypted secrets land under `/run/secrets/` (not the Nix store — anything placed in `/nix/store` is world-readable by design, so sops-nix's runtime-only placement is the whole point; double check no module accidentally pulls a secret path into a store-built config file).
- Re-run the grep/scanner sweep from Milestone 1 against the *working tree only* (not history yet) — it should now come back clean.
---
## Milestone 3 — Scrub git history
Do this only after Milestone 2 is merged to your main branch and confirmed working, since it rewrites every commit SHA from the point of the earliest offending commit onward.
**This is destructive and irreversible on your local clone. Back up first:**
- Value-based: for secrets embedded inline in files you're keeping (not deleting the whole file), use `--replace-text` with a file listing each literal secret string to replace with `***REMOVED***`:
```
git filter-repo --replace-text expressions.txt
```
3. After filtering, verify: run the Milestone 1 scanners again against full history (`--log-opts="--all"`). They must come back clean.
4. Force-push the rewritten history:
```
git push origin --force --all
git push origin --force --tags
```
5. **Every other clone of this repo (other machines, WSL instances, CI) must be deleted and re-cloned fresh** — a `git pull` against rewritten history will not work cleanly and risks resurrecting the old commits. Don't try to reconcile old clones; throw them away and re-clone.
6. If this repo has ever been pushed to a public host (GitHub, etc.) or a fork/mirror exists, treat every secret that was ever in history as **permanently compromised regardless of the rewrite** — caches, forks, and Wayback-style archives can retain old commits indefinitely. History scrubbing prevents *future* exposure via `git clone`; it does not undo past exposure.
---
## Milestone 4 — Rotate everything
Because the secrets were exposed in history (even briefly, even in a private repo), the migration is not complete until every credential in the inventory has been **rotated**, not just re-encrypted. Re-encrypting an already-leaked value protects it going forward but doesn't undo the leak.
For each row in the original inventory:
- Password hashes → change the actual account password, regenerate the hash, update the sops file.
- API tokens/auth keys → revoke the old token in the issuing service's dashboard (Cloudflare, Tailscale, backup provider, etc.) and generate a new one.
- SSH/WireGuard private keys → generate new keypairs, update the corresponding public key wherever it's trusted (authorized_keys, peer configs, etc.), retire the old ones.
- TLS certs → reissue if the private key was exposed.
Keep `secrets-inventory.md` open during this step and check off each row as rotated. Delete the file only once every row is checked off — it should not be committed.
---
## Ongoing prevention
Add a pre-commit hook (or a `nix flake check` step) running `gitleaks protect --staged` so a secret can't be committed again by accident. Document in the repo README (briefly) that new secrets go through `sops <file>` to edit, never as plaintext in a tracked file.
---
## Definition of done
- [ ] Milestone 1 inventory complete and reviewed
- [ ] All hosts have per-host age keys; admin key backed up outside the repo
- [ ] Every inventoried secret migrated to sops-nix, referenced via `*File`/`sops.secrets.*.path`, nothing plaintext in the working tree
- [ ] `nixos-rebuild dry-build` and at least one real `switch` verified per host
- [ ] Working-tree scanner sweep clean
- [ ] History rewritten with `git-filter-repo`, force-pushed, full-history scanner sweep clean
- [ ] All other clones deleted and re-cloned from the rewritten history
- [ ] Every credential in the original inventory rotated (not just re-encrypted)
- [ ] Pre-commit secret scanning hook added
- [ ] `secrets-inventory.md` deleted from the working directory (never committed)
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
beatzaplentyandClaude Sonnet 4.6