Adds static IP configuration to every NixOS host in the flake that
has a fixed LAN address, and centralises all network primitives
(IPs, gateway, prefix length, interface names) in variables.nix so
there is one place to update if any of them change.
variables.nix additions:
- lanGateway / lanPrefixLength — LAN gateway and /24 prefix, replacing
every hardcoded 192.168.2.254 / 24 across host files
- lxcLanInterface / vmLanInterface / vmStorageInterface — NIC names for
LXC containers (eth0), Proxmox VMs (ens18), and the HA storage NIC
(ens19), used as attribute keys so changing the name is a one-line edit
- haStoragePrefixLength — /29 for the storage subnet, mirrors haStorageCidr
- Per-host IP variables: nixCacheIp (.224), tailscaleRouterIp (.222),
torRelayIp (.221), serverIp (.226), dockerIp (.225)
host.nix changes:
- tailscale-router, tor-relay, nix-cache, pxe-boot: useDHCP = false,
static address on eth0 (lxcLanInterface), struct-form defaultGateway
(required when using systemd-networkd which LXC containers use)
- server, docker: useDHCP = false, static address on ens18 (vmLanInterface),
struct-form defaultGateway (works for both scripted networking and networkd)
- ha-server-1, ha-server-2: replace hardcoded 192.168.2.254 / 24 / 29
with the new variables; no functional change for these hosts
modules/build-types/pxe-boot.nix:
- Domain-controller kickstart template: replace hardcoded 192.168.2.138
and 192.168.2.254 with vars.domainControllerIp / vars.lanGateway /
vars.lanPrefixLength / vars.homeDomain so the template stays correct
if the DC IP or domain is ever changed again
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
dnsmasq binds port 53 by default even when only proxy DHCP is needed.
systemd-resolved already owns port 53 on the pxe-boot container, so
dnsmasq crashes at startup. Setting port=0 disables the DNS listener
entirely — pxe-boot only needs the proxy DHCP path for iPXE chainloading.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
dnsmasq config was placed inside systemd.services{} block, making Nix
interpret it as systemd.services.dnsmasq.settings (which doesn't exist).
Move to top-level services.dnsmasq and add UDP 67 to firewall for DHCP
proxy mode.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds proxmox-ha-server-1 and proxmox-ha-server-2 as real mkTarget entries
alongside the existing proxmox-server, backed by a new ha-server build type.
New modules
modules/ha/cluster-config.nix — DRBD resource + corosync nodelist sourced
from vars (haServer1Host/Ip, haServer2Host/Ip); resource-only fencing for
production STONITH; HA port firewall rules for DRBD, iSCSI, Corosync, pcsd
modules/build-types/ha-server.nix — imports pacemaker-stack + iscsi-target
+ cluster-config + beszel; NFS exports from vars.haStorageRoot (XFS-over-DRBD
mount); nfs-server.service.wantedBy force-cleared so Pacemaker controls
start/stop on the Active node only
New hosts
hosts/ha-server-{1,2}/host.nix — static IP from vars, unique hostId; sops
secrets (beszel, corosync authkey) are TODOs pending sync-host-keys.sh
variables.nix
haServer1/2Host, haServer1/2Ip, haServerVip, haStorageRoot, haIscsiIqn
ports.haServerDrbd/Iscsi/Corosync{1,2,Crypto}/PacemakerRemoted/Pcsd
scripts/ha/ (migrated + updated from test-lab/ha/)
cluster-init.sh — generates corosync authkey, initialises DRBD/XFS/iSCSI,
creates NFS dataset dirs, configures Pacemaker with DRBD + XFS + iSCSI
+ nfs-server + VIP; STONITH disabled initially (enable separately)
cluster-enable-stonith.sh — enables fence_pve_ssh STONITH after key deploy
fence-pve-ssh.py — Proxmox SSH fence agent (node names updated to ha-server-1/2)
acceptance-tests.sh — T1–T7 production acceptance tests
test-lab/ha/ removed — all Nix config moved to modules/ha/ and
modules/build-types/; scripts moved to scripts/ha/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
All 7 acceptance tests pass on live NixOS 25.11 VMs (VMIDs 200/201 on
pve1). Failover completes in ~5 s with data integrity verified.
modules/ha/pacemaker-stack.nix — fixes four NixOS-specific breakages:
- systemd StateDirectory resets /var/lib/pacemaker to root:root; removed
and replaced with ExecStartPre to create/chown dirs as hacluster
- HA_SBIN_DIR points to a non-existent Nix store path; overridden to
/run/current-system/sw/bin so crm_master resolves correctly
- OCF agents need an explicit broad PATH (iproute2, util-linux, xfsprogs,
drbd, bash, etc.) — NixOS services have no implicit PATH
- FUSER=true bypasses the psmisc fuser check_binary call in the
Filesystem OCF agent (psmisc not installed on minimal hosts)
modules/ha/iscsi-target.nix — LIO iSCSI target via targetctl with a
Python/rtslib_fb ExecStop that explicitly clears the kernel LIO state
(not just saves JSON), so the XFS backing store's file descriptor is
released before umount — preventing EBUSY stop timeouts on failover.
Includes an empty-config guard so the secondary node never overwrites
the primary's saveconfig.json with an empty one.
test-lab/ha/common.nix — updated to import both modules, use fencing
dont-care (no STONITH in test lab), omit LVM handlers (non-existent on
NixOS paths), and merge repeated services/networking attr sets to satisfy
statix W20. test-lab/ha/acceptance-tests.sh — final v4 with crm_standby
fix (pacemaker 3.x API).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Adds an unattended install option to the PXE boot menu that installs
Rocky Linux 9 and configures FreeIPA on the domain-controller.sweet.home
host without any operator interaction after selecting the menu entry.
How it works:
- fetch-rocky-pxeboot.service downloads the Rocky 9 Anaconda pxeboot
kernel and initrd from the Rocky mirror on first pxe-boot deploy
(idempotent, same pattern as fetch-debian-netboot)
- rocky-freeipa.ipxe boots Anaconda with inst.ks pointing at the
hosted Kickstart and net.ifnames=0 biosdevname=0 for stable eth0
- rocky-freeipa.ks (generated, includes vars.adminSshKey) performs:
- Minimal Rocky 9 install with ipa-server + ipa-server-dns
- Static IP 192.168.2.138 via NM connection file written in %post
- /etc/hosts fixed for FreeIPA FQDN requirement
- Random DM + admin passwords generated and saved to
/root/ipa-credentials.txt (chmod 600, never hardcoded)
- freeipa-first-boot.service oneshot enabled to run
ipa-server-install on the first real boot (~20 min)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Jbvxx4xbHVcx1NkK3vtmK
Add nixos@nixos session key so the Claude Code session can SSH into
test VMs directly. Also enable services.qemuGuest.enable so
qm guest exec works as a fallback for key injection.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SOPS_AGE_KEY_FILE was set in hosts/nixos/home.nix sessionVariables with a
literal ~ that Home Manager injects as-is into the environment. In bash,
tilde expansion does not happen inside double-quoted variable references, so
DEFAULT_SOPS_AGE_KEY_FILE resolved to ~/... literally and the -s file-existence
check in ensure_admin_decrypt_key silently failed. The script then generated
a brand-new age key (to ~/... relative to the repo root) while the real admin
key at ~/.config/sops/age/keys.txt went untouched -- making it appear the key
was lost when it was actually still intact.
Fix the home.nix root cause by using config.home.homeDirectory so the path
is fully resolved. Add tilde expansion in ensure_admin_decrypt_key as a
belt-and-suspenders guard for any caller whose environment has the same issue.
Also replace the auto-generate-a-new-key fallback with a hard failure: auto-
generating a new admin key is never useful (it cannot decrypt existing secrets)
and created serious confusion about whether the original key was lost.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the hardcoded hostname check (docker, pxe-boot) with a check
on config.fileSystems: any lxc-* host whose NixOS config declares an
NFS fileSystem entry is automatically made privileged. The script
already reads proxmoxLXC.privileged dynamically via
flake_target_lxc_privileged, so no logic change is needed there —
only the comment is updated to describe the new derivation.
Result: lxc-docker and lxc-pxe-boot (the two with NFS mounts) evaluate
as privileged=true; lxc-nix-cache, lxc-minimal, lxc-server,
lxc-tailscale-router, lxc-tor-relay evaluate as privileged=false.
Any future lxc-* host that declares an NFS mount gets the correct
privilege level for free without a separate manual edit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The kernel's NFS client (FS_USERNS_MOUNT not set) rejects NFS mounts
from inside any unprivileged container's user namespace with EPERM —
AppArmor's mount=nfs feature only whitelists the AppArmor layer; the
VFS-level rejection happens before AppArmor is consulted.
lxc-pxe-boot mounts server.sweet.home:/tank/pxe-boot/images at
/mnt/pxe-images so nginx can serve large ISOs without filling the
container's root disk. Same pattern as lxc-docker (already privileged
for the same reason since it also mounts several NFS shares).
Operator action required: VMID 103 must be recreated as a privileged
container (the UID mapping on disk differs between privileged and
unprivileged; changing it in-place with pct set is unsafe). Rebuild
the tarball and use create-proxmox-resource.sh to replace it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Correct the NFS path from /tank/proxmox/pxe-images to /tank/pxe-boot/images
(matches variables.nix's proxmoxPxeImages.subpath)
- Clarify that LXC uses NFSv3+nolock while VM uses NFSv4.2+automount
- Add explicit note that lxc-pxe-boot needs features: nesting=1,mount=nfs and
why: nesting=1 is required by systemd 260+ for userns/credential isolation
(AppArmor denies userns_create without it), mount=nfs for NFSv3 access.
pct set replaces the whole features string — include both or the container
will fail to boot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Filter non-attrset values from nfsShares in server.nix so the
poolDatasets loop skips the new `options` string entry
- Fix typo proxomoxLxcImages → proxmoxLxcImages in server.nix exports
- Rename proxmoxPxeImages → pxebootImages in mount-pxe-images.nix and
pxe-boot.nix to match the actual key in variables.nix
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Proxmox LXC containers block the sunrpc filesystem (rpc_pipefs) via AppArmor
unless the container has `features: mount=nfs` set. NFSv4 requires rpc_pipefs
for client state management, so the mount fails outright in a default LXC.
Two fixes for the LXC case (config.boot.isContainer):
- Switch from nfsvers=4.2 to nfsvers=3,proto=tcp,nolock,nofail: NFSv3 doesn't
need rpc_pipefs at the protocol level, and nofail keeps boot clean if the
NFS server is unreachable.
- Add ConditionVirtualization=!container to var-lib-nfs-rpc_pipefs.mount via
systemd drop-in: NixOS pulls this unit into nfs-client.target for any NFS
fileSystems entry. With the condition, systemd skips (not fails) the unit in
containers, keeping nfs-client.target green and activation reporting clean.
Proxmox VM hosts (not isContainer) continue to use nfsvers=4.2 with
x-systemd.automount unchanged.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
showmount and the NFSv3 mount protocol need mountd reachable after querying
portmapper on 111; the server was only opening TCP 111 and 2049, causing
clients (e.g. Proxmox GUI NFS storage scan) to time out connecting to
mountd on 20048. Also adds UDP for all three ports — portmapper, nfsd, and
mountd all use both protocols.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds modules/pxe-boot/mount-pxe-images.nix, which mounts
server.sweet.home:/tank/proxmox/pxe-images at /mnt/pxe-images via NFSv4.2
(x-systemd.automount on Proxmox VMs, nofail on LXC containers — same pattern
as docker/mount-data.nix). The pxe-boot build-type now imports this module and
replaces the previous local /srv/pxe/http/images directory rule with an L+
symlink pointing to /mnt/pxe-images, so large images (ISOs, disk images) live
on the NFS share rather than the host's own root disk.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a `fetch-debian-netboot.service` oneshot that downloads the Debian
bookworm netboot kernel and initrd from deb.debian.org on first boot,
stages them under /srv/pxe/http/debian/, and serves them via a generated
debian.ipxe chain script. The service is idempotent — it skips the
download if both files are already present.
Also merges the previously split systemd.tmpfiles.rules and
systemd.services blocks into a single systemd = { ... } attrset to
satisfy statix's repeated-keys lint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Wires beszel-agent into all tailscale-router variants (lxc/linode/proxmox)
by importing enable-agent.nix in the build type and host-token.nix in the
host file. Adds the sops creation rule for secrets/tailscale-router.yaml
(all three platform variants as recipients). The secrets file must be
created manually before deploying — see instructions in PR.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
extraCommands runs after nixos-nat-post is deleted but before it is
re-created, so -A nixos-nat-post silently fails every time. POSTROUTING
is a built-in chain that always exists; target it directly instead.
The -C idempotency check prevents duplicate rules on firewall reloads.
Drop networking.nat.enable -- it was only needed for the sub-chain that
turned out to be the wrong target.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
networking.nat.externalInterface without internalInterfaces creates the
nixos-nat-post chain but inserts no MASQUERADE rule into it — confirmed
by inspecting the live firewall-start script on the deployed host.
Add the rule explicitly via firewall.extraCommands targeting nixos-nat-post,
scoped to LAN source traffic (vars.lanCidr) going out tailscale0.
extraStopCommands removes it on firewall stop.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Without SNAT on tailscale0, Tailscale drops forwarded packets from LAN
source IPs (192.168.2.x) because they are not recognised Tailscale
addresses. With networking.nat.externalInterface = "tailscale0", all
traffic leaving through the Tailscale tunnel is masqueraded to the
router's own Tailscale IP (100.x.x.x), making it indistinguishable from
locally-originated traffic. Conntrack handles the return path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a oneshot systemd service that sets ethtool rx-udp-gro-forwarding on
and rx-gro-list off on the default-route interface at boot, silencing
Tailscale's warning about suboptimal UDP GRO forwarding on subnet routers.
Interface is discovered dynamically via `ip route get` so it works on all
platforms regardless of NIC naming.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--pre-format-files placed the key on the QEMU builder VM's rootfs, not the
target disk. nixos-install chroots into the target and runs sshd-keygen, which
found no key in the chroot and generated a fresh (unregistered) one. sops then
could not decrypt on first boot because the key didn't match .sops.yaml, leaving
both root and nixos with '!' in /etc/shadow even after mutableUsers = false was
set (hashedPasswordFile pointed to paths sops never wrote).
Fix modules/platforms/proxmox.nix to embed the clan SSH host key in
environment.etc via NIXOS_HOST_KEYS_DIR at eval time -- the same pattern
lxc.nix uses. nixos-install's own activation places the key on the target disk,
sshd-keygen finds it already present and skips generation, and sops decrypts
correctly on first boot. Includes the same preserveSshHostKey/restoreSshHostKey
activation scripts as lxc.nix so subsequent nixos-rebuild switch calls (without
NIXOS_HOST_KEYS_DIR) don't remove the key as "obsolete" from environment.etc.
Update create-proxmox-resource.sh: switch VM builds from
./result-<target> --pre-format-files ... --build-memory 2048
to
NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build --impure ... diskoImagesScript
./result-<target> --build-memory 2048
matching the LXC build path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
Three bugs combined to leave every VM build with a shell but no boot disk:
1. The remote build script moved the raw image to /var/lib/vz/import/ before
qm importdisk could use it. If the mv failed (cross-filesystem copy, sudo
path, or any other reason) the remote script exited non-zero -- but the
local script's set -e handling of the SSH heredoc was inconsistent, so
qm create sometimes ran anyway, leaving a diskless VM shell.
Fix: skip the mv entirely. The diskoImagesScript writes <hostname>.raw into
its CWD (the remote repo dir, $out = $PWD at invocation). Import directly
from that path; clean it up after a successful import.
2. The qm importdisk output regex expected "Successfully imported disk as '...'"
but current Proxmox emits "unusedN: successfully imported disk '...'"
(lowercase, no "as"). The grep returned no match and exited 1.
3. The disk_id assignment used $(... | grep ...) without || true inside the
substitution. With set -euo pipefail, a non-zero grep exit aborts the
script before the fallback could run -- so the VM was always left with an
unattached unused0 disk.
Fix: update the primary regex to match the actual PVE format; add || true
inside the substitution so set -e never fires on a grep miss; add a qm
config fallback (scan for unusedN: lines) that works regardless of PVE
output format changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
When a proxmox-* disk image is built, activation runs during the image
build without a valid sops age key (the SSH host key doesn't exist yet),
so root and nixos land in /etc/shadow with locked '!' entries. With the
default mutableUsers = true, update-users-groups.pl preserves existing
shadow entries for accounts that already exist, so hashedPasswordFile is
silently ignored on every subsequent boot — passwords are never fixed.
Setting mutableUsers = false forces update-users-groups.pl to apply
hashedPasswordFile unconditionally on every activation. On first real
boot the sops-decrypted hash is now written regardless of whether the
account already existed in shadow from the image build.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
qm importdisk in QEMU 11.x outputs lowercase "successfully imported disk
as '...'" rather than the capitalised form the original grep expected.
The case mismatch made disk_id always empty, which caused the script to
exit 1 after qm create had already run -- leaving the VM with only an
EFI disk, no scsi0, and boot order still set to net0.
Fix by adding -i (case-insensitive) to the grep. Both the old capitalised
format (where the disk id had an "unused0:" prefix inside the quotes) and
the new lowercase format are handled correctly: the sed strip of unused0:
is preserved for backward compatibility, and the regex result is identical
either way.
Also add an early warning when --disk-size is passed for --type vm: the
flag is LXC-only for create mode and was silently ignored, leaving users
expecting a different size than the proxmoxImageSize in variables.nix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the clan vars migration all keys are in vars/per-machine/, not
host-keys/. Update:
- File header: "existing clan var is never overwritten" (not host-keys/ file)
- Header --remove/--regenerate description: mention clan vars as primary
- usage() --remove, --regenerate-all-keys, --dry-run text
- cmd_remove/cmd_regenerate_all empty-guard messages
- README.md vars/per-machine/ row: "all deployed hosts" (not "LXC hosts")
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
locally_managed_hosts() only scanned host-keys/ (now empty for all
current targets), so --remove and --regenerate-all-keys silently did
nothing. Fix:
- locally_managed_hosts(): also yields targets from
vars/per-machine/*/openssh/ssh_host_ed25519_key/secret, deduped
- cmd_remove: shows [clan-vars] or [host-keys/] label per entry;
deletes vars/per-machine/<target>/openssh/ in addition to host-keys/
- cmd_regenerate_all: same -- removes clan vars dirs before regenerating
Also update CLAUDE.md and README.md to reflect that all flake targets
now use clan vars (not just lxc-*); host-keys/ is only for the
auto-installer's own pre-seeding path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Adds vars/per-machine/<target>/openssh/ for every flake target except
lxc-tor-relay and lxc-nix-cache (already committed). 18 targets recovered
from pve1 host-keys/ backup; lxc-gui and proxmox-minimal have no prior
live deployment and no backup key, so fresh ed25519 keys were generated —
their .sops.yaml anchors were updated to match.
All secrets are admin-only encrypted (matching clan_generate_ssh_key
convention). Age fingerprints verified against .sops.yaml anchors.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
The previous Phase 4 commit had sed-mangled fingerprints for these two
targets (old and new fingerprints concatenated into one line). The correct
new fingerprints are:
- proxmox-minimal: age19m0m7vdfg... (freshly generated, no prior key on pve1)
- lxc-gui: age1rrxqea6q6... (freshly generated, no prior key on pve1)
Re-run sops updatekeys on common.yaml and gui.yaml to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Remove four stale age key anchors from .sops.yaml that correspond to
non-lxc build-type variants that were never deployed (or are now
superseded by their lxc-* counterparts):
&docker → superseded by &lxc-docker (active running host)
&server → superseded by &lxc-server (active running host)
&nix-cache → superseded by &lxc-nix-cache (active running host)
&nix-minimal → superseded by &lxc-minimal (active running host)
Also remove the secrets/docker.yaml creation_rules block entirely since
that file does not exist.
Re-encrypt secrets/common.yaml, secrets/nix-cache.yaml, and
secrets/server.yaml with sops updatekeys to drop the stale recipients.
The four removed keys can no longer decrypt these files.
Update README.md and CLAUDE.md to clarify that deployed lxc-* hosts
now use clan vars (vars/per-machine/<target>/openssh/) rather than the
gitignored host-keys/ directory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
All actively deployed lxc-* hosts now have clan vars. Remove the legacy
scp -pr host-keys/ fallback in sync_remote_host_keys(): instead of
silently copying the gitignored directory, error clearly if no clan var
exists for the target and tell the operator how to generate one.
Also extend the uncommitted-changes check to cover vars/per-machine/ in
addition to .sops.yaml and secrets/, since clan vars must be committed
before the remote build git-pulls them.
Update the script header and sync_remote_host_keys comment to reflect
the new clan-only key flow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Encrypted private key matches the running container's key and the
vars.nixCacheHostKey in variables.nix (no rotation). Age fingerprint
age1ufg390... matches the &lxc-nix-cache anchor in .sops.yaml.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
nixos-lxc-sops-reinstall.service called switch-to-configuration test at
sysinit.target time (DefaultDependencies=false), before D-Bus was up.
D-Bus is required to restart systemd targets after activation scripts
run. The service reported failure on every boot (exit 1: "Failed to open
dbus connection") even though secrets were correctly installed, because
the D-Bus call happens after activation scripts complete.
Move the service to network.target so basic.target (which includes
dbus-broker.service) runs first. Also drop DefaultDependencies=false so
systemd auto-adds After=basic.target. Add SuccessExitStatus=11 to handle
the edge case where switch-to-configuration holds the lock during a
concurrent rebuild (exit 11 = "Could not acquire lock" -- the rebuild's
own activation already installed the secrets, so treating it as success
is correct).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
The key was generated in a prior session but not committed — the clan vars
files existed only in that session's working tree. Recovered the original
private key from pve1's host-keys/ backup (fingerprint age16kqf... matches
the &lxc-tor-relay anchor already in .sops.yaml), re-encrypted for admin
age key only, and stored in the canonical clan vars layout.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Two bugs prevented nixos-rebuild switch from working on lxc-* hosts after
first boot, both confirmed live on a deployed lxc-tor-relay container:
1. Ordering bug: preserveSshHostKey had no explicit deps, so the topological
sort placed it at position 7 — after etc at position 5. By the time it
tried to save the SSH key, etc had already removed it as "obsolete"
(absent from the current generation's environment.etc when built without
NIXOS_HOST_KEYS_DIR). Consolidate all four system.activationScripts entries
into one block and add etc = { deps = ["preserveSshHostKey"]; } and
setupSecrets = { deps = ["restoreSshHostKey"]; } to enforce the correct
save→etc→restore→sops chain.
2. No boot-time secrets: /run/secrets is a tmpfs cleared on every reboot, and
sops-nix does NOT generate a boot-time service in this configuration
(confirmed live: no sops-nix.service in systemctl list-unit-files).
Add nixos-lxc-sops-reinstall.service, modelled after sops-nix's own service
placement (wantedBy/before sysinit.target, DefaultDependencies=false), so
secrets are reinstalled before basic.target on every non-first boot.
ConditionPathExists skips it on first boot; nixos-lxc-first-boot-activate
handles that case.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Replaces the gitignored host-keys/ directory with clan vars as the
authoritative storage for SSH host keys. Keys are now generated as
sops-binary-encrypted clan var files (admin-key only) and checked into
vars/per-machine/<target>/openssh/, eliminating the plaintext private
key that previously had to live outside the repo.
Changes:
- modules/clan/ssh-host-key.nix: clan vars generator for the ed25519
SSH host key pair (neededFor="activation" — not mapped to sops.secrets,
delivered via tarball baking for LXC or --pre-format-files for VMs)
- flake.nix: add clanCore module + required settings to every mkTarget;
deduplicate bundled disko/sops-nix via follows; all 27 hosts eval clean
- flake.lock: updated to reflect the new follows constraints
- scripts/lib/clan-vars.sh: new helper library with
clan_ssh_key_exists / clan_ssh_pubkey_path / clan_decrypt_ssh_key /
clan_generate_ssh_key for use by the provisioning and sync scripts
- scripts/secrets/sync-host-keys.sh: queue_host_sync() now checks clan
vars first; generates via clan_generate_ssh_key if no key exists;
derives age fingerprint from clan pub key for .sops.yaml registration
- scripts/proxmox/create-proxmox-resource.sh: key management simplified
(sync-host-keys.sh now generates the key if missing, so the inline
prepare-host-key.sh call is gone); sync_remote_host_keys() decrypts
the clan key into a temp dir and scps just the two files to the node
when a clan key exists, falling back to the old host-keys/ scp for
any remaining legacy entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Introduces clan-core pinned to its 26.05 release alongside the existing nixpkgs
26.05 input. No host configuration is changed — this is a pure dependency
addition so Phase 2 (per-host vars/secret management migration) has the input
available without a separate flake.lock bump.
clan-core.inputs.nixpkgs.follows = "nixpkgs" keeps a single nixpkgs closure.
sops-nix remains as a flake input; vars layers on top of it rather than
replacing it (clan's sops storage backend still needs sops-nix).
All hosts evaluate cleanly (codex-maintenance.sh --full-check equivalent
triggered by the flake.nix change).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three ordering-related fixes to the Proxmox provisioning flow:
1. prepare-host-key.sh: make idempotent -- if the key already exists, print
a note and exit 0 instead of erroring. The caller (create-proxmox-resource.sh)
already guards standalone calls, but the script itself should be safe to
run directly on a host that was already keyed.
2. create-proxmox-resource.sh: after sync-host-keys.sh updates .sops.yaml /
secrets/, detect uncommitted changes and block with a prompt until the
operator confirms they've committed and pushed. The PVE node's git pull
only picks up committed+pushed state; without this gate, a new host's sops
recipient is missing from the secrets files the image build uses, so the
host can't decrypt secrets on first boot.
3. create-proxmox-resource.sh: add an explicit existence check for the host
key in both the LXC and VM remote build heredocs, before it's passed as
--pre-format-files / NIXOS_HOST_KEYS_DIR input. Gives a clear error
pointing at sync-host-keys.sh instead of a raw `cp: cannot stat` from
disko deep in the build.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The disko images script does `cd "$TMPDIR"` before parsing its arguments,
so relative paths passed to --pre-format-files resolve against the temp
dir instead of the repo root. Use $(pwd) to capture the absolute repo
path before the disko script changes directory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Complete the rename so all identifiers match networking.hostName:
- flake.nix: attribute names and buildType strings
(linode/proxmox/lxc-tailscale-subnet-router → *-tailscale-router)
- modules/build-types/tailscale-subnet-router.nix → tailscale-router.nix
- .sops.yaml: anchor and alias names (age keys unchanged, no re-encrypt needed)
- host-keys/: local gitignored key files renamed (not committed)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If host-keys/<target>_ssh_host_ed25519_key doesn't exist, run
prepare-host-key.sh to generate it before sync-host-keys.sh runs.
Prevents sync-host-keys.sh from hitting its SKIP/exit-1 path (anchor
in .sops.yaml but no local key) and the downstream disko build failure
when --pre-format-files can't find the key file.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The directory name was tailscale-subnet-router but networking.hostName
was already tailscale-router, causing a mismatch that confused scripts
comparing directory paths against actual hostnames. Flake attribute
names (linode/proxmox/lxc-tailscale-subnet-router) and .sops.yaml
anchors are unchanged — they describe the build type, not the hostname.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
systemd-resolved only uses LLMNR for single-label hostnames, never DNS —
same issue mount-data.nix already documented and fixed for NFS by switching
to server.sweet.home. Change the substituter URL, SSH knownHosts, and
remote-builder hostName from bare "nix-cache" to "nix-cache.sweet.home",
and update nginx's virtualHost to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
read exits non-zero when stdin is not a terminal (set -e killed the
script silently). Catch that and emit a clear error pointing to the
SUDO_PASS environment variable rather than crashing with no output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/dev/tty exists as a device node even without a controlling terminal,
so -r/-w tests pass but opening it fails. Plain 'read -r -s' from stdin
is enough: works interactively from a real terminal, and from a non-tty
context the caller should set SUDO_PASS in the environment instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Environments without a controlling terminal (containers, CI agents)
don't have /dev/tty. Try it first for the sudo password prompt, fall
back to plain stdin so the script works in both contexts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Instead of ssh -t (requires PTY on both sides), prompt for the sudo
password once at startup and pipe it to each remote invocation via
sudo -S. This works from any context -- interactive terminal, background
agent, or script -- with no PTY needed on either end.
Also accepts SUDO_PASS from the environment for non-interactive callers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ssh -t won't allocate a PTY when its own stdin is redirected (by a
heredoc). Replaced the heredoc-fed 'sudo bash -s' with commands passed
as an argument string so stdin stays free and -t can properly allocate
a PTY for the sudo password prompt.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
