Commit Graph
622 Commits
Author SHA1 Message Date
rootandClaude Sonnet 5 cafeb8853b Add scripts/rotate-admin-key.sh to automate sops admin key rotation
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m15s
Automates the manual steps sync-host-keys.sh/create-proxmox-resource.sh
print when they bootstrap a fresh, not-yet-trusted age key: verifies a
backed-up key matches the current &admin entry, swaps in a new key, and
re-encrypts every secrets/*.yaml. Explicitly cds into repo_root before any
sops call, since sops resolves .sops.yaml by walking up from cwd rather
than from the target file's path -- confirmed via a scratch-repo test that
running from elsewhere would otherwise silently rotate against the wrong
config.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 11:23:33 +00:00
beatzaplenty 2c2d464503 Merge pull request 'Fix stale documentation: outdated counts, missing build type, spec status' (#9) from fix-stale-docs into main
Check NixOS configurations / eval-hosts (push) Failing after 11m12s
2026-07-20 11:07:43 +00:00
rootandClaude Sonnet 5 5ec7033439 Fix stale documentation: outdated counts, missing build type, spec status
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m26s
Same class of problem as the deployedTargets/README fixes: hand-maintained
prose that drifted from reality and nobody was obligated to update.

- CLAUDE.md: "18 hosts" was a stale hardcoded count (actually 20); reworded
  to not need updating as hosts are added. Also added the missing
  tailscale-exit-node build type to a list that had it everywhere else in
  the file except one bullet.
- AGENTS.md: same missing tailscale-exit-node build type.
- docs/auto-installer.md: the hand-enumerated lxc-* list was missing
  lxc-tailscale-exit-node.
- flake-target-refactor-spec.md: added a "Status: implemented" note so this
  completed historical spec (referenced elsewhere purely for rationale)
  can't be mistaken for an open plan with unresolved Open Questions.
- remove-sensetive-info-refactor.md: the "Definition of done" checklist was
  entirely unchecked despite most of the work being done. Checked off what's
  actually done (sops-nix migration, history scrub just performed, the
  pre-commit gitleaks hook), and left rotation of the GitHub PAT found in
  history explicitly flagged as the one still-open item -- an operator
  action against GitHub, not something this repo can attest to itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 11:06:05 +00:00
beatzaplenty 9133afd444 Merge pull request 'Fix duplicate-host check reporting false SSH failures' (#8) from worktree-fix-duplicate-host-check-exitcode into main
Check NixOS configurations / eval-hosts (push) Failing after 11m36s
2026-07-20 11:00:18 +00:00
rootandClaude Sonnet 5 eeec9ce302 Fix duplicate-host check reporting false SSH failures
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m38s
The remote bash script run over SSH ended with a for-loop whose last
statement was `[[ "$n" == "$target" ]] && echo ...`. When the last
VM/CT checked on the node didn't match --host, that test evaluated
false and became the exit status of the whole remote script (1) --
which the wrapper then misreported as "couldn't reach the node",
even though SSH connectivity and the check itself were both fine.
The actual signal is the script's stdout, not its exit code, so end
it with an explicit exit 0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:55:53 +00:00
beatzaplenty a62c4fc023 Merge pull request 'Stop tracking deployment status in the README Hosts table' (#7) from remove-deploy-status-from-readme into main
Check NixOS configurations / eval-hosts (push) Failing after 11m20s
2026-07-20 10:51:59 +00:00
rootandClaude Sonnet 5 7e51168d1b Deduplicate reusable shell code in scripts/ into scripts/lib/
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m29s
Three chunks of copy-pasted logic were drifting across scripts/*.sh:
- codex-setup.sh and codex-maintenance.sh each carried an identical
  NIX_CONFIG bootstrap + ensure_nix_profile() -> scripts/lib/nix-bootstrap.sh
- sync-host-keys.sh and prepare-host-key.sh each ran the same
  ssh-keygen/ssh-to-age nix-shell invocations -> scripts/lib/ssh-host-keys.sh
  (prepare-host-key.sh now also calls env.sh's nix_extra_opts before using
  them, closing a gap where it alone skipped the nix-cache reachability
  check env.sh exists for)
- the "list nixosConfigurations attrNames" / "get one target's hostName"
  nix eval pattern was repeated across codex-setup.sh, codex-maintenance.sh,
  sync-host-keys.sh and create-proxmox-resource.sh (the latter twice, in
  its own --list and --host lookup) -> scripts/lib/nix-eval.sh, which also
  centralizes the --no-use-registries --no-accept-flake-config flag pair
  used on every such call

Verified against the real flake/node config (nix is available here):
create-proxmox-resource.sh --list for both --type lxc/vm, a full
--dry-run create, and prepare-host-key.sh generating and cleaning up a
real key/age-pubkey pair.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:35:49 +00:00
rootandClaude Sonnet 5 97ede62f6d Stop tracking deployment status in the Hosts table
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m24s
Same problem as the deployedTargets removal, just in markdown instead of
Nix: which variant of a buildtype is actually deployed is live
infrastructure state, and a committed table can't stay accurate as that
changes -- it already required a manual edit on every migration and had
drifted before. Keep only what doesn't rot: what each target is for, and
stable naming history. Point at the live node / /etc/flake-target instead
for actual deployment status.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:18:33 +00:00
beatzaplenty ab5206b1c7 Merge pull request 'Replace duplicate-host check with live Proxmox query; drop deployedTargets' (#6) from fix-duplicate-host-self-match into main
Check NixOS configurations / eval-hosts (push) Failing after 12m7s
2026-07-20 10:12:59 +00:00
rootandClaude Sonnet 5 2041557ab3 Replace the duplicate-host check with a live Proxmox query, drop deployedTargets
variables.nix's deployedTargets was a manually-maintained list with no
enforcement keeping it in sync with reality -- it caused two separate
false refusals in a row (naming a VM as deployed well after it had been
destroyed, then matching a target against itself once the list was
"corrected"). Static files can't track whether a resource still actually
exists.

create-proxmox-resource.sh's duplicate-host guard now queries the
Proxmox node directly (qm/pct's own name/hostname config, matched
against --host) instead. Also fixes a gap in that live check: it
originally swallowed ssh failures and would have silently treated "can't
reach the node" the same as "checked, nothing there" -- it now refuses
instead of guessing when the node can't be reached.

deployedTargets is removed entirely from variables.nix since nothing
else in the repo consumed it once this script no longer does; README.md's
Hosts table remains the sole source of truth for "(real, deployed)"
status. CLAUDE.md and the script's own --help/comments updated to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:03:00 +00:00
rootandClaude Sonnet 5 2fd483697b Don't refuse recreating the canonical already-deployed target itself
The duplicate-host check in create-proxmox-resource.sh compared by
hostName only, so it fired even when the target being created was
exactly the one variables.nix's deployedTargets already names (e.g.
rebuilding lxc-nix-cache after destroying its old container to pick up
new sops secrets) -- there's no other machine at risk of an identity
collision in that case, just the normal redeploy workflow. Skip the
check when dt == flake_target; the later VMID-existence check still
guards against clobbering a resource that's actually live on the node.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 09:49:31 +00:00
beatzaplenty 89186b0dee Merge pull request 'Track nix-cache real deployment as lxc-nix-cache, not proxmox-nix-cache' (#5) from worktree-nix-cache-lxc-migration into main 2026-07-20 09:46:41 +00:00
rootandClaude Sonnet 5 8e3606cbd3 Track nix-cache's real deployment as lxc-nix-cache, not proxmox-nix-cache
The old proxmox-nix-cache VM was destroyed and nix-cache is being
redeployed as an LXC container going forward. Without this update,
create-proxmox-resource.sh's duplicate-host check (which only reads this
static list, not live Proxmox state) kept refusing to create
lxc-nix-cache even though nothing named nix-cache actually exists on the
node anymore.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 09:40:03 +00:00
beatzaplenty a18dfb0127 Merge pull request 'Trust nix-cache's SSH host key declaratively on remote-builder clients' (#4) from worktree-magical-cooking-book into main 2026-07-20 07:26:18 +00:00
beatzaplentyandClaude Sonnet 5 75f1342339 Declaratively trust nix-cache's SSH host key on remote-builder clients
Distributed builds failed with "Host key verification failed" on any
client that had never manually SSH'd to nix-cache before, since
nothing populated root's known_hosts for it. Wire nix-cache's host
public key into programs.ssh.knownHosts via a new vars.nixCacheHostKey
so every client picks it up automatically on rebuild.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 17:21:48 +10:00
beatzaplenty 36ba99c9a1 Merge pull request 'Add buildImage shell function for building lxc-* tarballs with host keys' (#3) from worktree-fizzy-juggling-sedgewick into main
Reviewed-on: #3
2026-07-20 07:13:54 +00:00
beatzaplentyandClaude Sonnet 5 0cd8f15b48 Add buildImage shell function for building lxc-* tarballs with host keys
lxc-* hosts need NIXOS_HOST_KEYS_DIR + --impure to bake in a pre-seeded
SSH host key, otherwise sops-nix's .sops.yaml recipient never matches
and every secret permanently fails to decrypt on first boot. That
invocation is easy to forget, so wrap it as `buildImage <flake-target>`
alongside the existing Switch-nix/Test-nix helpers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 17:12:16 +10:00
beatzaplenty 8613b93fa8 Merge pull request 'Fix nix_extra_opts falsely reporting nix-cache's SSH remote builder down' (#2) from fix-nix-cache-probe-retry into main
Reviewed-on: #2
2026-07-20 07:02:14 +00:00
beatzaplentyandClaude Sonnet 5 20f9475a7d Fix nix_extra_opts falsely reporting nix-cache's SSH remote builder down
The reachability check used `cat < /dev/tcp/${NIX_CACHE_HOST}/22`, which
blocks forever reading for EOF that never comes -- sshd sends its banner
and then holds the connection open waiting for the client to speak next.
Every single check hit the 3s timeout and reported "unreachable"
unconditionally, regardless of whether the remote builder was actually up.
Confirmed live: a plain TCP connect (`exec 3<>/dev/tcp/...`, no read)
returns in ~60ms against a healthy nix-cache instead of always timing out.

Fixing that exposed a second, previously-dormant bug: `printf -v
NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"` on a genuinely empty NIX_OPTS array
still runs one format pass and yields the literal `'' ` rather than an
empty string. A subprocess (e.g. sync-host-keys.sh) reusing this
process's decision via `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` then rebuilt
a 1-element array holding an empty string instead of a 0-element array,
which broke `nix-shell "${NIX_OPTS[@]}" -p <pkg>` with a bogus positional
argument the moment NIX_OPTS was legitimately empty (nix-cache reachable)
-- something the first bug had made impossible to ever hit before.

Also adds a couple of retries (1s apart) to both checks as a secondary
safety net against genuine multi-second blips, on top of fixing the
checks themselves.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:57:43 +10:00
beatzaplenty 6babb3eec5 Merge pull request 'Fix create-proxmox-resource.sh --dry-run hiding nix-cache probe results' (#1) from worktree-starry-painting-whistle into main
Reviewed-on: #1
2026-07-20 06:50:40 +00:00
beatzaplentyandClaude Sonnet 5 33730e6ccf Fix create-proxmox-resource.sh --dry-run hiding nix-cache probe results
The tarball/disko-image build previews were hardcoded strings that never
included ${NIX_OPTS[@]}, so --dry-run always showed the same "would build"
command whether nix-cache's substituter/remote-builder got disabled by
nix_extra_opts's reachability probe or not -- the actual (non-dry-run)
build commands already applied it correctly, only the preview lied.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:36:14 +10:00
beatzaplentyandClaude Sonnet 5 65f89806cb Fix beszel-agent losing its hub-pairing fingerprint on every restart
services.beszel.agent runs under DynamicUser=true with ProtectSystem =
"strict" and no StateDirectory, so /var/lib/beszel-agent -- where the
agent persists the fingerprint that locks its hub pairing to this
machine (github.com/henrygd/beszel/discussions/1542) -- was never
actually writable. Every restart silently failed to persist it and
regenerated a fresh one in memory, permanently desyncing from whatever
the hub had on record after the very first successful pairing. Affects
every host importing modules/beszel/enable-agent.nix (nix-cache, server),
not just full container rebuilds.

Found via nix-cache showing "fingerprint mismatch" after being rebuilt
post-outage; confirmed server was silently exposed to the same bug, just
hadn't restarted since its first pairing. Fixed by declaring
StateDirectory so systemd gives the dynamic user real persistent storage.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:59:24 +10:00
beatzaplentyandClaude Sonnet 5 c3007097a6 Fix create-proxmox-resource.sh defaulting hostname to the flake target
--name (used as pct/qm create's --hostname/--name) defaulted to
$flake_target (e.g. "lxc-nix-cache"), not $host (e.g. "nix-cache"). Since
proxmoxLXC.manageHostName pulls the guest's real networking.hostName
straight from Proxmox's own container config, this silently overrode
host.nix's hostName with a build-type-specific name. Default --name to
--host instead, so the guest's identity matches host.nix regardless of
which platform variant built it.

Found by spinning up a fresh lxc-nix-cache test container and noticing its
hostname was "lxc-nix-cache" instead of "nix-cache".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:19:35 +10:00
beatzaplentyandClaude Sonnet 5 9724babcea Add tailscale-exit-node build type across all three platforms
New build type dedicated to Tailscale exit-node capability, wired up for
linode/proxmox/lxc like every other build type (the lxc variant is the one
actually intended for deployment). Kept separate from the "server" host
rather than bundling exit-node capability onto it.

Trimmed modules/tailscale/exit-node.nix down to pure exit-node behavior:
dropped the old --advertise-routes=${vars.lanCidr} bundling (meaningless
for a Linode-hosted VPS with no path to the LAN), and switched
extraUpFlags -> extraSetFlags. Confirmed against nixpkgs' tailscale.nix
that extraUpFlags is only applied by tailscaled-autoconnect, which itself
only runs when services.tailscale.authKeyFile is set -- nothing in this
repo sets one, so the old flags would never have actually been applied.
extraSetFlags runs unconditionally via tailscaled-set on every boot, so
--advertise-exit-node self-reapplies once the operator has done the
one-time manual `tailscale up` auth.

Verified: all three new targets eval cleanly, nixpkgs-fmt/statix clean,
and a dry-run build of lxc-tailscale-exit-node's tarball resolves its full
closure including tailscaled-set.service.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 13:38:39 +10:00
beatzaplentyandClaude Sonnet 5 7055bcdb97 Fix lxc-* hosts never completing first-boot user/secrets activation
virtualisation/proxmox-lxc.nix registers the Nix store DB via a systemd
service, never an activation script -- so neededForUsers sops secrets
(password hashes) and the user-creation step that consumes them never ran
on a real first boot, leaving /etc/shadow stuck with build-time placeholder
entries. boot.postBootCommands looked like the right hook (stage-2-init.sh
does invoke it) but switch-to-configuration behaves unreliably that early,
before systemd itself is up. Fixed with a genuine oneshot systemd service,
gated by ConditionPathExists so it only ever runs once.

Confirmed live via a from-scratch destroy+rebuild+redeploy of the
lxc-nix-cache test container: real password hashes applied automatically,
systemctl is-system-running -> running, zero failed units.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 13:38:21 +10:00
beatzaplentyandClaude Sonnet 5 d973da487c Fix lxc-* hosts having no host-key pre-seeding mechanism at all
The real root cause behind the original nix-cache 502, traced all the way
through: modules/installer/host-keys.nix (which NIXOS_HOST_KEYS_DIR=...
--impure actually wires up) is only ever imported by the installer's own
modules/installer/common.nix -- modules/platforms/lxc.nix, which every
real lxc-* host build actually uses, never imported anything like it.
docs/auto-installer.md previously claimed NIXOS_HOST_KEYS_DIR bakes a key
into lxc-* tarballs "the same way it does for the ISO/PXE installer
images" -- that was never actually true; I wrote it without verifying the
mechanism existed for lxc.nix specifically.

In practice this meant every lxc-* container booted with a freshly
self-generated SSH host key that could never match whatever .sops.yaml
actually trusts for that target, so *every* secret -- not just
cache-priv-key -- silently failed to decrypt. No error surfaces in the
boot log for this: the activation step that installs secrets only runs
on a genuinely fresh first activation and silently no-ops once
/run/current-system already exists, so by the time anyone looks the
window has closed. Found by manually invoking sops-install-secrets
directly: "Error getting data key: 0 successful groups required, got 0".

Fixed by giving modules/platforms/lxc.nix the same key-baking mechanism
the installer has, but keyed to its own exact flake target and placing
the key directly at /etc/ssh/ssh_host_ed25519_key (no copy step to stage
for, unlike the installer's /etc/host-keys/ staging area -- an lxc-*
tarball has no install step). The target name comes in via
specialArgs.flakeTarget (new, set by flake.nix's mkTarget) rather than
being read back from config.environment.etc."flake-target" -- reading
that back from within a module that also contributes to
environment.etc is circular (confirmed: "infinite recursion
encountered").

Verified live end-to-end against the real test container (lxc-nix-cache,
VMID 100 on pve.sweet.home): destroyed it, rebuilt the tarball fresh with
the fix, recreated it, and confirmed /run/secrets/ now has all three
secrets this host needs (beszel-token, cache-priv-key, nix-github-token),
nix-serve is active (running), and curl http://localhost/nix-cache-info
succeeds both directly and through nginx.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 12:29:01 +10:00
beatzaplentyandClaude Sonnet 5 c939454983 Fix tailscale exit-node module: enable routing, compose on enable-service
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m52s
exit-node.nix advertised --advertise-exit-node/--advertise-routes but
never enabled IP forwarding, so it wouldn't actually route traffic.
services.tailscale.useRoutingFeatures = "server" is the built-in way to
get the sysctls right without duplicating modules/networking/enable-ip-forwarding.nix.
Also opens the firewall for direct (non-DERP-relayed) peer connections,
and now imports enable-service.nix instead of duplicating `enable = true`,
so "plain tailscale" and "tailscale exit node" are two composable modules.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 12:24:18 +10:00
beatzaplentyandClaude Sonnet 5 274d54a774 Fix LXC container creation: unprivileged, nesting/keyctl, swap sizing
Found and fixed live against a real test container (VMID 100, lxc-nix-cache
on pve.sweet.home) after the previous pct-restore-to-pct-create fix still
produced a container that booted into garbled console output:

1. pct create's own CLI default for --unprivileged is privileged (unlike
   the web UI, whose checkbox defaults the other way), but
   modules/platforms/lxc.nix sets proxmoxLXC.privileged = false, so the
   image assumes it's running unprivileged. Real mismatch -- now passes
   --unprivileged 1 explicitly.

2. The actual root cause of the garbled console: modern (v247+) systemd
   routinely uses nested user namespaces and credential mounts (even
   plain getty units, via LoadCredential=-style mechanisms), which
   AppArmor's default LXC confinement denies without --features
   nesting=1,keyctl=1. Confirmed via the host's kernel audit log: every
   getty unit was crash-looping on a denied /run/credentials/* mount
   every ~3s, and core services like nsncd failed userns_create the same
   way -- the system never finished activating. Fixed live (pct set +
   restart on the running test container) before committing the script
   change: systemctl is-system-running went from never completing to
   "running" with zero failed units.

3. --memory doesn't touch swap -- confirmed live it silently stayed at
   Proxmox's own 512M default with --memory 2048. Now defaults --swap to
   whatever --memory resolves to.

docs/auto-installer.md's manual pct create walkthrough gets the same
fixes, with the "why" for each flag, since a human following it by hand
would hit the identical bugs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 11:33:53 +10:00
beatzaplentyandClaude Sonnet 5 ad274d99fb Fix nix-cache retry storms and lxc creation in create-proxmox-resource.sh
Two independent problems found while actually running the script:

1. nix build/nix-shell retry each unreachable substituter/builder up to
   5x with backoff, per store path -- with nix-cache down this compounds
   into minutes of noise. scripts/env.sh gains nix_extra_opts(), which
   probes http://nix-cache and nixremote@nix-cache:22 once via plain
   curl/TCP (bypassing Nix's own retry logic entirely -- confirmed
   nix store ping still retries 5x even with a short connect-timeout)
   and exports the decision so create-proxmox-resource.sh and the
   sync-host-keys.sh subprocess it shells out to both reuse it instead
   of probing independently.

2. The actual failure: "archive contains no configuration file". pct
   restore expects a vzdump backup archive with embedded config;
   config.system.build.tarball is a plain CT template tarball -- wrong
   Proxmox mechanism entirely. Fixed to pct create against it as a vztmpl
   template instead, uploaded to /var/lib/vz/template/cache/ rather than
   /var/lib/vz/dump/. This same wrong claim had propagated into
   docs/auto-installer.md, README.md, and CLAUDE.md from when the script
   was first written -- corrected everywhere.

Also: checks for an already-uploaded image on the node (fixed
<flake_target>.tar.xz/.raw naming) before building, skipping build+upload
entirely if found (--force-rebuild to always rebuild).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 11:05:26 +10:00
beatzaplentyandClaude Sonnet 5 bd8d93d890 Add sync-host-keys.sh and create-proxmox-resource.sh
sync-host-keys.sh: generates/registers SSH host keys and their
.sops.yaml/secrets/*.yaml recipients for flake targets, idempotently.
--all, <target>, --remove, --regenerate-all-keys, all with --dry-run
(verified zero-side-effect via a sandboxed git-status check across every
mode). Only ever touches anchors with a corresponding host-keys/ file --
&admin and any hand-registered real-host anchor are never listed,
removed, or regenerated. Supersedes running prepare-host-key.sh one host
at a time for any target that already has a flake entry.

create-proxmox-resource.sh: builds a lxc-*/proxmox-* target's
tarball/disk image and creates it on a real Proxmox node, or reconfigures
an existing resource's cores/memory/disk (--modify, always requires
typing the VMID back to confirm). Refuses to create a new resource for a
VMID that already exists, and refuses to duplicate a host identity that
already has a real deployment elsewhere (variables.nix's new
deployedTargets, checked by hostName so it also catches cross-platform
duplicates) unless --allow-duplicate-host is passed. --dry-run throughout.

scripts/env.sh centralizes the Proxmox connection config both scripts
(and future ones) share. Also fixes an unrelated gap found along the way:
proxmox-* Disko image builds write their .raw file straight into the
repo root, and .gitignore never covered it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 10:31:06 +10:00
beatzaplentyandClaude Sonnet 5 53b9a64826 Move nix-cache's binary cache signing key into sops
nix-serve's secretKeyFile was a manual, undocumented-outside-a-comment
`nix-store --generate-binary-cache-key` step per host -- easy to miss on
a fresh nix-cache instance (as lxc-nix-cache testing just found: systemd
fails the unit with EXIT_CREDENTIALS when LoadCredential can't find the
source file, which nginx then reports as a 502 from clients). It also
can't be regenerated per-host safely: modules/nix-cache/client.nix
hardcodes every client's trust in one specific public key, so every
nix-cache instance has to share the exact same keypair.

Sourced from secrets/nix-cache.yaml's new cache-priv-key entry instead,
via the same sops-nix pattern every other secret in this repo already
uses. Verified the added value derives to the exact public key
modules/nix-cache/client.nix already trusts before committing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 09:49:30 +10:00
beatzaplentyandClaude Sonnet 5 0ba837817e Expand variables.nix: NFS shares, ports, image size, GC/rotation, Pi host
Adds nested vars.nfsShares (subpath + mountpoint per dataset, previously
duplicated independently across server.nix's NFS exports, mount-data.nix's
client mounts, docker.nix's tmpfiles rules, traefik's log rotation path,
and hosts/server/host.nix's beszel config), vars.ports (every literal port
in modules/ and hosts/, kept as separate entries per service even where
numbers coincide so changing one can't silently change another), plus
vars.proxmoxImageSize, vars.nixCacheGcMaxAge, vars.traefikLogRotate, and
raspberryPiHost/raspiNfsPath/raspiMountpoint for the Pi's own NFS export.

Also fixes docker.nix/minimal.nix/gui.nix hardcoding the literal "nixos"
username instead of the existing vars.primaryUser, found during the sweep.

system.stateVersion is deliberately left untouched everywhere -- per
NixOS's own docs that value must stay fixed from first install, not
follow any shared variable.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 08:27:37 +10:00
beatzaplentyandClaude Sonnet 5 80f86b086b Add scripts/bump-nixpkgs-release.sh
flake.nix's nixpkgs.url/home-manager.url can't source a version string
from variables.nix -- flake input resolution requires a plain string
literal, confirmed empirically (nix flake metadata errors with
"expected a string or a path but got a thunk" otherwise). This script
is the one-command alternative: bump both release branches in flake.nix
via targeted substitution (never a blind repo-wide replace, so it can't
collide with stateVersion strings elsewhere), with an upstream branch
existence check before writing, plus an optional --tooling flag for
codex-maintenance.sh's separately-pinned nixpkgs-fmt/statix fetch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 08:04:44 +10:00
beatzaplentyandClaude Sonnet 5 a351cbcf80 Give linode-* hosts a real Disko config, simplify auto-install.sh
Linode provisions and sizes /dev/sda (root) and /dev/sdb (swap) itself
as whole, unpartitioned block devices before the OS ever boots.
modules/disko/linode.nix declares them with destroy = false (skips
Disko's wipe stage for these disks entirely) and a bare filesystem/swap
content type matching that existing layout, so re-running it against an
already-provisioned disk only mkfs/mkswaps if blkid shows it isn't
formatted yet -- never repartitions or destroys data.

With every host reachable through the installer menu now carrying a
Disko config, auto-install.sh no longer needs to probe the flake and
branch between `disko --mode destroy,format,mount` and a bind-mount
fallback -- it just always runs Disko.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 08:04:34 +10:00
beatzaplentyandClaude Sonnet 5 2aa625d566 Remove packages.all bundle and the installer's own proxmox-lxc target
Both existed only so the installer could boot as an LXC container and
nixos-install some other host from within it, but lxc-* targets are
already excluded from the install menu (nixos-install can't touch its
own running root filesystem), and now have their own direct tarball
path anyway. That left the installer's own LXC form with no real use
case, and packages.all with only two members worth bundling.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 08:04:08 +10:00
beatzaplentyandClaude Sonnet 5 288835db29 Consolidate CI on codex-maintenance.sh; clean up dead scripts and docs
Both check-nixos.yml workflows (GitHub + Gitea) now call
scripts/codex-maintenance.sh instead of a hand-rolled eval-only loop,
closing a real gap: CI previously enforced none of the secret grep,
nixpkgs-fmt, or statix checks that codex-maintenance.sh already runs
locally — nothing was stopping that from regressing. One script now
backs both, instead of two copies that can drift from each other.

codex-maintenance.sh itself is extended to cover buildable surface
that wasn't validated anywhere before: packages.x86_64-linux.*, plus
config.system.build.tarball (lxc-* hosts) and
config.system.build.diskoImagesScript (proxmox-*, excluding the
installer's own proxmox-lxc target, which has no disko config).

Also:
- scripts/prepare-host-key.sh: dropped the redundant
  [path-to-nixos-repo] parameter — it always defaults to the repo the
  script itself lives in now, so a second argument never made sense
  after the nix-auto-installer migration.
- Removed prepare.sh (dead pre-disko manual parted/mkfs/mkswap
  partitioning, fully superseded) and
  scripts/create-linode-installer-disk.sh (incomplete draft for an
  abandoned dd-via-rescue-mode approach; Linode hosts already deploy
  fine through the normal auto-installer flow).
- docs/pxe-boot.md: fixed a stale `nixosConfigurations.pxe-boot` eval
  command (pre-refactor flat name, not a real flake attribute
  anymore) and added a cross-reference to docs/auto-installer.md.
- CLAUDE.md/README.md: full documentation pass reconciling this
  session's changes — modules/installer/, modules/pxe-boot/, the
  LXC/Proxmox image-building deployment paths, corrected the
  password-hash/SSH-key locations in the safety-rules section (both
  had drifted to reference files/paths that no longer exist), and
  added session-workflow guidance to prefer targeted host evals over
  full-repo sweeps for incremental changes (explicitly scoped to
  interactive sessions, not CI).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 07:02:23 +10:00
beatzaplentyandClaude Sonnet 5 559c538a3d Register sops age key for proxmox-minimal
Adds the proxmox-minimal host's age key (derived from its SSH host
key via scripts/prepare-host-key.sh) as a recipient in .sops.yaml and
re-encrypts secrets/common.yaml for it via sops updatekeys.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 07:01:56 +10:00
beatzaplentyandClaude Sonnet 5 feee2f1679 Add Proxmox VM disk-image building; fix disko confirmation bypass
modules/disko/proxmox.nix gains imageSize (20G default) and a
per-host imageName (networking.hostName, so every proxmox-* host
produces a distinctly named image instead of an identical main.raw).
This is the same disko.devices config already used to format a real
disk on install, so it's available for every proxmox-* target with no
per-host changes needed:

  nix build .#nixosConfigurations.<host>.config.system.build.diskoImagesScript
  sudo ./result --build-memory 2048

docs/proxmox-images.md covers building, host-key pre-seeding via
disko's --pre-format-files (same host-keys/ workflow as the installer
and LXC tarball paths), and the qm import/attach sequence for
deploying the result to Proxmox.

Also fixes a real bug in auto-install.sh found while testing: the
disko confirmation bypass used --yes, which disko's CLI doesn't
recognize at all (the actual flag is --yes-wipe-all-disks) — so the
"skip confirmation" flag was silently a no-op and the interactive
prompt kept appearing regardless.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 07:01:33 +10:00
beatzaplentyandClaude Sonnet 5 120240f14a Fix LXC deployment path and clean up remaining eval warnings
LXC hosts (device busy fix):

modules/platforms/lxc.nix now imports nixpkgs' own
virtualisation/proxmox-lxc.nix, giving every lxc-* host a real
config.system.build.tarball output — a directly `pct restore`-able
Proxmox container image. This is the actual bug fix behind the
"cannot remove real root directory: device busy or in use" error:
lxc-* targets were only reachable through nixos-install, which
bind-mounts / onto /mnt for containers (no raw disk to partition)
and then correctly refuses to modify the filesystem it's currently
running on. auto-install.sh's menu now excludes lxc-* targets
entirely (they deploy via nix build + pct restore instead, see
docs/auto-installer.md) — and, on the same reasoning, also excludes
`installer`/`proxmox-lxc`, which are the installer image's own flake
targets, not deployable hosts.

manageHostName = true keeps host.nix's declared hostnames (upstream's
default would let Proxmox's ambient container config win instead);
privileged = false matches how these containers are actually created.

Eval warnings, now zero across all 19 nixosConfigurations + 4 packages:

- Multiple password options (root/nixos in the installer): nixpkgs'
  own installer profile sets initialHashedPassword = "" for
  passwordless login, conflicting with our explicit hashedPassword.
  Force-nulled the upstream option rather than adopting passwordless
  login, since this image now also boots over LAN PXE with
  PasswordAuthentication enabled.
- boot.zfs.forceImportRoot default value: set explicitly to false
  (matching the two places that already did) in
  modules/common/configuration.nix and modules/installer/common.nix,
  covering every host and the installer alike.
- Deprecated pkgs.system in modules/build-types/gui.nix: switched to
  pkgs.stdenv.hostPlatform.system.

All confirmed non-behavioral where it matters: unrelated hosts'
drvPaths are byte-identical to their pre-existing baselines throughout.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 05:58:56 +10:00
beatzaplentyandClaude Sonnet 5 b0ccbb1162 Migrate host-key tooling from nix-auto-installer, bake keys into images
Finishes the nix-auto-installer migration: scripts/prepare-host-key.sh
and the local host-keys/ directory (gitignored, private key material,
never committed — moved as plain files, not through git history)
weren't carried over in the initial migration.

Also implements automatic key staging, replacing the manual
scp-after-boot step:

- modules/installer/host-keys.nix reads host-keys/ via
  builtins.getEnv, which Nix silently returns as "" under normal
  (non---impure) evaluation — the module is a no-op by default, safe
  for CI, until explicitly opted into:

    NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build .#iso --impure

  When built this way every key present gets baked into the image at
  /etc/host-keys/, and auto-install.sh installs whichever one matches
  the flake target selected at install time — no manual per-host scp.

- This deliberately includes the PXE netboot variant, even though
  pxe-boot serves it unauthenticated over LAN HTTP: accepted
  explicitly as a reasonable trade-off for a network that sits behind
  LAN-only infrastructure, not the open internet. auto-install.sh
  still falls back to /root/host-keys (manual scp) if a key isn't
  baked in, so images built without --impure keep working exactly as
  before.

- docs/auto-installer.md replaces nix-auto-installer's README,
  updated for in-repo paths and the new build flow.

Verified: normal `nix eval` (no --impure) evaluates identically across
all 19 nixosConfigurations + 4 packages with zero host-keys/* entries
(CI-unaffected); with --impure + the env var set, all three installer
variants (installer/ISO, proxmox-lxc, pxe) correctly embed every key
in host-keys/.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 04:41:34 +10:00
beatzaplentyandClaude Sonnet 5 95d4db5609 Authorize the WSL workstation as a nix-cache remote-builder client
Adds debian@surface's existing SSH key (already used as its admin
key elsewhere in this repo) to vars.remoteBuilderAuthorizedKeys so
nix-cache will accept it as a distributed-build client once deployed.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 04:08:19 +10:00
beatzaplentyandClaude Sonnet 5 6f8c6c8ef1 Resolve all statix and nixpkgs-fmt warnings repo-wide
Zero W20 (repeated attribute keys), W10 (empty { ... }: variadic
pattern, use _: instead), and W04 (a = x.a instead of inherit)
warnings remain anywhere in the tree, and nixpkgs-fmt --check is
clean on all 46 .nix files.

Repeated-key merges go as deep as statix actually flags per file
(e.g. boot.loader.* nested under boot.loader = { ... } once the
outer boot.* merge exposed it as its own repeat) — every merge is a
pure attribute-path restructuring with no value changes, verified by
comparing config.system.build.toplevel.drvPath before/after for a
representative host per changed module plus a full 19-host + 4-package
eval sweep.

One indentation slip caught and fixed during this pass: nesting
modules/installer/common.nix's environment.etc."auto-install.sh".text
under an environment = { ... } block initially normalized the
script's shebang/set line indentation, which actually changes the
rendered file (Nix's '' string dedent treats it as real content, not
cosmetic whitespace) — reproduced the original's exact indentation
and reverified the rendered script is byte-identical to before.

modules/services/zfs/auto-mount-volumes.nix picked up formatting too;
worth noting it isn't imported by anything in this flake at all.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 03:47:42 +10:00
beatzaplentyandClaude Sonnet 5 627aad8c29 Migrate nix-auto-installer into this flake
Folds the separate nix-auto-installer repo's build outputs into this
flake so it can build every auto-installer artifact it used to,
negating the need for that repo:

- modules/installer/{common,iso,proxmox-lxc}.nix — migrated from
  nix-auto-installer's common.nix/installer.nix/proxmox-lxc.nix.
- flake.nix gains nixosConfigurations.{installer,proxmox-lxc} and
  packages.x86_64-linux.{iso,lxc,pxe,all}, matching the original
  repo's interface (nix build .#iso / .#lxc / .#pxe / .#all).
- Dropped the live Gitea PAT baked into every installer image via
  environment.etc."git-credentials" — gitea.lan.ddnsgeek.com/beatzaplenty/nixos.git
  allows anonymous read, so the installer (which only ever reads the
  flake, never pushes) doesn't need a credential at all.
- installer_old.nix wasn't migrated — dead code, unreferenced by the
  source repo's flake.nix, and carried a second stale leaked token.
- The installer environment's own hardcoded login password hash is
  preserved as-is: sops-nix has no stable per-boot host key to derive
  an age key from on ephemeral installer media, so it can't reuse the
  same per-host secret mechanism the rest of this repo uses without
  separate design work.
- vars.adminSshKey / vars.dockerHost-style dedup: the installer's SSH
  authorized key and its FLAKE_BASE_URL domain were exact duplicates
  of values already in variables.nix / modules/common/configuration.nix,
  so both now reference the single source of truth instead.

Verified eval-equivalent for every existing host (drvPath-identical)
and confirmed the migrated auto-install.sh script renders byte-for-byte
identical to the source repo's output.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 03:47:19 +10:00
beatzaplentyandClaude Sonnet 5 745f4d6fb4 Refresh stale architecture docs
CLAUDE.md's "Composition pattern" section still described the
pre-refactor layout (hosts/<host>/configuration.nix as a thin imports
list, hardware-configuration wired in from flake.nix) from before the
platform x build-type matrix landed. Rewrite it to match the current
mkTarget/host.nix architecture and the module moves from the prior
commit. Also fixes docs/nix-cache.md, which referenced a
modules/nix/ path that never existed in this repo.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 02:58:25 +10:00
beatzaplentyandClaude Sonnet 5 c5f8bb4d1d Reorganize loose modules into themed directories
Several single-purpose modules sat at modules/ root or in the
services/ catch-all despite the repo's established pattern of one
directory per concern (tailscale/, beszel/, docker/, nix-cache/):

- remote-builder-client.nix -> nix-cache/ (always co-included with
  nix-cache/client.nix in flake.nix's mkTarget, same buildType guard)
- set-locale.nix -> common/ (unconditionally imported by
  common/configuration.nix already)
- enable-ip-forwarding.nix -> networking/
- rotate-traefik-logs.nix -> traefik/rotate-logs.nix
- services/docker-health-to-gotify.nix and services/nextcloud-cron-job.nix
  -> docker/ (both only ever imported by the docker build type, same
  as the rest of modules/docker/*)

Pure path moves plus import-path updates in flake.nix,
common/configuration.nix, and build-types/docker.nix — verified
eval-equivalent (drvPath-identical) across representative hosts.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 02:58:09 +10:00
beatzaplentyandClaude Sonnet 5 e337063a95 Add parameterized beszel host-token helper module
hosts/server/host.nix and hosts/nix-cache/host.nix each hand-rolled the
same sops secret/template/environmentFile wiring for the beszel agent
token, differing only in the sops file path and template name. Factor
it into modules/beszel/host-token.nix ({ name, sopsFile }) so a third
host can adopt it without copy-pasting the boilerplate again. Also
drops two dead, stale commented-out HUB_URL lines left over from
before variables.nix grew a homeDomain var.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 02:57:55 +10:00
beatzaplentyandClaude Sonnet 5 d8d14db505 Extract ports, docker host, and remote-builder keys into variables.nix
Pulls the beszel hub / PVE / PBS ports, the docker-compose host's LAN
name, and the remote-builder client SSH keys out of scattered inline
literals across modules/hosts and into variables.nix as the single
source of truth, matching the existing pattern for other cross-host
references (nixCacheHost, nfsServerHost).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 02:57:38 +10:00
beatzaplentyandClaude Sonnet 5 be05c63a67 Switch LXC targets from NetworkManager to systemd-networkd
boot.isContainer disables services.udev, which NetworkManager depends on
to enumerate devices — this left NM unable to reliably manage the
container veth, breaking DHCP-hostname registration in Pi-hole. It also
defaulted networking.useHostResolvConf to true, which assumes a
systemd-nspawn-style resolv.conf bind-mount that real Proxmox LXC doesn't
provide (nixpkgs' own proxmox-lxc.nix module forces this false for the
same reason). Also guard the networkmanager extraGroups membership in the
minimal/gui build-types, since that group only exists when NM is enabled.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 01:16:29 +10:00
beatzaplenty ff695c1917 added lxc-nix-cache key 2026-07-19 17:22:20 +10:00
beatzaplenty fa2a9a595e updated sops keys 2026-07-19 16:59:44 +10:00