Commit Graph
733 Commits
Author SHA1 Message Date
beatzaplenty 75d09d57e3 Fix --allow-duplicate-host creating real duplicates in create-proxmox-resource.sh
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m24s
--allow-duplicate-host previously just skipped the existing-resource
check entirely, so re-running e.g. --type lxc --host docker while an
lxc-docker container already existed created a second container
sharing the same hostname/identity instead of replacing it -- both
then fight over DNS/DHCP for that hostname, and it's easy to end up
testing the stale one without realizing.

Now splits matches into "exact" (same --type as the one being
created, e.g. another lxc-docker) and "cross-type" (a different
platform sharing this host identity, e.g. a proxmox-docker VM
alongside an lxc-docker container -- a deliberate, valid coexistence
this script has never managed and still won't). Only an exact match is
destroyed and replaced, after typing the hostname back to confirm; a
cross-type match is always left untouched. Without
--allow-duplicate-host, both cases still refuse to run exactly as
before.

Verified live against pve.sweet.home: correctly split VMID 103 (a
stopped proxmox-docker VM, cross-type -- left untouched) from VMID 105
(the running lxc-docker container, exact-type -- flagged for
destroy+replace), and confirmed the destroy prompt safely aborts on a
non-matching confirmation, leaving both resources untouched.
2026-07-20 13:22:08 +00:00
beatzaplenty 6847a7a6f4 Fix NFS shares never mounting on lxc-docker
Two compounding bugs, confirmed live on the running lxc-docker
container (vmid 102 on pve.sweet.home):

1. x-systemd.automount never works inside any Linux container --
   systemd logs "Starting of <unit>.automount unsupported" for every
   share and never mounts them. modules/docker/mount-data.nix and
   modules/raspi/mount-data.nix now key off config.boot.isContainer
   (set true by nixpkgs' proxmox-lxc.nix) to mount eagerly with
   `nofail` there instead, while VM-based docker targets keep automount
   unchanged.

2. The container's Proxmox `features` never included `mount=nfs`, so
   AppArmor blanket-denies the nfs/rpc_pipefs mount syscalls NFS
   needs ("permission denied"). scripts/env.sh's
   PROXMOX_DEFAULT_LXC_FEATURES now includes mount=nfs;nfs4 for future
   lxc-* containers -- the semicolon required quoting the --features
   value in create-proxmox-resource.sh's remote pct-create command,
   since it's sent as a raw string for the remote shell to parse and an
   unquoted `;` would be read as a command separator.

The already-running container needs a matching `pct set --features`
plus a restart to pick this up -- that's an operator step outside this
repo.
2026-07-20 12:56:42 +00:00
beatzaplenty 17dd00bee1 updated sops keys
Check NixOS configurations / eval-hosts (push) Failing after 11m21s
2026-07-20 12:27:24 +00:00
beatzaplenty b3c81453e4 updated sops keys
Check NixOS configurations / eval-hosts (push) Failing after 11m17s
2026-07-20 12:03:30 +00:00
beatzaplenty 656dd975f0 Merge pull request 'Add scripts/backup-admin-key.sh to back up the local sops admin key' (#13) from add-backup-admin-key-script into main
Check NixOS configurations / eval-hosts (push) Failing after 11m27s
2026-07-20 11:47:22 +00:00
rootandClaude Sonnet 5 2661f6d271 Add scripts/backup-admin-key.sh to back up the local sops admin key
Check NixOS configurations / eval-hosts (pull_request) Failing after 12m7s
Companion to rotate-admin-key.sh: copies whatever age identity sops/age
itself would resolve (or an explicit --key-file) to a given destination
path with 0600 perms, validating it's a real identity and round-tripping
the derived public key before/after the write so a corrupted copy is
caught immediately rather than discovered later during a restore.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 11:42:02 +00:00
beatzaplenty 0532c3a282 Merge pull request 'Deduplicate reusable shell code in scripts/ into scripts/lib/' (#12) from worktree-scripts-dedup into main
Check NixOS configurations / eval-hosts (push) Failing after 12m16s
Reviewed-on: #12
2026-07-20 11:37:05 +00:00
beatzaplenty ac8c9a20e3 Merge pull request 'Fix tailscale exit-node module: enable routing, compose on enable-service' (#11) from worktree-tailscale-exit-node into main
Check NixOS configurations / eval-hosts (push) Failing after 12m10s
Reviewed-on: #11
2026-07-20 11:36:18 +00:00
root 0e66cdabc9 rotated sops age key
Check NixOS configurations / eval-hosts (push) Failing after 20m21s
2026-07-20 11:31:25 +00:00
beatzaplenty 9c892ce1c2 Merge pull request 'Add scripts/rotate-admin-key.sh to automate sops admin key rotation' (#10) from worktree-rotate-admin-key-script into main
Check NixOS configurations / eval-hosts (push) Failing after 47m28s
Reviewed-on: #10
2026-07-20 11:28:28 +00:00
beatzaplenty bfeea90597 Merge branch 'main' into worktree-rotate-admin-key-script
Check NixOS configurations / eval-hosts (pull_request) Failing after 1h6m12s
2026-07-20 11:27:45 +00:00
rootandClaude Sonnet 5 cafeb8853b Add scripts/rotate-admin-key.sh to automate sops admin key rotation
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m15s
Automates the manual steps sync-host-keys.sh/create-proxmox-resource.sh
print when they bootstrap a fresh, not-yet-trusted age key: verifies a
backed-up key matches the current &admin entry, swaps in a new key, and
re-encrypts every secrets/*.yaml. Explicitly cds into repo_root before any
sops call, since sops resolves .sops.yaml by walking up from cwd rather
than from the target file's path -- confirmed via a scratch-repo test that
running from elsewhere would otherwise silently rotate against the wrong
config.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 11:23:33 +00:00
beatzaplenty 2c2d464503 Merge pull request 'Fix stale documentation: outdated counts, missing build type, spec status' (#9) from fix-stale-docs into main
Check NixOS configurations / eval-hosts (push) Failing after 11m12s
2026-07-20 11:07:43 +00:00
rootandClaude Sonnet 5 5ec7033439 Fix stale documentation: outdated counts, missing build type, spec status
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m26s
Same class of problem as the deployedTargets/README fixes: hand-maintained
prose that drifted from reality and nobody was obligated to update.

- CLAUDE.md: "18 hosts" was a stale hardcoded count (actually 20); reworded
  to not need updating as hosts are added. Also added the missing
  tailscale-exit-node build type to a list that had it everywhere else in
  the file except one bullet.
- AGENTS.md: same missing tailscale-exit-node build type.
- docs/auto-installer.md: the hand-enumerated lxc-* list was missing
  lxc-tailscale-exit-node.
- flake-target-refactor-spec.md: added a "Status: implemented" note so this
  completed historical spec (referenced elsewhere purely for rationale)
  can't be mistaken for an open plan with unresolved Open Questions.
- remove-sensetive-info-refactor.md: the "Definition of done" checklist was
  entirely unchecked despite most of the work being done. Checked off what's
  actually done (sops-nix migration, history scrub just performed, the
  pre-commit gitleaks hook), and left rotation of the GitHub PAT found in
  history explicitly flagged as the one still-open item -- an operator
  action against GitHub, not something this repo can attest to itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 11:06:05 +00:00
beatzaplenty 9133afd444 Merge pull request 'Fix duplicate-host check reporting false SSH failures' (#8) from worktree-fix-duplicate-host-check-exitcode into main
Check NixOS configurations / eval-hosts (push) Failing after 11m36s
2026-07-20 11:00:18 +00:00
rootandClaude Sonnet 5 eeec9ce302 Fix duplicate-host check reporting false SSH failures
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m38s
The remote bash script run over SSH ended with a for-loop whose last
statement was `[[ "$n" == "$target" ]] && echo ...`. When the last
VM/CT checked on the node didn't match --host, that test evaluated
false and became the exit status of the whole remote script (1) --
which the wrapper then misreported as "couldn't reach the node",
even though SSH connectivity and the check itself were both fine.
The actual signal is the script's stdout, not its exit code, so end
it with an explicit exit 0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:55:53 +00:00
beatzaplenty a62c4fc023 Merge pull request 'Stop tracking deployment status in the README Hosts table' (#7) from remove-deploy-status-from-readme into main
Check NixOS configurations / eval-hosts (push) Failing after 11m20s
2026-07-20 10:51:59 +00:00
rootandClaude Sonnet 5 7e51168d1b Deduplicate reusable shell code in scripts/ into scripts/lib/
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m29s
Three chunks of copy-pasted logic were drifting across scripts/*.sh:
- codex-setup.sh and codex-maintenance.sh each carried an identical
  NIX_CONFIG bootstrap + ensure_nix_profile() -> scripts/lib/nix-bootstrap.sh
- sync-host-keys.sh and prepare-host-key.sh each ran the same
  ssh-keygen/ssh-to-age nix-shell invocations -> scripts/lib/ssh-host-keys.sh
  (prepare-host-key.sh now also calls env.sh's nix_extra_opts before using
  them, closing a gap where it alone skipped the nix-cache reachability
  check env.sh exists for)
- the "list nixosConfigurations attrNames" / "get one target's hostName"
  nix eval pattern was repeated across codex-setup.sh, codex-maintenance.sh,
  sync-host-keys.sh and create-proxmox-resource.sh (the latter twice, in
  its own --list and --host lookup) -> scripts/lib/nix-eval.sh, which also
  centralizes the --no-use-registries --no-accept-flake-config flag pair
  used on every such call

Verified against the real flake/node config (nix is available here):
create-proxmox-resource.sh --list for both --type lxc/vm, a full
--dry-run create, and prepare-host-key.sh generating and cleaning up a
real key/age-pubkey pair.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:35:49 +00:00
rootandClaude Sonnet 5 97ede62f6d Stop tracking deployment status in the Hosts table
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m24s
Same problem as the deployedTargets removal, just in markdown instead of
Nix: which variant of a buildtype is actually deployed is live
infrastructure state, and a committed table can't stay accurate as that
changes -- it already required a manual edit on every migration and had
drifted before. Keep only what doesn't rot: what each target is for, and
stable naming history. Point at the live node / /etc/flake-target instead
for actual deployment status.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:18:33 +00:00
beatzaplenty ab5206b1c7 Merge pull request 'Replace duplicate-host check with live Proxmox query; drop deployedTargets' (#6) from fix-duplicate-host-self-match into main
Check NixOS configurations / eval-hosts (push) Failing after 12m7s
2026-07-20 10:12:59 +00:00
rootandClaude Sonnet 5 2041557ab3 Replace the duplicate-host check with a live Proxmox query, drop deployedTargets
variables.nix's deployedTargets was a manually-maintained list with no
enforcement keeping it in sync with reality -- it caused two separate
false refusals in a row (naming a VM as deployed well after it had been
destroyed, then matching a target against itself once the list was
"corrected"). Static files can't track whether a resource still actually
exists.

create-proxmox-resource.sh's duplicate-host guard now queries the
Proxmox node directly (qm/pct's own name/hostname config, matched
against --host) instead. Also fixes a gap in that live check: it
originally swallowed ssh failures and would have silently treated "can't
reach the node" the same as "checked, nothing there" -- it now refuses
instead of guessing when the node can't be reached.

deployedTargets is removed entirely from variables.nix since nothing
else in the repo consumed it once this script no longer does; README.md's
Hosts table remains the sole source of truth for "(real, deployed)"
status. CLAUDE.md and the script's own --help/comments updated to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:03:00 +00:00
rootandClaude Sonnet 5 2fd483697b Don't refuse recreating the canonical already-deployed target itself
The duplicate-host check in create-proxmox-resource.sh compared by
hostName only, so it fired even when the target being created was
exactly the one variables.nix's deployedTargets already names (e.g.
rebuilding lxc-nix-cache after destroying its old container to pick up
new sops secrets) -- there's no other machine at risk of an identity
collision in that case, just the normal redeploy workflow. Skip the
check when dt == flake_target; the later VMID-existence check still
guards against clobbering a resource that's actually live on the node.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 09:49:31 +00:00
beatzaplenty 89186b0dee Merge pull request 'Track nix-cache real deployment as lxc-nix-cache, not proxmox-nix-cache' (#5) from worktree-nix-cache-lxc-migration into main 2026-07-20 09:46:41 +00:00
rootandClaude Sonnet 5 8e3606cbd3 Track nix-cache's real deployment as lxc-nix-cache, not proxmox-nix-cache
The old proxmox-nix-cache VM was destroyed and nix-cache is being
redeployed as an LXC container going forward. Without this update,
create-proxmox-resource.sh's duplicate-host check (which only reads this
static list, not live Proxmox state) kept refusing to create
lxc-nix-cache even though nothing named nix-cache actually exists on the
node anymore.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 09:40:03 +00:00
beatzaplenty a18dfb0127 Merge pull request 'Trust nix-cache's SSH host key declaratively on remote-builder clients' (#4) from worktree-magical-cooking-book into main 2026-07-20 07:26:18 +00:00
beatzaplentyandClaude Sonnet 5 75f1342339 Declaratively trust nix-cache's SSH host key on remote-builder clients
Distributed builds failed with "Host key verification failed" on any
client that had never manually SSH'd to nix-cache before, since
nothing populated root's known_hosts for it. Wire nix-cache's host
public key into programs.ssh.knownHosts via a new vars.nixCacheHostKey
so every client picks it up automatically on rebuild.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 17:21:48 +10:00
beatzaplenty 36ba99c9a1 Merge pull request 'Add buildImage shell function for building lxc-* tarballs with host keys' (#3) from worktree-fizzy-juggling-sedgewick into main
Reviewed-on: #3
2026-07-20 07:13:54 +00:00
beatzaplentyandClaude Sonnet 5 0cd8f15b48 Add buildImage shell function for building lxc-* tarballs with host keys
lxc-* hosts need NIXOS_HOST_KEYS_DIR + --impure to bake in a pre-seeded
SSH host key, otherwise sops-nix's .sops.yaml recipient never matches
and every secret permanently fails to decrypt on first boot. That
invocation is easy to forget, so wrap it as `buildImage <flake-target>`
alongside the existing Switch-nix/Test-nix helpers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 17:12:16 +10:00
beatzaplenty 8613b93fa8 Merge pull request 'Fix nix_extra_opts falsely reporting nix-cache's SSH remote builder down' (#2) from fix-nix-cache-probe-retry into main
Reviewed-on: #2
2026-07-20 07:02:14 +00:00
beatzaplentyandClaude Sonnet 5 20f9475a7d Fix nix_extra_opts falsely reporting nix-cache's SSH remote builder down
The reachability check used `cat < /dev/tcp/${NIX_CACHE_HOST}/22`, which
blocks forever reading for EOF that never comes -- sshd sends its banner
and then holds the connection open waiting for the client to speak next.
Every single check hit the 3s timeout and reported "unreachable"
unconditionally, regardless of whether the remote builder was actually up.
Confirmed live: a plain TCP connect (`exec 3<>/dev/tcp/...`, no read)
returns in ~60ms against a healthy nix-cache instead of always timing out.

Fixing that exposed a second, previously-dormant bug: `printf -v
NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"` on a genuinely empty NIX_OPTS array
still runs one format pass and yields the literal `'' ` rather than an
empty string. A subprocess (e.g. sync-host-keys.sh) reusing this
process's decision via `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` then rebuilt
a 1-element array holding an empty string instead of a 0-element array,
which broke `nix-shell "${NIX_OPTS[@]}" -p <pkg>` with a bogus positional
argument the moment NIX_OPTS was legitimately empty (nix-cache reachable)
-- something the first bug had made impossible to ever hit before.

Also adds a couple of retries (1s apart) to both checks as a secondary
safety net against genuine multi-second blips, on top of fixing the
checks themselves.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:57:43 +10:00
beatzaplenty 6babb3eec5 Merge pull request 'Fix create-proxmox-resource.sh --dry-run hiding nix-cache probe results' (#1) from worktree-starry-painting-whistle into main
Reviewed-on: #1
2026-07-20 06:50:40 +00:00
beatzaplentyandClaude Sonnet 5 33730e6ccf Fix create-proxmox-resource.sh --dry-run hiding nix-cache probe results
The tarball/disko-image build previews were hardcoded strings that never
included ${NIX_OPTS[@]}, so --dry-run always showed the same "would build"
command whether nix-cache's substituter/remote-builder got disabled by
nix_extra_opts's reachability probe or not -- the actual (non-dry-run)
build commands already applied it correctly, only the preview lied.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:36:14 +10:00
beatzaplentyandClaude Sonnet 5 65f89806cb Fix beszel-agent losing its hub-pairing fingerprint on every restart
services.beszel.agent runs under DynamicUser=true with ProtectSystem =
"strict" and no StateDirectory, so /var/lib/beszel-agent -- where the
agent persists the fingerprint that locks its hub pairing to this
machine (github.com/henrygd/beszel/discussions/1542) -- was never
actually writable. Every restart silently failed to persist it and
regenerated a fresh one in memory, permanently desyncing from whatever
the hub had on record after the very first successful pairing. Affects
every host importing modules/beszel/enable-agent.nix (nix-cache, server),
not just full container rebuilds.

Found via nix-cache showing "fingerprint mismatch" after being rebuilt
post-outage; confirmed server was silently exposed to the same bug, just
hadn't restarted since its first pairing. Fixed by declaring
StateDirectory so systemd gives the dynamic user real persistent storage.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:59:24 +10:00
beatzaplentyandClaude Sonnet 5 c3007097a6 Fix create-proxmox-resource.sh defaulting hostname to the flake target
--name (used as pct/qm create's --hostname/--name) defaulted to
$flake_target (e.g. "lxc-nix-cache"), not $host (e.g. "nix-cache"). Since
proxmoxLXC.manageHostName pulls the guest's real networking.hostName
straight from Proxmox's own container config, this silently overrode
host.nix's hostName with a build-type-specific name. Default --name to
--host instead, so the guest's identity matches host.nix regardless of
which platform variant built it.

Found by spinning up a fresh lxc-nix-cache test container and noticing its
hostname was "lxc-nix-cache" instead of "nix-cache".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:19:35 +10:00
beatzaplentyandClaude Sonnet 5 9724babcea Add tailscale-exit-node build type across all three platforms
New build type dedicated to Tailscale exit-node capability, wired up for
linode/proxmox/lxc like every other build type (the lxc variant is the one
actually intended for deployment). Kept separate from the "server" host
rather than bundling exit-node capability onto it.

Trimmed modules/tailscale/exit-node.nix down to pure exit-node behavior:
dropped the old --advertise-routes=${vars.lanCidr} bundling (meaningless
for a Linode-hosted VPS with no path to the LAN), and switched
extraUpFlags -> extraSetFlags. Confirmed against nixpkgs' tailscale.nix
that extraUpFlags is only applied by tailscaled-autoconnect, which itself
only runs when services.tailscale.authKeyFile is set -- nothing in this
repo sets one, so the old flags would never have actually been applied.
extraSetFlags runs unconditionally via tailscaled-set on every boot, so
--advertise-exit-node self-reapplies once the operator has done the
one-time manual `tailscale up` auth.

Verified: all three new targets eval cleanly, nixpkgs-fmt/statix clean,
and a dry-run build of lxc-tailscale-exit-node's tarball resolves its full
closure including tailscaled-set.service.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 13:38:39 +10:00
beatzaplentyandClaude Sonnet 5 7055bcdb97 Fix lxc-* hosts never completing first-boot user/secrets activation
virtualisation/proxmox-lxc.nix registers the Nix store DB via a systemd
service, never an activation script -- so neededForUsers sops secrets
(password hashes) and the user-creation step that consumes them never ran
on a real first boot, leaving /etc/shadow stuck with build-time placeholder
entries. boot.postBootCommands looked like the right hook (stage-2-init.sh
does invoke it) but switch-to-configuration behaves unreliably that early,
before systemd itself is up. Fixed with a genuine oneshot systemd service,
gated by ConditionPathExists so it only ever runs once.

Confirmed live via a from-scratch destroy+rebuild+redeploy of the
lxc-nix-cache test container: real password hashes applied automatically,
systemctl is-system-running -> running, zero failed units.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 13:38:21 +10:00
beatzaplentyandClaude Sonnet 5 d973da487c Fix lxc-* hosts having no host-key pre-seeding mechanism at all
The real root cause behind the original nix-cache 502, traced all the way
through: modules/installer/host-keys.nix (which NIXOS_HOST_KEYS_DIR=...
--impure actually wires up) is only ever imported by the installer's own
modules/installer/common.nix -- modules/platforms/lxc.nix, which every
real lxc-* host build actually uses, never imported anything like it.
docs/auto-installer.md previously claimed NIXOS_HOST_KEYS_DIR bakes a key
into lxc-* tarballs "the same way it does for the ISO/PXE installer
images" -- that was never actually true; I wrote it without verifying the
mechanism existed for lxc.nix specifically.

In practice this meant every lxc-* container booted with a freshly
self-generated SSH host key that could never match whatever .sops.yaml
actually trusts for that target, so *every* secret -- not just
cache-priv-key -- silently failed to decrypt. No error surfaces in the
boot log for this: the activation step that installs secrets only runs
on a genuinely fresh first activation and silently no-ops once
/run/current-system already exists, so by the time anyone looks the
window has closed. Found by manually invoking sops-install-secrets
directly: "Error getting data key: 0 successful groups required, got 0".

Fixed by giving modules/platforms/lxc.nix the same key-baking mechanism
the installer has, but keyed to its own exact flake target and placing
the key directly at /etc/ssh/ssh_host_ed25519_key (no copy step to stage
for, unlike the installer's /etc/host-keys/ staging area -- an lxc-*
tarball has no install step). The target name comes in via
specialArgs.flakeTarget (new, set by flake.nix's mkTarget) rather than
being read back from config.environment.etc."flake-target" -- reading
that back from within a module that also contributes to
environment.etc is circular (confirmed: "infinite recursion
encountered").

Verified live end-to-end against the real test container (lxc-nix-cache,
VMID 100 on pve.sweet.home): destroyed it, rebuilt the tarball fresh with
the fix, recreated it, and confirmed /run/secrets/ now has all three
secrets this host needs (beszel-token, cache-priv-key, nix-github-token),
nix-serve is active (running), and curl http://localhost/nix-cache-info
succeeds both directly and through nginx.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 12:29:01 +10:00
beatzaplentyandClaude Sonnet 5 c939454983 Fix tailscale exit-node module: enable routing, compose on enable-service
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m52s
exit-node.nix advertised --advertise-exit-node/--advertise-routes but
never enabled IP forwarding, so it wouldn't actually route traffic.
services.tailscale.useRoutingFeatures = "server" is the built-in way to
get the sysctls right without duplicating modules/networking/enable-ip-forwarding.nix.
Also opens the firewall for direct (non-DERP-relayed) peer connections,
and now imports enable-service.nix instead of duplicating `enable = true`,
so "plain tailscale" and "tailscale exit node" are two composable modules.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 12:24:18 +10:00
beatzaplentyandClaude Sonnet 5 274d54a774 Fix LXC container creation: unprivileged, nesting/keyctl, swap sizing
Found and fixed live against a real test container (VMID 100, lxc-nix-cache
on pve.sweet.home) after the previous pct-restore-to-pct-create fix still
produced a container that booted into garbled console output:

1. pct create's own CLI default for --unprivileged is privileged (unlike
   the web UI, whose checkbox defaults the other way), but
   modules/platforms/lxc.nix sets proxmoxLXC.privileged = false, so the
   image assumes it's running unprivileged. Real mismatch -- now passes
   --unprivileged 1 explicitly.

2. The actual root cause of the garbled console: modern (v247+) systemd
   routinely uses nested user namespaces and credential mounts (even
   plain getty units, via LoadCredential=-style mechanisms), which
   AppArmor's default LXC confinement denies without --features
   nesting=1,keyctl=1. Confirmed via the host's kernel audit log: every
   getty unit was crash-looping on a denied /run/credentials/* mount
   every ~3s, and core services like nsncd failed userns_create the same
   way -- the system never finished activating. Fixed live (pct set +
   restart on the running test container) before committing the script
   change: systemctl is-system-running went from never completing to
   "running" with zero failed units.

3. --memory doesn't touch swap -- confirmed live it silently stayed at
   Proxmox's own 512M default with --memory 2048. Now defaults --swap to
   whatever --memory resolves to.

docs/auto-installer.md's manual pct create walkthrough gets the same
fixes, with the "why" for each flag, since a human following it by hand
would hit the identical bugs.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 11:33:53 +10:00
beatzaplentyandClaude Sonnet 5 ad274d99fb Fix nix-cache retry storms and lxc creation in create-proxmox-resource.sh
Two independent problems found while actually running the script:

1. nix build/nix-shell retry each unreachable substituter/builder up to
   5x with backoff, per store path -- with nix-cache down this compounds
   into minutes of noise. scripts/env.sh gains nix_extra_opts(), which
   probes http://nix-cache and nixremote@nix-cache:22 once via plain
   curl/TCP (bypassing Nix's own retry logic entirely -- confirmed
   nix store ping still retries 5x even with a short connect-timeout)
   and exports the decision so create-proxmox-resource.sh and the
   sync-host-keys.sh subprocess it shells out to both reuse it instead
   of probing independently.

2. The actual failure: "archive contains no configuration file". pct
   restore expects a vzdump backup archive with embedded config;
   config.system.build.tarball is a plain CT template tarball -- wrong
   Proxmox mechanism entirely. Fixed to pct create against it as a vztmpl
   template instead, uploaded to /var/lib/vz/template/cache/ rather than
   /var/lib/vz/dump/. This same wrong claim had propagated into
   docs/auto-installer.md, README.md, and CLAUDE.md from when the script
   was first written -- corrected everywhere.

Also: checks for an already-uploaded image on the node (fixed
<flake_target>.tar.xz/.raw naming) before building, skipping build+upload
entirely if found (--force-rebuild to always rebuild).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 11:05:26 +10:00
beatzaplentyandClaude Sonnet 5 bd8d93d890 Add sync-host-keys.sh and create-proxmox-resource.sh
sync-host-keys.sh: generates/registers SSH host keys and their
.sops.yaml/secrets/*.yaml recipients for flake targets, idempotently.
--all, <target>, --remove, --regenerate-all-keys, all with --dry-run
(verified zero-side-effect via a sandboxed git-status check across every
mode). Only ever touches anchors with a corresponding host-keys/ file --
&admin and any hand-registered real-host anchor are never listed,
removed, or regenerated. Supersedes running prepare-host-key.sh one host
at a time for any target that already has a flake entry.

create-proxmox-resource.sh: builds a lxc-*/proxmox-* target's
tarball/disk image and creates it on a real Proxmox node, or reconfigures
an existing resource's cores/memory/disk (--modify, always requires
typing the VMID back to confirm). Refuses to create a new resource for a
VMID that already exists, and refuses to duplicate a host identity that
already has a real deployment elsewhere (variables.nix's new
deployedTargets, checked by hostName so it also catches cross-platform
duplicates) unless --allow-duplicate-host is passed. --dry-run throughout.

scripts/env.sh centralizes the Proxmox connection config both scripts
(and future ones) share. Also fixes an unrelated gap found along the way:
proxmox-* Disko image builds write their .raw file straight into the
repo root, and .gitignore never covered it.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 10:31:06 +10:00
beatzaplentyandClaude Sonnet 5 53b9a64826 Move nix-cache's binary cache signing key into sops
nix-serve's secretKeyFile was a manual, undocumented-outside-a-comment
`nix-store --generate-binary-cache-key` step per host -- easy to miss on
a fresh nix-cache instance (as lxc-nix-cache testing just found: systemd
fails the unit with EXIT_CREDENTIALS when LoadCredential can't find the
source file, which nginx then reports as a 502 from clients). It also
can't be regenerated per-host safely: modules/nix-cache/client.nix
hardcodes every client's trust in one specific public key, so every
nix-cache instance has to share the exact same keypair.

Sourced from secrets/nix-cache.yaml's new cache-priv-key entry instead,
via the same sops-nix pattern every other secret in this repo already
uses. Verified the added value derives to the exact public key
modules/nix-cache/client.nix already trusts before committing.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 09:49:30 +10:00
beatzaplentyandClaude Sonnet 5 0ba837817e Expand variables.nix: NFS shares, ports, image size, GC/rotation, Pi host
Adds nested vars.nfsShares (subpath + mountpoint per dataset, previously
duplicated independently across server.nix's NFS exports, mount-data.nix's
client mounts, docker.nix's tmpfiles rules, traefik's log rotation path,
and hosts/server/host.nix's beszel config), vars.ports (every literal port
in modules/ and hosts/, kept as separate entries per service even where
numbers coincide so changing one can't silently change another), plus
vars.proxmoxImageSize, vars.nixCacheGcMaxAge, vars.traefikLogRotate, and
raspberryPiHost/raspiNfsPath/raspiMountpoint for the Pi's own NFS export.

Also fixes docker.nix/minimal.nix/gui.nix hardcoding the literal "nixos"
username instead of the existing vars.primaryUser, found during the sweep.

system.stateVersion is deliberately left untouched everywhere -- per
NixOS's own docs that value must stay fixed from first install, not
follow any shared variable.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 08:27:37 +10:00
beatzaplentyandClaude Sonnet 5 80f86b086b Add scripts/bump-nixpkgs-release.sh
flake.nix's nixpkgs.url/home-manager.url can't source a version string
from variables.nix -- flake input resolution requires a plain string
literal, confirmed empirically (nix flake metadata errors with
"expected a string or a path but got a thunk" otherwise). This script
is the one-command alternative: bump both release branches in flake.nix
via targeted substitution (never a blind repo-wide replace, so it can't
collide with stateVersion strings elsewhere), with an upstream branch
existence check before writing, plus an optional --tooling flag for
codex-maintenance.sh's separately-pinned nixpkgs-fmt/statix fetch.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 08:04:44 +10:00
beatzaplentyandClaude Sonnet 5 a351cbcf80 Give linode-* hosts a real Disko config, simplify auto-install.sh
Linode provisions and sizes /dev/sda (root) and /dev/sdb (swap) itself
as whole, unpartitioned block devices before the OS ever boots.
modules/disko/linode.nix declares them with destroy = false (skips
Disko's wipe stage for these disks entirely) and a bare filesystem/swap
content type matching that existing layout, so re-running it against an
already-provisioned disk only mkfs/mkswaps if blkid shows it isn't
formatted yet -- never repartitions or destroys data.

With every host reachable through the installer menu now carrying a
Disko config, auto-install.sh no longer needs to probe the flake and
branch between `disko --mode destroy,format,mount` and a bind-mount
fallback -- it just always runs Disko.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 08:04:34 +10:00
beatzaplentyandClaude Sonnet 5 2aa625d566 Remove packages.all bundle and the installer's own proxmox-lxc target
Both existed only so the installer could boot as an LXC container and
nixos-install some other host from within it, but lxc-* targets are
already excluded from the install menu (nixos-install can't touch its
own running root filesystem), and now have their own direct tarball
path anyway. That left the installer's own LXC form with no real use
case, and packages.all with only two members worth bundling.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 08:04:08 +10:00
beatzaplentyandClaude Sonnet 5 288835db29 Consolidate CI on codex-maintenance.sh; clean up dead scripts and docs
Both check-nixos.yml workflows (GitHub + Gitea) now call
scripts/codex-maintenance.sh instead of a hand-rolled eval-only loop,
closing a real gap: CI previously enforced none of the secret grep,
nixpkgs-fmt, or statix checks that codex-maintenance.sh already runs
locally — nothing was stopping that from regressing. One script now
backs both, instead of two copies that can drift from each other.

codex-maintenance.sh itself is extended to cover buildable surface
that wasn't validated anywhere before: packages.x86_64-linux.*, plus
config.system.build.tarball (lxc-* hosts) and
config.system.build.diskoImagesScript (proxmox-*, excluding the
installer's own proxmox-lxc target, which has no disko config).

Also:
- scripts/prepare-host-key.sh: dropped the redundant
  [path-to-nixos-repo] parameter — it always defaults to the repo the
  script itself lives in now, so a second argument never made sense
  after the nix-auto-installer migration.
- Removed prepare.sh (dead pre-disko manual parted/mkfs/mkswap
  partitioning, fully superseded) and
  scripts/create-linode-installer-disk.sh (incomplete draft for an
  abandoned dd-via-rescue-mode approach; Linode hosts already deploy
  fine through the normal auto-installer flow).
- docs/pxe-boot.md: fixed a stale `nixosConfigurations.pxe-boot` eval
  command (pre-refactor flat name, not a real flake attribute
  anymore) and added a cross-reference to docs/auto-installer.md.
- CLAUDE.md/README.md: full documentation pass reconciling this
  session's changes — modules/installer/, modules/pxe-boot/, the
  LXC/Proxmox image-building deployment paths, corrected the
  password-hash/SSH-key locations in the safety-rules section (both
  had drifted to reference files/paths that no longer exist), and
  added session-workflow guidance to prefer targeted host evals over
  full-repo sweeps for incremental changes (explicitly scoped to
  interactive sessions, not CI).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 07:02:23 +10:00
beatzaplentyandClaude Sonnet 5 559c538a3d Register sops age key for proxmox-minimal
Adds the proxmox-minimal host's age key (derived from its SSH host
key via scripts/prepare-host-key.sh) as a recipient in .sops.yaml and
re-encrypts secrets/common.yaml for it via sops updatekeys.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 07:01:56 +10:00
beatzaplentyandClaude Sonnet 5 feee2f1679 Add Proxmox VM disk-image building; fix disko confirmation bypass
modules/disko/proxmox.nix gains imageSize (20G default) and a
per-host imageName (networking.hostName, so every proxmox-* host
produces a distinctly named image instead of an identical main.raw).
This is the same disko.devices config already used to format a real
disk on install, so it's available for every proxmox-* target with no
per-host changes needed:

  nix build .#nixosConfigurations.<host>.config.system.build.diskoImagesScript
  sudo ./result --build-memory 2048

docs/proxmox-images.md covers building, host-key pre-seeding via
disko's --pre-format-files (same host-keys/ workflow as the installer
and LXC tarball paths), and the qm import/attach sequence for
deploying the result to Proxmox.

Also fixes a real bug in auto-install.sh found while testing: the
disko confirmation bypass used --yes, which disko's CLI doesn't
recognize at all (the actual flag is --yes-wipe-all-disks) — so the
"skip confirmation" flag was silently a no-op and the interactive
prompt kept appearing regardless.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 07:01:33 +10:00
beatzaplentyandClaude Sonnet 5 120240f14a Fix LXC deployment path and clean up remaining eval warnings
LXC hosts (device busy fix):

modules/platforms/lxc.nix now imports nixpkgs' own
virtualisation/proxmox-lxc.nix, giving every lxc-* host a real
config.system.build.tarball output — a directly `pct restore`-able
Proxmox container image. This is the actual bug fix behind the
"cannot remove real root directory: device busy or in use" error:
lxc-* targets were only reachable through nixos-install, which
bind-mounts / onto /mnt for containers (no raw disk to partition)
and then correctly refuses to modify the filesystem it's currently
running on. auto-install.sh's menu now excludes lxc-* targets
entirely (they deploy via nix build + pct restore instead, see
docs/auto-installer.md) — and, on the same reasoning, also excludes
`installer`/`proxmox-lxc`, which are the installer image's own flake
targets, not deployable hosts.

manageHostName = true keeps host.nix's declared hostnames (upstream's
default would let Proxmox's ambient container config win instead);
privileged = false matches how these containers are actually created.

Eval warnings, now zero across all 19 nixosConfigurations + 4 packages:

- Multiple password options (root/nixos in the installer): nixpkgs'
  own installer profile sets initialHashedPassword = "" for
  passwordless login, conflicting with our explicit hashedPassword.
  Force-nulled the upstream option rather than adopting passwordless
  login, since this image now also boots over LAN PXE with
  PasswordAuthentication enabled.
- boot.zfs.forceImportRoot default value: set explicitly to false
  (matching the two places that already did) in
  modules/common/configuration.nix and modules/installer/common.nix,
  covering every host and the installer alike.
- Deprecated pkgs.system in modules/build-types/gui.nix: switched to
  pkgs.stdenv.hostPlatform.system.

All confirmed non-behavioral where it matters: unrelated hosts'
drvPaths are byte-identical to their pre-existing baselines throughout.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 05:58:56 +10:00