If neither /etc/host-keys nor /root/host-keys has the target's SSH
host key, auto-install.sh previously went straight to "continue
without one anyway?". Added a third option in between, gated on
[[ -t 0 ]] (only offered when there's an actual operator at stdin, never
in an unattended/non-interactive run): prompt for an arbitrary
directory (USB stick, other mount, etc.), and if the key pair is
there, copy it into /root/host-keys and install it to /mnt same as the
existing pre-seeded-key path. Falls through to the original
warning+confirm if the prompt is skipped, the path doesn't have the
key, or the run isn't interactive at all.
docs/auto-installer.md updated to mention the new fallback. Quick
bash -n + shellcheck pass only, per request.
Two real bugs, both hit live:
1. Shebang: #!/run/current-system/sw/bin/bash only resolves on an
already-activated NixOS system -- running the checked-out script
directly (e.g. from a stock ISO, cloned repo) failed with "cannot
execute: required file not found" on a non-NixOS box. Switched to
#!/usr/bin/env bash, which resolves identically on NixOS
(environment.usrbinenv's own default) and any normal Linux distro.
Also fixed the file's missing executable bit.
2. FLAKE_BASE_URL: previously depended on pkgs.replaceVars substituting
a Nix-templated @lanDomain@ placeholder at build time -- meaning it
only ever worked when baked into the built installer image, not when
run straight from a checkout (the literal, unexpanded "@lanDomain@"
string reached git as a bogus hostname). Replaced with LAN_DOMAIN in
scripts/env.sh (manually kept in sync with variables.nix's lanDomain,
same pattern as NIX_CACHE_HOST/nixCacheHost already), sourced by the
script itself like every other script in scripts/. Dropped
pkgs.replaceVars from modules/installer/common.nix entirely --
scripts/env.sh is now baked into the image alongside auto-install.sh
at a matching relative path (/etc/nixos-installer/env.sh next to
/etc/nixos-installer/installer/auto-install.sh) so the script's own
relative `source` line resolves the same way in both contexts.
loginShellInit's invocation path and docs/auto-installer.md updated to
match. Verified: shellcheck clean on both scripts, the baked files are
byte-identical to their checked-in sources (no templating left to
verify), and codex-maintenance.sh (secret grep, fmt, statix, full eval
of every host/package including the installer/pxe artifacts) passes
clean.
The iPXE menu's "nixos" entry actually chain-loads this flake's own
custom auto-installer image, not a stock NixOS image — rename it to
"auto-installer" (label "NixOS Auto-Installer") so the menu says what it
boots, and set networking.hostName on netbootSystem to match, so the
generated system name (nixos-system-auto-installer-*) and staged
directory (/srv/pxe/http/auto-installer) agree with the menu entry too.
Add a second, genuinely vanilla NixOS minimal netboot image
(netbootMinimalSystem in flake.nix — nixpkgs' netboot-minimal.nix on its
own, none of modules/installer/common.nix's auto-installer wiring),
built from source the same way as the auto-installer image and exposed
as packages.x86_64-linux.pxe-minimal. Staged and menu-wired the same
way, as "nixos-minimal" (item, hostname, and directory all matching).
modules/pxe-boot/stage-installer-artifacts.nix is generalized to stage
both images via a shared rule-builder instead of one hardcoded set of
paths.
Verified: nix eval confirms both images' config.system.name matches
their menu entry/directory names, the pxe-boot host itself builds
clean with the new menu.ipxe, and the new pxe-minimal image was booted
directly under QEMU (kernel+initrd, no KVM) to a working login shell
with hostname nixos-minimal, no hang.
scripts/ had grown to 10 top-level scripts covering three distinct
concerns (sops/age + SSH host-key management, Proxmox deployment, and
repo-wide bootstrap/CI) with no grouping. Move the key-management scripts
(backup-admin-key.sh, rotate-admin-key.sh, prepare-host-key.sh,
sync-host-keys.sh) into scripts/secrets/, and the Proxmox scripts
(create-proxmox-resource.sh, configure-nix-cache-client.sh) into
scripts/proxmox/; leave env.sh, codex-setup.sh, codex-maintenance.sh, and
bump-nixpkgs-release.sh at the top level (frequently hand-typed or pure
shared config) and scripts/lib/ as-is.
Updates every cross-reference: each moved script's repo_root computation
(now one directory deeper), shellcheck source= directives, inter-script
paths (create-proxmox-resource.sh's call into sync-host-keys.sh and its
remote bootstrap of configure-nix-cache-client.sh on the Proxmox node),
and every doc/module mention (CLAUDE.md's Scripts section reorganized to
match, README.md, docs/auto-installer.md, docs/proxmox-images.md,
modules/installer/common.nix, modules/platforms/lxc.nix). CI workflows
need no change -- they only invoke codex-maintenance.sh, which didn't
move. Verified via bash -n, shellcheck (no new warnings beyond the
pre-existing SC1091/SC2029/SC2095 baseline), and live dry-runs of
sync-host-keys.sh --all and create-proxmox-resource.sh --list from their
new paths.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New tor-relay build type (currently lxc-only) running a plain Tor
middle relay via modules/tor/enable-relay.nix, plus nyx for
interactive monitoring over the relay's control socket.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
create-proxmox-resource.sh no longer builds locally and scp's a
multi-gigabyte image over -- it now clones/pulls this repo onto the
Proxmox node itself (bootstrapping build tooling via the existing
codex-setup.sh on first use) and runs the nix build / disko image
script there, staging the result straight into the node's own import
directory. host-keys/ (gitignored) is copied over separately since a
git pull doesn't carry it. --image still uploads an explicit local
file for the case where you don't want a build at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same class of problem as the deployedTargets/README fixes: hand-maintained
prose that drifted from reality and nobody was obligated to update.
- CLAUDE.md: "18 hosts" was a stale hardcoded count (actually 20); reworded
to not need updating as hosts are added. Also added the missing
tailscale-exit-node build type to a list that had it everywhere else in
the file except one bullet.
- AGENTS.md: same missing tailscale-exit-node build type.
- docs/auto-installer.md: the hand-enumerated lxc-* list was missing
lxc-tailscale-exit-node.
- flake-target-refactor-spec.md: added a "Status: implemented" note so this
completed historical spec (referenced elsewhere purely for rationale)
can't be mistaken for an open plan with unresolved Open Questions.
- remove-sensetive-info-refactor.md: the "Definition of done" checklist was
entirely unchecked despite most of the work being done. Checked off what's
actually done (sops-nix migration, history scrub just performed, the
pre-commit gitleaks hook), and left rotation of the GitHub PAT found in
history explicitly flagged as the one still-open item -- an operator
action against GitHub, not something this repo can attest to itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The real root cause behind the original nix-cache 502, traced all the way
through: modules/installer/host-keys.nix (which NIXOS_HOST_KEYS_DIR=...
--impure actually wires up) is only ever imported by the installer's own
modules/installer/common.nix -- modules/platforms/lxc.nix, which every
real lxc-* host build actually uses, never imported anything like it.
docs/auto-installer.md previously claimed NIXOS_HOST_KEYS_DIR bakes a key
into lxc-* tarballs "the same way it does for the ISO/PXE installer
images" -- that was never actually true; I wrote it without verifying the
mechanism existed for lxc.nix specifically.
In practice this meant every lxc-* container booted with a freshly
self-generated SSH host key that could never match whatever .sops.yaml
actually trusts for that target, so *every* secret -- not just
cache-priv-key -- silently failed to decrypt. No error surfaces in the
boot log for this: the activation step that installs secrets only runs
on a genuinely fresh first activation and silently no-ops once
/run/current-system already exists, so by the time anyone looks the
window has closed. Found by manually invoking sops-install-secrets
directly: "Error getting data key: 0 successful groups required, got 0".
Fixed by giving modules/platforms/lxc.nix the same key-baking mechanism
the installer has, but keyed to its own exact flake target and placing
the key directly at /etc/ssh/ssh_host_ed25519_key (no copy step to stage
for, unlike the installer's /etc/host-keys/ staging area -- an lxc-*
tarball has no install step). The target name comes in via
specialArgs.flakeTarget (new, set by flake.nix's mkTarget) rather than
being read back from config.environment.etc."flake-target" -- reading
that back from within a module that also contributes to
environment.etc is circular (confirmed: "infinite recursion
encountered").
Verified live end-to-end against the real test container (lxc-nix-cache,
VMID 100 on pve.sweet.home): destroyed it, rebuilt the tarball fresh with
the fix, recreated it, and confirmed /run/secrets/ now has all three
secrets this host needs (beszel-token, cache-priv-key, nix-github-token),
nix-serve is active (running), and curl http://localhost/nix-cache-info
succeeds both directly and through nginx.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Found and fixed live against a real test container (VMID 100, lxc-nix-cache
on pve.sweet.home) after the previous pct-restore-to-pct-create fix still
produced a container that booted into garbled console output:
1. pct create's own CLI default for --unprivileged is privileged (unlike
the web UI, whose checkbox defaults the other way), but
modules/platforms/lxc.nix sets proxmoxLXC.privileged = false, so the
image assumes it's running unprivileged. Real mismatch -- now passes
--unprivileged 1 explicitly.
2. The actual root cause of the garbled console: modern (v247+) systemd
routinely uses nested user namespaces and credential mounts (even
plain getty units, via LoadCredential=-style mechanisms), which
AppArmor's default LXC confinement denies without --features
nesting=1,keyctl=1. Confirmed via the host's kernel audit log: every
getty unit was crash-looping on a denied /run/credentials/* mount
every ~3s, and core services like nsncd failed userns_create the same
way -- the system never finished activating. Fixed live (pct set +
restart on the running test container) before committing the script
change: systemctl is-system-running went from never completing to
"running" with zero failed units.
3. --memory doesn't touch swap -- confirmed live it silently stayed at
Proxmox's own 512M default with --memory 2048. Now defaults --swap to
whatever --memory resolves to.
docs/auto-installer.md's manual pct create walkthrough gets the same
fixes, with the "why" for each flag, since a human following it by hand
would hit the identical bugs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Two independent problems found while actually running the script:
1. nix build/nix-shell retry each unreachable substituter/builder up to
5x with backoff, per store path -- with nix-cache down this compounds
into minutes of noise. scripts/env.sh gains nix_extra_opts(), which
probes http://nix-cache and nixremote@nix-cache:22 once via plain
curl/TCP (bypassing Nix's own retry logic entirely -- confirmed
nix store ping still retries 5x even with a short connect-timeout)
and exports the decision so create-proxmox-resource.sh and the
sync-host-keys.sh subprocess it shells out to both reuse it instead
of probing independently.
2. The actual failure: "archive contains no configuration file". pct
restore expects a vzdump backup archive with embedded config;
config.system.build.tarball is a plain CT template tarball -- wrong
Proxmox mechanism entirely. Fixed to pct create against it as a vztmpl
template instead, uploaded to /var/lib/vz/template/cache/ rather than
/var/lib/vz/dump/. This same wrong claim had propagated into
docs/auto-installer.md, README.md, and CLAUDE.md from when the script
was first written -- corrected everywhere.
Also: checks for an already-uploaded image on the node (fixed
<flake_target>.tar.xz/.raw naming) before building, skipping build+upload
entirely if found (--force-rebuild to always rebuild).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
sync-host-keys.sh: generates/registers SSH host keys and their
.sops.yaml/secrets/*.yaml recipients for flake targets, idempotently.
--all, <target>, --remove, --regenerate-all-keys, all with --dry-run
(verified zero-side-effect via a sandboxed git-status check across every
mode). Only ever touches anchors with a corresponding host-keys/ file --
&admin and any hand-registered real-host anchor are never listed,
removed, or regenerated. Supersedes running prepare-host-key.sh one host
at a time for any target that already has a flake entry.
create-proxmox-resource.sh: builds a lxc-*/proxmox-* target's
tarball/disk image and creates it on a real Proxmox node, or reconfigures
an existing resource's cores/memory/disk (--modify, always requires
typing the VMID back to confirm). Refuses to create a new resource for a
VMID that already exists, and refuses to duplicate a host identity that
already has a real deployment elsewhere (variables.nix's new
deployedTargets, checked by hostName so it also catches cross-platform
duplicates) unless --allow-duplicate-host is passed. --dry-run throughout.
scripts/env.sh centralizes the Proxmox connection config both scripts
(and future ones) share. Also fixes an unrelated gap found along the way:
proxmox-* Disko image builds write their .raw file straight into the
repo root, and .gitignore never covered it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Linode provisions and sizes /dev/sda (root) and /dev/sdb (swap) itself
as whole, unpartitioned block devices before the OS ever boots.
modules/disko/linode.nix declares them with destroy = false (skips
Disko's wipe stage for these disks entirely) and a bare filesystem/swap
content type matching that existing layout, so re-running it against an
already-provisioned disk only mkfs/mkswaps if blkid shows it isn't
formatted yet -- never repartitions or destroys data.
With every host reachable through the installer menu now carrying a
Disko config, auto-install.sh no longer needs to probe the flake and
branch between `disko --mode destroy,format,mount` and a bind-mount
fallback -- it just always runs Disko.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Both existed only so the installer could boot as an LXC container and
nixos-install some other host from within it, but lxc-* targets are
already excluded from the install menu (nixos-install can't touch its
own running root filesystem), and now have their own direct tarball
path anyway. That left the installer's own LXC form with no real use
case, and packages.all with only two members worth bundling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
LXC hosts (device busy fix):
modules/platforms/lxc.nix now imports nixpkgs' own
virtualisation/proxmox-lxc.nix, giving every lxc-* host a real
config.system.build.tarball output — a directly `pct restore`-able
Proxmox container image. This is the actual bug fix behind the
"cannot remove real root directory: device busy or in use" error:
lxc-* targets were only reachable through nixos-install, which
bind-mounts / onto /mnt for containers (no raw disk to partition)
and then correctly refuses to modify the filesystem it's currently
running on. auto-install.sh's menu now excludes lxc-* targets
entirely (they deploy via nix build + pct restore instead, see
docs/auto-installer.md) — and, on the same reasoning, also excludes
`installer`/`proxmox-lxc`, which are the installer image's own flake
targets, not deployable hosts.
manageHostName = true keeps host.nix's declared hostnames (upstream's
default would let Proxmox's ambient container config win instead);
privileged = false matches how these containers are actually created.
Eval warnings, now zero across all 19 nixosConfigurations + 4 packages:
- Multiple password options (root/nixos in the installer): nixpkgs'
own installer profile sets initialHashedPassword = "" for
passwordless login, conflicting with our explicit hashedPassword.
Force-nulled the upstream option rather than adopting passwordless
login, since this image now also boots over LAN PXE with
PasswordAuthentication enabled.
- boot.zfs.forceImportRoot default value: set explicitly to false
(matching the two places that already did) in
modules/common/configuration.nix and modules/installer/common.nix,
covering every host and the installer alike.
- Deprecated pkgs.system in modules/build-types/gui.nix: switched to
pkgs.stdenv.hostPlatform.system.
All confirmed non-behavioral where it matters: unrelated hosts'
drvPaths are byte-identical to their pre-existing baselines throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Finishes the nix-auto-installer migration: scripts/prepare-host-key.sh
and the local host-keys/ directory (gitignored, private key material,
never committed — moved as plain files, not through git history)
weren't carried over in the initial migration.
Also implements automatic key staging, replacing the manual
scp-after-boot step:
- modules/installer/host-keys.nix reads host-keys/ via
builtins.getEnv, which Nix silently returns as "" under normal
(non---impure) evaluation — the module is a no-op by default, safe
for CI, until explicitly opted into:
NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build .#iso --impure
When built this way every key present gets baked into the image at
/etc/host-keys/, and auto-install.sh installs whichever one matches
the flake target selected at install time — no manual per-host scp.
- This deliberately includes the PXE netboot variant, even though
pxe-boot serves it unauthenticated over LAN HTTP: accepted
explicitly as a reasonable trade-off for a network that sits behind
LAN-only infrastructure, not the open internet. auto-install.sh
still falls back to /root/host-keys (manual scp) if a key isn't
baked in, so images built without --impure keep working exactly as
before.
- docs/auto-installer.md replaces nix-auto-installer's README,
updated for in-repo paths and the new build flow.
Verified: normal `nix eval` (no --impure) evaluates identically across
all 19 nixosConfigurations + 4 packages with zero host-keys/* entries
(CI-unaffected); with --impure + the env var set, all three installer
variants (installer/ISO, proxmox-lxc, pxe) correctly embed every key
in host-keys/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
