If neither /etc/host-keys nor /root/host-keys has the target's SSH
host key, auto-install.sh previously went straight to "continue
without one anyway?". Added a third option in between, gated on
[[ -t 0 ]] (only offered when there's an actual operator at stdin, never
in an unattended/non-interactive run): prompt for an arbitrary
directory (USB stick, other mount, etc.), and if the key pair is
there, copy it into /root/host-keys and install it to /mnt same as the
existing pre-seeded-key path. Falls through to the original
warning+confirm if the prompt is skipped, the path doesn't have the
key, or the run isn't interactive at all.
docs/auto-installer.md updated to mention the new fallback. Quick
bash -n + shellcheck pass only, per request.
Running the script standalone (its whole point per the last commit)
failed with "disko: command not found" -- jq/disko/nixos-install are
only guaranteed present via the built installer image's
environment.systemPackages, not on a plain checkout.
Added a #!/usr/bin/env nix-shell / #!nix-shell -i bash -p jq disko
nixos-install-tools shebang instead of a per-tool fallback: disko's own
generated scripts already hardcode absolute Nix store paths for
everything they shell out to internally (parted/sgdisk/mkfs.*/zfs/...
confirmed by inspecting a generated system.build.formatScript earlier),
so these three are the only genuinely external dependencies the script
itself has. This is a fast no-op on the built installer image (already
has all three) and what makes it also work standalone.
Quick syntax + shellcheck pass only this round (bash -n, shellcheck
with a `shellcheck shell=bash` directive since it doesn't recognize
nix-shell shebangs natively) -- skipping the full codex-maintenance.sh
sweep per request, to get this out for a real hardware test.
Two real bugs, both hit live:
1. Shebang: #!/run/current-system/sw/bin/bash only resolves on an
already-activated NixOS system -- running the checked-out script
directly (e.g. from a stock ISO, cloned repo) failed with "cannot
execute: required file not found" on a non-NixOS box. Switched to
#!/usr/bin/env bash, which resolves identically on NixOS
(environment.usrbinenv's own default) and any normal Linux distro.
Also fixed the file's missing executable bit.
2. FLAKE_BASE_URL: previously depended on pkgs.replaceVars substituting
a Nix-templated @lanDomain@ placeholder at build time -- meaning it
only ever worked when baked into the built installer image, not when
run straight from a checkout (the literal, unexpanded "@lanDomain@"
string reached git as a bogus hostname). Replaced with LAN_DOMAIN in
scripts/env.sh (manually kept in sync with variables.nix's lanDomain,
same pattern as NIX_CACHE_HOST/nixCacheHost already), sourced by the
script itself like every other script in scripts/. Dropped
pkgs.replaceVars from modules/installer/common.nix entirely --
scripts/env.sh is now baked into the image alongside auto-install.sh
at a matching relative path (/etc/nixos-installer/env.sh next to
/etc/nixos-installer/installer/auto-install.sh) so the script's own
relative `source` line resolves the same way in both contexts.
loginShellInit's invocation path and docs/auto-installer.md updated to
match. Verified: shellcheck clean on both scripts, the baked files are
byte-identical to their checked-in sources (no templating left to
verify), and codex-maintenance.sh (secret grep, fmt, statix, full eval
of every host/package including the installer/pxe artifacts) passes
clean.
Moves the auto-installer's shell script from an inline Nix string in
modules/installer/common.nix to scripts/installer/auto-install.sh, a
real, version-controlled, directly-editable/shellcheck-able file.
common.nix now wires it in with pkgs.replaceVars, substituting the one
value that actually needs to come from variables.nix (lanDomain) --
every other `${...}` in the script is a literal bash reference, left
untouched. replaceVars fails the build if any @name@-shaped placeholder
is left unsubstituted, so a typo'd or renamed variable is caught at
eval time rather than silently shipping broken.
Verified: built the substituted derivation and diffed it against the
source template -- identical except for the one substituted line, no
leftover unsubstituted placeholders. Full codex-maintenance.sh (secret
grep, fmt, statix, full eval of every host/package including the
installer/pxe artifacts that consume this) passes clean.
codex-maintenance.sh evaluated each affected host/package one at a time,
even though those calls are independent. Added scripts/lib/nix-parallel.sh
(run_nix_parallel) and wired it into the host-eval, package-eval, and
dry-run-build loops.
Concurrency defaults to core count capped by available memory (~1GB/job)
rather than plain nproc: empirically, nproc-many concurrent full-flake
evals OOM-killed each other on a 4GB/6-core box, while 3-4 ran clean and
were still ~2x faster than serial. Override via NIX_PARALLEL_JOBS.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Backs up a VM/CT on pve1 (snapshot mode by default, so the source stays
online), relays the archive to pve-test, restores it there with fresh
MAC addresses (--unique), and deletes both the source and relayed
backup copies afterward -- no ad hoc backup files left behind on
either node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pve1.sweet.home is production; scripts/env.sh now also defines
PVE_TEST_HOST for a separate sandbox node, individually targetable via
--node/PROXMOX_HOST. Tooling defaults are unchanged (still pve1) -- the
new restriction (Claude defaults to pve-test unless explicitly told to
use pve1) is documented as policy in CLAUDE.md, not enforced in the
scripts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The kernel's NFS client filesystem doesn't set FS_USERNS_MOUNT, so mounting
NFS from inside any non-init user namespace -- exactly what an unprivileged
LXC container's UID-mapped root runs in -- is rejected at the VFS layer
with EPERM, regardless of Proxmox's mount=nfs;nfs4 container feature (which
only patches the AppArmor layer). Confirmed live on the redeployed lxc-docker
container: TCP to the NFS server's port 2049 succeeds, the server's export
table matches the container's IP, and mount.nfs: Operation not permitted
still fires immediately with no corresponding denial anywhere in the
server's own logs -- a kernel-level rejection that no amount of DNS/
automount/export tweaking (this branch's earlier commits) could ever fix.
modules/platforms/lxc.nix now keys proxmoxLXC.privileged off hostName
("docker" -> true) rather than a blanket false, since build-types/docker.nix
is also composed for linode-docker/proxmox-docker, which don't import
proxmox-lxc.nix at all -- setting this option there would break their eval.
create-proxmox-resource.sh reads the value back via a new
flake_target_lxc_privileged helper instead of hardcoding --unprivileged 1,
so the two stay in sync automatically for every lxc-* target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
create-proxmox-resource.sh's `qm create` never passed --agent, so despite
services.qemuGuest.enable = true being set on every host, Proxmox never
created the virtio-serial channel the guest agent needs -- qm guest exec
and the UI's IP-address display silently never worked for any VM this
script created. Found while live-testing every lxc-*/proxmox-* build type
against pve.sweet.home for an end-to-end flake audit.
Also registers a fresh sops age key for lxc-gui (no prior registration
existed), generated while testing that target live -- needed before
lxc-gui can be deployed with working secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZHwwAM7cacQKkqKfQ5e8a
CI was running a full eval of every host + package on every push/PR,
which was slow enough to routinely time out the Gitea runner. Default
mode now diffs against a base ref and scopes nixpkgs-fmt/statix/eval to
the files that changed and the hosts/packages they can affect; a change
to flake.nix/flake.lock/variables.nix/modules/common/* (or any other
modules/*.nix outside platforms//build-types, whose blast radius isn't
inferable from the path) falls back to evaluating everything. The old
full sweep moves behind --full-check, which CI never passes; --dry-run
adds build-planning on top of whichever scope is active.
Also trims codex-setup.sh's redundant full host eval loop -- that's
what codex-maintenance.sh is for; setup should just install tooling.
variables.nix's nixCacheHostKey no longer matched nix-cache's actual SSH
host key (confirmed via ssh-keyscan against the live container), so every
declaratively-configured client's programs.ssh.knownHosts trusted the
wrong key -- distributed builds would fail host-key verification. Also,
modules/nix-cache/remote-builder-client.nix hardcoded sshKey to
/root/.ssh/nixremote, but the `server` host only has its own default
/root/.ssh/id_ed25519 installed (confirmed live via qm guest-agent) --
that file was never even present, so the build machine config pointed at
nothing. Standardize on each client's own default identity, matching the
per-host-key pattern vars.remoteBuilderAuthorizedKeys already uses instead
of a shared/differently-named keypair, and add
scripts/secrets/sync-nix-cache-host-key.sh (wired into
codex-maintenance.sh's --check) so the host-key drift doesn't silently
recur next time nix-cache is rebuilt or recreated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V7yVH71vGrDVzovh9UaMu8
scripts/ had grown to 10 top-level scripts covering three distinct
concerns (sops/age + SSH host-key management, Proxmox deployment, and
repo-wide bootstrap/CI) with no grouping. Move the key-management scripts
(backup-admin-key.sh, rotate-admin-key.sh, prepare-host-key.sh,
sync-host-keys.sh) into scripts/secrets/, and the Proxmox scripts
(create-proxmox-resource.sh, configure-nix-cache-client.sh) into
scripts/proxmox/; leave env.sh, codex-setup.sh, codex-maintenance.sh, and
bump-nixpkgs-release.sh at the top level (frequently hand-typed or pure
shared config) and scripts/lib/ as-is.
Updates every cross-reference: each moved script's repo_root computation
(now one directory deeper), shellcheck source= directives, inter-script
paths (create-proxmox-resource.sh's call into sync-host-keys.sh and its
remote bootstrap of configure-nix-cache-client.sh on the Proxmox node),
and every doc/module mention (CLAUDE.md's Scripts section reorganized to
match, README.md, docs/auto-installer.md, docs/proxmox-images.md,
modules/installer/common.nix, modules/platforms/lxc.nix). CI workflows
need no change -- they only invoke codex-maintenance.sh, which didn't
move. Verified via bash -n, shellcheck (no new warnings beyond the
pre-existing SC1091/SC2029/SC2095 baseline), and live dry-runs of
sync-host-keys.sh --all and create-proxmox-resource.sh --list from their
new paths.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scripts/backup-admin-key.sh, rotate-admin-key.sh, and sync-host-keys.sh
each independently resolved sops/age's default key-file path, derived an
age pubkey from an identity file, and (two of them) ran `sops updatekeys`
the same way -- now shared via scripts/lib/sops-age.sh. Also extracted the
"type X to confirm" prompt duplicated across create-proxmox-resource.sh
and sync-host-keys.sh into scripts/lib/confirm.sh. Pure extraction, no
behavior change -- each call site produces identical commands/output to
before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Run it once, right after a node's first-time Nix bootstrap (not on every
invocation, and not inside codex-setup.sh/codex-maintenance.sh themselves),
so a freshly-bootstrapped Proxmox node substitutes from and can offload
builds to nix-cache on every subsequent run. Non-fatal on failure -- the
build still proceeds, just without nix-cache.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Non-NixOS machines with just the Nix package manager installed have no
module system to pick up modules/nix-cache/client.nix, so this edits
/etc/nix/nix.conf directly (extra-substituters/extra-trusted-public-keys,
plus the SSH remote-builder config once the nixremote key is installed),
falling back to cache.nixos.org when nix-cache is unreachable.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-run tooling verification (added in the previous commit) was hardcoded
to checking just `nix`. Turn it into a small array instead, so a future
remote build step needing another tool extends that list rather than
growing a second parallel check.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two bugs found running scripts/create-proxmox-resource.sh against a real
Proxmox node for the first time:
- The Nix installer's single-user root path still shells out to `sudo`
to create /nix even though it already detected it's running as root,
which fails outright on a minimal, sudo-less Debian/Proxmox node.
codex-setup.sh now pre-creates /nix itself so that branch of the
installer is skipped.
- ensure_remote_repo() only ran scripts/codex-setup.sh right after a
fresh git clone, so a bootstrap that cloned the repo but then failed
installing Nix (exactly the failure above) left every subsequent run
silently building with a `nix` that was never actually installed,
since the repo already existing skipped tooling setup entirely. It
now checks `command -v nix` (via the same ensure_nix_profile used
elsewhere, since a non-interactive ssh session won't otherwise have a
single-user install on PATH) on every run and re-bootstraps if it's
missing. Both remote build heredocs also now source
scripts/lib/nix-bootstrap.sh themselves for the same PATH reason.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
create-proxmox-resource.sh no longer builds locally and scp's a
multi-gigabyte image over -- it now clones/pulls this repo onto the
Proxmox node itself (bootstrapping build tooling via the existing
codex-setup.sh on first use) and runs the nix build / disko image
script there, staging the result straight into the node's own import
directory. host-keys/ (gitignored) is copied over separately since a
git pull doesn't carry it. --image still uploads an explicit local
file for the case where you don't want a build at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
--allow-duplicate-host previously just skipped the existing-resource
check entirely, so re-running e.g. --type lxc --host docker while an
lxc-docker container already existed created a second container
sharing the same hostname/identity instead of replacing it -- both
then fight over DNS/DHCP for that hostname, and it's easy to end up
testing the stale one without realizing.
Now splits matches into "exact" (same --type as the one being
created, e.g. another lxc-docker) and "cross-type" (a different
platform sharing this host identity, e.g. a proxmox-docker VM
alongside an lxc-docker container -- a deliberate, valid coexistence
this script has never managed and still won't). Only an exact match is
destroyed and replaced, after typing the hostname back to confirm; a
cross-type match is always left untouched. Without
--allow-duplicate-host, both cases still refuse to run exactly as
before.
Verified live against pve.sweet.home: correctly split VMID 103 (a
stopped proxmox-docker VM, cross-type -- left untouched) from VMID 105
(the running lxc-docker container, exact-type -- flagged for
destroy+replace), and confirmed the destroy prompt safely aborts on a
non-matching confirmation, leaving both resources untouched.
Two compounding bugs, confirmed live on the running lxc-docker
container (vmid 102 on pve.sweet.home):
1. x-systemd.automount never works inside any Linux container --
systemd logs "Starting of <unit>.automount unsupported" for every
share and never mounts them. modules/docker/mount-data.nix and
modules/raspi/mount-data.nix now key off config.boot.isContainer
(set true by nixpkgs' proxmox-lxc.nix) to mount eagerly with
`nofail` there instead, while VM-based docker targets keep automount
unchanged.
2. The container's Proxmox `features` never included `mount=nfs`, so
AppArmor blanket-denies the nfs/rpc_pipefs mount syscalls NFS
needs ("permission denied"). scripts/env.sh's
PROXMOX_DEFAULT_LXC_FEATURES now includes mount=nfs;nfs4 for future
lxc-* containers -- the semicolon required quoting the --features
value in create-proxmox-resource.sh's remote pct-create command,
since it's sent as a raw string for the remote shell to parse and an
unquoted `;` would be read as a command separator.
The already-running container needs a matching `pct set --features`
plus a restart to pick this up -- that's an operator step outside this
repo.
Companion to rotate-admin-key.sh: copies whatever age identity sops/age
itself would resolve (or an explicit --key-file) to a given destination
path with 0600 perms, validating it's a real identity and round-tripping
the derived public key before/after the write so a corrupted copy is
caught immediately rather than discovered later during a restore.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Automates the manual steps sync-host-keys.sh/create-proxmox-resource.sh
print when they bootstrap a fresh, not-yet-trusted age key: verifies a
backed-up key matches the current &admin entry, swaps in a new key, and
re-encrypts every secrets/*.yaml. Explicitly cds into repo_root before any
sops call, since sops resolves .sops.yaml by walking up from cwd rather
than from the target file's path -- confirmed via a scratch-repo test that
running from elsewhere would otherwise silently rotate against the wrong
config.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The remote bash script run over SSH ended with a for-loop whose last
statement was `[[ "$n" == "$target" ]] && echo ...`. When the last
VM/CT checked on the node didn't match --host, that test evaluated
false and became the exit status of the whole remote script (1) --
which the wrapper then misreported as "couldn't reach the node",
even though SSH connectivity and the check itself were both fine.
The actual signal is the script's stdout, not its exit code, so end
it with an explicit exit 0.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Three chunks of copy-pasted logic were drifting across scripts/*.sh:
- codex-setup.sh and codex-maintenance.sh each carried an identical
NIX_CONFIG bootstrap + ensure_nix_profile() -> scripts/lib/nix-bootstrap.sh
- sync-host-keys.sh and prepare-host-key.sh each ran the same
ssh-keygen/ssh-to-age nix-shell invocations -> scripts/lib/ssh-host-keys.sh
(prepare-host-key.sh now also calls env.sh's nix_extra_opts before using
them, closing a gap where it alone skipped the nix-cache reachability
check env.sh exists for)
- the "list nixosConfigurations attrNames" / "get one target's hostName"
nix eval pattern was repeated across codex-setup.sh, codex-maintenance.sh,
sync-host-keys.sh and create-proxmox-resource.sh (the latter twice, in
its own --list and --host lookup) -> scripts/lib/nix-eval.sh, which also
centralizes the --no-use-registries --no-accept-flake-config flag pair
used on every such call
Verified against the real flake/node config (nix is available here):
create-proxmox-resource.sh --list for both --type lxc/vm, a full
--dry-run create, and prepare-host-key.sh generating and cleaning up a
real key/age-pubkey pair.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
variables.nix's deployedTargets was a manually-maintained list with no
enforcement keeping it in sync with reality -- it caused two separate
false refusals in a row (naming a VM as deployed well after it had been
destroyed, then matching a target against itself once the list was
"corrected"). Static files can't track whether a resource still actually
exists.
create-proxmox-resource.sh's duplicate-host guard now queries the
Proxmox node directly (qm/pct's own name/hostname config, matched
against --host) instead. Also fixes a gap in that live check: it
originally swallowed ssh failures and would have silently treated "can't
reach the node" the same as "checked, nothing there" -- it now refuses
instead of guessing when the node can't be reached.
deployedTargets is removed entirely from variables.nix since nothing
else in the repo consumed it once this script no longer does; README.md's
Hosts table remains the sole source of truth for "(real, deployed)"
status. CLAUDE.md and the script's own --help/comments updated to match.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The duplicate-host check in create-proxmox-resource.sh compared by
hostName only, so it fired even when the target being created was
exactly the one variables.nix's deployedTargets already names (e.g.
rebuilding lxc-nix-cache after destroying its old container to pick up
new sops secrets) -- there's no other machine at risk of an identity
collision in that case, just the normal redeploy workflow. Skip the
check when dt == flake_target; the later VMID-existence check still
guards against clobbering a resource that's actually live on the node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The reachability check used `cat < /dev/tcp/${NIX_CACHE_HOST}/22`, which
blocks forever reading for EOF that never comes -- sshd sends its banner
and then holds the connection open waiting for the client to speak next.
Every single check hit the 3s timeout and reported "unreachable"
unconditionally, regardless of whether the remote builder was actually up.
Confirmed live: a plain TCP connect (`exec 3<>/dev/tcp/...`, no read)
returns in ~60ms against a healthy nix-cache instead of always timing out.
Fixing that exposed a second, previously-dormant bug: `printf -v
NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"` on a genuinely empty NIX_OPTS array
still runs one format pass and yields the literal `'' ` rather than an
empty string. A subprocess (e.g. sync-host-keys.sh) reusing this
process's decision via `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` then rebuilt
a 1-element array holding an empty string instead of a 0-element array,
which broke `nix-shell "${NIX_OPTS[@]}" -p <pkg>` with a bogus positional
argument the moment NIX_OPTS was legitimately empty (nix-cache reachable)
-- something the first bug had made impossible to ever hit before.
Also adds a couple of retries (1s apart) to both checks as a secondary
safety net against genuine multi-second blips, on top of fixing the
checks themselves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The tarball/disko-image build previews were hardcoded strings that never
included ${NIX_OPTS[@]}, so --dry-run always showed the same "would build"
command whether nix-cache's substituter/remote-builder got disabled by
nix_extra_opts's reachability probe or not -- the actual (non-dry-run)
build commands already applied it correctly, only the preview lied.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
--name (used as pct/qm create's --hostname/--name) defaulted to
$flake_target (e.g. "lxc-nix-cache"), not $host (e.g. "nix-cache"). Since
proxmoxLXC.manageHostName pulls the guest's real networking.hostName
straight from Proxmox's own container config, this silently overrode
host.nix's hostName with a build-type-specific name. Default --name to
--host instead, so the guest's identity matches host.nix regardless of
which platform variant built it.
Found by spinning up a fresh lxc-nix-cache test container and noticing its
hostname was "lxc-nix-cache" instead of "nix-cache".
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Found and fixed live against a real test container (VMID 100, lxc-nix-cache
on pve.sweet.home) after the previous pct-restore-to-pct-create fix still
produced a container that booted into garbled console output:
1. pct create's own CLI default for --unprivileged is privileged (unlike
the web UI, whose checkbox defaults the other way), but
modules/platforms/lxc.nix sets proxmoxLXC.privileged = false, so the
image assumes it's running unprivileged. Real mismatch -- now passes
--unprivileged 1 explicitly.
2. The actual root cause of the garbled console: modern (v247+) systemd
routinely uses nested user namespaces and credential mounts (even
plain getty units, via LoadCredential=-style mechanisms), which
AppArmor's default LXC confinement denies without --features
nesting=1,keyctl=1. Confirmed via the host's kernel audit log: every
getty unit was crash-looping on a denied /run/credentials/* mount
every ~3s, and core services like nsncd failed userns_create the same
way -- the system never finished activating. Fixed live (pct set +
restart on the running test container) before committing the script
change: systemctl is-system-running went from never completing to
"running" with zero failed units.
3. --memory doesn't touch swap -- confirmed live it silently stayed at
Proxmox's own 512M default with --memory 2048. Now defaults --swap to
whatever --memory resolves to.
docs/auto-installer.md's manual pct create walkthrough gets the same
fixes, with the "why" for each flag, since a human following it by hand
would hit the identical bugs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Two independent problems found while actually running the script:
1. nix build/nix-shell retry each unreachable substituter/builder up to
5x with backoff, per store path -- with nix-cache down this compounds
into minutes of noise. scripts/env.sh gains nix_extra_opts(), which
probes http://nix-cache and nixremote@nix-cache:22 once via plain
curl/TCP (bypassing Nix's own retry logic entirely -- confirmed
nix store ping still retries 5x even with a short connect-timeout)
and exports the decision so create-proxmox-resource.sh and the
sync-host-keys.sh subprocess it shells out to both reuse it instead
of probing independently.
2. The actual failure: "archive contains no configuration file". pct
restore expects a vzdump backup archive with embedded config;
config.system.build.tarball is a plain CT template tarball -- wrong
Proxmox mechanism entirely. Fixed to pct create against it as a vztmpl
template instead, uploaded to /var/lib/vz/template/cache/ rather than
/var/lib/vz/dump/. This same wrong claim had propagated into
docs/auto-installer.md, README.md, and CLAUDE.md from when the script
was first written -- corrected everywhere.
Also: checks for an already-uploaded image on the node (fixed
<flake_target>.tar.xz/.raw naming) before building, skipping build+upload
entirely if found (--force-rebuild to always rebuild).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
sync-host-keys.sh: generates/registers SSH host keys and their
.sops.yaml/secrets/*.yaml recipients for flake targets, idempotently.
--all, <target>, --remove, --regenerate-all-keys, all with --dry-run
(verified zero-side-effect via a sandboxed git-status check across every
mode). Only ever touches anchors with a corresponding host-keys/ file --
&admin and any hand-registered real-host anchor are never listed,
removed, or regenerated. Supersedes running prepare-host-key.sh one host
at a time for any target that already has a flake entry.
create-proxmox-resource.sh: builds a lxc-*/proxmox-* target's
tarball/disk image and creates it on a real Proxmox node, or reconfigures
an existing resource's cores/memory/disk (--modify, always requires
typing the VMID back to confirm). Refuses to create a new resource for a
VMID that already exists, and refuses to duplicate a host identity that
already has a real deployment elsewhere (variables.nix's new
deployedTargets, checked by hostName so it also catches cross-platform
duplicates) unless --allow-duplicate-host is passed. --dry-run throughout.
scripts/env.sh centralizes the Proxmox connection config both scripts
(and future ones) share. Also fixes an unrelated gap found along the way:
proxmox-* Disko image builds write their .raw file straight into the
repo root, and .gitignore never covered it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
flake.nix's nixpkgs.url/home-manager.url can't source a version string
from variables.nix -- flake input resolution requires a plain string
literal, confirmed empirically (nix flake metadata errors with
"expected a string or a path but got a thunk" otherwise). This script
is the one-command alternative: bump both release branches in flake.nix
via targeted substitution (never a blind repo-wide replace, so it can't
collide with stateVersion strings elsewhere), with an upstream branch
existence check before writing, plus an optional --tooling flag for
codex-maintenance.sh's separately-pinned nixpkgs-fmt/statix fetch.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Both existed only so the installer could boot as an LXC container and
nixos-install some other host from within it, but lxc-* targets are
already excluded from the install menu (nixos-install can't touch its
own running root filesystem), and now have their own direct tarball
path anyway. That left the installer's own LXC form with no real use
case, and packages.all with only two members worth bundling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Both check-nixos.yml workflows (GitHub + Gitea) now call
scripts/codex-maintenance.sh instead of a hand-rolled eval-only loop,
closing a real gap: CI previously enforced none of the secret grep,
nixpkgs-fmt, or statix checks that codex-maintenance.sh already runs
locally — nothing was stopping that from regressing. One script now
backs both, instead of two copies that can drift from each other.
codex-maintenance.sh itself is extended to cover buildable surface
that wasn't validated anywhere before: packages.x86_64-linux.*, plus
config.system.build.tarball (lxc-* hosts) and
config.system.build.diskoImagesScript (proxmox-*, excluding the
installer's own proxmox-lxc target, which has no disko config).
Also:
- scripts/prepare-host-key.sh: dropped the redundant
[path-to-nixos-repo] parameter — it always defaults to the repo the
script itself lives in now, so a second argument never made sense
after the nix-auto-installer migration.
- Removed prepare.sh (dead pre-disko manual parted/mkfs/mkswap
partitioning, fully superseded) and
scripts/create-linode-installer-disk.sh (incomplete draft for an
abandoned dd-via-rescue-mode approach; Linode hosts already deploy
fine through the normal auto-installer flow).
- docs/pxe-boot.md: fixed a stale `nixosConfigurations.pxe-boot` eval
command (pre-refactor flat name, not a real flake attribute
anymore) and added a cross-reference to docs/auto-installer.md.
- CLAUDE.md/README.md: full documentation pass reconciling this
session's changes — modules/installer/, modules/pxe-boot/, the
LXC/Proxmox image-building deployment paths, corrected the
password-hash/SSH-key locations in the safety-rules section (both
had drifted to reference files/paths that no longer exist), and
added session-workflow guidance to prefer targeted host evals over
full-repo sweeps for incremental changes (explicitly scoped to
interactive sessions, not CI).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Finishes the nix-auto-installer migration: scripts/prepare-host-key.sh
and the local host-keys/ directory (gitignored, private key material,
never committed — moved as plain files, not through git history)
weren't carried over in the initial migration.
Also implements automatic key staging, replacing the manual
scp-after-boot step:
- modules/installer/host-keys.nix reads host-keys/ via
builtins.getEnv, which Nix silently returns as "" under normal
(non---impure) evaluation — the module is a no-op by default, safe
for CI, until explicitly opted into:
NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build .#iso --impure
When built this way every key present gets baked into the image at
/etc/host-keys/, and auto-install.sh installs whichever one matches
the flake target selected at install time — no manual per-host scp.
- This deliberately includes the PXE netboot variant, even though
pxe-boot serves it unauthenticated over LAN HTTP: accepted
explicitly as a reasonable trade-off for a network that sits behind
LAN-only infrastructure, not the open internet. auto-install.sh
still falls back to /root/host-keys (manual scp) if a key isn't
baked in, so images built without --impure keep working exactly as
before.
- docs/auto-installer.md replaces nix-auto-installer's README,
updated for in-repo paths and the new build flow.
Verified: normal `nix eval` (no --impure) evaluates identically across
all 19 nixosConfigurations + 4 packages with zero host-keys/* entries
(CI-unaffected); with --impure + the env var set, all three installer
variants (installer/ISO, proxmox-lxc, pxe) correctly embed every key
in host-keys/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Audited the working tree and full git history for committed secrets
(gitleaks + trufflehog + manual grep, see secrets-inventory.md, kept
local/gitignored per the spec). Found: a password hash shared by root
and the nixos user across every host, two live Beszel monitoring
tokens, and a GitHub fine-grained PAT embedded in a home-manager
nix.conf.
Migrates all of them to sops-nix:
- .sops.yaml + secrets/*.yaml, encrypted for admin + the age keys
derived (via ssh-to-age) from each live host's existing SSH host
key — no new key material transferred to any machine.
- users.users.{root,nixos}.hashedPasswordFile replaces the inline
hashedPassword shared by every target.
- The GitHub PAT moves from a home-manager-managed, store-visible
nix.conf to a sops.templates-rendered file included via nix.conf's
native !include, system-wide instead of per-user.
- Beszel TOKEN moves from `environment` (store-visible) to
`environmentFile` (runtime-only via sops.templates); the dead
commented-out docker token is removed from the tree entirely.
Added a tracked pre-commit hook (gitleaks protect --staged, wired via
core.hooksPath) so a secret can't be committed by accident again, and
documented the sops workflow in README.md.
Structural verification only: all 17 flake targets evaluate, and
`nix build --dry-run --no-link` succeeds for the three currently
deployed hosts. Per CLAUDE.md, actual `nixos-rebuild switch` — the
step that confirms secrets decrypt and services start on a real
machine — is left for manual verification.
Git history still contains the original plaintext secrets; scrubbing
history (Milestone 3) and rotating every credential (Milestone 4) are
separate, deliberately gated steps per remove-sensetive-info-refactor.md.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
beatzaplenty
