Compare commits

...
Author SHA1 Message Date
beatzaplentyandClaude Sonnet 5 013b2c7009 Add ad hoc pve1 -> pve-test clone script (vzdump + qmrestore/pct restore)
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m21s
Backs up a VM/CT on pve1 (snapshot mode by default, so the source stays
online), relays the archive to pve-test, restores it there with fresh
MAC addresses (--unique), and deletes both the source and relayed
backup copies afterward -- no ad hoc backup files left behind on
either node.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-21 20:22:56 +00:00
beatzaplenty 12f9153957 claude audit report
Check NixOS configurations / eval-hosts (push) Successful in 10m36s
2026-07-21 20:07:30 +00:00
beatzaplenty 1004538f00 updated sops secrets 2026-07-21 20:07:14 +00:00
beatzaplentyandClaude Sonnet 5 87873300e1 Add pve-test.sweet.home as a second Proxmox target
pve1.sweet.home is production; scripts/env.sh now also defines
PVE_TEST_HOST for a separate sandbox node, individually targetable via
--node/PROXMOX_HOST. Tooling defaults are unchanged (still pve1) -- the
new restriction (Claude defaults to pve-test unless explicitly told to
use pve1) is documented as policy in CLAUDE.md, not enforced in the
scripts.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-21 19:57:15 +00:00
beatzaplenty e9e2312163 Merge pull request 'Make lxc-docker a privileged container: unprivileged can't NFS-mount at all' (#28) from worktree-fix-container-dns-search-domain into main
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
Reviewed-on: #28
2026-07-21 08:50:14 +00:00
beatzaplentyandClaude Sonnet 5 6b09a808ed Make lxc-docker a privileged container: unprivileged can't NFS-mount at all
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
The kernel's NFS client filesystem doesn't set FS_USERNS_MOUNT, so mounting
NFS from inside any non-init user namespace -- exactly what an unprivileged
LXC container's UID-mapped root runs in -- is rejected at the VFS layer
with EPERM, regardless of Proxmox's mount=nfs;nfs4 container feature (which
only patches the AppArmor layer). Confirmed live on the redeployed lxc-docker
container: TCP to the NFS server's port 2049 succeeds, the server's export
table matches the container's IP, and mount.nfs: Operation not permitted
still fires immediately with no corresponding denial anywhere in the
server's own logs -- a kernel-level rejection that no amount of DNS/
automount/export tweaking (this branch's earlier commits) could ever fix.

modules/platforms/lxc.nix now keys proxmoxLXC.privileged off hostName
("docker" -> true) rather than a blanket false, since build-types/docker.nix
is also composed for linode-docker/proxmox-docker, which don't import
proxmox-lxc.nix at all -- setting this option there would break their eval.
create-proxmox-resource.sh reads the value back via a new
flake_target_lxc_privileged helper instead of hardcoding --unprivileged 1,
so the two stay in sync automatically for every lxc-* target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
2026-07-21 05:57:06 +00:00
beatzaplenty 9496efdd22 Merge pull request 'Add pve.sweet.home guard rails to CLAUDE.md' (#26) from worktree-claude-md-pve-guardrails into main
Check NixOS configurations / eval-hosts (push) Successful in 10m13s
Reviewed-on: #26
2026-07-21 04:28:14 +00:00
beatzaplenty 33c9506c7d Merge pull request 'Fix NFS mount device strings on lxc-docker: use FQDN, not search domain' (#25) from worktree-fix-container-dns-search-domain into main
Check NixOS configurations / eval-hosts (push) Failing after 7m3s
Reviewed-on: #25
2026-07-21 04:26:32 +00:00
beatzaplentyandClaude Sonnet 5 abe3763cb3 Add pve.sweet.home guard rails to CLAUDE.md
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m24s
Codifies read-only access to existing Proxmox config/VMs/containers,
allows scratch test VMs/containers as long as they're torn down again,
and forbids any change to production on the node.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-21 02:01:15 +00:00
beatzaplenty 744904b19f Merge pull request 'Enable QEMU guest agent on Proxmox VMs; register lxc-gui sops key' (#24) from worktree-flake-e2e-audit into main
Check NixOS configurations / eval-hosts (push) Successful in 16m3s
Reviewed-on: #24
2026-07-21 01:46:06 +00:00
beatzaplentyandClaude Sonnet 5 9cbaf1a070 Enable QEMU guest agent on Proxmox VMs; register lxc-gui sops key
Check NixOS configurations / eval-hosts (pull_request) Successful in 16m19s
create-proxmox-resource.sh's `qm create` never passed --agent, so despite
services.qemuGuest.enable = true being set on every host, Proxmox never
created the virtio-serial channel the guest agent needs -- qm guest exec
and the UI's IP-address display silently never worked for any VM this
script created. Found while live-testing every lxc-*/proxmox-* build type
against pve.sweet.home for an end-to-end flake audit.

Also registers a fresh sops age key for lxc-gui (no prior registration
existed), generated while testing that target live -- needed before
lxc-gui can be deployed with working secrets.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZHwwAM7cacQKkqKfQ5e8a
2026-07-21 01:25:16 +00:00
beatzaplentyandClaude Sonnet 5 42da626397 Fix NFS mount device strings on lxc-docker: use FQDN, not search domain
Check NixOS configurations / eval-hosts (pull_request) Successful in 17m29s
The previous commit's networking.search fix was wrong. Confirmed live
on lxc-docker (vmid 105) after redeploying with it: `resolvectl query
server.sweet.home` started failing again, even though
`resolvectl query --interface=eth0 server.sweet.home` still resolved
correctly to the right IP via the LAN's real DNS server. The debug log
showed why -- adding a *global* search domain via networking.search
gave systemd-resolved a domain-matched but server-less "global" scope,
which it now prioritizes over eth0's correctly-configured scope for
every "*.sweet.home" query, silently sending them to public fallback
DNS (1.1.1.1 et al) instead, which of course returns NXDOMAIN for an
internal-only name. Bare single-label names (e.g. "server") were never
going to work either way -- systemd-resolved only ever tries LLMNR for
those, never DNS search-suffixing, regardless of configuration.

Reverts the networking.search addition and instead has
modules/docker/mount-data.nix build each NFS device string from
"${vars.nfsServerHost}.${vars.homeDomain}" (a plain FQDN, no dependency
on search-domain behavior at all) -- the same pattern
modules/raspi/mount-data.nix already uses for the Raspberry Pi's share
and for the identical reason.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
2026-07-21 01:19:42 +00:00
beatzaplenty d7aba8554d Merge pull request 'Add global DNS search domain to fix NFS mounts on lxc-docker' (#23) from worktree-fix-container-dns-search-domain into main
Check NixOS configurations / eval-hosts (push) Successful in 18m29s
Reviewed-on: #23
2026-07-21 00:11:16 +00:00
beatzaplentyandClaude Sonnet 5 e176ff723d Add global DNS search domain to fix NFS mounts on lxc-docker
Check NixOS configurations / eval-hosts (pull_request) Failing after 40m2s
Cross-host references throughout this repo (vars.nfsServerHost,
vars.nixCacheHost, vars.dockerHost) are bare short names, not FQDNs.
Resolving them has always depended on whatever network stack happens
to be in play picking up the DHCP-advertised domain as a search
suffix -- NetworkManager does this by default, which is why it went
unnoticed everywhere else, but LXC containers force-disable
NetworkManager and get their systemd-networkd config written directly
by Proxmox instead, which never sets one.

Confirmed live on lxc-docker (vmid 105) after today's earlier fix for
the automount/mount=nfs bugs: systemd-resolved had no search domain
for eth0, so "server" failed to resolve ("Name or service not known")
while "server.sweet.home" resolved fine via the same DNS server --
every NFS mount in modules/docker/mount-data.nix was still failing.

networking.search sets systemd-resolved's own Domains= globally
rather than depending on the per-link DHCP path, so it isn't at the
mercy of whichever component owns a given host's interface file.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
2026-07-20 18:22:12 +00:00
beatzaplenty 61bbe5e6da clean up in ailse 3
Check NixOS configurations / eval-hosts (push) Failing after 8m41s
2026-07-20 17:49:35 +00:00
beatzaplenty ab719cc8eb Merge branch 'worktree-maintenance-script-changed-files-only' into worktree-resolve-maintenance-merge
Check NixOS configurations / eval-hosts (push) Successful in 21m49s
# Conflicts:
#	scripts/codex-maintenance.sh
2026-07-20 17:46:07 +00:00
beatzaplenty 91c977e5e7 Merge pull request 'Fix nix-cache remote-builder trust: stale host key + wrong sshKey path' (#21) from worktree-fix-nix-cache-host-key into main
Check NixOS configurations / eval-hosts (push) Failing after 12m42s
Reviewed-on: #21
2026-07-20 17:32:25 +00:00
beatzaplenty 7e9c0c2a6f Rewrite codex-maintenance.sh to scope CI checks to changed files
Check NixOS configurations / eval-hosts (pull_request) Canceled after 0s
CI was running a full eval of every host + package on every push/PR,
which was slow enough to routinely time out the Gitea runner. Default
mode now diffs against a base ref and scopes nixpkgs-fmt/statix/eval to
the files that changed and the hosts/packages they can affect; a change
to flake.nix/flake.lock/variables.nix/modules/common/* (or any other
modules/*.nix outside platforms//build-types, whose blast radius isn't
inferable from the path) falls back to evaluating everything. The old
full sweep moves behind --full-check, which CI never passes; --dry-run
adds build-planning on top of whichever scope is active.

Also trims codex-setup.sh's redundant full host eval loop -- that's
what codex-maintenance.sh is for; setup should just install tooling.
2026-07-20 17:25:22 +00:00
beatzaplentyandClaude Sonnet 5 fd773b65da Fix nix-cache remote-builder trust: stale host key + wrong sshKey path
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m12s
variables.nix's nixCacheHostKey no longer matched nix-cache's actual SSH
host key (confirmed via ssh-keyscan against the live container), so every
declaratively-configured client's programs.ssh.knownHosts trusted the
wrong key -- distributed builds would fail host-key verification. Also,
modules/nix-cache/remote-builder-client.nix hardcoded sshKey to
/root/.ssh/nixremote, but the `server` host only has its own default
/root/.ssh/id_ed25519 installed (confirmed live via qm guest-agent) --
that file was never even present, so the build machine config pointed at
nothing. Standardize on each client's own default identity, matching the
per-host-key pattern vars.remoteBuilderAuthorizedKeys already uses instead
of a shared/differently-named keypair, and add
scripts/secrets/sync-nix-cache-host-key.sh (wired into
codex-maintenance.sh's --check) so the host-key drift doesn't silently
recur next time nix-cache is rebuilt or recreated.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V7yVH71vGrDVzovh9UaMu8
2026-07-20 17:21:12 +00:00
beatzaplenty d340aca403 Merge pull request 'Authorize this box's SSH key as a nix-cache remote-builder client' (#20) from add-claude-builder-key into main
Check NixOS configurations / eval-hosts (push) Failing after 18m59s
Reviewed-on: #20
2026-07-20 16:31:25 +00:00
beatzaplenty bf8ee3ce48 Merge pull request 'Worktree refactor scripts shared lib' (#19) from worktree-refactor-scripts-shared-lib into main
Check NixOS configurations / eval-hosts (push) Failing after 11m3s
Reviewed-on: #19
2026-07-20 16:29:42 +00:00
beatzaplentyandClaude Sonnet 5 98d4545e8f Authorize this box's SSH key as a nix-cache remote-builder client
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m4s
Following the existing per-client-key pattern in
vars.remoteBuilderAuthorizedKeys (one entry per host's own default root
key, not a shared dedicated keypair -- docs/nix-cache.md's "./nixremote"
filename is just a generic placeholder in the instructions). This box
had no entry yet, which is why its earlier `ssh-copy-id nixremote@nix-cache`
never stuck: nix-cache's nixremote authorized_keys is NixOS-declarative
and gets overwritten on every rebuild.

Public key only; requires a nix-cache rebuild to take effect.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:29:42 +00:00
beatzaplentyandClaude Sonnet 5 d8687d979c Reorganize scripts/ into secrets/, proxmox/, and lib/ subfolders
Check NixOS configurations / eval-hosts (pull_request) Failing after 30m6s
scripts/ had grown to 10 top-level scripts covering three distinct
concerns (sops/age + SSH host-key management, Proxmox deployment, and
repo-wide bootstrap/CI) with no grouping. Move the key-management scripts
(backup-admin-key.sh, rotate-admin-key.sh, prepare-host-key.sh,
sync-host-keys.sh) into scripts/secrets/, and the Proxmox scripts
(create-proxmox-resource.sh, configure-nix-cache-client.sh) into
scripts/proxmox/; leave env.sh, codex-setup.sh, codex-maintenance.sh, and
bump-nixpkgs-release.sh at the top level (frequently hand-typed or pure
shared config) and scripts/lib/ as-is.

Updates every cross-reference: each moved script's repo_root computation
(now one directory deeper), shellcheck source= directives, inter-script
paths (create-proxmox-resource.sh's call into sync-host-keys.sh and its
remote bootstrap of configure-nix-cache-client.sh on the Proxmox node),
and every doc/module mention (CLAUDE.md's Scripts section reorganized to
match, README.md, docs/auto-installer.md, docs/proxmox-images.md,
modules/installer/common.nix, modules/platforms/lxc.nix). CI workflows
need no change -- they only invoke codex-maintenance.sh, which didn't
move. Verified via bash -n, shellcheck (no new warnings beyond the
pre-existing SC1091/SC2029/SC2095 baseline), and live dry-runs of
sync-host-keys.sh --all and create-proxmox-resource.sh --list from their
new paths.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:20:45 +00:00
beatzaplentyandClaude Sonnet 5 a2b557c034 Lift duplicated sops/age and confirm-prompt logic into scripts/lib/
scripts/backup-admin-key.sh, rotate-admin-key.sh, and sync-host-keys.sh
each independently resolved sops/age's default key-file path, derived an
age pubkey from an identity file, and (two of them) ran `sops updatekeys`
the same way -- now shared via scripts/lib/sops-age.sh. Also extracted the
"type X to confirm" prompt duplicated across create-proxmox-resource.sh
and sync-host-keys.sh into scripts/lib/confirm.sh. Pure extraction, no
behavior change -- each call site produces identical commands/output to
before.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:38:32 +00:00
beatzaplenty 1d44523181 Merge pull request 'Re-encrypt secrets/nix-cache.yaml for rotated lxc-nix-cache key' (#18) from worktree-nix-cache-502-fix into main
Check NixOS configurations / eval-hosts (push) Failing after 11m15s
Reviewed-on: #18
2026-07-20 15:33:31 +00:00
beatzaplentyandClaude Sonnet 5 222a3ced69 Re-encrypt secrets/nix-cache.yaml for current lxc-nix-cache key
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m53s
.sops.yaml's &lxc-nix-cache anchor was rotated (commit b3c8145) after
today's LXC rebuild, but secrets/nix-cache.yaml was never re-run through
sops updatekeys -- it stayed encrypted for the pre-rotation key. Per
docs/nix-cache.md and docs/auto-installer.md, sops-nix decrypt failures
are silent: cache-priv-key never materialized, nix-serve had nothing to
bind its secretKeyFile to and failed to start, and nginx proxy_pass had
no upstream -- the 502 Bad Gateway seen from clients.

Confirmed the new key matches the currently-deployed host by deriving
the age identity from host-keys/lxc-nix-cache_ssh_host_ed25519_key.pub
via ssh-to-age and comparing to .sops.yaml.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:29:34 +00:00
beatzaplenty 03137eef9a Merge pull request 'Add nix-cache client config script for non-NixOS Debian machines' (#17) from worktree-nix-cache-client-script into main
Check NixOS configurations / eval-hosts (push) Failing after 37m6s
Reviewed-on: #17
2026-07-20 15:14:52 +00:00
beatzaplentyandClaude Sonnet 5 af0fe5bdfd Wire configure-nix-cache-client.sh into create-proxmox-resource.sh's tooling bootstrap
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m15s
Run it once, right after a node's first-time Nix bootstrap (not on every
invocation, and not inside codex-setup.sh/codex-maintenance.sh themselves),
so a freshly-bootstrapped Proxmox node substitutes from and can offload
builds to nix-cache on every subsequent run. Non-fatal on failure -- the
build still proceeds, just without nix-cache.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:09:57 +00:00
beatzaplentyandClaude Sonnet 5 23b910a011 Add script to configure nix-cache as substituter/remote builder on Debian clients
Non-NixOS machines with just the Nix package manager installed have no
module system to pick up modules/nix-cache/client.nix, so this edits
/etc/nix/nix.conf directly (extra-substituters/extra-trusted-public-keys,
plus the SSH remote-builder config once the nixremote key is installed),
falling back to cache.nixos.org when nix-cache is unreachable.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:07:04 +00:00
beatzaplenty 19f076bba1 Merge pull request 'Fix proxmox remote build bootstrap' (#16) from fix-proxmox-remote-build-bootstrap into main
Check NixOS configurations / eval-hosts (push) Failing after 12m2s
Reviewed-on: #16
2026-07-20 14:43:31 +00:00
beatzaplentyandClaude Sonnet 5 9a1d6842d7 Generalize the remote tooling check to a list of required commands
Check NixOS configurations / eval-hosts (pull_request) Failing after 12m13s
Per-run tooling verification (added in the previous commit) was hardcoded
to checking just `nix`. Turn it into a small array instead, so a future
remote build step needing another tool extends that list rather than
growing a second parallel check.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 14:42:00 +00:00
beatzaplentyandClaude Sonnet 5 f5ef3194d4 Fix remote-build bootstrap: retry on partial failure, fix sudo-less root Nix install
Two bugs found running scripts/create-proxmox-resource.sh against a real
Proxmox node for the first time:

- The Nix installer's single-user root path still shells out to `sudo`
  to create /nix even though it already detected it's running as root,
  which fails outright on a minimal, sudo-less Debian/Proxmox node.
  codex-setup.sh now pre-creates /nix itself so that branch of the
  installer is skipped.

- ensure_remote_repo() only ran scripts/codex-setup.sh right after a
  fresh git clone, so a bootstrap that cloned the repo but then failed
  installing Nix (exactly the failure above) left every subsequent run
  silently building with a `nix` that was never actually installed,
  since the repo already existing skipped tooling setup entirely. It
  now checks `command -v nix` (via the same ensure_nix_profile used
  elsewhere, since a non-interactive ssh session won't otherwise have a
  single-user install on PATH) on every run and re-bootstraps if it's
  missing. Both remote build heredocs also now source
  scripts/lib/nix-bootstrap.sh themselves for the same PATH reason.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 14:39:43 +00:00
beatzaplenty 90e3397b42 worktree deleted
Check NixOS configurations / eval-hosts (push) Failing after 12m2s
2026-07-20 14:34:32 +00:00
beatzaplenty be5812d5bb Merge pull request 'Build Proxmox images directly on the node instead of transferring them' (#15) from worktree-proxmox-remote-build into main
Check NixOS configurations / eval-hosts (push) Failing after 11m29s
Reviewed-on: #15
2026-07-20 14:17:56 +00:00
beatzaplenty 84f7e038cb removed old work specs
Check NixOS configurations / eval-hosts (push) Failing after 11m42s
2026-07-20 13:36:03 +00:00
beatzaplentyandClaude Sonnet 5 91d8f8fab1 Add lxc-tor-relay build type with nyx monitoring
New tor-relay build type (currently lxc-only) running a plain Tor
middle relay via modules/tor/enable-relay.nix, plus nyx for
interactive monitoring over the relay's control socket.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 13:36:03 +00:00
beatzaplenty 723212a81f Merge pull request 'Worktree fix lxc docker nfs mounts' (#14) from worktree-fix-lxc-docker-nfs-mounts into main
Check NixOS configurations / eval-hosts (push) Failing after 11m25s
Reviewed-on: #14
2026-07-20 13:31:28 +00:00
beatzaplenty 75d09d57e3 Fix --allow-duplicate-host creating real duplicates in create-proxmox-resource.sh
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m24s
--allow-duplicate-host previously just skipped the existing-resource
check entirely, so re-running e.g. --type lxc --host docker while an
lxc-docker container already existed created a second container
sharing the same hostname/identity instead of replacing it -- both
then fight over DNS/DHCP for that hostname, and it's easy to end up
testing the stale one without realizing.

Now splits matches into "exact" (same --type as the one being
created, e.g. another lxc-docker) and "cross-type" (a different
platform sharing this host identity, e.g. a proxmox-docker VM
alongside an lxc-docker container -- a deliberate, valid coexistence
this script has never managed and still won't). Only an exact match is
destroyed and replaced, after typing the hostname back to confirm; a
cross-type match is always left untouched. Without
--allow-duplicate-host, both cases still refuse to run exactly as
before.

Verified live against pve.sweet.home: correctly split VMID 103 (a
stopped proxmox-docker VM, cross-type -- left untouched) from VMID 105
(the running lxc-docker container, exact-type -- flagged for
destroy+replace), and confirmed the destroy prompt safely aborts on a
non-matching confirmation, leaving both resources untouched.
2026-07-20 13:22:08 +00:00
beatzaplenty 6847a7a6f4 Fix NFS shares never mounting on lxc-docker
Two compounding bugs, confirmed live on the running lxc-docker
container (vmid 102 on pve.sweet.home):

1. x-systemd.automount never works inside any Linux container --
   systemd logs "Starting of <unit>.automount unsupported" for every
   share and never mounts them. modules/docker/mount-data.nix and
   modules/raspi/mount-data.nix now key off config.boot.isContainer
   (set true by nixpkgs' proxmox-lxc.nix) to mount eagerly with
   `nofail` there instead, while VM-based docker targets keep automount
   unchanged.

2. The container's Proxmox `features` never included `mount=nfs`, so
   AppArmor blanket-denies the nfs/rpc_pipefs mount syscalls NFS
   needs ("permission denied"). scripts/env.sh's
   PROXMOX_DEFAULT_LXC_FEATURES now includes mount=nfs;nfs4 for future
   lxc-* containers -- the semicolon required quoting the --features
   value in create-proxmox-resource.sh's remote pct-create command,
   since it's sent as a raw string for the remote shell to parse and an
   unquoted `;` would be read as a command separator.

The already-running container needs a matching `pct set --features`
plus a restart to pick this up -- that's an operator step outside this
repo.
2026-07-20 12:56:42 +00:00
42 changed files with 1894 additions and 629 deletions
+9 -1
View File
@@ -13,9 +13,17 @@ jobs:
steps:
- name: Check out repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install Nix
uses: DeterminateSystems/nix-installer-action@v19
- name: Run maintenance checks (secrets, fmt, lint, eval)
# Scoped to files changed since the PR base / previous push -- see
# scripts/codex-maintenance.sh. CI never passes --full-check: that
# full sweep is for local/manual use, since it's slow enough to time
# out this runner.
- name: Run maintenance checks (secrets, fmt, lint, eval -- changed files only)
env:
MAINT_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: bash scripts/codex-maintenance.sh
+9 -1
View File
@@ -13,9 +13,17 @@ jobs:
steps:
- name: Check out repository
uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Install Nix
uses: DeterminateSystems/nix-installer-action@v19
- name: Run maintenance checks (secrets, fmt, lint, eval)
# Scoped to files changed since the PR base / previous push -- see
# scripts/codex-maintenance.sh. CI never passes --full-check: that
# full sweep is for local/manual use, since it's slow enough to time
# out this runner.
- name: Run maintenance checks (secrets, fmt, lint, eval -- changed files only)
env:
MAINT_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
run: bash scripts/codex-maintenance.sh
+11 -1
View File
@@ -3,11 +3,15 @@ keys:
- &docker age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e
- &server age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
- &nix-cache age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
- &lxc-minimal age1qz9d4ka4xgexujyd247s7lp737sulp5fhxl5d65fj2ykvc4j4edqrsdks8
- &nix-minimal age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu
- &proxmox-minimal age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg
- &lxc-nix-cache age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
- &lxc-docker age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs
- &lxc-minimal age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep
- &lxc-pxe-boot age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp
- &lxc-gui age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05
- &proxmox-server age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
- &vm-server age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
creation_rules:
# Shared across every currently-deployed host: root/nixos password hash,
@@ -26,6 +30,10 @@ creation_rules:
- *lxc-nix-cache
- *proxmox-minimal
- *lxc-docker
- *lxc-pxe-boot
- *lxc-gui
- *proxmox-server
- *vm-server
- path_regex: secrets/nix-cache\.yaml$
key_groups:
@@ -39,6 +47,8 @@ creation_rules:
- age:
- *admin
- *server
- *proxmox-server
- *vm-server
- path_regex: secrets/docker\.yaml$
key_groups:
+7 -2
View File
@@ -7,7 +7,7 @@ servers and workstation.
The flake exposes NixOS configurations named `<platform>-<buildtype>`
(platforms: `linode`, `proxmox`, `lxc`; build types: `minimal`, `nix-cache`,
`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`), generated from `modules/platforms/*`
`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`, `tor-relay`), generated from `modules/platforms/*`
and `modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not
every combination is built — `pxe-boot` has no `linode` variant. See
`README.md` for the full current target list; treat `flake.nix` as the
@@ -35,9 +35,14 @@ Use these commands when validating changes:
```bash
bash scripts/codex-setup.sh
bash scripts/codex-maintenance.sh
bash scripts/codex-maintenance.sh dry-run
```
With no flags, `codex-maintenance.sh` scopes fmt-check/statix/eval to files
changed against a base ref — this is what CI runs on every push/PR. For the
full sweep (every host, every package — slow; CI never runs this), use
`bash scripts/codex-maintenance.sh --full-check` (add `--dry-run` for build
planning on top of whichever scope is active).
Host evaluation is safe when limited to drvPath checks:
```bash
+150
View File
@@ -0,0 +1,150 @@
# Flake End-to-End Audit Report
**Date:** 2026-07-21
**Scope:** Full static lint/eval sweep + live build/deploy/interrogate/destroy testing of every `lxc-*` and `proxmox-*` flake target against `pve.sweet.home`, plus an audit of the operator's ability to manage the flake/secrets tooling.
**Branch:** `worktree-flake-e2e-audit` (this session's isolated worktree)
## Executive Summary
The flake itself is in good shape: `nixpkgs-fmt`, `statix`, and a full eval + dry-run build of every host and package are all clean. Every `lxc-*`/`proxmox-*` target's NixOS configuration builds successfully — no target has a broken derivation graph.
The issues found are **operational, not code-level**:
1. **pve.sweet.home is critically low on disk space** (91-95% full during this session) and cannot currently build the two largest closures (`gui`, `pxe-boot`) to completion — this actively blocks deploying/redeploying those hosts via the documented workflow.
2. **A real, reproducible secrets-decryption failure** was caught live: a stale cached container image (built before a same-day sops-key fix) boots with sshd never starting and every secret failing to decrypt. This is a **general hazard in `create-proxmox-resource.sh`'s "reuse the cached image if present" default**, not a one-off.
3. **sops key/anchor drift**: `proxmox-minimal` has a `.sops.yaml` recipient anchor with no corresponding private key anywhere in this environment; several `lxc-*`/`proxmox-*` targets have no sops registration at all yet.
4. One concrete script bug was found and **fixed in this session**: `create-proxmox-resource.sh` never enabled the QEMU guest agent channel on VMs it creates, despite the guest OS already running it.
5. A management-surface audit (of the operator's ability to run this repo day to day) found 5 process gaps, detailed below.
Nothing here required or received a `nixos-rebuild switch/boot/test`, `nixos-install`, or any disk-formatting command — all validation was `nix build`/`nix eval`, plus disposable `pct`/`qm` create-then-destroy cycles via the repo's own `create-proxmox-resource.sh`.
---
## 1. Static Analysis Results — all clean
`bash scripts/codex-maintenance.sh --full-check --dry-run` (whole-tree sweep, not just changed files):
| Check | Result |
|---|---|
| Secret grep | Clean — only the documented exceptions (installer's own hashed passwords, `access-tokens` comment references) |
| `nixpkgs-fmt --check` | 0/53 files would be reformatted |
| `statix` | No lint warnings |
| nix-cache host key drift check | Up to date |
| Full eval of every host's `system.build.toplevel` | All 19 `nixosConfigurations` targets evaluate cleanly |
| Dry-run build of every host + package | All succeed, no derivation errors |
No drift, no formatting issues, no lint findings anywhere in the tree.
---
## 2. Per-Target Test Results
Legend: **LIVE** = built on pve, `pct`/`qm` create → interrogated → destroyed. **BUILD-ONLY** = `nix build` validated the config (mostly `.config.system.build.toplevel`, occasionally `.tarball`), no resource created on pve.
| Target | Test type | Result | Notes |
|---|---|---|---|
| `lxc-docker` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT105 is already running this identity in production; `--allow-duplicate-host` would have destroyed it. |
| `lxc-minimal` | **LIVE** | ✅ PASS (after retry) | First attempt reused a stale cached tarball predating a same-day sops-key commit → activation failed, sshd never started (see Finding #2). Redeployed with `--force-rebuild`: clean boot, `systemctl is-system-running` = `running`, secrets decrypted, sshd listening, users correct. |
| `lxc-nix-cache` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — CT101 is already running this identity. First local build attempt appeared to hang on a remote-builder handoff to nix-cache; killed and retried with `--builders ""` (local-only), succeeded. |
| `lxc-gui` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Registered a fresh sops key (no prior registration existed), built successfully through the full NixOS system closure, then **failed packaging the tarball**: `No space left on device` on pve's root filesystem. Not a flake defect. |
| `lxc-pxe-boot` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Same failure as `lxc-gui` — this target additionally builds a full nested installer/netboot image (`stage-installer-artifacts.nix`), making it similarly large. Failed with the same `No space left on device` error, immediately after the gui attempt had already consumed pve's remaining headroom. |
| `lxc-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; live deploy also would have hit `boot.zfs.extraPools` trying to import a real ZFS pool that doesn't exist in an isolated test container — an expected limitation of testing this build type outside its real hardware, not a bug. |
| `lxc-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
| `lxc-tor-relay` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT106 already holds this identity in production. |
| `proxmox-docker` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — both CT105 *and* VM103 already hold `docker` identities. Combined `toplevel` + `diskoImagesScript` build crashed with a **Nix-internal assertion failure** (`worker.cc:360`) under this session's memory pressure (see Finding #6) — not a flake bug. Retried with `toplevel` alone: clean. |
| `proxmox-minimal` | **LIVE (attempted)** | ⚠️ BLOCKED by key drift → BUILD-ONLY | `.sops.yaml` has a registered `&proxmox-minimal` anchor but **no corresponding private key exists anywhere in this environment** — the script correctly refused to generate a mismatched replacement. Fell back to `toplevel` build: ✅ PASS. |
| `proxmox-nix-cache` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
| `proxmox-gui` | BUILD-ONLY | ⚠️ Killed after ~40min (resource-limited) | This session's local build machine has only 2GB RAM; swap filled completely (2.0/2.0GB) and the build stalled, so it was killed rather than risk destabilizing the session further. **Not a flake defect** — the equivalent `gui` NixOS configuration already proved fully buildable during the `lxc-gui` live attempt above (it built the entire system closure successfully and only failed at the pve-side tarball-packaging step due to disk space, not the config). |
| `proxmox-pxe-boot` | BUILD-ONLY | ⚠️ Killed after ~35min (resource-limited) | Was deep into building the nested installer's kernel initrd (this build type bundles a full netboot installer image via `stage-installer-artifacts.nix`) when killed to keep the audit moving. **Not a flake defect** — this target's own module logic was already effectively validated via the earlier *live* pve deploy attempt (`lxc-pxe-boot` above), which built the complete image and only failed at the final tarball-packaging step due to pve's disk space (Finding 1). |
| `proxmox-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; same ZFS-pool caveat as `lxc-server` would apply to a live deploy. |
| `proxmox-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
**Not tested at all:** `linode-*` targets (not deployable to Proxmox) and `installer` (not a normal host) — both were still covered by the static eval/dry-run-build sweep above.
---
## 3. Findings, Ranked by Severity
### Finding 1 — pve.sweet.home is critically low on disk space (blocks real deployments)
At session start: `/dev/mapper/pve-root` was **95% full, 5.3GB free** (of 94GB). After two failed large builds it recovered slightly to **91% full, 8.2GB free** (nix cleans up its own failed-build scratch space). `/nix/store` alone is 26GB; `nix-store --gc --print-dead` reports **zero** reclaimable garbage — everything currently in the store is a live GC root, so `nix-collect-garbage` won't help without first removing old roots.
**Why it matters:** `create-proxmox-resource.sh` builds every VM/CT image **directly on pve**, not on a build machine and transferred over. With <10GB headroom, any closure approaching a few GB (the `gui` build type: full Cinnamon desktop + Firefox + LibreOffice + GIMP + VS Code + xrdp; the `pxe-boot` build type: nginx/atftpd *plus* an entire nested installer/netboot image) cannot currently be built there at all. Both `lxc-gui` and `lxc-pxe-boot` failed live with `No space left on device` during this audit.
**Recommended action:** Expand `pve-root`'s LV, or free space by pruning old container templates in `/var/lib/vz/template/cache` (1.5GB) / old backups in `/var/lib/vz/dump` (306MB) / auditing what's pinning 26GB of `/nix/store` as live GC roots (likely `result-*` symlinks — see below). This is real production disk state; **not something this session touched or fixed** — it needs the operator's judgment on what's safe to remove.
**Secondary, smaller finding:** every `create-proxmox-resource.sh` run leaves a `result-<target>` symlink in the node's repo checkout as a permanent GC root (`ls /root/nixos/result-*` on pve showed 3 from this session alone: `lxc-docker`, `lxc-minimal`, `lxc-nix-cache`). These accumulate forever and pin their entire closures in the store. Consider having the script clean up its own `result-*` link after staging the built artifact (or use a temp `--out-link` under `/tmp`), so `nix-collect-garbage` can actually reclaim old build outputs.
### Finding 2 — Stale cached images can silently ship broken secrets (reproduced live)
`create-proxmox-resource.sh`'s default behavior is: if the node already has `<target>.tar.xz`/`.raw` staged, **reuse it** — only `--force-rebuild` forces a fresh build. This session hit exactly the failure mode `docs/auto-installer.md` already warns about: `lxc-minimal`'s cached tarball (built 2026-07-20T15:57Z) predated a same-day sops-key fix commit (2026-07-20T17:49Z, "clean up in ailse 3"). The deployed container booted with:
```
sops-install-secrets: failed to decrypt '.../common.yaml': Error getting data key: 0 successful groups required, got 0
Activation script snippet 'setupSecrets' failed (1)
```
— every secret permanently failed to decrypt, `sshd` never started (though the container otherwise looked "running"). This was **not a code bug**: the currently-committed `secrets/common.yaml` decrypts fine for that host's key when checked independently; the *cached artifact on pve* simply reflected an older commit's ciphertext. Redeploying with `--force-rebuild` fixed it immediately.
**Why it matters:** this is silent and easy to trigger by accident — any operator who redeploys a host without remembering `--force-rebuild` after a secrets change gets a container that looks like it started (`pct start` succeeds, `pct status` = running) but is completely inaccessible.
**Recommended action:** Have `create-proxmox-resource.sh` compare the cached image's build timestamp (or embed the source commit hash in the staged filename) against current HEAD, and warn (or refuse without `--force-rebuild`) if they differ — rather than silently trusting presence alone.
### Finding 3 — sops key/anchor drift
Two concrete instances hit live during this session:
- **`proxmox-minimal`**: `.sops.yaml` already has a registered `&proxmox-minimal` age recipient, but this environment's `host-keys/` directory has no corresponding private key file. `sync-host-keys.sh` correctly refused to generate a replacement (it would silently mismatch whatever's already registered/deployed) — but this means **no environment currently has this host's private key**, unless it exists on some other machine that was never backed up here.
- **`lxc-gui`**, and by the same logic `lxc-server`/`lxc-tailscale-exit-node`/most `proxmox-*` targets, have **no sops registration at all yet** — expected for undeployed hosts per `docs/auto-installer.md`, but this session's live-testing needed to register `lxc-gui`'s key on the fly, which immediately hit **Finding 3b**: registering a key locally does nothing for pve's build until it's pushed to `origin/main` (pve builds via `git pull`, not from this uncommitted worktree). This is exactly gap #4 the management-surface audit (below) already flagged in the abstract — this session hit it concretely.
**Recommended action:** for `proxmox-minimal`, decide whether to regenerate its key (destroying old-key decrypt access, if anything still holds it) or track down wherever the original private key lives and back it up here. For the general pattern, see the management-surface audit's recommendation to pre-flight-check key registration before building.
### Finding 4 — QEMU guest agent never wired up (found and fixed this session)
`modules/common/configuration.nix:44` sets `services.qemuGuest.enable = true` on every host — the guest-side agent daemon is correctly enabled everywhere. But `scripts/proxmox/create-proxmox-resource.sh`'s `qm create` call never passed `--agent 1`, so **Proxmox never created the virtio-serial channel** the agent needs. Every `proxmox-*` VM this script ever created was silently missing `qm guest exec`/IP-address reporting in the Proxmox UI, despite the guest daemon actually running.
**Status: fixed in this session's worktree** (`scripts/proxmox/create-proxmox-resource.sh`, `qm create` now includes `--agent enabled=1`) — see the diff, included in the PR from this session.
### Finding 5 — Orphaned container on pve (CT102)
`pve.sweet.home` has a stopped LXC container, **VMID 102**, with an essentially empty config (`lock: create` and nothing else — no hostname, no rootfs, no network) — the leftover of a `pct create` that started and never finished. It predates this session (not created by any of this audit's activity) and wasn't touched. **Recommend the operator confirm it's abandoned and remove it** (`pct destroy 102 --purge 1`) — left as-is it may be someone's genuine in-progress work, so it wasn't assumed safe to delete autonomously.
### Finding 6 — Nix-internal crash under memory pressure (tooling, not flake)
Building `proxmox-docker`'s `toplevel` and `diskoImagesScript` together crashed with a Nix-internal assertion failure (`Assertion '!awake.empty()' failed ... worker.cc:360`, a known class of bug in Nix's multi-goal build scheduler) while this session's 2GB-RAM build container was under heavy swap pressure (1.8-2.0/2GB swap in use) from a separate concurrent build. Retrying the same target alone (no concurrency) succeeded cleanly. **Not a flake defect** — purely an artifact of this session's constrained build environment; noted for completeness since it looked alarming in isolation.
### Finding 7 — Management-surface audit: 5 operability gaps
A focused audit of "can the operator actually run this repo day to day" (flake, home-manager, sops, related scripts) found:
1. **No documented recovery path if the `&admin` sops age key is lost without a backup.** `scripts/secrets/backup-admin-key.sh` exists and works but is referenced nowhere in `README.md`/`docs/` — no forcing function ensures a backup was ever taken. `rotate-admin-key.sh` requires the *old* key to re-key; there's no bootstrap-from-nothing path documented (the real fallback — deriving an age identity from any still-live host's own SSH key — isn't written down anywhere).
2. **home-manager has no standalone iteration path.** It's wired only inside `nixosConfigurations` (`flake.nix`) — no `homeConfigurations` output. The fastest real shortcut (`nix build .#nixosConfigurations.<target>.config.home-manager.users.nixos.home.activationPackage`) isn't documented anywhere, so the practical workflow is a full host rebuild to test one HM tweak.
3. **Gitea's flake-lock-update workflow pushes straight to `main` with no pre-merge validation.** `.gitea/workflows/update-flake-lock.yml` commits and pushes `nix flake update`'s result directly; `codex-maintenance.sh` only runs *after*, on the resulting push — a genuinely broken lockfile bump lands on `main` before anything catches it. (The GitHub-side workflow is safer — PR-based — but has the opposite gap: nothing alerts if the PR sits unmerged.)
4. **No pre-flight check that a build target has a registered sops key before building it.** `docs/auto-installer.md` documents the failure mode (silent, total secrets-decrypt failure) but nothing in `create-proxmox-resource.sh` refuses to proceed when it's about to build a target with no `.sops.yaml` anchor — it's on the operator to remember. This session's `lxc-gui` test hit close to this exact gap (needed the key added on the fly, mid-session).
5. **`vars.remoteBuilderAuthorizedKeys` has the same drift risk as `vars.nixCacheHostKey`, but no checker script.** `sync-nix-cache-host-key.sh --check` guards the latter; the former (and `vars.pxeServerIp`/`vars.pbsIp`) has no equivalent — a rotated/revoked client key just silently stops working with no diagnostic pointing back here.
---
## 4. Action Plan (priority order)
1. **Free up disk space on pve.sweet.home** (or expand `pve-root`). Blocking: `lxc-gui`, `proxmox-gui`, `lxc-pxe-boot`, `proxmox-pxe-boot` cannot currently be built/redeployed on this node at all.
2. **Decide on `proxmox-minimal`'s orphaned sops key**: locate the original private key and back it up here, or accept regenerating it (breaks decrypt access for whoever/whatever currently holds the old one).
3. **Merge this session's PR** (see below) to get the `--agent 1` fix and `lxc-gui`'s new sops registration onto `main` — required before `lxc-gui` can be live-redeployed with working secrets.
4. **Add a staleness guard to `create-proxmox-resource.sh`'s cache-reuse path** (Finding 2) — highest-leverage fix, since it silently produces a broken-but-"running" host.
5. **Add a pre-flight sops-anchor check to `create-proxmox-resource.sh`** (management-surface gap #4) — same root cause class as #4 above, catch it before building instead of at first boot.
6. Investigate/clean up **CT102** on pve (Finding 5) — confirm abandoned, then remove.
7. Document `backup-admin-key.sh` in `README.md`'s Security Notes and add the live-host-key bootstrap-recovery procedure to `docs/` (management-surface gap #1).
8. Add pre-push validation to the Gitea flake-lock-update workflow (management-surface gap #3).
9. Lower-priority: document the home-manager `activationPackage` shortcut (gap #2); extend `sync-nix-cache-host-key.sh`'s drift-check pattern to `remoteBuilderAuthorizedKeys` (gap #5).
10. Follow-up session: finish build-validating `proxmox-gui` and `proxmox-pxe-boot` (both killed here after 35-40min on this session's 2GB-RAM machine — not failures, just unfinished) once pve has headroom (item 1) — ideally from a machine with more RAM. `proxmox-server` and `proxmox-tailscale-exit-node` already passed build-only validation in this session, no follow-up needed.
---
## 5. Uncommitted Changes From This Session
This worktree (`worktree-flake-e2e-audit`) currently has:
- `scripts/proxmox/create-proxmox-resource.sh` — the `--agent enabled=1` fix (Finding 4).
- `.sops.yaml` / `secrets/common.yaml``lxc-gui`'s new age key registered as a recipient (generated live during this session's testing).
Per this session's standard workflow, these will be committed, pushed, and opened as a draft PR rather than pushed to `main` directly — merging it is the operator's call, and is also **prerequisite to live-redeploying `lxc-gui` successfully** (its build will keep hitting the sops-staleness failure from Finding 2 on pve until this registration is on `origin/main`).
+217 -53
View File
@@ -31,17 +31,96 @@ machines when deployed.
the auto-installer (see `docs/auto-installer.md`). Never commit its
contents; if `git status` ever shows it as trackable, something is wrong.
### Two Proxmox nodes: `pve1.sweet.home` (production) and `pve-test.sweet.home` (sandbox)
There are two SSH-reachable Proxmox nodes on the LAN, both defined in
`scripts/env.sh` (`PVE1_HOST` / `PVE_TEST_HOST`), individually targetable
via `scripts/proxmox/create-proxmox-resource.sh --node <host>` or by
overriding `PROXMOX_HOST`. `PROXMOX_HOST` itself still defaults to
`PVE1_HOST` (production) — that default, and every other script behavior,
is unchanged from before `pve-test` existed; the only thing new is that
`pve-test` can now be reached at all. They are **not interchangeable**
one is real production infrastructure, the other exists specifically so
there's somewhere safe to test. The restriction below is a policy for
Claude specifically, not a change to the tooling's own default or
anything the operator needs to opt into.
#### `pve1.sweet.home` (production — off-limits to Claude)
A real, live Proxmox node hosting production VMs/containers — not a
sandbox, and not Claude's to touch by default.
- **Off-limits at all times unless the operator has given explicit,
same-session instructions to act on this specific host.** That
authorization is scoped to the task it was given for — don't carry it
forward to unrelated later work in the same conversation, and never
assume it from a previous session.
- **Read-only for existing state is always fine, authorization or not.**
You may SSH in (or use `pvesm`, `qm list`, `pct list`, `qm config`, `pct
config`, the Proxmox API, etc.) to inspect the node's config, storage,
and any existing VM/container — including ones this repo didn't create.
- **Never** modify, stop, restart, delete, reconfigure, or create anything
on this node (`qm set`, `pct set`, `qm destroy`, `pct destroy`, `qm
stop`, `pct stop`, `qm create`, `pct create`, snapshot operations,
storage changes, etc.) — including scratch/test resources — without
that explicit go-ahead. Use `pve-test.sweet.home` for anything
exploratory instead; it exists precisely so `pve1` never has to be the
answer to "where do I test this."
- **This is a Claude-specific policy, not something the scripts enforce.**
`scripts/env.sh`/`create-proxmox-resource.sh` default to `pve1` exactly
as they did before `pve-test` existed, with no extra flag or prompt
required — that's deliberate, so the operator's own existing workflows
don't change. Claude, however, must never rely on that default: every
Proxmox action Claude takes on its own initiative — not explicitly
pointed at `pve1` by the operator this session — targets `pve-test`
instead (e.g. `--node "$PVE_TEST_HOST"`, or `PROXMOX_HOST=$PVE_TEST_HOST`).
Claude's own default is `pve-test`, full stop, regardless of what the
tooling's own unqualified default happens to be.
#### `pve-test.sweet.home` (sandbox — Claude's default target)
A separate Proxmox node set aside for testing. The *tooling's* default is
still production (`PROXMOX_HOST``PVE1_HOST`, see above) — but
**Claude's own default is this node**: absent an explicit, same-session
instruction to use `pve1`, every Proxmox action Claude initiates targets
`pve-test`. Once targeted, it's safe to create, interrogate, and destroy
resources on without asking first.
- **Test VMs/containers are allowed, but must be torn down.** Create a
scratch VM or container here (e.g. via
`scripts/proxmox/create-proxmox-resource.sh` or raw `qm`/`pct create`)
to validate something. Anything created this way must be destroyed
again in the same session, before ending the task — never leave a test
resource running. Use a VMID/name that's obviously scratch (and doesn't
collide with a real flake target) so it's unambiguous what's safe to
remove.
- **Node-level config is still not yours to change.** Creating/destroying
your own scratch guests is fine; Proxmox host config, storage pools, and
networking on `pve-test` itself are still the operator's call to make
manually, same as on `pve1`.
## Commands
```bash
# One-time environment bootstrap (installs Nix if missing, prints hosts)
bash scripts/codex-setup.sh
# Full validation: secret grep, nixpkgs-fmt --check, statix lint, eval all hosts
# Changed-files-only validation: secret grep (whole repo), nixpkgs-fmt --check
# and statix on changed *.nix files, eval of the hosts/packages those changes
# can affect. This is what CI runs on every push/PR.
bash scripts/codex-maintenance.sh
# Same, plus a dry-run build (no result symlink) of every host's toplevel
bash scripts/codex-maintenance.sh dry-run
# Full sweep: nixpkgs-fmt --check/statix over the whole tree, eval every host
# and package. Slow (minutes) -- CI never runs this; use it locally before a
# release or after touching modules/common/*, flake.nix, or variables.nix for
# extra confidence beyond the automatic full-fallback those paths already
# trigger in the default mode (see below).
bash scripts/codex-maintenance.sh --full-check
# Either mode, plus a dry-run build (no result symlink) of every host/package
# in whichever scope is active
bash scripts/codex-maintenance.sh --dry-run
bash scripts/codex-maintenance.sh --full-check --dry-run
# List the hosts the flake currently exposes
nix eval --json .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]'
@@ -58,36 +137,96 @@ maintenance script pulls them via `nix run github:NixOS/nixpkgs/nixos-25.11#<too
There is no test suite — "correctness" here means the flake evaluates and
`nixpkgs-fmt`/`statix` are clean.
**In an interactive agent session**, prefer targeted checks over full-repo
sweeps: after editing one or two hosts/modules, evaluate just the
`nixosConfigurations.<host>` you touched (plus any `config.system.build.tarball`
/`diskoImagesScript`/package output affected) rather than looping over every
host`codex-maintenance.sh` evaluates every `nixosConfigurations` host plus
every package/tarball/image variant and is slow to run after each small
change. Reserve a full
`codex-maintenance.sh` run for changes that plausibly affect every host
(`modules/common/*`, `flake.nix`, `variables.nix`) or as a final check before
committing. This is a session-workflow preference only — it does not apply to
CI, which should keep running the full script on every push/PR regardless of
diff size; that's the point of it.
With no flags, `codex-maintenance.sh` diffs against a base ref (env
`MAINT_BASE_SHA`, else the PR base SHA in CI, else `HEAD^` locally) and scopes
fmt-check/statix to the changed `*.nix` files and eval to the hosts/packages
those changes can affect — a `hosts/<name>/host.nix` edit only evals that
host's targets, a `modules/platforms/<platform>.nix` edit only evals that
platform's hosts, and so on. A change to `flake.nix`, `flake.lock`,
`variables.nix`, `modules/common/*`, or any other `modules/*.nix` file outside
`platforms/`/`build-types/` (whose blast radius isn't safely inferable from
the path alone) falls back to evaluating every host and package, same as
`--full-check` would, just without the whole-tree fmt/statix sweep. This
exists because the whole-tree sweep is what was timing out CI; **CI always
runs the plain, no-flag form and never passes `--full-check`.**
The default mode's diff is against the working tree (uncommitted and staged
edits included, not just committed ones), so it's already the right tool for
an interactive session too: after editing one or two hosts/modules, plain
`bash scripts/codex-maintenance.sh` naturally scopes to just what you
touched. Reserve `--full-check` for changes that plausibly affect every host
(`modules/common/*`, `flake.nix`, `variables.nix` — though the default mode
already falls back to evaluating everything for those paths, `--full-check`
additionally re-checks fmt/statix over the whole tree) or as a final check
before committing.
## Scripts
Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` is
organized by purpose: `scripts/secrets/` (sops/age + SSH host-key
management), `scripts/proxmox/` (Proxmox deployment), `scripts/lib/`
(shared helpers, sourced by the scripts below — not run directly), and a
handful of repo-wide scripts left at the top level (`env.sh`,
`bump-nixpkgs-release.sh`, plus `codex-setup.sh`/`codex-maintenance.sh`
above). When adding a new script, put it in the matching subfolder rather
than the top level, and if it duplicates logic another script already has,
lift the shared part into `scripts/lib/` instead of copying it.
- `scripts/sync-host-keys.sh` — generates/registers SSH host keys and their
`.sops.yaml`/`secrets/*.yaml` recipients for flake targets, idempotently
(`--all`, `<target>`, `--remove`, `--regenerate-all-keys`, all with
`--dry-run`). The primary tool for provisioning a new host's secrets
access — see "Creating a new machine" in `docs/auto-installer.md`.
- `scripts/prepare-host-key.sh` — narrower predecessor: generates a key by
an arbitrary name without touching `.sops.yaml`. Still useful to
### `scripts/secrets/`
- `scripts/secrets/sync-host-keys.sh` — generates/registers SSH host keys
and their `.sops.yaml`/`secrets/*.yaml` recipients for flake targets,
idempotently (`--all`, `<target>`, `--remove`, `--regenerate-all-keys`,
all with `--dry-run`). The primary tool for provisioning a new host's
secrets access — see "Creating a new machine" in `docs/auto-installer.md`.
- `scripts/secrets/prepare-host-key.sh` — narrower predecessor: generates a
key by an arbitrary name without touching `.sops.yaml`. Still useful to
pre-generate a key before its flake target exists yet, since
`sync-host-keys.sh` can only act on targets `nixosConfigurations` already
has.
- `scripts/create-proxmox-resource.sh` — builds a `lxc-*`/`proxmox-*`
target's tarball/disk image and creates it on a real Proxmox node
(`pct create` against the tarball as a CT template / `qm create`+
- `scripts/secrets/rotate-admin-key.sh <backup-admin-key> [--new-key-file
<path>] [--dry-run]` — rotates `.sops.yaml`'s `&admin` age key: decrypts
with a backed-up copy of the key currently trusted as `&admin` (verified
by deriving its public key and comparing, not taken on faith), replaces
the `&admin` line with a new key already present in the environment
(defaults to wherever sops/age itself would look), and runs
`sops updatekeys` on every `secrets/*.yaml`. One-way: the old key can no
longer decrypt anything re-encrypted this way. This is the automation
for the manual steps `sync-host-keys.sh`/`create-proxmox-resource.sh`
print when they bootstrap a brand-new, not-yet-trusted key on a machine
with no prior admin access.
- `scripts/secrets/backup-admin-key.sh <dest-path> [--key-file <path>]
[--force] [--dry-run]` — copies the local sops age key (source
resolution matches sops/age itself: `$SOPS_AGE_KEY` inline, then
`--key-file`, then `$SOPS_AGE_KEY_FILE`, then the XDG default) to an
arbitrary destination path with `0600` permissions, validating it's a
real age identity and round-tripping the public key before and after the
write. Refuses to overwrite an existing `<dest-path>` without `--force`.
Purely a local filesystem copy — never touches `.sops.yaml`/
`secrets/*.yaml` or the repo at all. The resulting file is exactly what
`rotate-admin-key.sh` expects as its backup-key argument.
- `scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run]
[--host <name>]` — detects drift between the ed25519 SSH host key
nix-cache is actually serving right now (via `ssh-keyscan`) and
`vars.nixCacheHostKey` (`variables.nix`), the value
`modules/nix-cache/remote-builder-client.nix` bakes into every real
client's declarative `programs.ssh.knownHosts` and
`configure-nix-cache-client.sh` hardcodes as its own default for
non-NixOS clients. That value has no automatic source of truth — it's
set once from whatever nix-cache's host key happened to be at the time,
and silently goes stale if the host is ever rebuilt/recreated with a new
key, breaking every client's distributed-build SSH trust with no error
that points back here. `--check` (used by `codex-maintenance.sh`, which
treats an unreachable nix-cache — e.g. from a non-LAN CI runner — as a
silent skip rather than a failure) only reports drift; the no-flags form
updates both files in place. Declarative clients still need a rebuild to
pick up the fix.
### `scripts/proxmox/`
- `scripts/proxmox/create-proxmox-resource.sh` — builds a `lxc-*`/
`proxmox-*` target's tarball/disk image and creates it on a real Proxmox
node (`pct create` against the tarball as a CT template / `qm create`+
`importdisk`), or reconfigures an existing resource's cores/memory/disk
size (`--modify`, always requires typing the VMID back to confirm).
Checks for an already-uploaded image on the node before building
@@ -97,7 +236,51 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
Refuses to create a target whose host identity already exists live on
the node (checked directly via `qm`/`pct`, not any file in this repo)
unless `--allow-duplicate-host` is passed. `--dry-run` throughout both
modes.
modes. The first time it has to bootstrap build tooling on a node (i.e.
`nix` wasn't already on its `PATH`), it also runs
`scripts/proxmox/configure-nix-cache-client.sh` there (non-fatally — a
failure just falls back to building from source / `cache.nixos.org`) so
the node substitutes from and can offload builds to nix-cache on every
subsequent run, not just this one.
- `scripts/proxmox/configure-nix-cache-client.sh [--dry-run]
[--no-remote-builder] [--no-restart]` — the non-NixOS equivalent of
`modules/nix-cache/client.nix`/`remote-builder-client.nix`, for a plain
Debian machine with the Nix package manager (not NixOS) already
installed: run as root *on that machine* to add nix-cache as a
substituter in `/etc/nix/nix.conf` (`https://cache.nixos.org/` kept as
fallback) via `extra-substituters`/`extra-trusted-public-keys` so it
layers on top of whatever's already there instead of clobbering it, and,
if `/root/.ssh/nixremote` is already present (see docs/nix-cache.md
"Remote builder SSH keys"), configures it as a distributed-build
machine too and trusts nix-cache's SSH host key in
`/etc/ssh/ssh_known_hosts`. Idempotent (re-running replaces its own
marked block rather than duplicating it); restarts `nix-daemon` by
default so the change takes effect immediately.
### `scripts/lib/`
Sourced by the scripts above, never run directly:
- `nix-bootstrap.sh` — `NIX_CONFIG`/`ensure_nix_profile`, shared by
`codex-setup.sh`/`codex-maintenance.sh` and the remote build commands
`create-proxmox-resource.sh` runs over SSH.
- `nix-eval.sh` — `NIX_EVAL_FLAGS` plus `list_flake_targets`/
`flake_target_hostname` flake-introspection helpers.
- `ssh-host-keys.sh` — `generate_host_ed25519_key`/`ssh_pubkey_to_age`,
shared by `sync-host-keys.sh` and `prepare-host-key.sh`.
- `sops-age.sh` — `age_pubkey_from_identity_file`/`sops_yaml_admin_pubkey`/
`sops_updatekeys` plus the shared sops/age default key-file resolution,
shared by `backup-admin-key.sh`, `rotate-admin-key.sh`, and
`sync-host-keys.sh`.
- `confirm.sh` — `confirm_typed`, the "type X back to confirm" destructive-
action prompt shared by `create-proxmox-resource.sh` and
`sync-host-keys.sh`.
- `sync-host-keys-edit-sops.py` — the `.sops.yaml` anchor/key_groups editor
`sync-host-keys.sh` shells out to (see that script for why: precise,
idempotent YAML edits are impractical in bash).
### Top level
- `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
cross-script config here instead of duplicating it per-script.
@@ -106,27 +289,6 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
reference `variables.nix` (confirmed empirically — `nix flake metadata`
errors on it), so this is the closest equivalent to a single source of
truth for the tracked release.
- `scripts/rotate-admin-key.sh <backup-admin-key> [--new-key-file <path>]
[--dry-run]` — rotates `.sops.yaml`'s `&admin` age key: decrypts with a
backed-up copy of the key currently trusted as `&admin` (verified by
deriving its public key and comparing, not taken on faith), replaces the
`&admin` line with a new key already present in the environment
(defaults to wherever sops/age itself would look), and runs
`sops updatekeys` on every `secrets/*.yaml`. One-way: the old key can no
longer decrypt anything re-encrypted this way. This is the automation
for the manual steps `sync-host-keys.sh`/`create-proxmox-resource.sh`
print when they bootstrap a brand-new, not-yet-trusted key on a machine
with no prior admin access.
- `scripts/backup-admin-key.sh <dest-path> [--key-file <path>] [--force]
[--dry-run]` — copies the local sops age key (source resolution matches
sops/age itself: `$SOPS_AGE_KEY` inline, then `--key-file`, then
`$SOPS_AGE_KEY_FILE`, then the XDG default) to an arbitrary destination
path with `0600` permissions, validating it's a real age identity and
round-tripping the public key before and after the write. Refuses to
overwrite an existing `<dest-path>` without `--force`. Purely a local
filesystem copy — never touches `.sops.yaml`/`secrets/*.yaml` or the
repo at all. The resulting file is exactly what `rotate-admin-key.sh`
expects as its backup-key argument.
`sync-host-keys.sh`, `create-proxmox-resource.sh`, and
`rotate-admin-key.sh` genuinely mutate real state when run for real (not
@@ -160,9 +322,10 @@ nixosSystem {
```
Platforms: `linode`, `proxmox`, `lxc`. Build types: `minimal`, `nix-cache`,
`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`. Not every
combination is built — e.g. `pxe-boot` has no `linode` variant (PXE/DHCP/TFTP
need LAN L2 adjacency a Linode VPS doesn't have). Treat `flake.nix`'s
`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`, `tor-relay`. Not
every combination is built — e.g. `pxe-boot` has no `linode` variant
(PXE/DHCP/TFTP need LAN L2 adjacency a Linode VPS doesn't have), and
`tor-relay` currently only exists as `lxc-tor-relay`. Treat `flake.nix`'s
`generatedTargets` as the source
of truth for which hosts exist — `README.md`, `AGENTS.md`,
`docs/flake-lock-automation.md`, and the CI eval workflows
@@ -190,7 +353,7 @@ removing a host.
`vzdump` backup-archive metadata this doesn't have), no install step —
see `docs/auto-installer.md`.
- `modules/build-types/*.nix` — what a system is for:
minimal/server/docker/gui/pxe-boot/nix-cache/tailscale-exit-node.
minimal/server/docker/gui/pxe-boot/nix-cache/tailscale-exit-node/tor-relay.
- `modules/common/configuration.nix` — base NixOS config imported by every
host: locale, users, nix settings, git.
- `modules/common/home.nix` / `hosts/nixos/home.nix` — Home Manager config for
@@ -225,7 +388,8 @@ removing a host.
and `environmentFile`; used by `hosts/server/host.nix` and
`hosts/nix-cache/host.nix` to avoid duplicating that boilerplate.
- `modules/tailscale/`, `modules/docker/`, `modules/networking/`,
`modules/traefik/`, `modules/services/*` — single-purpose, single-host
`modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose,
single-host
feature modules (e.g. `docker/enable-service.nix`,
`services/zfs/enable-service.nix`). Grep `modules/build-types/*.nix` for
each build type's `imports` list to see which modules apply where.
+21 -6
View File
@@ -10,11 +10,11 @@ pieces composed in `flake.nix`:
- **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`
- **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
`gui`, `pxe-boot`, `tailscale-exit-node`
`gui`, `pxe-boot`, `tailscale-exit-node`, `tor-relay`
Not every combination exists — `pxe-boot` has no `linode` variant, since
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have. The full
list:
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have, and
`tor-relay` currently only exists as `lxc-tor-relay`. The full list:
| Target | Purpose |
| --- | --- |
@@ -27,14 +27,20 @@ list:
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target |
| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host — previously the flat `pxe-boot` target |
| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node |
| `lxc-tor-relay` | Tor middle relay |
Which variant of a given buildtype is actually deployed isn't tracked
anywhere in this repo — that's live infrastructure state, not something a
committed file can keep accurate, and it changes independently of the code.
Check the Proxmox node itself, or `/etc/flake-target` on a running host (see
below), if you need to know what's really out there right now.
`scripts/create-proxmox-resource.sh`'s duplicate-host guard works the same
`scripts/proxmox/create-proxmox-resource.sh`'s duplicate-host guard works the same
way: it checks the Proxmox node directly rather than any file here.
Real, production deployments live on `pve1.sweet.home`; there's a second
node, `pve-test.sweet.home`, set aside purely for scratch/test resources —
see `scripts/env.sh` (`PVE1_HOST` / `PVE_TEST_HOST`, and the
`--node`/`PROXMOX_HOST` targeting they feed into) and CLAUDE.md's Proxmox
section for which is which.
Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
(hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay
@@ -73,10 +79,19 @@ Safe validation commands for Codex and local review:
```bash
bash scripts/codex-setup.sh
bash scripts/codex-maintenance.sh dry-run
bash scripts/codex-maintenance.sh
```
`codex-maintenance.sh` with no flags (what CI runs on every push/PR) scopes
fmt-check/statix/eval to files changed against a base ref — fast, but only
as thorough as the diff. For the full sweep (every host, every package,
fmt-check and statix over the whole tree — slow, CI never runs this):
```bash
bash scripts/codex-maintenance.sh --full-check
bash scripts/codex-maintenance.sh --full-check --dry-run
```
For individual host evaluation:
```bash
@@ -113,7 +128,7 @@ Three different paths depending on target, none of them involving a manual
disk image and attached to a new VM with no install step — see
`docs/proxmox-images.md`.
`scripts/create-proxmox-resource.sh --type lxc|vm --host <name>` automates
`scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --host <name>` automates
either of the last two end to end (host-key registration, building the
image directly on the Proxmox node itself, `pct create`/`qm create`), with
`--dry-run` and a guard against duplicating an already-deployed host's
+7 -7
View File
@@ -19,7 +19,7 @@ see "LXC hosts" immediately below for why those are different.**
## LXC hosts
`lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`,
`lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-exit-node`) are **not** installed via `auto-install.sh` — the
`lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-exit-node`, `lxc-tor-relay`) are **not** installed via `auto-install.sh` — the
interactive menu deliberately excludes them. Don't try to select one there;
`nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw
disk to partition) and then refuse to touch the filesystem it's currently
@@ -73,7 +73,7 @@ booting one:
First boot runs `boot.postBootCommands` (registers the Nix store DB and
system profile) — there's no separate activation step to run yourself.
`scripts/create-proxmox-resource.sh --type lxc --host <name>` automates all
`scripts/proxmox/create-proxmox-resource.sh --type lxc --host <name>` automates all
of this (host-key handling, building the tarball directly on the Proxmox
node itself, `pct create` with the flags above) — see its `--help`.
@@ -102,7 +102,7 @@ groups required, got 0`, and *every* secret (including this host's own
login) permanently fails to decrypt, silently — no error in the boot log
at all, since the activation step that would install secrets only runs on
a from-scratch first activation and skips silently once `/run/current-system`
already exists. `scripts/create-proxmox-resource.sh` always builds with
already exists. `scripts/proxmox/create-proxmox-resource.sh` always builds with
`NIXOS_HOST_KEYS_DIR` set for this reason.
## Layout
@@ -116,10 +116,10 @@ already exists. `scripts/create-proxmox-resource.sh` always builds with
`docs/pxe-boot.md`).
- `modules/installer/host-keys.nix` — optionally bakes pre-generated SSH
host keys into the image; see "Host keys" below.
- `scripts/sync-host-keys.sh` — admin-workstation tool that generates,
- `scripts/secrets/sync-host-keys.sh` — admin-workstation tool that generates,
registers, and (via `--remove`/`--regenerate-all-keys`) retires host
keys; see "Creating a New Machine" below.
- `scripts/prepare-host-key.sh` — narrower predecessor: generates a single
- `scripts/secrets/prepare-host-key.sh` — narrower predecessor: generates a single
key by an arbitrary name without touching `.sops.yaml`. Still useful for
pre-generating a key *before* its flake target exists (`sync-host-keys.sh`
can only act on targets `nixosConfigurations` already has); otherwise
@@ -238,7 +238,7 @@ GitHub token behind sops-nix for all of them).
2. **On your admin workstation, generate and register its host key:**
```sh
./scripts/sync-host-keys.sh <flake-target>
./scripts/secrets/sync-host-keys.sh <flake-target>
```
This generates `host-keys/<flake-target>_ssh_host_ed25519_key(.pub)`,
@@ -250,7 +250,7 @@ GitHub token behind sops-nix for all of them).
Doing this for every host that needs one at once — after adding several
new targets, or just to catch up any that were missed — is
`./scripts/sync-host-keys.sh --all`. See `scripts/sync-host-keys.sh --help`
`./scripts/secrets/sync-host-keys.sh --all`. See `scripts/secrets/sync-host-keys.sh --help`
for its other modes (`--remove`, `--regenerate-all-keys`).
3. **Commit and push.** The flake build the installer uses has to see the
+8 -3
View File
@@ -8,9 +8,14 @@ and to verify that declared NixOS hosts still evaluate after dependency updates.
- A scheduled workflow runs `nix flake update` once per week.
- On GitHub, any resulting `flake.lock` change is proposed through a pull request.
- On Gitea, the workflow can commit and push `flake.lock` directly when PR automation is not configured.
- A separate CI workflow evaluates every configured host before merge, listed
dynamically via `nix eval --json .#nixosConfigurations --apply builtins.attrNames`
rather than hand-enumerated, so it can't drift as `<platform>-<buildtype>`
- A separate CI workflow runs `scripts/codex-maintenance.sh` before merge.
Its default mode scopes eval to the hosts/packages a change can affect,
determined from a git diff against the PR base — but a `flake.lock` change
is treated as repo-wide and always falls back to evaluating every host, so
a lock-file update PR still gets full coverage. Hosts are still listed
dynamically via
`nix eval --json .#nixosConfigurations --apply builtins.attrNames` rather
than hand-enumerated, so that fallback can't drift as `<platform>-<buildtype>`
targets are added or removed. See `README.md` for the current target list.
## Why hosts should stop using `--upgrade-all`
+17 -9
View File
@@ -46,18 +46,26 @@ the new key up automatically on next activation — no more manual
## Remote builder SSH keys
On each client, install the private key used to authenticate as `nixremote`:
Each client authenticates as `nixremote` using its **own default root SSH
identity** (`/root/.ssh/id_ed25519`) — not a separately-named or shared
keypair. If a client doesn't have one yet:
```bash
sudo install -d -m 0700 /root/.ssh
sudo install -m 0600 ./nixremote /root/.ssh/nixremote
sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
sudo ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
```
On `nix-cache`, install the matching public key used by `nixremote` authorized keys.
Then add its `.pub` contents as a new entry in `vars.remoteBuilderAuthorizedKeys`
(`variables.nix`) and rebuild `nix-cache` to pick it up (that list is
declarative — an imperative `ssh-copy-id nixremote@nix-cache` won't stick;
it gets overwritten on every rebuild). Verify with:
The committed `nixremote` authorized keys are public SSH keys only. Keep the
matching private keys on client hosts and out of the repository.
```bash
sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
```
The committed `remoteBuilderAuthorizedKeys` entries are public SSH keys
only. Keep the matching private keys on client hosts and out of the
repository.
nix-cache's own SSH *host* key is trusted declaratively via
`programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
@@ -76,8 +84,8 @@ After deployment:
curl http://nix-cache/nix-cache-info
nix store ping --store http://nix-cache
nix show-config | grep -E 'substituters|trusted-public-keys|builders-use-substitutes'
sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/nixremote 4 2 big-parallel,kvm,nixos-test,benchmark' -L
sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/id_ed25519 4 2 big-parallel,kvm,nixos-test,benchmark' -L
nix path-info -r nixpkgs#hello
curl -I "http://nix-cache/$(basename "$(nix path-info nixpkgs#hello)").narinfo"
```
+2 -2
View File
@@ -7,7 +7,7 @@ config (`modules/disko/proxmox.nix`) already used to format a real disk on
install, so there's nothing host-specific to write; it's available for every
`proxmox-*` target automatically.
`scripts/create-proxmox-resource.sh --type vm --host <name>` automates the
`scripts/proxmox/create-proxmox-resource.sh --type vm --host <name>` automates the
whole walkthrough below (and the equivalent LXC one) end to end, including
host-key handling and building the image directly on the Proxmox node
itself (no local build, no image transfer) — see its `--help`. The steps
@@ -51,7 +51,7 @@ sudo ./result \
--build-memory 2048
```
Generate the key first with `scripts/sync-host-keys.sh <hostname>`, same
Generate the key first with `scripts/secrets/sync-host-keys.sh <hostname>`, same
as any other host — see `docs/auto-installer.md` for the full walkthrough
(it registers the new key in `.sops.yaml` and re-encrypts the affected
`secrets/*.yaml` files too, no manual editing needed).
-151
View File
@@ -1,151 +0,0 @@
# Spec: Refactor Flake Targets into Platform × Build-Type Matrix
**Status: implemented.** `flake.nix`'s `generatedTargets`/`mkTarget` and
`modules/platforms/*`/`modules/build-types/*` are the result of this spec —
kept here for historical rationale only (referenced from `CLAUDE.md`'s
"Composition pattern" section), not as an active or open plan. The "Open
Questions" below were resolved during implementation; don't treat them as
outstanding. A `tailscale-exit-node` build type was added later, beyond this
spec's original scope.
## Context
The flake at `~/nixos` currently defines these output targets (flat, ad-hoc naming):
- `docker`
- `linode-minimal`
- `nix-cache`
- `nix-minimal`
- `nixos`
- `server`
- `pxe-boot`
Some already follow a `platform-buildtype` convention (`linode-minimal`), most don't.
`~/nix-auto-installer` is a related repo and should be checked for any coupling to
these target names (scripts, docs, CI, or install automation that reference them by
name) before renaming anything.
## Goal
Restructure the flake so targets are generated from two orthogonal concepts:
**Build types** (what the system is for):
- `minimal`
- `nix-cache`
- `server`
- `docker`
- `pxe-boot`
- `gui`
**Platforms** (what it's deployed on):
- `linode` (Linode VM)
- `proxmox` (Proxmox VM)
- `lxc` (Proxmox LXC container)
Final targets should be named consistently as `<platform>-<buildtype>`, e.g.:
```
linode-minimal proxmox-minimal lxc-minimal
linode-nix-cache proxmox-nix-cache lxc-nix-cache
linode-server proxmox-server lxc-server
linode-docker proxmox-docker lxc-docker
linode-pxe-boot proxmox-pxe-boot lxc-pxe-boot
linode-gui proxmox-gui lxc-gui
```
That's the full matrix (18 targets) if every build type applies to every platform.
See **Open Questions** below — some combinations may not make sense and should be
confirmed with me before being built out, not silently included or dropped.
## Migration mapping (old → new)
| Old target | New target | Notes |
|--------------------|------------------------------------------------------|-------|
| `linode-minimal` | `linode-minimal` | Already correct, keep as-is |
| `nix-minimal` | likely `proxmox-minimal` or a platform-less base module | Ambiguous — see Open Questions |
| `nix-cache` | base module consumed by `linode-nix-cache`, `proxmox-nix-cache`, `lxc-nix-cache` | Currently platform-less; needs to become a build-type module, not a standalone target |
| `server` | base module consumed by `linode-server`, `proxmox-server`, `lxc-server` | Same as above |
| `docker` | base module consumed by `linode-docker`, `proxmox-docker`, `lxc-docker` | Confirm docker actually makes sense as an LXC/VM guest build vs. a standalone container image — see Open Questions |
| `pxe-boot` | TBD — may stay a single target rather than a per-platform one | See Open Questions |
| `nixos` | TBD — unclear what this maps to in the new scheme | See Open Questions |
## Open Questions (Claude Code: raise these with me before implementing, don't guess)
1. **`nixos` target** — what is this currently used for (bare metal install, dev
shell, template)? It doesn't obviously map to any of the six build types.
2. **`nix-minimal` vs `linode-minimal`** — are these two different things, or is
`nix-minimal` a leftover/duplicate?
3. **`pxe-boot` and `gui` across all three platforms** — does PXE boot make sense
for an LXC container or a cloud VM (Linode), or is it inherently bare-metal/
network-boot only and should remain a single non-platform target? Does `gui`
make sense inside an LXC container?
4. **`docker` as a build type** — is this "a NixOS host configured to run Docker"
(which would sensibly have linode/proxmox/lxc variants), or "a Docker container
image built by the flake" (which wouldn't take a platform prefix at all, since
it doesn't run on Linode/Proxmox/LXC as a guest OS)? These are structurally
different and change how it should be wired in.
5. Confirm whether all 18 combinations should actually exist, or whether this is
meant to produce only the combinations that are genuinely useful (e.g. maybe no
one needs `lxc-pxe-boot`).
## Implementation approach
1. **Inventory first.** Read the current `flake.nix` and any `nixosConfigurations`/
`modules` structure. Map every existing target to what module(s) it actually
pulls in. Don't assume — confirm against the real file contents.
2. **Separate build-type and platform into their own module directories**, e.g.:
```
modules/build-types/minimal.nix
modules/build-types/nix-cache.nix
modules/build-types/server.nix
modules/build-types/docker.nix
modules/build-types/pxe-boot.nix
modules/build-types/gui.nix
modules/platforms/linode.nix
modules/platforms/proxmox.nix
modules/platforms/lxc.nix
```
Build-type modules should contain only what makes a system "minimal" vs
"server" vs "gui", etc. Platform modules should contain only what's specific
to running as a Linode VM vs Proxmox VM vs LXC container (virtualisation
guest tools, boot method, filesystem/image format, LXC-specific constraints
like no kernel modules, etc).
3. **Generate the target matrix programmatically** in `flake.nix` rather than
hand-writing 18 near-identical `nixosConfigurations` entries — e.g. a small
function that takes a platform name and build-type name, composes the two
modules plus any shared base module, and produces the named output. This
keeps future build types/platforms a one-line addition rather than a copy-paste
job.
4. **Only build combinations we've confirmed make sense** (see Open Questions) —
don't emit all 18 by default if some are structurally invalid.
5. **Preserve existing working configs during the transition.** Don't delete the
old target names until their replacements build successfully — rename/alias
at the end, not the start, so there's no window where the flake is broken.
## Verification
For every new target produced:
```bash
nix flake check
nix build .#nixosConfigurations.<target>.config.system.build.toplevel
```
Confirm each builds without evaluation errors before considering it done. If a
target fails to build, report which one and why rather than silently skipping it.
## Deliverables
- Refactored `flake.nix` using the composed module + generated-matrix approach.
- New `modules/build-types/*.nix` and `modules/platforms/*.nix` files.
- Old flat target names removed only after their replacements are verified.
- A short `README.md` (or section in existing docs) listing the final target
names and what each one is for.
- A summary at the end of what changed, what was removed, and any of the Open
Questions above that got resolved differently than expected.
## Out of scope
- Don't touch `~/nix-auto-installer` contents beyond checking it for references
to the old target names — if changes there are needed, flag them, don't make
them without confirming.
- Don't add new build types or platforms beyond the ones listed here.
+2
View File
@@ -98,6 +98,8 @@
linode-tailscale-exit-node = mkTarget { platform = "linode"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
proxmox-tailscale-exit-node = mkTarget { platform = "proxmox"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
lxc-tailscale-exit-node = mkTarget { platform = "lxc"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
lxc-tor-relay = mkTarget { platform = "lxc"; buildType = "tor-relay"; hostPath = ./hosts/tor-relay/host.nix; };
};
# Auto-install environments (migrated from the former nix-auto-installer
+12
View File
@@ -0,0 +1,12 @@
_:
{
networking.hostName = "tor-relay";
# No networking.hostId: only ZFS-touching hosts (server, docker) need one
# for pool-import safety, and this host does neither.
# A genuinely new host (not a pre-refactor carry-over), so it tracks the
# flake's current nixpkgs release rather than being pinned to an older one.
system.stateVersion = "26.05";
}
+7
View File
@@ -0,0 +1,7 @@
{ ... }:
{
imports = [
../tor/enable-relay.nix
];
}
+34 -15
View File
@@ -1,65 +1,84 @@
{ config, lib, pkgs, vars, ... }:
let
# `x-systemd.automount` never works inside a Linux container (LXC
# included, regardless of privilege) -- confirmed live on lxc-docker:
# systemd logs "Starting of <unit>.automount unsupported" for every
# share and never mounts them. Mount eagerly there instead, with
# `nofail` so a boot with the NFS server unreachable doesn't hang
# (the VM platforms rely on automount itself to get that same
# non-blocking behavior, so they don't need `nofail` too).
automountOpts = if config.boot.isContainer then [ "nofail" ] else [ "x-systemd.automount" ];
# A bare hostname here never resolves reliably: systemd-resolved only
# ever tries LLMNR for single-label names (never DNS, regardless of any
# configured search domain), and a *global* search domain (the first fix
# attempted here) backfires worse -- confirmed live on lxc-docker, adding
# `networking.search` made systemd-resolved prioritize its domain-matched
# but server-less global scope over eth0's correctly-configured one for
# every "*.sweet.home" query, silently sending them to public fallback
# DNS instead. `resolvectl query --interface=eth0 server.sweet.home`
# resolved fine throughout, proving the LAN DNS server was never the
# problem -- only the ambient, unqualified device string was. Using the
# FQDN directly sidesteps all of that, matching the pattern
# ../raspi/mount-data.nix already uses for the same reason.
nfsServer = "${vars.nfsServerHost}.${vars.homeDomain}";
in
{
fileSystems = {
${vars.nfsShares.dockerConfig.mountpoint} = {
device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath}";
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath}";
fsType = "nfs";
options = [
"nfsvers=4.2"
"_netdev"
"x-systemd.automount"
"noatime"
];
] ++ automountOpts;
};
${vars.nfsShares.dockerDatabases.mountpoint} = {
device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath}";
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath}";
fsType = "nfs";
options = [
"nfsvers=4.2"
"_netdev"
"x-systemd.automount"
"noatime"
];
] ++ automountOpts;
};
${vars.nfsShares.dockerVolumes.mountpoint} = {
device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
fsType = "nfs";
options = [
"nfsvers=4.2"
"_netdev"
"x-systemd.automount"
"noatime"
];
] ++ automountOpts;
};
${vars.nfsShares.nextcloudData.mountpoint} = {
device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath}";
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath}";
fsType = "nfs";
options = [
"nfsvers=4.2"
"_netdev"
"x-systemd.automount"
"noatime"
];
] ++ automountOpts;
};
${vars.nfsShares.raspiVolumes.mountpoint} = {
device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath}";
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath}";
fsType = "nfs";
options = [
"nfsvers=4.2"
"_netdev"
"x-systemd.automount"
"noatime"
];
] ++ automountOpts;
};
};
}
+2 -2
View File
@@ -130,7 +130,7 @@
# at *activation* time, which runs before systemd would otherwise
# generate one on first boot. Without pre-seeding it here, secrets
# (including the login password) fail to decrypt on first boot.
# Generate the key with scripts/prepare-host-key.sh first.
# Generate the key with scripts/secrets/prepare-host-key.sh first.
#
# Two places a key can come from, checked in order:
# /etc/host-keys baked into this image at build time (see
@@ -150,7 +150,7 @@
else
echo "WARNING: no SSH host key found for ''${choice} (checked /etc/host-keys and /root/host-keys)"
echo "sops-nix secrets (including the login password) will NOT decrypt on first boot."
echo "Run scripts/prepare-host-key.sh for host ''${choice} on your admin workstation first,"
echo "Run scripts/secrets/prepare-host-key.sh for host ''${choice} on your admin workstation first,"
echo "then either rebuild this image with NIXOS_HOST_KEYS_DIR set, or scp the result to"
echo "/root/host-keys/ on this machine."
read -rp "Continue without a pre-seeded key anyway? (y/N): " skip_key
+9 -5
View File
@@ -1,10 +1,14 @@
{ pkgs, vars, ... }:
{
# Install the remote builder key on each client host (do not commit private keys):
# sudo install -d -m 0700 /root/.ssh
# sudo install -m 0600 ./nixremote /root/.ssh/nixremote
# sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
# Authenticate as nixremote using the client host's own default root SSH
# identity (/root/.ssh/id_ed25519) rather than a separately-named key --
# matches vars.remoteBuilderAuthorizedKeys, which already authorizes
# each host's own default key (one entry per host, not a shared
# dedicated keypair). If this host doesn't have one yet:
# sudo -u root ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
# # then add its .pub to vars.remoteBuilderAuthorizedKeys and rebuild nix-cache
# sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
# Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
# can connect the first time without a manual ssh-keyscan/known_hosts
# step on every new client.
@@ -20,7 +24,7 @@
{
hostName = vars.nixCacheHost;
sshUser = vars.remoteBuilderUser;
sshKey = "/root/.ssh/${vars.remoteBuilderUser}";
sshKey = "/root/.ssh/id_ed25519";
inherit (pkgs.stdenv.hostPlatform) system;
maxJobs = 4;
speedFactor = 2;
+25 -5
View File
@@ -1,4 +1,4 @@
{ lib, modulesPath, flakeTarget, ... }:
{ config, lib, modulesPath, flakeTarget, ... }:
let
# Bakes this exact flake target's pre-generated SSH host key straight
@@ -14,7 +14,7 @@ let
# Without this, config.system.build.tarball's built-in system just
# generates a fresh host key at first boot like any other host would --
# but sops-nix derives its decryption key from *this* file, and
# .sops.yaml only trusts whatever key scripts/sync-host-keys.sh already
# .sops.yaml only trusts whatever key scripts/secrets/sync-host-keys.sh already
# registered for this exact target name. A freshly-generated key can
# never match that, so every secret (including this host's own login)
# permanently fails to decrypt. Confirmed live: sops-install-secrets
@@ -26,7 +26,7 @@ let
hostKeysDir = /. + hostKeysDirStr;
# flakeTarget ("${platform}-${buildType}") comes in via specialArgs from
# flake.nix's mkTarget -- exactly the name scripts/sync-host-keys.sh
# flake.nix's mkTarget -- exactly the name scripts/secrets/sync-host-keys.sh
# registers keys under. Deliberately not read back from
# config.environment.etc."flake-target" (which is set to the same value)
# -- this module also *contributes* to environment.etc below, and a
@@ -58,8 +58,28 @@ in
# host.nix declares each host's real hostname (networking.hostName);
# keep that instead of letting Proxmox's ambient container config win.
manageHostName = true;
# Unprivileged matches how these containers are actually created.
privileged = false;
# Unprivileged by default -- matches how these containers are actually
# created (scripts/proxmox/create-proxmox-resource.sh reads this value
# back to decide `pct create`'s --unprivileged flag, so the two stay
# in sync).
#
# lxc-docker is the one exception: the kernel's NFS client doesn't set
# FS_USERNS_MOUNT, so mounting NFS from inside *any* non-init user
# namespace -- which is exactly what an unprivileged container's
# UID-mapped root runs in -- is rejected at the VFS layer with EPERM,
# no matter what Proxmox's own `mount=nfs;nfs4` container feature
# allows at the AppArmor layer (confirmed live: TCP to the NFS server
# succeeds, the server's export table matches the container's IP, and
# `mount.nfs: Operation not permitted` still fires immediately with no
# corresponding denial anywhere in the server's logs -- a kernel-level
# rejection, not a network or export-permission one). Keying off
# hostName rather than something docker-build-type-specific because
# modules/build-types/docker.nix is also composed for linode-docker/
# proxmox-docker, which don't import proxmox-lxc.nix at all --setting
# this option there would break their eval with "option does not
# exist" regardless of any mkIf guard, since mkIf only makes a value
# conditional, not whether the option needs to exist somewhere.
privileged = config.networking.hostName == "docker";
};
boot.loader = {
+10 -4
View File
@@ -1,4 +1,4 @@
{ vars, ... }:
{ config, lib, vars, ... }:
{
fileSystems.${vars.raspiMountpoint} = {
@@ -9,6 +9,15 @@
"_netdev"
"noatime"
# Explicitly use NFSv4.2 if supported
"nfsvers=4.2"
] ++ lib.optionals (!config.boot.isContainer) [
# `x-systemd.automount` never works inside a Linux container (LXC
# included) -- confirmed live on lxc-docker: systemd logs "Starting
# of <unit>.automount unsupported" and never mounts it. `nofail`
# above already keeps boot non-blocking there, so plain eager
# mounting is fine.
# Don't mount until first access
"x-systemd.automount"
@@ -17,9 +26,6 @@
# Give the Pi/Tailscale a little time to appear
"x-systemd.device-timeout=10s"
# Explicitly use NFSv4.2 if supported
"nfsvers=4.2"
];
};
+35
View File
@@ -0,0 +1,35 @@
{ pkgs, vars, ... }:
{
services.tor = {
enable = true;
# Opens settings.ORPort (and DirPort, unset here) in the firewall —
# see the nixpkgs tor module's own networking.firewall.mkIf block.
openFirewall = true;
relay = {
enable = true;
# Plain middle/guard relay, not "exit" — relays onion traffic between
# other Tor nodes without ever making requests to the public internet
# on a user's behalf, avoiding the abuse complaints and legal exposure
# an exit node invites.
role = "relay";
};
settings.ORPort = vars.ports.torRelayOrPort;
# Unix control socket at /run/tor/control (GroupWritable, group "tor")
# -- what nyx below actually monitors the relay through. Nyx's own
# default control-socket path (/var/run/tor/control) resolves to the
# same place, so no extra nyx config is needed.
controlSocket.enable = true;
};
# Lets the primary user's shell session read/write the control socket
# above without being root -- otherwise nyx fails to authenticate against
# it at all.
users.users.${vars.primaryUser}.extraGroups = [ "tor" ];
environment.systemPackages = [ pkgs.nyx ];
}
-146
View File
@@ -1,146 +0,0 @@
# Spec: Remove Sensitive Information from NixOS Flake
## Goal
Every secret currently readable in plaintext anywhere in this repo (working tree *and* git history) gets removed, replaced with `sops-nix`-managed encrypted references, and rotated. When this is done, the repo should be safe to make public without exposing anything about the systems it configures.
Treat this as three sequential milestones. Do not start git history rewriting (Milestone 3) until Milestones 1 and 2 are fully verified and the flake still builds. This should be its own branch (`refactor/secrets`) until fully verified, then merged.
---
## Milestone 1 — Audit
Before touching anything, produce a complete inventory. Do not guess at scope — grep the whole tree and the whole history.
1. Run a secret scanner across the working tree and full history. Use both, since they catch different things:
- `gitleaks detect --source . -v --log-opts="--all"` (scans history too)
- `trufflehog git file://. --since-commit=$(git rev-list --max-parents=0 HEAD) --only-verified=false`
If neither is installed, add them via a temporary `nix-shell -p gitleaks trufflehog` — don't install anything globally on the host.
2. Manually grep for the categories below, since scanners miss config-specific patterns:
- `hashedPassword`, `password`, `initialPassword`, `initialHashedPassword` in any `users.users.*` block
- `age.secrets`, `sops.secrets` (if any partial secrets work already exists — check for it)
- PSK / `preSharedKey`, `privateKeyFile` inline values (vs. file references) for WireGuard
- `authKey`, `apiToken`, `api_key`, `token =`, `secret =` in service modules (Tailscale, Cloudflare, backup tools, etc.)
- SSH private key material: search for `BEGIN OPENSSH PRIVATE KEY` / `BEGIN RSA PRIVATE KEY` literals
- TLS cert/key pairs committed under e.g. `secrets/`, `certs/`, `pki/`
- Real name, personal email, home address, or anything in comments/hostnames that maps a machine to your physical identity or network layout (e.g. hostnames like `wayne-desktop`, static LAN IPs, ISP-identifying info)
- `.env` files, `secrets.nix`, `secrets.yaml`, or any file that looks like it was meant to be gitignored but wasn't
3. Produce `secrets-inventory.md` (temporary, delete before finishing) listing: file path, line, secret type, and which host/service it belongs to. This becomes the checklist for Milestone 2 — every row must be either migrated to sops or deleted, with nothing left unaccounted for.
---
## Milestone 2 — Migrate to sops-nix
### 2.1 Set up sops-nix
1. Add the flake input:
```nix
sops-nix.url = "github:Mic92/sops-nix";
sops-nix.inputs.nixpkgs.follows = "nixpkgs";
```
2. Import `sops-nix.nixosModules.sops` into each host's module list (or into a shared `common.nix` if all hosts use it).
3. Generate an age keypair **per host** (not one shared key for everything — a compromised host shouldn't decrypt every other host's secrets):
```
nix-shell -p age --run "age-keygen -o /var/lib/sops-nix/key.txt"
```
Print the public key (`age-keygen -y`) for each host — you'll need it for `.sops.yaml`.
4. Also generate one age key for yourself (your admin workstation) so you can edit secrets without needing to SSH into a host: store it at `~/.config/sops/age/keys.txt`, back it up somewhere outside this repo (password manager, offline). **If this key is lost, every secret encrypted with it is unrecoverable — losing the age key is equivalent to losing the secrets.**
5. Create `.sops.yaml` at the repo root defining creation rules: which age public keys can decrypt which secrets files, keyed by path regex, so e.g. `secrets/hostA.yaml` is decryptable by your admin key + hostA's key, `secrets/hostB.yaml` by your admin key + hostB's key.
### 2.2 Migrate each secret category from the inventory
For each row in `secrets-inventory.md`:
- **Password hashes**: generate hash with `mkpasswd -m sha-512` (or `bcrypt` if your setup wants that), store under `sops.secrets."<name>/hashedPassword"`, reference via `users.users.<name>.hashedPasswordFile = config.sops.secrets."<name>/hashedPassword".path;`. Do not put the *plaintext* password anywhere, only the hash, and only the hash goes into the encrypted sops file.
- **API tokens / auth keys**: move the raw value into the per-host sops YAML, reference in the module via `config.sops.secrets."<service>/token".path` — most NixOS service modules that take a token also accept a `*File` variant (e.g. `environmentFile`, `tokenFile`); use that instead of passing the value directly.
- **Private keys / certs**: move the PEM/key content wholesale into a sops secret, output as a file with appropriate `sops.secrets.<name>.path`, `owner`, `mode`, `restartUnits` so the depending service (sshd, wireguard, nginx) reloads when the secret changes.
- **Personal/identifying info**: this doesn't belong in sops (it's not "secret," it's just information you don't want public). Replace real names/emails with placeholders or move to a small untracked `local.nix` that's `.gitignore`'d and imported conditionally, with a documented template (`local.nix.example`) committed instead.
### 2.3 Verify before moving on
- `nixos-rebuild dry-build --flake .#<host>` succeeds for every host.
- `sudo nixos-rebuild switch --flake .#<host>` on at least one real machine (or a VM) confirms secrets decrypt and services start.
- Confirm decrypted secrets land under `/run/secrets/` (not the Nix store — anything placed in `/nix/store` is world-readable by design, so sops-nix's runtime-only placement is the whole point; double check no module accidentally pulls a secret path into a store-built config file).
- Re-run the grep/scanner sweep from Milestone 1 against the *working tree only* (not history yet) — it should now come back clean.
---
## Milestone 3 — Scrub git history
Do this only after Milestone 2 is merged to your main branch and confirmed working, since it rewrites every commit SHA from the point of the earliest offending commit onward.
**This is destructive and irreversible on your local clone. Back up first:**
```
cp -r /path/to/nixos-repo /path/to/nixos-repo-backup-$(date +%F)
```
1. Install `git-filter-repo` (not the older `git filter-branch` / BFG — filter-repo is the currently maintained, faster, safer tool):
```
nix-shell -p git-filter-repo
```
2. Use the `secrets-inventory.md` list to build a list of literal strings/paths to strip. Two approaches, use both:
- Path-based: if whole files were secret (e.g. `secrets.nix`, a `.env`, a private key file), remove them entirely from history:
```
git filter-repo --path secrets.nix --path .env --invert-paths
```
- Value-based: for secrets embedded inline in files you're keeping (not deleting the whole file), use `--replace-text` with a file listing each literal secret string to replace with `***REMOVED***`:
```
git filter-repo --replace-text expressions.txt
```
3. After filtering, verify: run the Milestone 1 scanners again against full history (`--log-opts="--all"`). They must come back clean.
4. Force-push the rewritten history:
```
git push origin --force --all
git push origin --force --tags
```
5. **Every other clone of this repo (other machines, WSL instances, CI) must be deleted and re-cloned fresh** — a `git pull` against rewritten history will not work cleanly and risks resurrecting the old commits. Don't try to reconcile old clones; throw them away and re-clone.
6. If this repo has ever been pushed to a public host (GitHub, etc.) or a fork/mirror exists, treat every secret that was ever in history as **permanently compromised regardless of the rewrite** — caches, forks, and Wayback-style archives can retain old commits indefinitely. History scrubbing prevents *future* exposure via `git clone`; it does not undo past exposure.
---
## Milestone 4 — Rotate everything
Because the secrets were exposed in history (even briefly, even in a private repo), the migration is not complete until every credential in the inventory has been **rotated**, not just re-encrypted. Re-encrypting an already-leaked value protects it going forward but doesn't undo the leak.
For each row in the original inventory:
- Password hashes → change the actual account password, regenerate the hash, update the sops file.
- API tokens/auth keys → revoke the old token in the issuing service's dashboard (Cloudflare, Tailscale, backup provider, etc.) and generate a new one.
- SSH/WireGuard private keys → generate new keypairs, update the corresponding public key wherever it's trusted (authorized_keys, peer configs, etc.), retire the old ones.
- TLS certs → reissue if the private key was exposed.
Keep `secrets-inventory.md` open during this step and check off each row as rotated. Delete the file only once every row is checked off — it should not be committed.
---
## Ongoing prevention
Add a pre-commit hook (or a `nix flake check` step) running `gitleaks protect --staged` so a secret can't be committed again by accident. Document in the repo README (briefly) that new secrets go through `sops <file>` to edit, never as plaintext in a tracked file.
---
## Definition of done
**Status as of 2026-07-20:** Milestones 13 are done — sops-nix is fully
wired (`.sops.yaml`, `secrets/*.yaml`, referenced via `hashedPasswordFile`/
`*File`/`sops.secrets.*.path` throughout), and history has been scrubbed
with `git-filter-repo` + force-push (this removed a GitHub fine-grained PAT
that had been committed in plaintext in `flake.nix`/`common/home.nix`
between 2025-07-16 and 2026-02-09, later migrated to sops but never scrubbed
from history until now). **Milestone 4 is not confirmed** — whether that PAT
(or any other historically-plaintext credential) was actually rotated, not
just re-encrypted, isn't something this repo can attest to; that's an
operator action against the issuing service (GitHub, etc.), not a repo
change. Do that before considering this fully closed.
- [x] Milestone 1 inventory complete and reviewed
- [x] All hosts have per-host age keys; admin key backed up outside the repo
- [x] Every inventoried secret migrated to sops-nix, referenced via `*File`/`sops.secrets.*.path`, nothing plaintext in the working tree
- [x] `nixos-rebuild dry-build` and at least one real `switch` verified per host
- [x] Working-tree scanner sweep clean
- [x] History rewritten with `git-filter-repo`, force-pushed, full-history scanner sweep clean
- [ ] All other clones deleted and re-cloned from the rewritten history — every clone that existed before 2026-07-20's rewrite (any other machine, WSL instance, or CI checkout) needs this
- [ ] Every credential in the original inventory rotated (not just re-encrypted) — **the GitHub PAT found in history specifically still needs this**
- [x] Pre-commit secret scanning hook added (`.githooks/pre-commit`, `gitleaks protect --staged`)
- [x] `secrets-inventory.md` deleted from the working directory (never committed)
+3 -2
View File
@@ -129,5 +129,6 @@ echo "flake.lock still points at the old input revisions until refreshed. Either
echo " nix flake update nixpkgs home-manager # just these two inputs"
echo " nix flake update # everything — see docs/flake-lock-automation.md"
echo
echo "Then run 'bash scripts/codex-maintenance.sh dry-run' before committing —"
echo "a channel bump can shift option defaults across every host."
echo "Then run 'bash scripts/codex-maintenance.sh --full-check --dry-run' before"
echo "committing — a channel bump can shift option defaults across every host,"
echo "and only --dry-run actually builds anything to catch that."
+255 -45
View File
@@ -1,4 +1,21 @@
#!/usr/bin/env bash
# Validation entry point for CI and local/agent review.
#
# Default mode (what CI runs on every push/PR): fmt-check, statix, and eval
# are scoped to files that actually changed against a base ref, plus
# whichever hosts/packages those changes can affect. This exists because
# the unscoped sweep below is slow enough to time out CI runners -- see
# --full-check.
#
# --full-check: the historical full sweep (every host, every package,
# fmt --check ./statix check . over the whole tree). Slow -- minutes, not
# seconds. CI never passes this; run it locally before a release or after
# touching modules/common/*, flake.nix, or variables.nix if you want extra
# confidence beyond what the changed-files scope already covers for those
# paths (see below).
#
# --dry-run: adds `nix build --dry-run --no-link` for whatever scope is
# active (changed-files scope by default, full scope under --full-check).
set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -7,7 +24,39 @@ source "${script_dir}/lib/nix-bootstrap.sh"
# shellcheck source=lib/nix-eval.sh
source "${script_dir}/lib/nix-eval.sh"
MODE="${1:-validate}"
repo_root="$(cd "${script_dir}/.." && pwd)"
cd "$repo_root"
full_check=false
dry_run=false
usage() {
cat <<'EOF'
Usage: scripts/codex-maintenance.sh [--full-check] [--dry-run]
--full-check Run the full sweep: fmt-check and statix over the whole
repo, eval every host and package. Slow. Never run by CI.
--dry-run Additionally run `nix build --dry-run --no-link` for
whatever scope is active.
With neither flag (the CI default), fmt-check/statix/eval are scoped to
files changed against a base ref (env MAINT_BASE_SHA, else the PR base,
else HEAD^), plus the hosts/packages those changes can affect.
EOF
}
for arg in "$@"; do
case "$arg" in
--full-check) full_check=true ;;
--dry-run) dry_run=true ;;
-h|--help) usage; exit 0 ;;
*)
echo "Unknown argument: $arg" >&2
usage >&2
exit 1
;;
esac
done
ensure_nix_profile
@@ -16,12 +65,6 @@ if ! command -v nix >/dev/null 2>&1; then
exit 127
fi
hosts="$(list_flake_targets .)"
echo "Hosts:"
echo "$hosts"
echo
echo "Checking for obvious committed secrets..."
if grep -RInE 'github_pat_|ghp_|access-tokens|hashedPassword[[:space:]]*=' \
--exclude-dir=.git \
@@ -33,49 +76,214 @@ else
echo "No obvious token patterns found."
fi
mapfile -t all_hosts < <(list_flake_targets .)
mapfile -t all_packages < <(nix eval --json "${NIX_EVAL_FLAGS[@]}" .#packages.x86_64-linux --apply builtins.attrNames | jq -r '.[]')
# host_targets_for_dir <hosts-subdir-name>
# Prints the nixosConfigurations target names whose hostPath is
# ./hosts/<dir>/host.nix, derived straight from flake.nix's generatedTargets
# (one mkTarget { ... } call per line) rather than a hand-maintained table,
# so it can't drift the way a copied mapping would.
host_targets_for_dir() {
local dir="$1"
grep -oE '^[[:space:]]*[A-Za-z0-9_-]+ = mkTarget \{[^}]*hostPath = \./hosts/'"${dir}"'/host\.nix;[^}]*\};' flake.nix \
| sed -E 's/^[[:space:]]*([A-Za-z0-9_-]+) = mkTarget.*/\1/' \
|| true
}
declare -a changed_files=()
scope_desc="full repo"
if ! $full_check; then
resolve_base_ref() {
if [[ -n "${MAINT_BASE_SHA:-}" ]] && git cat-file -e "${MAINT_BASE_SHA}^{commit}" 2>/dev/null; then
echo "$MAINT_BASE_SHA"
return
fi
if git rev-parse --verify -q HEAD^ >/dev/null 2>&1; then
echo "HEAD^"
return
fi
git hash-object -t tree /dev/null
}
base_ref="$(resolve_base_ref)"
echo
echo "Changed-files scope: diffing against ${base_ref}"
mapfile -t changed_files < <(git diff --name-only --diff-filter=ACMR "$base_ref" -- . | sort -u)
if [[ ${#changed_files[@]} -eq 0 ]]; then
echo "No changed files detected."
else
printf ' %s\n' "${changed_files[@]}"
fi
scope_desc="changed files only (base: ${base_ref})"
fi
# Whole-tree fmt/lint always run under --full-check; otherwise scoped below.
declare -a changed_nix_files=()
for f in "${changed_files[@]:-}"; do
[[ "$f" == *.nix && -f "$f" ]] && changed_nix_files+=("$f")
done
echo
echo "Checking Nix formatting with nixpkgs-fmt..."
nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check .
if $full_check; then
nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check .
elif [[ ${#changed_nix_files[@]} -gt 0 ]]; then
nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check "${changed_nix_files[@]}"
else
echo "No changed .nix files; skipping."
fi
echo
echo "Running statix lint..."
nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check .
if $full_check; then
nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check .
elif [[ ${#changed_nix_files[@]} -gt 0 ]]; then
for f in "${changed_nix_files[@]}"; do
nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check "$f"
done
else
echo "No changed .nix files; skipping."
fi
# Figure out which hosts/packages this run needs to eval (and, under
# --dry-run, build). full_check always means "everything"; otherwise a
# change to flake.nix/flake.lock/variables.nix/modules/common/* (repo-wide
# inputs) or to any other modules/*.nix outside platforms//build-types
# (whose blast radius isn't safely inferable from the path alone -- see
# CLAUDE.md's "Grep modules/build-types/*.nix for each build type's imports
# list") also falls back to everything, on the same reasoning CLAUDE.md
# already gives interactive sessions for when to run the full sweep.
# Anything more targeted -- a host.nix, a platform module, a build-type
# module -- narrows to just the hosts it can affect.
declare -A affected_hosts=()
eval_packages=false
if $full_check; then
for h in "${all_hosts[@]}"; do affected_hosts[$h]=1; done
eval_packages=true
else
full_fallback=false
for f in "${changed_files[@]:-}"; do
case "$f" in
flake.nix|flake.lock|variables.nix|modules/common/*)
full_fallback=true
;;
esac
done
if ! $full_fallback; then
for f in "${changed_files[@]:-}"; do
case "$f" in
hosts/*/*)
hostdir="${f#hosts/}"
hostdir="${hostdir%%/*}"
while IFS= read -r t; do
[[ -n "$t" ]] && affected_hosts[$t]=1
done < <(host_targets_for_dir "$hostdir")
;;
modules/platforms/*.nix)
platform="$(basename "$f" .nix)"
for h in "${all_hosts[@]}"; do
[[ "$h" == "${platform}-"* ]] && affected_hosts[$h]=1
done
;;
modules/build-types/*.nix)
buildtype="$(basename "$f" .nix)"
for h in "${all_hosts[@]}"; do
[[ "$h" == *"-${buildtype}" ]] && affected_hosts[$h]=1
done
;;
modules/installer/*)
# iso.nix (imported by both the "installer" nixosConfigurations
# target and netbootSystem, which backs packages.pxe) pulls in
# common.nix, so a common.nix change reaches all three.
affected_hosts[installer]=1
eval_packages=true
;;
modules/pxe-boot/*)
# stage-installer-artifacts.nix is imported by
# modules/build-types/pxe-boot.nix only -- same blast radius as a
# build-types/*.nix change, not a packages one.
for h in "${all_hosts[@]}"; do
[[ "$h" == *"-pxe-boot" ]] && affected_hosts[$h]=1
done
;;
modules/*)
full_fallback=true
;;
esac
done
fi
if $full_fallback; then
echo
echo "Changed files affect shared config; falling back to evaluating every host/package."
for h in "${all_hosts[@]}"; do affected_hosts[$h]=1; done
eval_packages=true
fi
fi
mapfile -t hosts < <(for h in "${!affected_hosts[@]}"; do echo "$h"; done | sort)
echo
echo "Evaluating host toplevel derivations..."
for host in $hosts; do
echo "==> $host"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
# lxc-* hosts deploy via a directly pct-restore-able tarball instead of
# nixos-install (see docs/auto-installer.md); proxmox-* hosts can
# alternatively be built as a standalone disk image (see
# docs/proxmox-images.md). Both are otherwise-unvalidated buildable
# surface, easy to silently break without this.
case "$host" in
lxc-*)
echo "==> $host (tarball)"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.tarball.drvPath"
;;
proxmox-*)
echo "==> $host (diskoImagesScript)"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript.drvPath"
;;
esac
done
echo "Checking nix-cache host key for drift..."
if bash "${script_dir}/secrets/sync-nix-cache-host-key.sh" --check; then
:
else
drift_status=$?
if [[ "$drift_status" -eq 2 ]]; then
echo "nix-cache unreachable from here -- skipping host-key drift check."
else
echo "WARNING: nix-cache's host key has drifted from variables.nix (see above)." >&2
echo " Run 'bash scripts/secrets/sync-nix-cache-host-key.sh' to fix." >&2
fi
fi
echo
echo "Evaluating buildable packages..."
packages="$(nix eval --json "${NIX_EVAL_FLAGS[@]}" .#packages.x86_64-linux --apply builtins.attrNames | jq -r '.[]')"
for pkg in $packages; do
echo "==> packages.x86_64-linux.${pkg}"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
done
if [[ ${#hosts[@]} -eq 0 ]]; then
echo "No hosts affected by changed files; skipping host eval."
else
echo "Evaluating host toplevel derivations (${scope_desc})..."
for host in "${hosts[@]}"; do
echo "==> $host"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
if [[ "$MODE" == "dry-run" ]]; then
# lxc-* hosts deploy via a directly pct-restore-able tarball instead of
# nixos-install (see docs/auto-installer.md); proxmox-* hosts can
# alternatively be built as a standalone disk image (see
# docs/proxmox-images.md). Both are otherwise-unvalidated buildable
# surface, easy to silently break without this.
case "$host" in
lxc-*)
echo "==> $host (tarball)"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.tarball.drvPath"
;;
proxmox-*)
echo "==> $host (diskoImagesScript)"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript.drvPath"
;;
esac
done
fi
echo
if ! $eval_packages; then
echo "No packages affected by changed files; skipping package eval."
else
echo "Evaluating buildable packages..."
for pkg in "${all_packages[@]}"; do
echo "==> packages.x86_64-linux.${pkg}"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
done
fi
if $dry_run; then
echo
echo "Running dry-run builds for all hosts. This will not create result symlinks."
for host in $hosts; do
echo "Running dry-run builds for the active scope. This will not create result symlinks."
for host in "${hosts[@]:-}"; do
echo "==> Dry-run build: $host"
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel"
@@ -91,12 +299,14 @@ if [[ "$MODE" == "dry-run" ]]; then
esac
done
echo
echo "Running dry-run builds for all packages."
for pkg in $packages; do
echo "==> Dry-run build: packages.x86_64-linux.${pkg}"
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
done
if $eval_packages; then
echo
echo "Running dry-run builds for packages."
for pkg in "${all_packages[@]}"; do
echo "==> Dry-run build: packages.x86_64-linux.${pkg}"
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
done
fi
fi
echo
+13 -9
View File
@@ -41,6 +41,17 @@ warn-dirty = false
build-users-group = nixbld
EOF
# The official installer's single-user root path still shells out to
# `sudo` to create /nix even though it already knows it's running as
# root -- confirmed live against a sudo-less minimal Debian/Proxmox
# node, where it fails with "sudo: not found" and prints this exact
# mkdir/chown as the manual fix. Pre-create it so that branch of the
# installer is skipped entirely.
if [ ! -d /nix ]; then
mkdir -m 0755 /nix
chown root /nix
fi
sh <(curl -L https://nixos.org/nix/install) --no-daemon
else
sh <(curl -L https://nixos.org/nix/install) --no-daemon
@@ -71,13 +82,6 @@ if ! command -v jq >/dev/null 2>&1; then
fi
echo "Available NixOS hosts:"
hosts="$(list_flake_targets .)"
echo "$hosts"
list_flake_targets .
echo "Evaluating all host toplevel derivations..."
for host in $hosts; do
echo "==> Evaluating $host"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
done
echo "Codex setup complete."
echo "Codex setup complete. Run bash scripts/codex-maintenance.sh to validate changes."
+32 -14
View File
@@ -3,19 +3,29 @@
# second copy of these values in every script:
# source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh"
# Every variable can still be overridden per-invocation via the
# environment (e.g. PROXMOX_STORAGE=tank-nvme ./scripts/create-proxmox-resource.sh ...)
# environment (e.g. PROXMOX_STORAGE=tank-nvme ./scripts/proxmox/create-proxmox-resource.sh ...)
# since each one only sets a default if unset.
# SSH-reachable Proxmox node that scripts/create-proxmox-resource.sh runs
# pct/qm on. Matches the Proxmox web UI hostname already used in
# hosts/nixos/home.nix's desktop shortcuts (pve.<homeDomain> from
# variables.nix) -- change this if that's not actually reachable over SSH,
# or if you're targeting a different node in a multi-node cluster.
: "${PROXMOX_HOST:=pve.sweet.home}"
# Two SSH-reachable Proxmox nodes exist on the LAN:
# - pve1.sweet.home -- production. Real, live VMs/containers.
# - pve-test.sweet.home -- sandbox/test node, for scratch VMs/containers
# that don't belong on production.
#
# PROXMOX_HOST is what scripts/proxmox/create-proxmox-resource.sh actually
# targets by default -- overridable per-invocation with --node <hostname>,
# or per-variable as usual (e.g. PROXMOX_HOST=$PVE_TEST_HOST). It defaults
# to production, matching this repo's behavior before pve-test existed --
# see CLAUDE.md's "Two Proxmox nodes" section for the policy on which
# situations should target which node (in particular: Claude defaults to
# pve-test, not this variable's own default, unless explicitly told
# otherwise).
: "${PVE1_HOST:=pve1.sweet.home}"
: "${PVE_TEST_HOST:=pve-test.sweet.home}"
: "${PROXMOX_HOST:=$PVE1_HOST}"
: "${PROXMOX_SSH_USER:=root}"
# Where this flake repo lives on the Proxmox node itself.
# scripts/create-proxmox-resource.sh builds images directly on the node
# scripts/proxmox/create-proxmox-resource.sh builds images directly on the node
# instead of transferring them over the network -- it clones the repo here
# (from this checkout's own `origin` remote) the first time it doesn't
# find it, installing build tooling via scripts/codex-setup.sh, then
@@ -25,7 +35,7 @@
# Storage pool names -- Proxmox's own stock-install defaults, but this
# varies a lot by setup (ZFS pool name, custom LVM-thin volume, etc.).
# Verify with `pvesm status` on the node and correct these if wrong.
: "${PROXMOX_STORAGE:=local-lvm}" # VM disks / CT rootfs
: "${PROXMOX_STORAGE:=local-zfs}" # VM disks / CT rootfs
: "${PROXMOX_ISO_STORAGE:=local}" # uploaded images/ISOs/CT templates
: "${PROXMOX_BRIDGE:=vmbr0}"
@@ -53,12 +63,20 @@
# crash-loops on a denied `/run/credentials/*` mount every ~3s (visible
# as garbage on the console) and core services like nsncd fail the same
# way on userns_create; system.build.tarball never finishes activating.
: "${PROXMOX_DEFAULT_LXC_FEATURES:=nesting=1,keyctl=1}"
#
# mount=nfs;nfs4: without it, AppArmor blanket-denies the `nfs`/
# `rpc_pipefs` mount syscalls any NFS client share needs -- confirmed
# live on lxc-docker (which mounts several, see modules/docker/mount-data.nix
# and modules/raspi/mount-data.nix): `mount: /var/lib/nfs/rpc_pipefs:
# permission denied`. Harmless to grant on lxc targets that don't mount
# NFS at all -- it only widens what the container is *allowed* to mount,
# nothing here forces a mount to happen.
: "${PROXMOX_DEFAULT_LXC_FEATURES:=nesting=1,keyctl=1,mount=nfs;nfs4}"
export PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE PROXMOX_ISO_STORAGE \
PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES PROXMOX_DEFAULT_MEMORY_MB \
PROXMOX_DEFAULT_LXC_DISK_GB PROXMOX_DEFAULT_LXC_FEATURES \
PROXMOX_REMOTE_REPO_DIR
export PVE1_HOST PVE_TEST_HOST PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE \
PROXMOX_ISO_STORAGE PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES \
PROXMOX_DEFAULT_MEMORY_MB PROXMOX_DEFAULT_LXC_DISK_GB \
PROXMOX_DEFAULT_LXC_FEATURES PROXMOX_REMOTE_REPO_DIR
# Matches variables.nix's nixCacheHost -- update both if it ever changes.
: "${NIX_CACHE_HOST:=nix-cache}"
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env bash
# Shared "type X to confirm" prompt for scripts/proxmox/create-proxmox-resource.sh
# (--modify, and replacing an existing --allow-duplicate-host resource) and
# scripts/secrets/sync-host-keys.sh (--regenerate-all-keys) -- three destructive
# confirmations that all work the same way (echo the expected value back
# exactly), kept in one place so the prompt/comparison logic can't drift.
# Source alongside env.sh:
# source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/confirm.sh"
#
# Deliberately does NOT print anything on mismatch or decide exit-vs-return
# -- callers vary on both (a top-level script exits, a subcommand function
# returns; wording differs too), so that stays at the call site.
# confirm_typed <expected> <prompt>
# Prints <prompt> via `read -rp`, then reports (via exit status) whether the
# typed input matched <expected> exactly.
confirm_typed() {
local expected="$1" prompt="$2" input
read -rp "$prompt" input
[[ "$input" == "$expected" ]]
}
+18
View File
@@ -34,3 +34,21 @@ flake_target_hostname() {
nix eval --raw "${NIX_EVAL_FLAGS[@]}" \
"${flake_ref}#nixosConfigurations.${target}.config.networking.hostName" 2>/dev/null
}
# flake_target_lxc_privileged <flake_ref> <target>
# Prints "true" or "false" for one lxc-* target's config.proxmoxLXC.privileged
# (modules/platforms/lxc.nix is the single source of truth -- e.g.
# lxc-docker sets this true so it can NFS-mount; every other lxc-* host
# stays unprivileged). Only meaningful for lxc-* targets -- the option
# doesn't exist for linode-*/proxmox-* (nixpkgs' proxmox-lxc.nix, which
# declares it, is only ever imported by modules/platforms/lxc.nix). Empty
# (not an error under set -e) if the eval fails.
flake_target_lxc_privileged() {
local flake_ref="$1" target="$2"
# Not --raw: the option is a Nix boolean, and --raw can only coerce
# strings ("cannot coerce a Boolean to a string"). Plain `nix eval`
# prints a bare `true`/`false` for a boolean, which is exactly the
# string this needs.
nix eval "${NIX_EVAL_FLAGS[@]}" \
"${flake_ref}#nixosConfigurations.${target}.config.proxmoxLXC.privileged" 2>/dev/null
}
+52
View File
@@ -0,0 +1,52 @@
#!/usr/bin/env bash
# Shared sops/age helpers for scripts/secrets/backup-admin-key.sh,
# scripts/secrets/rotate-admin-key.sh, and scripts/secrets/sync-host-keys.sh -- all three
# derive an age public key from a private identity file the same way, two
# of them resolve the same sops/age default key-file path, and two of them
# run `sops updatekeys` the same way. Kept in one place so they can't drift
# apart. Source alongside env.sh:
# source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/sops-age.sh"
#
# Uses NIX_OPTS (an array of extra `nix-shell` options -- see env.sh's
# nix_extra_opts) if the caller has already set it, same convention as
# lib/ssh-host-keys.sh. Falls back to no extra options if the caller never
# sourced env.sh.
if ! declare -p NIX_OPTS >/dev/null 2>&1; then
declare -a NIX_OPTS=()
fi
# sops/age's own default identity-file resolution order, minus $SOPS_AGE_KEY
# itself (an inline identity, not a path -- callers that accept it check it
# separately, before falling back to this).
: "${DEFAULT_SOPS_AGE_KEY_FILE:=${SOPS_AGE_KEY_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/sops/age/keys.txt}}"
# age_pubkey_from_identity_file <identity-file>
# Prints the age public key for a private identity file (age-keygen -y).
age_pubkey_from_identity_file() {
local identity_file="$1"
nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '${identity_file}'"
}
# sops_yaml_admin_pubkey <sops-yaml-path>
# Prints .sops.yaml's current &admin age public key, or empty (not an error
# under set -e) if no such anchor line exists -- callers that need to treat
# "missing" as fatal check for an empty result themselves.
sops_yaml_admin_pubkey() {
local sops_yaml="$1"
grep -E '^ - &admin age1' "$sops_yaml" 2>/dev/null | awk '{print $NF}' || true
}
# sops_updatekeys <secrets-file> [key-file]
# Re-encrypts <secrets-file> for .sops.yaml's current recipient set. If
# <key-file> is given, decrypts with that identity (SOPS_AGE_KEY_FILE)
# instead of whatever's ambient -- needed when the ambient default key
# doesn't match yet (e.g. mid-rotation, decrypting with the outgoing key).
sops_updatekeys() {
local secrets_file="$1" key_file="${2:-}"
if [[ -n "$key_file" ]]; then
SOPS_AGE_KEY_FILE="$key_file" nix-shell "${NIX_OPTS[@]}" -p sops --run \
"sops updatekeys --yes '${secrets_file}'"
else
nix-shell "${NIX_OPTS[@]}" -p sops --run "sops updatekeys --yes '${secrets_file}'"
fi
}
+2 -2
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env bash
# Shared SSH-host-key / age-conversion helpers for scripts/sync-host-keys.sh
# and scripts/prepare-host-key.sh -- both generate the same kind of key
# Shared SSH-host-key / age-conversion helpers for scripts/secrets/sync-host-keys.sh
# and scripts/secrets/prepare-host-key.sh -- both generate the same kind of key
# (ed25519, no passphrase, the sops-nix age-derivation input) and convert it
# to an age recipient the same way; kept in one place so the two can't
# drift apart.
+221
View File
@@ -0,0 +1,221 @@
#!/usr/bin/env bash
# Ad hoc clone of a single VM/CT from pve1 (production) to pve-test
# (sandbox), via vzdump + qmrestore/pct restore -- not a general-purpose
# backup tool, just a quick "give me a disposable copy of this thing on
# pve-test" for testing against real-ish data without touching prod.
#
# Flow:
# 1. vzdump the resource on pve1 into its "local" storage (--mode
# snapshot by default, so the source keeps running throughout --
# see --mode below for when that's not possible).
# 2. Stream the resulting archive straight from pve1 to pve-test
# (ssh pve1 cat ... | ssh pve-test cat > ...) -- this machine is
# just the relay, no separate on-disk staging copy here.
# 3. qmrestore / pct restore it on pve-test under --new-vmid (default:
# same VMID as the source -- pve-test is a separate node/cluster, so
# no collision unless that VMID is already in use there too).
# Always restored with --unique 1 (fresh MAC addresses) since the
# source is typically still running on the same LAN -- restoring
# with the *same* MAC would put two live guests on the wire with
# identical hardware addresses.
# 4. Delete the vzdump archive from pve1's local storage and the
# relayed copy on pve-test, so neither node accumulates ad hoc
# backup files from this script. Only the pve1 original is
# preserved on any failure after step 1, so a failed
# transfer/restore can be retried without re-running the backup.
#
# This script's own defaults are pve1 -> pve-test, unlike
# create-proxmox-resource.sh's --node (which defaults to production) --
# see CLAUDE.md's "Two Proxmox nodes" section. pve1 is only ever touched
# here after typing the source VMID back to confirm; pve-test is treated
# as disposable, matching this repo's usual policy for that node.
#
# See --help for the full option list.
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=../lib/confirm.sh
source "${repo_root}/scripts/lib/confirm.sh"
usage() {
cat <<EOF
Usage: $0 --vmid <n> [options]
--vmid <n> Required: VMID on the source node to clone.
Kind (qemu VM vs LXC CT) is auto-detected.
--new-vmid <n> VMID to restore as on the target node
(default: same as --vmid).
--mode snapshot|suspend|stop
vzdump backup mode (default: snapshot -- the
source resource keeps running throughout;
requires snapshot-capable storage, e.g.
ZFS/LVM-thin/Ceph/qcow2). Fall back to
"suspend" (brief pause) or "stop" (source
goes down for the duration) if the source's
storage doesn't support live snapshots --
vzdump's own error will say so.
--source-node <host> (default: \$PVE1_HOST, ${PVE1_HOST})
--target-node <host> (default: \$PVE_TEST_HOST, ${PVE_TEST_HOST})
--source-storage <pool> Where vzdump writes the backup on the
source node (default: local).
--target-storage <pool> Where the restored disk/rootfs lands on
the target node (default: \$PROXMOX_STORAGE, ${PROXMOX_STORAGE}).
--keep-backup Don't delete the vzdump archive from
either node afterward (debugging aid).
--yes Skip the typed VMID confirmation
before touching the source node.
--dry-run Print the full plan and skip every
mutating step (vzdump, transfer,
restore, delete) and the confirm
prompt. Still makes read-only SSH
calls to look up the source kind
and check the target VMID is free
-- harmless on either node.
-h, --help
EOF
}
vmid=""
new_vmid=""
mode="snapshot"
source_node="$PVE1_HOST"
target_node="$PVE_TEST_HOST"
source_storage="local"
target_storage="$PROXMOX_STORAGE"
keep_backup=0
skip_confirm=0
dry_run=0
while [[ $# -gt 0 ]]; do
case "$1" in
--vmid) vmid="$2"; shift 2 ;;
--new-vmid) new_vmid="$2"; shift 2 ;;
--mode) mode="$2"; shift 2 ;;
--source-node) source_node="$2"; shift 2 ;;
--target-node) target_node="$2"; shift 2 ;;
--source-storage) source_storage="$2"; shift 2 ;;
--target-storage) target_storage="$2"; shift 2 ;;
--keep-backup) keep_backup=1; shift ;;
--yes) skip_confirm=1; shift ;;
--dry-run) dry_run=1; shift ;;
-h | --help) usage; exit 0 ;;
*) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;;
esac
done
if [[ -z "$vmid" ]]; then
echo "ERROR: --vmid is required." >&2
usage >&2
exit 1
fi
if [[ "$mode" != "snapshot" && "$mode" != "suspend" && "$mode" != "stop" ]]; then
echo "ERROR: --mode must be snapshot, suspend, or stop." >&2
exit 1
fi
[[ -z "$new_vmid" ]] && new_vmid="$vmid"
source_target="${PROXMOX_SSH_USER}@${source_node}"
target_target="${PROXMOX_SSH_USER}@${target_node}"
# No dry-run wrapper needed for the calls below: every mutating step
# (vzdump, transfer, restore, delete) is reached only after the --dry-run
# early-exit further down, so a plain `ssh` call is never in the dry-run
# path.
# --- identify the resource kind on the source node -----------------------
echo "==> Looking up VMID ${vmid} on ${source_node}..."
kind=""
if ssh "$source_target" "qm status ${vmid}" >/dev/null 2>&1; then
kind="vm"
elif ssh "$source_target" "pct status ${vmid}" >/dev/null 2>&1; then
kind="lxc"
else
echo "ERROR: VMID ${vmid} doesn't exist on ${source_node} as either a VM or CT." >&2
exit 1
fi
echo "VMID ${vmid} on ${source_node} is a ${kind}."
# --- refuse to clobber an existing resource on the target node -----------
if ssh "$target_target" "qm status ${new_vmid}" >/dev/null 2>&1 \
|| ssh "$target_target" "pct status ${new_vmid}" >/dev/null 2>&1; then
echo "ERROR: VMID ${new_vmid} already exists on ${target_node}. Pass --new-vmid" >&2
echo "with a free ID, or remove the existing resource there first." >&2
exit 1
fi
echo
echo "Plan:"
echo " source: ${kind} VMID ${vmid} on ${source_node} (storage: ${source_storage}, mode: ${mode})"
echo " target: VMID ${new_vmid} on ${target_node} (storage: ${target_storage}, fresh MAC via --unique)"
[[ "$keep_backup" -eq 1 ]] && echo " backup archives are kept on both nodes afterward (--keep-backup)"
if [[ "$dry_run" -eq 1 ]]; then
echo
echo "[dry-run] No backup, transfer, restore, or delete was performed."
exit 0
fi
if [[ "$skip_confirm" -ne 1 ]]; then
echo
if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) to confirm backing it up from ${source_node}: "; then
echo "Cancelled -- input didn't match ${vmid}." >&2
exit 1
fi
fi
# --- vzdump on the source node --------------------------------------------
echo
echo "==> Backing up VMID ${vmid} on ${source_node} (mode=${mode}, storage=${source_storage})..."
vzdump_log="$(ssh "$source_target" \
"vzdump ${vmid} --mode ${mode} --storage ${source_storage} --compress zstd" 2>&1)" \
|| {
echo "$vzdump_log" >&2
echo "ERROR: vzdump failed on ${source_node}." >&2
exit 1
}
echo "$vzdump_log"
archive="$(echo "$vzdump_log" | grep -oP "creating vzdump archive '\K[^']+" | tail -n1)"
if [[ -z "$archive" ]]; then
echo "ERROR: couldn't find the archive path in vzdump's output above." >&2
exit 1
fi
archive_basename="$(basename "$archive")"
target_tmp_archive="/var/tmp/${archive_basename}"
echo "Archive: ${archive}"
# Always clean up the relayed copy on the target node, success or failure
# -- it's only ever a working copy, restored or not.
cleanup_target_tmp() {
if [[ "$keep_backup" -ne 1 ]]; then
ssh "$target_target" "rm -f '${target_tmp_archive}'" >/dev/null 2>&1 || true
fi
}
trap cleanup_target_tmp EXIT
# --- relay the archive from source to target ------------------------------
echo
echo "==> Transferring archive to ${target_node}..."
ssh "$source_target" "cat '${archive}'" | ssh "$target_target" "cat > '${target_tmp_archive}'"
# --- restore on the target node --------------------------------------------
echo
echo "==> Restoring as VMID ${new_vmid} on ${target_node} (storage=${target_storage})..."
if [[ "$kind" == "vm" ]]; then
ssh "$target_target" "qmrestore '${target_tmp_archive}' ${new_vmid} --storage ${target_storage} --unique 1"
else
ssh "$target_target" "pct restore ${new_vmid} '${target_tmp_archive}' --storage ${target_storage} --unique 1"
fi
# --- clean up the source backup now that the restore succeeded -----------
if [[ "$keep_backup" -ne 1 ]]; then
echo
echo "==> Deleting backup archive from ${source_node}'s ${source_storage} storage..."
ssh "$source_target" "rm -f '${archive}' '${archive}.notes' '${archive}.log'" >/dev/null 2>&1 || true
fi
echo
echo "Done. VMID ${new_vmid} (${kind}) is now on ${target_node}, cloned from" \
"VMID ${vmid} on ${source_node}."
+182
View File
@@ -0,0 +1,182 @@
#!/usr/bin/env bash
# Points a non-NixOS Debian machine's Nix install at nix-cache: adds it as
# a substituter (with cache.nixos.org kept as fallback) and, once the
# remote-builder private key is installed, as a distributed-build machine
# too.
#
# This is the non-NixOS equivalent of modules/nix-cache/client.nix +
# modules/nix-cache/remote-builder-client.nix -- those two only apply to
# hosts built from this flake. A plain Debian box with Nix installed
# (single- or multi-user install, nix-daemon running) has no NixOS module
# system to pick that config up, so this edits /etc/nix/nix.conf by hand
# instead. Run this ON the target Debian machine, as root.
#
# The values below mirror variables.nix / modules/nix-cache/client.nix in
# this repo -- update both if nix-cache is ever rebuilt with a new host
# key or the cache signing key is rotated (see docs/nix-cache.md).
#
# REMOTE_BUILDER_KEY defaults to this machine's own default root SSH
# identity (matches modules/nix-cache/remote-builder-client.nix's
# convention for real NixOS clients: authenticate as nixremote with the
# host's own default key, added individually to
# vars.remoteBuilderAuthorizedKeys, rather than a separately-named or
# shared keypair) -- generate one with
# `ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519` if this machine
# doesn't have one yet, then add its .pub to vars.remoteBuilderAuthorizedKeys
# and rebuild nix-cache.
#
# Usage:
# sudo ./configure-nix-cache-client.sh [--dry-run] [--no-remote-builder] [--no-restart]
#
# Env overrides (defaults match variables.nix):
# NIX_CACHE_HOST, NIX_CACHE_HOST_KEY, REMOTE_BUILDER_USER, REMOTE_BUILDER_KEY
set -euo pipefail
: "${NIX_CACHE_HOST:=nix-cache}"
: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache}"
: "${REMOTE_BUILDER_USER:=nixremote}"
: "${REMOTE_BUILDER_KEY:=/root/.ssh/id_ed25519}"
CACHE_PUB_KEY="cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4="
FALLBACK_URL="https://cache.nixos.org/"
FALLBACK_PUB_KEY="cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
NIX_CONF="/etc/nix/nix.conf"
KNOWN_HOSTS="/etc/ssh/ssh_known_hosts"
MARKER_BEGIN="# BEGIN nix-cache client config (configure-nix-cache-client.sh)"
MARKER_END="# END nix-cache client config"
dry_run=0
with_remote_builder=1
restart_daemon=1
for arg in "$@"; do
case "$arg" in
--dry-run) dry_run=1 ;;
--no-remote-builder) with_remote_builder=0 ;;
--no-restart) restart_daemon=0 ;;
-h|--help)
sed -n '2,20p' "$0"
exit 0
;;
*)
echo "ERROR: unknown argument: $arg" >&2
exit 1
;;
esac
done
if [[ "$dry_run" -eq 0 && "$EUID" -ne 0 ]]; then
echo "ERROR: must run as root (writes $NIX_CONF and, unless --no-remote-builder, $KNOWN_HOSTS)." >&2
exit 1
fi
if ! command -v nix >/dev/null 2>&1; then
echo "ERROR: no 'nix' binary on PATH -- install the Nix package manager first." >&2
exit 1
fi
if [[ ! -f "$NIX_CONF" ]]; then
echo "ERROR: $NIX_CONF not found -- expected an existing multi-user Nix install." >&2
exit 1
fi
builder_line=""
if [[ "$with_remote_builder" -eq 1 ]]; then
if [[ -f "$REMOTE_BUILDER_KEY" ]]; then
case "$(uname -m)" in
x86_64) nix_system="x86_64-linux" ;;
aarch64) nix_system="aarch64-linux" ;;
*)
echo "WARNING: unrecognized architecture '$(uname -m)' -- skipping remote builder, keeping substituter config." >&2
with_remote_builder=0
;;
esac
if [[ "$with_remote_builder" -eq 1 ]]; then
builder_line="builders = ssh://${REMOTE_BUILDER_USER}@${NIX_CACHE_HOST} ${nix_system} ${REMOTE_BUILDER_KEY} 4 2 big-parallel,kvm,nixos-test,benchmark"
fi
else
echo "WARNING: $REMOTE_BUILDER_KEY not found -- skipping remote builder config (substituter still configured)." >&2
echo " See docs/nix-cache.md 'Remote builder SSH keys' for how to install it, then re-run this script." >&2
with_remote_builder=0
fi
fi
block="$(cat <<EOF
$MARKER_BEGIN
extra-substituters = http://${NIX_CACHE_HOST} ${FALLBACK_URL}
extra-trusted-public-keys = ${CACHE_PUB_KEY} ${FALLBACK_PUB_KEY}
EOF
)"
if [[ "$with_remote_builder" -eq 1 ]]; then
block="${block}
builders-use-substitutes = true
${builder_line}"
fi
block="${block}
$MARKER_END"
echo "== nix.conf block to install =="
echo "$block"
echo "================================"
if [[ "$dry_run" -eq 1 ]]; then
echo "(--dry-run: not writing $NIX_CONF)"
else
tmp_conf="$(mktemp)"
trap 'rm -f "$tmp_conf"' EXIT
if grep -qF "$MARKER_BEGIN" "$NIX_CONF"; then
awk -v begin="$MARKER_BEGIN" -v end="$MARKER_END" -v block="$block" '
$0 == begin { print block; skip = 1; next }
$0 == end { skip = 0; next }
skip { next }
{ print }
' "$NIX_CONF" > "$tmp_conf"
else
cp "$NIX_CONF" "$tmp_conf"
printf '\n%s\n' "$block" >> "$tmp_conf"
fi
cp "$NIX_CONF" "${NIX_CONF}.bak.$(date +%Y%m%d%H%M%S)"
install -m 0644 "$tmp_conf" "$NIX_CONF"
echo "Updated $NIX_CONF (backup saved alongside it)."
fi
if [[ "$with_remote_builder" -eq 1 ]]; then
known_hosts_line="${NIX_CACHE_HOST} ${NIX_CACHE_HOST_KEY}"
if [[ "$dry_run" -eq 1 ]]; then
echo "(--dry-run: would ensure this line is present in $KNOWN_HOSTS)"
echo " $known_hosts_line"
else
mkdir -p "$(dirname "$KNOWN_HOSTS")"
touch "$KNOWN_HOSTS"
if ! grep -qF "$known_hosts_line" "$KNOWN_HOSTS" 2>/dev/null; then
echo "$known_hosts_line" >> "$KNOWN_HOSTS"
echo "Added nix-cache's SSH host key to $KNOWN_HOSTS."
fi
fi
fi
if [[ "$dry_run" -eq 0 && "$restart_daemon" -eq 1 ]]; then
if command -v systemctl >/dev/null 2>&1 && systemctl is-active --quiet nix-daemon 2>/dev/null; then
systemctl restart nix-daemon
echo "Restarted nix-daemon to pick up the new config."
else
echo "nix-daemon not managed by systemd (or not running) -- restart it manually to pick up the new config."
fi
fi
cat <<EOF
Done. Verify with:
curl http://${NIX_CACHE_HOST}/nix-cache-info
nix show-config | grep -E 'substituters|trusted-public-keys|builders'
EOF
if [[ "$with_remote_builder" -eq 1 ]]; then
cat <<EOF
ssh -i ${REMOTE_BUILDER_KEY} ${REMOTE_BUILDER_USER}@${NIX_CACHE_HOST} nix-store --version
nix build nixpkgs#hello -L
EOF
fi
@@ -12,33 +12,55 @@
# copies over the locally-managed host-keys/ (gitignored, so a git pull
# alone wouldn't carry it) before building.
#
# --node (default: $PROXMOX_HOST, see scripts/env.sh) picks which of the two
# LAN Proxmox nodes this runs against: production, pve1.sweet.home
# ($PVE1_HOST, PROXMOX_HOST's own default), or the sandbox node,
# pve-test.sweet.home ($PVE_TEST_HOST) -- pass --node "$PVE_TEST_HOST" (or
# set PROXMOX_HOST=$PVE_TEST_HOST) to target the sandbox instead. See
# CLAUDE.md's "Two Proxmox nodes" section: an agent session should default
# to pve-test and only touch pve1 when the operator has explicitly said so
# for the current task -- this script itself doesn't enforce that (its own
# default is production, matching this repo's behavior before pve-test
# existed), it's a policy for whoever/whatever is driving it.
#
# Usage:
# scripts/create-proxmox-resource.sh --type lxc|vm --host <name> [options]
# scripts/create-proxmox-resource.sh --type lxc|vm --list
# scripts/create-proxmox-resource.sh --modify --vmid <n> [--cores N] [--memory MB] [--grow-disk GB]
# scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --host <name> [options]
# scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --list
# scripts/proxmox/create-proxmox-resource.sh --modify --vmid <n> [--cores N] [--memory MB] [--grow-disk GB]
#
# SAFETY:
# - The default (create) mode only ever creates a NEW resource -- it
# refuses to run if the target VMID already exists on the node, or if
# a VM/CT identified as --host already exists under any other VMID
# (checked live against the node; --allow-duplicate-host overrides).
# - --allow-duplicate-host distinguishes an exact match (same --type
# *and* --host, e.g. re-running --type lxc --host docker while an
# lxc-docker container already exists -- almost always a redeploy of
# the same target to pick up a rebuilt image) from a cross-type match
# (a different platform sharing the same host identity, e.g. a
# proxmox-docker VM coexisting with lxc-docker). Only the exact match
# is destroyed and replaced, after typing the hostname back to
# confirm (outside --dry-run) -- a cross-type match is always left
# untouched, matching-or-not.
# - --modify only ever touches a resource you name explicitly via
# --vmid, shows exactly what will change first, and (outside
# --dry-run) always requires typing that VMID back to confirm before
# anything is sent to the node. There is no bulk/implicit modify.
# - Neither mode can start/stop/delete a resource. Not implemented on
# purpose -- ask before adding it.
# - Outside of --allow-duplicate-host's exact-match replace above,
# neither mode can start/stop/delete a resource.
#
# See --help for the full option list.
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
# shellcheck source=env.sh
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=lib/nix-eval.sh
# shellcheck source=../lib/nix-eval.sh
source "${repo_root}/scripts/lib/nix-eval.sh"
# shellcheck source=../lib/confirm.sh
source "${repo_root}/scripts/lib/confirm.sh"
sync_keys="${repo_root}/scripts/sync-host-keys.sh"
sync_keys="${repo_root}/scripts/secrets/sync-host-keys.sh"
usage() {
cat <<EOF
@@ -77,7 +99,13 @@ Create mode (default):
already exists on the node (checked live via
qm/pct, not any file in this repo) --
otherwise refused, since it'd share that
host's hostName/hostId.
host's hostName/hostId. An existing resource
of this *same* --type (e.g. re-running --type
lxc --host docker over an existing lxc-docker)
is destroyed and replaced, after confirming --
a different --type sharing the same --host
(e.g. a proxmox-docker VM) is always left
untouched.
Modify mode (reconfigure an EXISTING resource -- requires --modify):
--modify Switch to modify mode.
@@ -103,7 +131,9 @@ Shared:
--iso-storage <pool> (default: \$PROXMOX_ISO_STORAGE, ${PROXMOX_ISO_STORAGE})
--bridge <bridge> (default: \$PROXMOX_BRIDGE, ${PROXMOX_BRIDGE})
--node <host> Proxmox node to SSH into (default:
\$PROXMOX_HOST, ${PROXMOX_HOST})
\$PROXMOX_HOST, ${PROXMOX_HOST} --
production; the sandbox node is
\$PVE_TEST_HOST, ${PVE_TEST_HOST}).
--dry-run Print the full plan; touch nothing
local or remote, no prompts.
-h, --help
@@ -215,8 +245,7 @@ cmd_modify() {
fi
echo
read -rp "Type the VMID (${vmid}) to confirm these changes: " confirm
if [[ "$confirm" != "$vmid" ]]; then
if ! confirm_typed "$vmid" "Type the VMID (${vmid}) to confirm these changes: "; then
echo "Cancelled -- input didn't match ${vmid}."
exit 1
fi
@@ -319,12 +348,13 @@ fi
# this script itself always uses unless --name is overridden) -- a guest
# manually renamed on the node afterwards wouldn't match, but nothing here
# creates guests that way.
if [[ "$allow_duplicate_host" -eq 1 ]]; then
echo
echo "--allow-duplicate-host: skipping the check for an existing '${host}' on ${node}."
elif [[ "$dry_run" -eq 1 ]]; then
if [[ "$dry_run" -eq 1 ]]; then
echo
echo "[dry-run] would check ${node} for an existing VM/CT identified as '${host}'"
if [[ "$allow_duplicate_host" -eq 1 ]]; then
echo "[dry-run] --allow-duplicate-host: an existing ${type} named '${host}' would be" \
"destroyed and replaced; a different-type match would be left untouched"
fi
else
echo
echo "==> Checking ${node} for an existing VM/CT identified as '${host}'..."
@@ -349,17 +379,89 @@ REMOTE_SCRIPT
echo "check entirely)." >&2
exit 1
fi
# Split into "exact" (same resource kind as --type -- i.e. literally this
# same host+platform combo already exists, almost always a redeploy of
# the same target to test a rebuilt image) vs "cross-type" (a different
# platform sharing this host identity, e.g. a stopped proxmox-docker VM
# coexisting with an lxc-docker container -- a deliberate, valid setup
# this script has never managed and still won't). Read via a herestring
# (not a pipe) so the appends below survive outside the loop.
this_kind="$type"
exact_matches=""
cross_matches=""
if [[ -n "$existing" ]]; then
echo "ERROR: '${host}' already exists on ${node}:" >&2
echo "$existing" | while read -r kind id n; do
while read -r kind id n; do
[[ -z "$kind" ]] && continue
if [[ "$kind" == "$this_kind" ]]; then
exact_matches+="${kind} ${id} ${n}"$'\n'
else
cross_matches+="${kind} ${id} ${n}"$'\n'
fi
done <<<"$existing"
fi
if [[ -n "$exact_matches" && "$allow_duplicate_host" -ne 1 ]]; then
echo "ERROR: '${host}' already exists on ${node} as this same resource type:" >&2
echo "$exact_matches" | while read -r kind id n; do
[[ -z "$kind" ]] && continue
echo " - ${kind} VMID ${id} (${n})" >&2
done
echo "Refusing to create a second ${this_kind} sharing this identity. Pass" >&2
echo "--allow-duplicate-host to destroy it and create a fresh one in its place" >&2
echo "(after confirming), or use --modify to reconfigure the existing one instead." >&2
exit 1
fi
if [[ -n "$cross_matches" && "$allow_duplicate_host" -ne 1 ]]; then
echo "ERROR: '${host}' already exists on ${node} as a different resource type:" >&2
echo "$cross_matches" | while read -r kind id n; do
[[ -z "$kind" ]] && continue
echo " - ${kind} VMID ${id} (${n})" >&2
done
echo "Refusing to create a second resource sharing this identity. Pass" >&2
echo "--allow-duplicate-host to create one anyway (it gets its own distinct" >&2
echo "sops key and VMID -- the existing resource(s) above are left untouched)," >&2
echo "sops key and VMID -- the existing resource above is left untouched)," >&2
echo "or use --modify to reconfigure the existing one instead." >&2
exit 1
fi
if [[ -n "$cross_matches" ]]; then
echo "--allow-duplicate-host: '${host}' also exists on ${node} as a different resource" \
"type -- leaving it untouched:"
echo "$cross_matches" | while read -r kind id n; do
[[ -z "$kind" ]] && continue
echo " - ${kind} VMID ${id} (${n})"
done
fi
if [[ -n "$exact_matches" ]]; then
echo "--allow-duplicate-host: '${host}' already exists on ${node} as this same resource" \
"type -- it will be destroyed and replaced:"
echo "$exact_matches" | while read -r kind id n; do
[[ -z "$kind" ]] && continue
echo " - ${kind} VMID ${id} (${n})"
done
echo
if ! confirm_typed "$host" "Type the hostname (${host}) to confirm destroying the above and replacing it: "; then
echo "Cancelled -- input didn't match ${host}." >&2
exit 1
fi
echo "$exact_matches" | while read -r kind id n; do
[[ -z "$kind" ]] && continue
echo "==> Destroying ${kind} VMID ${id} (${n})..."
if [[ "$kind" == "vm" ]]; then
# qm destroy has no --force to stop-then-destroy in one call (pct's
# does) -- stop explicitly first if it's running.
if ssh "$ssh_target" "qm status ${id}" 2>/dev/null | grep -q running; then
ssh "$ssh_target" "qm stop ${id}"
fi
ssh "$ssh_target" "qm destroy ${id} --purge 1"
else
ssh "$ssh_target" "pct destroy ${id} --force 1 --purge 1"
fi
done
fi
fi
echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '${name}'"
@@ -435,7 +537,7 @@ ensure_remote_repo() {
echo
echo "==> Ensuring ${remote_repo_dir} exists and is current on ${node}..."
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would ensure ${remote_repo_dir} exists on ${node} (clone + scripts/codex-setup.sh if missing, git pull if present)"
echo "[dry-run] would ensure ${remote_repo_dir} exists on ${node} (clone if missing, git pull if present), and would verify/bootstrap build tooling there (scripts/codex-setup.sh) if \`nix\` isn't already on PATH -- and if that bootstrap actually ran, would also configure ${node} as a nix-cache client (scripts/proxmox/configure-nix-cache-client.sh)"
return
fi
@@ -453,8 +555,50 @@ ensure_remote_repo() {
fi
echo "Not present -- cloning from ${origin_url}..."
ssh "$ssh_target" "git clone '${origin_url}' '${remote_repo_dir}'"
fi
# Trivial check, run every time (not just right after a fresh clone) --
# confirmed live: a first bootstrap can clone the repo successfully and
# still leave the node without a working `nix` (e.g. the node had no
# `sudo`, which the Nix installer's root path depends on -- see the fix
# in scripts/codex-setup.sh), and a later run with the repo already
# present would otherwise never retry it. Sources
# scripts/lib/nix-bootstrap.sh's ensure_nix_profile first -- a
# single-user Nix install typically only gets sourced into login shells,
# and ssh's non-interactive command execution is neither, so a
# freshly-installed `nix` still wouldn't be on PATH here without it.
#
# Just `nix` today -- the only thing the remote build commands below
# actually invoke -- but a list (not a single hardcoded check) so a
# future remote step needing another tool can add itself here instead of
# growing a parallel check.
local remote_required_cmds=(nix)
local tooling_check_cmd="cd '${remote_repo_dir}' && . scripts/lib/nix-bootstrap.sh && ensure_nix_profile"
local cmd
for cmd in "${remote_required_cmds[@]}"; do
tooling_check_cmd="${tooling_check_cmd} && command -v ${cmd}"
done
if ssh "$ssh_target" "$tooling_check_cmd" >/dev/null 2>&1; then
echo "Build tooling already present on ${node}."
else
echo "==> Bootstrapping build tooling on ${node} (scripts/codex-setup.sh)..."
ssh "$ssh_target" "cd '${remote_repo_dir}' && bash scripts/codex-setup.sh"
# Only on this first-time bootstrap, not every run -- a node that
# already has tooling either already went through this once, or had
# it configured some other way, and re-running is harmless but
# pointless. Non-fatal: this only makes the node's own builds faster
# (substitute from nix-cache instead of building from source) and
# offloadable to it as a remote builder -- worth trying, not worth
# aborting the image build over if nix-cache happens to be down right
# now. Needs ensure_nix_profile first, same as the tooling_check_cmd
# above -- ssh's non-interactive command execution won't have picked
# up a freshly single-user-installed `nix` otherwise.
echo "==> Configuring ${node} as a nix-cache substituter/remote-builder client..."
if ! ssh "$ssh_target" "cd '${remote_repo_dir}' && . scripts/lib/nix-bootstrap.sh && ensure_nix_profile && bash scripts/proxmox/configure-nix-cache-client.sh"; then
echo "WARNING: configure-nix-cache-client.sh failed on ${node} -- continuing without it (${node} will build from source / against cache.nixos.org only)." >&2
fi
fi
}
@@ -536,6 +680,11 @@ repo_dir="$1"; target="$2"; dest_dir="$3"; dest_name="$4"; nix_extra_opts_str="$
declare -a NIX_OPTS=()
[[ -n "$nix_extra_opts_str" ]] && eval "NIX_OPTS=(${nix_extra_opts_str})"
cd "$repo_dir"
# A single-user Nix install only gets sourced into login shells; this ssh
# session is neither, so `nix` wouldn't otherwise be on PATH here even
# right after a successful install.
. scripts/lib/nix-bootstrap.sh
ensure_nix_profile
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
--no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \
".#nixosConfigurations.${target}.config.system.build.tarball" \
@@ -583,6 +732,8 @@ repo_dir="$1"; target="$2"; dest_dir="$3"; dest_name="$4"; nix_extra_opts_str="$
declare -a NIX_OPTS=()
[[ -n "$nix_extra_opts_str" ]] && eval "NIX_OPTS=(${nix_extra_opts_str})"
cd "$repo_dir"
. scripts/lib/nix-bootstrap.sh
ensure_nix_profile
nix build --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \
".#nixosConfigurations.${target}.config.system.build.diskoImagesScript" \
--out-link "result-${target}"
@@ -628,13 +779,22 @@ if [[ "$type" == "lxc" ]]; then
# 512M default otherwise (confirmed live: --memory 2048 left swap at
# 512). Default to matching whatever --memory resolved to above.
local_swap="${swap:-$memory}"
# --unprivileged 1: modules/platforms/lxc.nix sets proxmoxLXC.privileged
# = false, so the NixOS config inside the image assumes it's running as
# an unprivileged container (cgroup/capability/mount expectations baked
# in at boot). `pct create`'s own CLI default for this flag is
# privileged (unlike the web UI, which defaults its checkbox the other
# way) -- leaving it unset creates a privileged container running a
# NixOS config that assumes unprivileged, a real mismatch.
# --unprivileged: read back from modules/platforms/lxc.nix's own
# proxmoxLXC.privileged (via flake_target_lxc_privileged) rather than
# hardcoded, since that's no longer the same for every lxc-* target --
# lxc-docker sets it true so the container's NFS mounts work at all (the
# kernel's NFS client can't mount from inside any unprivileged
# container's user namespace, no matter what AppArmor allows -- see that
# option's own comment). The NixOS config inside the image bakes in
# cgroup/capability/mount expectations matching whichever value it was
# built with, so this must stay in sync with it -- `pct create`'s own
# CLI default for this flag is privileged (unlike the web UI, which
# defaults its checkbox the other way), so leaving it unset would create
# a privileged container running a NixOS config that assumes
# unprivileged for every target except lxc-docker, a real mismatch.
privileged_eval="$(flake_target_lxc_privileged "$repo_root" "$flake_target")"
unprivileged_flag=1
[[ "$privileged_eval" == "true" ]] && unprivileged_flag=0
#
# --features nesting=1,keyctl=1: required for a modern (v247+) systemd
# guest to actually boot unprivileged -- confirmed live: without this,
@@ -642,16 +802,32 @@ if [[ "$type" == "lxc" ]]; then
# systemd routinely uses (even plain getty units), and every getty
# crash-loops on a denied mount every ~3s (visible as garbage on the
# console) while core services like nsncd fail the same way.
create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --unprivileged 1 --features ${PROXMOX_DEFAULT_LXC_FEATURES} --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --swap ${local_swap} --net0 name=eth0,bridge=${bridge},ip=dhcp"
#
# ...,mount=nfs;nfs4: without it AppArmor blanket-denies the `nfs`/
# `rpc_pipefs` mount syscalls any NFS client share needs -- confirmed
# live on lxc-docker: `mount: /var/lib/nfs/rpc_pipefs: permission
# denied`. The value's `;` (Proxmox's own multi-fstype separator for
# this one feature, per PVE::LXC's use of PVE::ParseUtils::split_list)
# must stay single-quoted here: create_cmd is sent to `remote()`, which
# hands the whole string to `ssh` as a single command for the *remote*
# shell to parse -- unquoted, that `;` would be read as a remote
# command separator and silently truncate this into two commands.
create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --unprivileged ${unprivileged_flag} --features '${PROXMOX_DEFAULT_LXC_FEATURES}' --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --swap ${local_swap} --net0 name=eth0,bridge=${bridge},ip=dhcp"
remote "$create_cmd"
remote "pct start ${vmid}"
else
echo "==> Creating VM ${vmid} (${name})..."
# pre-enrolled-keys=0 disables OVMF's Secure Boot key pre-enrollment --
# required, or systemd-boot (unsigned) can't be trusted by the firmware.
# --agent 1: wires up the virtio-serial channel QEMU exposes to the guest.
# modules/common/configuration.nix sets services.qemuGuest.enable = true
# on every host, so the guest-side qemu-ga daemon is already running --
# without this flag Proxmox never creates the channel it listens on, so
# `qm guest exec`/`qm agent` and the UI's IP-address display silently
# never work for any VM this script creates.
remote "qm create ${vmid} --name ${name} --memory ${memory} --cores ${cores} \
--net0 virtio,bridge=${bridge} --bios ovmf --machine q35 --scsihw virtio-scsi-pci \
--efidisk0 ${storage}:1,efitype=4m,pre-enrolled-keys=0"
--efidisk0 ${storage}:1,efitype=4m,pre-enrolled-keys=0 --agent enabled=1"
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] ssh ${ssh_target} -- qm importdisk ${vmid} ${remote_path} ${storage}"
@@ -6,7 +6,7 @@
# copy is ever lost, or to run either script from a different machine.
#
# Usage:
# scripts/backup-admin-key.sh <dest-path> [--key-file <path>] [--force] [--dry-run]
# scripts/secrets/backup-admin-key.sh <dest-path> [--key-file <path>] [--force] [--dry-run]
#
# Source key resolution matches sops/age's own default order:
# $SOPS_AGE_KEY (inline identity text) if set, else
@@ -15,11 +15,13 @@
# ${XDG_CONFIG_HOME:-$HOME/.config}/sops/age/keys.txt
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
sops_yaml="${repo_root}/.sops.yaml"
# shellcheck source=env.sh
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=../lib/sops-age.sh
source "${repo_root}/scripts/lib/sops-age.sh"
# Pin cwd for the same reason rotate-admin-key.sh does: age/sops calls
# below should never depend on wherever the caller's shell happened to be.
@@ -43,7 +45,7 @@ EOF
dry_run=0
force=0
key_file="${SOPS_AGE_KEY_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/sops/age/keys.txt}"
key_file="$DEFAULT_SOPS_AGE_KEY_FILE"
args=()
while [[ $# -gt 0 ]]; do
@@ -103,13 +105,13 @@ scratch="$(mktemp)"
trap 'rm -f "$scratch"' EXIT
( umask 077; printf '%s\n' "$src_content" > "$scratch" )
src_pub="$(nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '$scratch'")" || {
src_pub="$(age_pubkey_from_identity_file "$scratch")" || {
echo "ERROR: source doesn't look like a valid age identity (age-keygen -y failed)." >&2
exit 1
}
echo " public key: ${src_pub}"
current_admin_pub="$(grep -E '^ - &admin age1' "$sops_yaml" 2>/dev/null | awk '{print $NF}' || true)"
current_admin_pub="$(sops_yaml_admin_pubkey "$sops_yaml")"
if [[ -n "$current_admin_pub" && "$current_admin_pub" != "$src_pub" ]]; then
echo "NOTE: this key does not match .sops.yaml's current &admin entry (${current_admin_pub})."
echo " Backing it up anyway -- this script doesn't require it to be the admin key."
@@ -131,7 +133,7 @@ fi
mkdir -p "$(dirname "$dest")"
install -m 600 "$scratch" "$dest"
dest_pub="$(nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '$dest'")"
dest_pub="$(age_pubkey_from_identity_file "$dest")"
if [[ "$dest_pub" != "$src_pub" ]]; then
echo "ERROR: ${dest} was written but its public key doesn't match the source -- investigate before relying on this backup." >&2
exit 1
@@ -144,5 +146,5 @@ Done. Backed up to: ${dest}
This is a private key -- store it somewhere offline/secure, not in this
repo or anywhere it'd get committed. Restore it with:
scripts/rotate-admin-key.sh ${dest}
scripts/secrets/rotate-admin-key.sh ${dest}
EOF
@@ -2,7 +2,7 @@
# Generates a new machine's SSH host key by an arbitrary name, before it
# necessarily has a flake target yet -- prints the .sops.yaml snippet to
# add by hand. For any host that already has a flake target,
# scripts/sync-host-keys.sh <target> does this same job plus the
# scripts/secrets/sync-host-keys.sh <target> does this same job plus the
# .sops.yaml/key_groups registration and re-encryption automatically; use
# this script only to pre-generate a key ahead of adding the flake target
# itself.
@@ -22,13 +22,13 @@
# new machine.
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
# shellcheck source=env.sh
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=lib/ssh-host-keys.sh
# shellcheck source=../lib/ssh-host-keys.sh
source "${repo_root}/scripts/lib/ssh-host-keys.sh"
hostname="${1:?usage: scripts/prepare-host-key.sh <hostname>}"
hostname="${1:?usage: scripts/secrets/prepare-host-key.sh <hostname>}"
sops_yaml="${repo_root}/.sops.yaml"
if [[ ! -f "$sops_yaml" ]]; then
@@ -10,7 +10,7 @@
# sync-host-keys.sh print when they bootstrap a brand-new, not-yet-trusted
# age key on a machine that's never had admin access before:
#
# scripts/rotate-admin-key.sh /path/to/backed-up/admin/keys.txt
# scripts/secrets/rotate-admin-key.sh /path/to/backed-up/admin/keys.txt
#
# The backup key's *public* key must match .sops.yaml's current &admin
# entry -- this script verifies that by deriving it, it doesn't just trust
@@ -19,11 +19,13 @@
# the common case is just pointing this at the restored backup.
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
sops_yaml="${repo_root}/.sops.yaml"
# shellcheck source=env.sh
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=../lib/sops-age.sh
source "${repo_root}/scripts/lib/sops-age.sh"
# sops resolves .sops.yaml by walking up from the process's cwd, not from
# the target file's own path -- if this script were invoked from somewhere
@@ -53,7 +55,7 @@ EOF
}
dry_run=0
new_key_file="${SOPS_AGE_KEY_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/sops/age/keys.txt}"
new_key_file="$DEFAULT_SOPS_AGE_KEY_FILE"
args=()
while [[ $# -gt 0 ]]; do
@@ -93,13 +95,9 @@ backup_key="${args[0]}"
nix_extra_opts
age_pub() {
nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '$1'"
}
echo "==> Deriving public keys..."
old_pub="$(age_pub "$backup_key")"
new_pub="$(age_pub "$new_key_file")"
old_pub="$(age_pubkey_from_identity_file "$backup_key")"
new_pub="$(age_pubkey_from_identity_file "$new_key_file")"
echo " backup (old admin) key: ${old_pub}"
echo " new admin key: ${new_pub}"
@@ -108,12 +106,11 @@ if [[ "$old_pub" == "$new_pub" ]]; then
exit 1
fi
current_admin_line="$(grep -E '^ - &admin age1' "$sops_yaml" || true)"
if [[ -z "$current_admin_line" ]]; then
current_admin_pub="$(sops_yaml_admin_pubkey "$sops_yaml")"
if [[ -z "$current_admin_pub" ]]; then
echo "ERROR: couldn't find a '&admin age1...' line in ${sops_yaml}." >&2
exit 1
fi
current_admin_pub="$(awk '{print $NF}' <<<"$current_admin_line")"
if [[ "$current_admin_pub" != "$old_pub" ]]; then
echo "ERROR: ${backup_key} doesn't match the current &admin key in .sops.yaml." >&2
@@ -129,9 +126,17 @@ if [[ "${#secrets_files[@]}" -eq 0 ]]; then
exit 1
fi
# sops_can_decrypt <key-file> <secrets-file>: used both to confirm the
# backup key still works before touching anything, and again after
# rotation to confirm the new key does too.
sops_can_decrypt() {
local key_file="$1" secrets_file="$2"
SOPS_AGE_KEY_FILE="$key_file" nix-shell "${NIX_OPTS[@]}" -p sops --run \
"sops -d '${secrets_file}'" >/dev/null
}
echo "==> Confirming the backup key can actually decrypt..."
if ! SOPS_AGE_KEY_FILE="$backup_key" nix-shell "${NIX_OPTS[@]}" -p sops --run \
"sops -d '${secrets_files[0]}'" >/dev/null; then
if ! sops_can_decrypt "$backup_key" "${secrets_files[0]}"; then
echo "ERROR: backup key failed to decrypt $(basename "${secrets_files[0]}") -- aborting." >&2
exit 1
fi
@@ -162,14 +167,12 @@ echo " Updated."
echo "==> Re-encrypting secrets/*.yaml for the new recipient set..."
for f in "${secrets_files[@]}"; do
echo "==> $(basename "$f")"
SOPS_AGE_KEY_FILE="$backup_key" nix-shell "${NIX_OPTS[@]}" -p sops --run \
"sops updatekeys --yes '${f}'"
sops_updatekeys "$f" "$backup_key"
done
echo "==> Verifying the new key can decrypt everything..."
for f in "${secrets_files[@]}"; do
if ! SOPS_AGE_KEY_FILE="$new_key_file" nix-shell "${NIX_OPTS[@]}" -p sops --run \
"sops -d '${f}'" >/dev/null; then
if ! sops_can_decrypt "$new_key_file" "$f"; then
echo "ERROR: new key failed to decrypt $(basename "$f") after rotation -- investigate before committing." >&2
exit 1
fi
@@ -24,17 +24,21 @@
# ever touches keys it itself manages.
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
sops_yaml="${repo_root}/.sops.yaml"
keydir="${repo_root}/host-keys"
editor="${repo_root}/scripts/lib/sync-host-keys-edit-sops.py"
# shellcheck source=env.sh
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=lib/nix-eval.sh
# shellcheck source=../lib/nix-eval.sh
source "${repo_root}/scripts/lib/nix-eval.sh"
# shellcheck source=lib/ssh-host-keys.sh
# shellcheck source=../lib/ssh-host-keys.sh
source "${repo_root}/scripts/lib/ssh-host-keys.sh"
# shellcheck source=../lib/sops-age.sh
source "${repo_root}/scripts/lib/sops-age.sh"
# shellcheck source=../lib/confirm.sh
source "${repo_root}/scripts/lib/confirm.sh"
mkdir -p "$keydir"
@@ -78,7 +82,7 @@ ensure_admin_decrypt_key() {
return
fi
local key_file="${SOPS_AGE_KEY_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/sops/age/keys.txt}"
local key_file="$DEFAULT_SOPS_AGE_KEY_FILE"
if [[ -s "$key_file" ]]; then
echo "Found existing sops age key at ${key_file}."
@@ -97,7 +101,7 @@ ensure_admin_decrypt_key() {
mkdir -p "$(dirname "$key_file")"
nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -o '${key_file}'" 2>&1 | grep -v "^Public key:" || true
local new_pub
new_pub="$(nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '${key_file}'")"
new_pub="$(age_pubkey_from_identity_file "$key_file")"
cat <<EOF
@@ -239,7 +243,7 @@ apply_edit_plan() {
while IFS= read -r basename; do
[[ -z "$basename" ]] && continue
echo "==> secrets/${basename}"
nix-shell "${NIX_OPTS[@]}" -p sops --run "sops updatekeys --yes '${repo_root}/secrets/${basename}'"
sops_updatekeys "${repo_root}/secrets/${basename}"
done <<<"$changed"
fi
fi
@@ -359,8 +363,7 @@ cmd_regenerate_all() {
echo "image/tarball before it can decrypt secrets again."
if [[ "$dry_run" -ne 1 ]]; then
read -rp "Type REGENERATE to confirm: " confirm
if [[ "$confirm" != "REGENERATE" ]]; then
if ! confirm_typed "REGENERATE" "Type REGENERATE to confirm: "; then
echo "Cancelled."
return
fi
+110
View File
@@ -0,0 +1,110 @@
#!/usr/bin/env bash
# Detects and fixes drift between the ed25519 SSH host key nix-cache is
# actually serving right now and vars.nixCacheHostKey (variables.nix) --
# the value modules/nix-cache/remote-builder-client.nix bakes into every
# client's declarative programs.ssh.knownHosts, and
# scripts/proxmox/configure-nix-cache-client.sh hardcodes as its own
# default for non-NixOS clients.
#
# This value has no automatic source of truth: nix-cache's host key is
# generated once (first boot / container recreate) and never touches this
# repo again unless someone remembers to update it by hand afterwards. It
# drifted silently once already -- confirmed live: variables.nix recorded
# a key that no longer matched what nix-cache actually presented, which
# would fail every real client's SSH host-key verification for
# distributed builds without ever producing an obvious error pointing
# back here (a client just sees "Host key verification failed" against
# *some* key, with no hint that the trusted value itself was stale).
#
# codex-maintenance.sh runs this in --check mode on every invocation so
# that drift surfaces as a warning instead of a future debugging session.
#
# Usage:
# scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run] [--host <name>]
#
# --check Only report drift (exit 1 if found, 2 if nix-cache is
# unreachable); never writes. For CI/maintenance use.
# --dry-run Show what would change; never writes.
# --host Override the hostname to scan (default: variables.nix's
# nixCacheHost / env.sh's NIX_CACHE_HOST).
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
variables_nix="${repo_root}/variables.nix"
client_script="${repo_root}/scripts/proxmox/configure-nix-cache-client.sh"
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
check_only=0
dry_run=0
host="${NIX_CACHE_HOST}"
while [[ $# -gt 0 ]]; do
case "$1" in
--check) check_only=1; shift ;;
--dry-run) dry_run=1; shift ;;
--host)
host="${2:?--host requires a hostname}"
shift 2
;;
-h|--help)
sed -n '2,23p' "$0"
exit 0
;;
*)
echo "ERROR: unknown argument: $1" >&2
exit 1
;;
esac
done
current_value="$(grep -oE 'nixCacheHostKey = "[^"]+"' "$variables_nix" | sed -E 's/nixCacheHostKey = "(.*)"/\1/')"
if [[ -z "$current_value" ]]; then
echo "ERROR: couldn't find nixCacheHostKey in $variables_nix" >&2
exit 1
fi
current_type_blob="$(awk '{print $1, $2}' <<<"$current_value")"
current_label="$(awk '{print $3}' <<<"$current_value")"
echo "Scanning ${host} for its current ed25519 SSH host key..."
nix_extra_opts
scanned="$(nix-shell "${NIX_OPTS[@]}" -p openssh --run "ssh-keyscan -t ed25519 -T 5 '${host}'" 2>/dev/null | grep -v '^#' | head -1 || true)"
if [[ -z "$scanned" ]]; then
echo "ERROR: couldn't reach ${host} (or got no ed25519 host key back) via ssh-keyscan." >&2
exit 2
fi
scanned_type_blob="$(awk '{print $2, $3}' <<<"$scanned")"
if [[ "$current_type_blob" == "$scanned_type_blob" ]]; then
echo "Up to date: ${host}'s host key matches variables.nix's nixCacheHostKey."
exit 0
fi
echo "DRIFT DETECTED:"
echo " variables.nix has: $current_type_blob"
echo " ${host} is now: $scanned_type_blob"
if [[ "$check_only" -eq 1 ]]; then
echo
echo "Run 'scripts/secrets/sync-nix-cache-host-key.sh' (no flags) to fix." >&2
exit 1
fi
new_value="${scanned_type_blob} ${current_label}"
if [[ "$dry_run" -eq 1 ]]; then
echo "(--dry-run: would update variables.nix and ${client_script##*/} to:)"
echo " $new_value"
exit 0
fi
sed -i "s|nixCacheHostKey = \"[^\"]*\"|nixCacheHostKey = \"${new_value}\"|" "$variables_nix"
sed -i "s|NIX_CACHE_HOST_KEY:=[^}]*}|NIX_CACHE_HOST_KEY:=${new_value}}|" "$client_script"
echo "Updated variables.nix and ${client_script##*/} to:"
echo " $new_value"
echo
echo "This only takes effect on already-deployed NixOS clients after their"
echo "next rebuild (programs.ssh.knownHosts is declarative). Review with"
echo "'git diff', then run 'bash scripts/codex-maintenance.sh' before committing."
+82 -46
View File
@@ -5,85 +5,121 @@ sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaMUpxYmUzaVY4d1pRY3g3
empaVUJSN0JaSlNQQlJtZENwc0VMYVhJSkFvCmJQMkpSd3RoYzlKQjBSZWErQzR2
aG5SVndOQWpSRTBDSWJVQkg0c0hiNFUKLS0tIHhiaEdpY3gwZkpCcHl2TW5CNThn
aVVUQy9Qd0trb0RNdUpVTXdrdGlrTmcK1uphQAyDV+Gk5+K1YOqw1Z8ynGP5sAPF
q5icujja/SGexX18hPYXbkyUtOrBYjW62gCuGJinSBPROoFUJbiP7g==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQUFhWVFVlVlBnNE5FTnMz
VkxkTmxpRXlzZ3pSNTVZWFUrSllsYWo0alRnCkJSc25TYktSTFFJdkQydHcxOUlj
ajhQU1ZIb1lodEpHTnVhQjJ6WEthaDQKLS0tIDJCY1E2UVBaU3BoMzhXUXlIdnMv
djZTcE1rcWNTOXFPMmFDYTVoRGo4ZTQKYy8g6pqP3VpTKDIBPbnC8NzCdDvOCKnL
14kSrKmKlzefTrbkVyriz2Jdl2s0F374yfQQFreZ3m4AffSACCxziQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKOWZxcFlhRHhKelhDS29M
NGRlTEhJUGUzcFV6QVFCdTNkdEN3MlljWnl3CnI3LzVxSHBwSi9TYlRUcFdyN2py
cVZBV0Q3Z3FhRXA0T1NFQzd2R00yeEUKLS0tIHkxL2dHV3RkVFYxZTJNTVZvVFIr
TXpJZzdnYlpJaXBmcjdWWUtxNkc4dWMKVsImJiavzUzSFn78pciNJPHaS3KWqJer
VkxF6kF3tl2HmW46eVXtsQowdu+zVR9HS35i/HvQ1r1TyP6qT5ofcQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBVQnpVWC9wcnIreG9GeE9U
UWhuRytkc2Flc0hyQm5yMjZnelNwaXhlWWc4CjBkWnd4cHNRRXQ0UXFkZGp4QlR1
eW5NNnE1WFhnb054M1pac2ZidFg4Y3MKLS0tIFdTNmk2V1l2WC9rUk8yd0ZnOEJS
VkNnejVGVUZPZkorQkltVEplN2FmdTAKRY7DPP5HeFQntn2f/fXLjU6M1V6iug86
BD09PI+T2DbIBQPotRZisw8IzHu9gY/O3+h0TccyIsXjI9wy/XPCAQ==
-----END AGE ENCRYPTED FILE-----
recipient: age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyWjdSanRRaEI4VHFZVkdw
NkxMODRuakFJTFNoTHZIRHptZFNxaGZ3K3lnCjJGR1FJalRsYmtLcHFxZW8yNU5W
MzBjWmltbzI3MTByUjB6djlEdllHNXMKLS0tIEhEOVhLVjZkRi9vUEtDWUhxT0NX
eUZ3bGdEaHdGbXZIYkowYUZuTWJ3d2MKMwgxOqlMH7GfLlseD4J277Dcg0KCD3d5
jwJDb82kRWoBMicTJZXoq/5oe5blJNa7dWYoqgkYxMA2O+0igT+I6g==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkczFSTVhxWHlIWjhRcWlV
V2JPQXd5Wnk5R3NwWC81T3Z0MW4vYnd5S1ZZClV1NlU1Tzd6UkxPQ2M4MmhLV01G
d3VIb0RhR1RiNTZqNjlQcmg2YjdPeGsKLS0tIDQ1RTFTWGN4MnEvWkRUR3VnN204
WVdFOXdmNC9FVFhBSGNEUUgyYWpYYzAKfdpeaFL/RrIbqpD9hNj8L7UxpmiBjE2I
go/dR2E1LLXsDPtnSuJb2EZYoFvSsjsIQQQDt+YwRv0fplRtssKdxQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsZzdRMm9zdlJHckFoREVs
UkVXL3dCeWRuN2NqcnVsOWVNT1FLUHY3andrCmNFY0ZCby9SdnUyYlV6R05YNDB0
em43SjRCNVhYbmpRZUVWT0cxQlhGdmsKLS0tIDZHazNyck5VNHBuNVM5bmRZUlpR
QzBSNy84VDdLVkZZbnNlUFYydXlreEEKYZaR2b7tyRAhPdP+ytpP0veUTi7pY9Nw
pK0h4hcegLNYJL1AfOYwFQoW7vb256GdmwdcuuBl1YBGXWGraaBnZg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB6TjhmOTJ0bUpMQWowb0hB
T3Era1loU1pMdmkxdnAvRkViekpqZWZjaWgwCkFRQXhFUy9PRVBma2JMUDhqY2F1
VVFDRFNVbWpNaEczY1JVQUMyck9XdEkKLS0tIGpxc0tGdVFKK3FteVJKM1Fxa2ky
a21WLy9qV05hUURCTVBvcVh3cE45Z3cKXCYfXSjhApBoLbHDu2OOd57Y1zN54yy+
WDQvz8PpMxhc1nU5Kw/cI+WmL1KvN0qQZfOx/7D4W+dy/ZDWX27TpA==
-----END AGE ENCRYPTED FILE-----
recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFSnZiOGpsL1BESWVEdUxx
dXc5LzBHTlNodGJ1S3hOeHd5Wmgxb3BPSHl3CnBEWDNTc3NDdzN3RHQxUlNLOEs5
RU1SS2tVT29XbDJCWVliWnVkekN4L1kKLS0tIGFqY2pNK2h2S2ZWMndrRVN6eUdN
M0NLY2EvTTVIYlZvdk9XY0NwNE14ZlEKcOwgCK3g56kId/4tEt+2iCcylisn+Fca
5VWamsXdbwxbxmCEEUbgN3aHrdVz3mV4+8FZqA34yXz8pc45/PUcFg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZK0FYQnBHMHZ6dlpMYTlC
WFdOWDFkRVBuY1pmdTFiUndLV3JXcndZa3pNCnJsd0tHN0FveWV6UUNQSEdpdWw5
dWZITkxWelNIRlpKS1pnN0ZmVlQvZjAKLS0tIEUwMXdtNFdkUWdIRjlxc0owdTRr
c1o1TmptWWd1ZGxzcWJJNzJ0K25PTTAKoos5rnkyQBCm+ZuhCCaMJwqJBo1fpnsl
G74wu5vbTBG4VjVhI5KqyiuiTRU4jPcGxysECqe7AyZUBGp7ndewgw==
-----END AGE ENCRYPTED FILE-----
recipient: age1qz9d4ka4xgexujyd247s7lp737sulp5fhxl5d65fj2ykvc4j4edqrsdks8
recipient: age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJU1cydFFuRnpCUU5mbjFu
TTIxREkxZ1hMalVtNjNjcHVaUXpPNWdib2tzCkRScWRTWHhxN3lhckI4UFpMSFRa
RGxWcldlWnd5SVJFNWxQWjB3R0pMb3MKLS0tIGdjLzlhTHgyT2hjTmRTK1RiZmxS
OWRWYklBWkoyUG8raFNzd3JlRUtEUEUKw+NdAp1Mz0dEOUGPbqCV8y7029I3Kye0
keU2T29JGCN1D6x//1NcMUYSaFZKgv0ZVSjVUCl1EnmlJ0nBdbaDXA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTVkVXaCtGTXVkRXRZMFZ1
ZUdXM1hRcDhaSURqcGh2eksxTUNCckk5SVJjCnhkSDRKdDFSckxUWXd4SmVxWG5p
bU5OV0hzaWR6VDFwcDY2WlY4WnN3cFEKLS0tIElOVzRCcXR4U0dhajJySUhaZGps
MW9rQk1JVDFWRnFxVzhCUkRIS09EamsK1rVidD48PqwlEWQyjF7iQWU7aBdPqQHy
z5LaSi3LvJX3rNE/+q0E8/gbZyjGpbEn3AUI5mBF64GY3IZkRxZSXQ==
-----END AGE ENCRYPTED FILE-----
recipient: age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnNFlzYVBqaGFDVE9xbmt5
OVd4ZlZRdEVJd0JFdkRIYVA5ZUEwYjJjWVJjCnZzTExNbDlYbGN6c1dYU2ZlNEVs
ZUo4MXdpc0tzbDI0MHl5eXc3d09VRDQKLS0tIGs4SGMya1RZQ0hpa2NtQk9Wa29C
bmxIYWk1VThsaFhxRDlOOGhGQ2tYamsKs+PcOiaeNFujCWwZBr+nq1MzrGohl+ch
TASI7eNsiHmSVRzSMHv4mX+8yXDHIRgHbFuUty2gdgGFLRjYXe4gDA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvbnJCWU9UMXJXMG92b3hm
WWNBTFpQamVWQThITmt5QUVwR3h2OHI2SlVRCjcxOEJTVkFjN0NhamFZQ1plK29w
dk5XYkYxOXQ0YkVzcVc3VnhCQWlsV0UKLS0tIHZ5cWtFZUhDKzZkOE1BK2Y5TStR
SFlDRjE4ZHpiVEJOQk5TUGNEN1B4amMKUCJ8CL8QpmRpFs83HD9TUn7NrPguuP8S
JQH/bzPorXTXJuyOKuKAZq1hK8BmiMUFksaZ03yN6YaFVIOeelEEMg==
-----END AGE ENCRYPTED FILE-----
recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkcmVQYXV1NzRrQ2IxTElt
QUJRMkhiUUJqRDdwQW03K0lCTEtkSTU5UDJrCjh6SldpQkF1NzIybTJTdnBlOHdQ
VlREMy9hM0ZaL21Pa0VPUzcvb0pQQjQKLS0tIFcrK0I3Z3Nhem45eHpZYmdWeit6
eXFtRHVxODhaNTVDdVR1a2tOb1N1U1EKZBdLHq8PRMKX2ndFr3AxFVAZRyvhFa1u
72R5tordo4IR3HSxG3Z06rokOITd+KKhaQ8NEWEC8qioAAMxEC9QWw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxWHV0S2dxeHZEMDNRSkhO
ZFJHaiszd0tzYzlzd3gvbXNSQmlMVlJUNGcwCnVrM29MdFZCR1NBYnpkQ1k5VFZQ
b3Z2Q3ZGekVQZkZKWGlka3NDOHJ0R1EKLS0tIDlXTmNzUk0wVXo0UWhkd0ZvK3FI
UzJxU3RkdWs4aTZYVVkrS056bTN1ek0KgKJNz8GvynX5pK33aW9x3v6yr2Ox0LCT
GGrt+ddbKLcwpBpYjfWkFhffO330EKui73S+c/qMf8N9j6wzalOTpQ==
-----END AGE ENCRYPTED FILE-----
recipient: age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0WjJPbXhXNTNyQUZzMmxS
Rkd3Yk5Ga3pPRXFtMFpXcmpBZ3RJYnJlaHhjClRDbFBHRVZ0SDdrNFdVWmUwalpT
WUt5a2RUTnQ4VFBGMlowR3JycDM3aXMKLS0tIERSeFd4cGdSbzRpSmpSQnB1Yng5
OHBma3lYeWEwQ2RvelpzZHRkN1JPWFEKLXkJfEkH3lo5Z4mj8PZXTfAfZn6B46To
U0G54tUh0U1EeMnI5ZPRGwDxi5K0eD68HjeE9MBvFBysOChP/ANDPQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3ZWJlbDl1TEkvWEpBMC9Y
ejY1MnFMUExBeDhITndxNy9YUi9hU0tXM1JJCjcwNEppcDdxYzlCSVMrMExWa3A4
cmFPSjV3LzkyMXZCUDU2QmtHRmpHRmsKLS0tIFFTdXBOaDJJTERseXlGbmdrQzhD
TWtnRFdIRXpsNkY0U1BiczNsdUk1V1kKGpndKmT8kj/oIxQuxQALfzscw+CsVmnj
cyPC3bF+tG6LcqqoKLjPSJfcIgzhnX7cAr/wwESavemLn8L/zQMe4w==
-----END AGE ENCRYPTED FILE-----
recipient: age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpQVVyY3UrSHFWSmpDdmRE
dlh5akFqVFdDVFNMSHE1eVJnZzR6YzFHSVFRCko4UE9EdXNxZzF2MW5PTTN6dEdU
ZHM1MGowcVB2Y1ZlOTVHdnNtY3diM2cKLS0tIDBYSmh5dVVPaTM3d0ErcC8wMDNB
eUpHWnZlYnJsbHZuS3pwbG15UGtwN2MKVPQA1MpjIfYAsNacoAbpvZNuAIkvx7ER
CvWBKEHUVm6m8905BXzv8MdGTAk0EyCIP3aMmYqTIYfv2k9pP0T08A==
-----END AGE ENCRYPTED FILE-----
recipient: age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0MTQ4MzN0bDJNL1l1bmpX
ekxDUXRNa0JHWWltZFNGTVltTFdSSE82SVZvCmdUWUdja3JIajMzY09IMUE5elox
MDdEakFJTmtkRWF2R1BGNkQ4U3grNWsKLS0tIE9hZUhkVGI1ZEpzdDhRU21EZm91
VnJNb1kyQ05MM0RJa1lLUEtjWWxkSTAKHVAKcGcWl6LncJALRBU9RKP7ot6C6GSE
1iZtj1SNX6wzEWrhOEnV37aQ8bKZj6u+Y/q6/vJ4qiBs78y/drdIzA==
-----END AGE ENCRYPTED FILE-----
recipient: age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBNUEVWY2NpVEU4OTZPZTZR
UzdKcktpUGJOcnJ0aEhkQlhvUWdUYkV2SkdzCm51eGJVeHJMcVRRRld0dFRCYUxr
TTN2WEhOVjRqV0FtQXowZWNTbkJneEUKLS0tIFpUazZpTUNWZUZBSFE0VDZZbkJu
SFVlUVhySnNqUENYOG9qUm5ZMDc1ZW8Kv0lY5dhnCEheM0sttfr4p7IL+EVog16T
OapUdbuXL2l7t7URzHnvfG/nbOtJIjH8a0XFsWyJChtNXpF2d/vf2g==
-----END AGE ENCRYPTED FILE-----
recipient: age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvTzc2aExGTHFrc0ZtL3pq
YlZLbzd3MVZHZTB0VUQvSXZ4NGVsZk42c2hBCk1vbTk0Tnl3b01vbVZaMkJ5aldE
WmZBMGFjb2pjQXpYcnBxWmp0UUsrdXcKLS0tIG16SG9JbFdkbmVidCsxUnpBR3V3
RzNOY3hIRWk4UXh6N3NrcjNSU3ZwWTgKaExY4U2s8E6ojljJ+4TU+YJhcLXyuVA1
ROB70jQCjFvQOeo6thjQohSSUoPKhxSl1/nr4ZiGBO3/VskzihckKg==
-----END AGE ENCRYPTED FILE-----
recipient: age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
lastmodified: "2026-07-19T02:30:40Z"
mac: ENC[AES256_GCM,data:UiL3VMDF6rq4Nr87KspcDx434q3tfNXeb5pwH2O+4ssNQ6xzcYDdzXBnhAY3zLBsqPMKrvHBd4Ot/gEMcq3FMIVe7Q6p9yWKpep66KZ/yWEhAlwIVhD79Oj8VS+1CHKjf25zpRdhZorp04oeFQQd9VfjJB4EE/Q1aVbwTGlpIic=,iv:i/0conaFgFia+wzNTdUL6tlSTw35HTK3Ap1Sr5RGHf8=,tag:ULbz5FllShA/JjlSRdxA0g==,type:str]
unencrypted_suffix: _unencrypted
+16 -16
View File
@@ -4,31 +4,31 @@ sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmOHlPcGZvN3o5aEZCcFdz
WlYydFdoQ0ZHbElMdTRkZWljMmoxTHZqNjJZCnE3cmgzKzZya3FROEczbHVveDND
VnEzRW12cnRKdzhkZm5uTXpkTEtrNUEKLS0tIHFDbkc4Mk4wVlM2R29zZXYwR2Ri
ODBML1p4eUZiZldYUERQTUhTU1llV0UKxjvH6zbW6wKghzR1o34CyKPEa2FqZmo0
PxgqyuXkIwas9soXVAkScx7ElaV09Fjaj+mDrKwi4a+DwdoSP7czyA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBPWE1HTUhiSUp5ZEUwWEpI
bGpkZlBIMUo5ZlYrQ09SN3Q1a0ZkQ0ZnOEhVCnJPNEZQenVWWGZiODlzQzNEc1Zq
c3l4OWZJTElJc2Y2UE15OGtEUzhyY1EKLS0tIHNJUStyWnlQWjZBbEZjQ3UwdUpz
ZndoUDR6bisrNGJCUHk3TGI4bTZaMFUK87fFsm9ne9s+PK2pcwtrDjqyGBss2r2E
8lhqoeiKZ2j96z8kP/7ChzovwTCmqdcmAQuyNQD+ZAFijseipSvfbQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0VDBXYUZNbnpEZHlqV05C
WWxQZjVYcWlZeXlGc0RiaUpERzR0T21CWWtBCkM0dW5kYlFNK1RGSzRudFZ2Z0t3
aVBreElGM1BIT0RzQkxTVTA3S2NhQncKLS0tIERyTU8zWWRTRm95SmRZQ1BhalVV
SVdBajN1V1BuM2s4K216S3c0VHczNlkKM5jvsSEfCBA5uZRjBJNbM91lLRQkj+jK
rM5uSfGLTvSjPgXIMIq03OXxH1CE7GoKxAPwFrJdAFMMQcutIethhw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUYi9SRFFGV3Z6cFd2Znk5
b2FLbWtzTllJMDBUaGk0NTViOTNBa2hQclVZClZHKzNhbGVjQUJhWkFWdTFBMG5a
cUFJdUdyVG5HQXJRRnJId3hqRTN2cXMKLS0tIEFMRjh3WE1ON0U2TTNTZ3hxMTR4
ZGRlemlIbDZKeExmVHROc3Eyak5DdzQKaLwIVDi6BN4cxpVxJoqTYvJETPOp4thc
l9uVMvIGuEsEZgDsvShw1dYLljd+uGy/A+dXbcxIUCP/mmPkwmd1Pw==
-----END AGE ENCRYPTED FILE-----
recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvbk9tRE9jYkpxNThQeElH
dWpKYThDb1ZkUWtMWlRxc2tYbjhBbEFEcWtVCkNiK0NBRko2TkJQdUVtdGtHNGcv
andSaUlKSFA4THRyZXNTYmI0Yk5WbmsKLS0tIHlQTUtpWDdPeTVZL0M1RElRdFNk
QWlGdFo5NkZWSmY0YXdpNzNUQnlsK3MKtzC0bM7Ek+K73nMranOA1Mc98RUnYnq1
hAt0QEFKWK4QVKubaN/rG3AzE0U7qPKWHTzoxgnAiL3WyV9teLW+iA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArOWovSW9DeFpxL0VDUDQ3
SHUwTzJVZUtPV01ZRkdCUXZGL2lTRCtCNFNnCjBSNExqRW5mTEN5SFVucHJHSzZt
cDlNc3BjY3M1c1k1Z2tkVEg4R1pacGsKLS0tIEFWbHNKZW0vbVh1Y2VhQW93OUwx
MWV0eW9sOXdQd0l2ZjlWOEVVc1dwcTgK2s4p9xoNkawH2OkGsl80bNIo3ad5vn4W
Z2w+jwppSoUmbQnD3WFbLmSSxmuobmU8HILwElv6SZu+KE3aspF6XA==
-----END AGE ENCRYPTED FILE-----
recipient: age164px2a8e48ptsf9ngtan38aa6jls4jdl26mzrgzf6sn3vcvt49hqjrgr8w
recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
lastmodified: "2026-07-19T23:30:21Z"
mac: ENC[AES256_GCM,data:kLGE2xawQT7mx+sfw68hmGk5nCEGiEjZrqTEl9B1dtQmTrMwmoVr/1RISi4LfJrwxy31mDgff4lcIL4wIJuM373uk3X8j4RNyYQNTfKEkORT6r8NHeepNs267O77pKGd7OmcM4MT/BqOnB8ELS7Wlf2ect7CAlvUUVyc8icxgZE=,iv:EYLDsHYHZ1XOQXafOTqHHWpk/OBNq/R6IJnOBYV33E4=,tag:thxrCPC5oGvDjhK7Dz87YA==,type:str]
unencrypted_suffix: _unencrypted
+28 -10
View File
@@ -3,22 +3,40 @@ sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBURmMzN3hrSlNrUkkvVWNl
L3M1dEhWeW14N0RFNVRPci9QK1YyTFdqRVVJCk5WaWswT2NicldkYzZjbVhYU2xu
MGFsNmUzeTN2TS9wOEdvRURpVUVYZXMKLS0tIDZ6MEdPTVhCaTQ2UXFWTUFtc0pm
MFlJb0c2WXJtMGRLZEZYY0pZWWpFWm8K/mlYZIe8UC0QU+1mq3NtrtTF5b2m5hCK
+K0QiZLTKmmDcr4bRhZ32VE7R7GRwtMNnOP/mElZvPAyWyHHhRiOHg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaYU9HR0lETDhkYXR5NWlj
L0diSE5JQU9KWjRoU3hrVjdqZ2tPNUtOdFZvCjdUVGNFbDVYa3pVSC9ZWVNORER4
QWozUlhoSEtjTk9IRlM3VkZoYlc1RTAKLS0tIDlHS01WVWlOMWFEQU5GTVRLZVhV
VTcwekhrRHB3SlVYT2MzOW5GbE52dkkKKCWehPhpdGapdyzpll20NJUcZwvW/7X8
KQ1EqAgI2fewnbwuIDYCleN0b0SLJNUeSV/tFKDDoTMnHWCdeD4ECg==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBURHlUL0RNMUtNallEcG5p
eG4xby8yVzdQUTBaNkl4ano0YjBMcDd0Wm5NCkZvaXNPZm9wemJkMmNSdGdOaTI4
Z1RwUnhiRUpCMWZaeWtlSVBmNW5KOXMKLS0tIDk0R0k3ZHczTFNCWUZxSWF0M0FJ
MGlZMmtuSFYrcG1meDNMWDNqSjFxcE0KDu2dAc0gqmmPkpbpBe4YohM7rYmUwEkI
V2FUQwjlvh50svtjCVdYbx2xuq4sQLnKelk/q1onLw60FwsVfzD8sQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4YndybFBTQ2p4SGZ4SDJs
a2p0eFRQOVVWcGd2a25ESW9ESGx1RWw0Zno4CnhpVVh3cGI5UjY0YmFINFFPMTh4
b1B5SjJ3NTNvUE1QUmJjVFozY1dYS2MKLS0tIC9HZGNpOFZhZGNFZGt5blJuZXVV
SXpkRzV4d2ppV3ZQZSt1dmxYNGVFMUEKmSe9dkrmkND81Hw2/ATAmFvcmhk1tUC1
LxxBw54IVHUqwYKgRYUYRNu+pykDT5OnFDPiskd49Xso99LY87PyiA==
-----END AGE ENCRYPTED FILE-----
recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLd1FSSHVTSGdHWm8wMVQz
d0dlOEsxeGwxdHU0eUlFSUxka0ZmcVpnOTM0CndYNUVjVy92QVhNY2orQmpSQjYy
VS9KM0NUTXhuM0lCSDBZMWtISEdtWTgKLS0tIExTL2wvS3FEdVViUmRYZEFsR3R0
YTJFM200RjF6MjNxOFA4eHRpWVhtRkUKOqBIT445HnPXrrH/qV6FIuAhAuJmSL6V
+PQopM/m3PAnK5m5Mu3cfjYfDiB8+GWTABhljfT+GbcoK7CqWLehrw==
-----END AGE ENCRYPTED FILE-----
recipient: age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvVXF4a2NmaW15R2VXS3FP
SDVPUGpUWTlIWjl2N2N2SXR1UlRkdllWNVRJClZDbVo4dUhrZytqZkMrYWtpOEZx
c0dSZGViN04zQ1B1WEZEWm1QM1lsejQKLS0tIE0wa0k5Rm1xZmw3OHFESkVXc25j
NlFpYTJSckQ4MVlZQ01reDlkaWY3TkEKsmQlreRhRAjVZ/q5x52FNATDF2sLhbHo
djOZigZx2rs6shqQ6It/XRJ8CiPaXZBPOt529Gwmplu+hWlAU1+l2Q==
-----END AGE ENCRYPTED FILE-----
recipient: age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
lastmodified: "2026-07-19T02:30:40Z"
mac: ENC[AES256_GCM,data:rKHZjU/MH08ASTlu32HZO9uWmsBYuMCEC6M8gwVhzuWvmablnP05tS2z13XfaWaCEUXk6kmGJKuU0zu5+IKVZgamCF6DAMtxQb6bVCaLsoAm/GSqWQ5VI9eHqgnSSdN/o3ul/33Rf8iBQo4aw8FFAmDVuNz8bfAn0QefFTj0ByI=,iv:JD2gtqRinOY77etg6PUmZNovkYl1Q3F6ZvRi4x7RznQ=,tag:/5IMpWKRVt+l1luCTQE0BA==,type:str]
unencrypted_suffix: _unencrypted
+10 -1
View File
@@ -26,7 +26,7 @@
# fresh client that has never manually ssh'd to nix-cache before. Update
# this if nix-cache's host key is ever rotated or the host is rebuilt
# from scratch.
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache";
# Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
# host (modules/nix-cache/server.nix) — one per client host that's allowed
@@ -38,6 +38,7 @@
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGtOWOCS+ImHc7NehguoyD7PbonGosKMZqc9+QR3v/h root@nixos"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHxXTQxFnArK5HXG7czeoybZebCGfxpUdusJkPn+BCSp root@server"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEj26SL/emsVjW2YhRucJVp2kTz8WgcEQgjBEBLRikk root@claude"
];
# Admin SSH public key, authorized on the primary user of every host and
@@ -137,6 +138,14 @@
# shortcuts on the gui build type (hosts/nixos/home.nix).
pveWeb = 8006;
pbsWeb = 8007;
# Tor relay's ORPort — the port other Tor relays connect to for onion
# routing traffic (modules/tor/enable-relay.nix). Tor's own conventional
# default; opened via services.tor.openFirewall rather than
# networking.firewall.allowedTCPPorts directly, but kept here anyway so
# it's not a bare literal duplicated between the relay's settings and
# anything else that ever needs to reference it.
torRelayOrPort = 9001;
};
# .raw disk image size for every proxmox-* host's standalone Disko image