Fix nix-cache remote-builder trust: stale host key + wrong sshKey path

variables.nix's nixCacheHostKey no longer matched nix-cache's actual SSH
host key (confirmed via ssh-keyscan against the live container), so every
declaratively-configured client's programs.ssh.knownHosts trusted the
wrong key -- distributed builds would fail host-key verification. Also,
modules/nix-cache/remote-builder-client.nix hardcoded sshKey to
/root/.ssh/nixremote, but the `server` host only has its own default
/root/.ssh/id_ed25519 installed (confirmed live via qm guest-agent) --
that file was never even present, so the build machine config pointed at
nothing. Standardize on each client's own default identity, matching the
per-host-key pattern vars.remoteBuilderAuthorizedKeys already uses instead
of a shared/differently-named keypair, and add
scripts/secrets/sync-nix-cache-host-key.sh (wired into
codex-maintenance.sh's --check) so the host-key drift doesn't silently
recur next time nix-cache is rebuilt or recreated.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V7yVH71vGrDVzovh9UaMu8

CLAUDE.md

@@ -73,21 +73,71 @@ diff size; that's the point of it.
 
 ## Scripts
 
-Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
+Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` is
+organized by purpose: `scripts/secrets/` (sops/age + SSH host-key
+management), `scripts/proxmox/` (Proxmox deployment), `scripts/lib/`
+(shared helpers, sourced by the scripts below — not run directly), and a
+handful of repo-wide scripts left at the top level (`env.sh`,
+`bump-nixpkgs-release.sh`, plus `codex-setup.sh`/`codex-maintenance.sh`
+above). When adding a new script, put it in the matching subfolder rather
+than the top level, and if it duplicates logic another script already has,
+lift the shared part into `scripts/lib/` instead of copying it.
 
-- `scripts/sync-host-keys.sh` — generates/registers SSH host keys and their
+### `scripts/secrets/`
-  `.sops.yaml`/`secrets/*.yaml` recipients for flake targets, idempotently
+
-  (`--all`, `<target>`, `--remove`, `--regenerate-all-keys`, all with
+- `scripts/secrets/sync-host-keys.sh` — generates/registers SSH host keys
-  `--dry-run`). The primary tool for provisioning a new host's secrets
+  and their `.sops.yaml`/`secrets/*.yaml` recipients for flake targets,
-  access — see "Creating a new machine" in `docs/auto-installer.md`.
+  idempotently (`--all`, `<target>`, `--remove`, `--regenerate-all-keys`,
-- `scripts/prepare-host-key.sh` — narrower predecessor: generates a key by
+  all with `--dry-run`). The primary tool for provisioning a new host's
-  an arbitrary name without touching `.sops.yaml`. Still useful to
+  secrets access — see "Creating a new machine" in `docs/auto-installer.md`.
+- `scripts/secrets/prepare-host-key.sh` — narrower predecessor: generates a
+  key by an arbitrary name without touching `.sops.yaml`. Still useful to
   pre-generate a key before its flake target exists yet, since
   `sync-host-keys.sh` can only act on targets `nixosConfigurations` already
   has.
-- `scripts/create-proxmox-resource.sh` — builds a `lxc-*`/`proxmox-*`
+- `scripts/secrets/rotate-admin-key.sh <backup-admin-key> [--new-key-file
-  target's tarball/disk image and creates it on a real Proxmox node
+  <path>] [--dry-run]` — rotates `.sops.yaml`'s `&admin` age key: decrypts
-  (`pct create` against the tarball as a CT template / `qm create`+
+  with a backed-up copy of the key currently trusted as `&admin` (verified
+  by deriving its public key and comparing, not taken on faith), replaces
+  the `&admin` line with a new key already present in the environment
+  (defaults to wherever sops/age itself would look), and runs
+  `sops updatekeys` on every `secrets/*.yaml`. One-way: the old key can no
+  longer decrypt anything re-encrypted this way. This is the automation
+  for the manual steps `sync-host-keys.sh`/`create-proxmox-resource.sh`
+  print when they bootstrap a brand-new, not-yet-trusted key on a machine
+  with no prior admin access.
+- `scripts/secrets/backup-admin-key.sh <dest-path> [--key-file <path>]
+  [--force] [--dry-run]` — copies the local sops age key (source
+  resolution matches sops/age itself: `$SOPS_AGE_KEY` inline, then
+  `--key-file`, then `$SOPS_AGE_KEY_FILE`, then the XDG default) to an
+  arbitrary destination path with `0600` permissions, validating it's a
+  real age identity and round-tripping the public key before and after the
+  write. Refuses to overwrite an existing `<dest-path>` without `--force`.
+  Purely a local filesystem copy — never touches `.sops.yaml`/
+  `secrets/*.yaml` or the repo at all. The resulting file is exactly what
+  `rotate-admin-key.sh` expects as its backup-key argument.
+- `scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run]
+  [--host <name>]` — detects drift between the ed25519 SSH host key
+  nix-cache is actually serving right now (via `ssh-keyscan`) and
+  `vars.nixCacheHostKey` (`variables.nix`), the value
+  `modules/nix-cache/remote-builder-client.nix` bakes into every real
+  client's declarative `programs.ssh.knownHosts` and
+  `configure-nix-cache-client.sh` hardcodes as its own default for
+  non-NixOS clients. That value has no automatic source of truth — it's
+  set once from whatever nix-cache's host key happened to be at the time,
+  and silently goes stale if the host is ever rebuilt/recreated with a new
+  key, breaking every client's distributed-build SSH trust with no error
+  that points back here. `--check` (used by `codex-maintenance.sh`, which
+  treats an unreachable nix-cache — e.g. from a non-LAN CI runner — as a
+  silent skip rather than a failure) only reports drift; the no-flags form
+  updates both files in place. Declarative clients still need a rebuild to
+  pick up the fix.
+
+### `scripts/proxmox/`
+
+- `scripts/proxmox/create-proxmox-resource.sh` — builds a `lxc-*`/
+  `proxmox-*` target's tarball/disk image and creates it on a real Proxmox
+  node (`pct create` against the tarball as a CT template / `qm create`+
   `importdisk`), or reconfigures an existing resource's cores/memory/disk
   size (`--modify`, always requires typing the VMID back to confirm).
   Checks for an already-uploaded image on the node before building
@@ -97,7 +147,51 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
   Refuses to create a target whose host identity already exists live on
   the node (checked directly via `qm`/`pct`, not any file in this repo)
   unless `--allow-duplicate-host` is passed. `--dry-run` throughout both
-  modes.
+  modes. The first time it has to bootstrap build tooling on a node (i.e.
+  `nix` wasn't already on its `PATH`), it also runs
+  `scripts/proxmox/configure-nix-cache-client.sh` there (non-fatally — a
+  failure just falls back to building from source / `cache.nixos.org`) so
+  the node substitutes from and can offload builds to nix-cache on every
+  subsequent run, not just this one.
+- `scripts/proxmox/configure-nix-cache-client.sh [--dry-run]
+  [--no-remote-builder] [--no-restart]` — the non-NixOS equivalent of
+  `modules/nix-cache/client.nix`/`remote-builder-client.nix`, for a plain
+  Debian machine with the Nix package manager (not NixOS) already
+  installed: run as root *on that machine* to add nix-cache as a
+  substituter in `/etc/nix/nix.conf` (`https://cache.nixos.org/` kept as
+  fallback) via `extra-substituters`/`extra-trusted-public-keys` so it
+  layers on top of whatever's already there instead of clobbering it, and,
+  if `/root/.ssh/nixremote` is already present (see docs/nix-cache.md
+  "Remote builder SSH keys"), configures it as a distributed-build
+  machine too and trusts nix-cache's SSH host key in
+  `/etc/ssh/ssh_known_hosts`. Idempotent (re-running replaces its own
+  marked block rather than duplicating it); restarts `nix-daemon` by
+  default so the change takes effect immediately.
+
+### `scripts/lib/`
+
+Sourced by the scripts above, never run directly:
+
+- `nix-bootstrap.sh` — `NIX_CONFIG`/`ensure_nix_profile`, shared by
+  `codex-setup.sh`/`codex-maintenance.sh` and the remote build commands
+  `create-proxmox-resource.sh` runs over SSH.
+- `nix-eval.sh` — `NIX_EVAL_FLAGS` plus `list_flake_targets`/
+  `flake_target_hostname` flake-introspection helpers.
+- `ssh-host-keys.sh` — `generate_host_ed25519_key`/`ssh_pubkey_to_age`,
+  shared by `sync-host-keys.sh` and `prepare-host-key.sh`.
+- `sops-age.sh` — `age_pubkey_from_identity_file`/`sops_yaml_admin_pubkey`/
+  `sops_updatekeys` plus the shared sops/age default key-file resolution,
+  shared by `backup-admin-key.sh`, `rotate-admin-key.sh`, and
+  `sync-host-keys.sh`.
+- `confirm.sh` — `confirm_typed`, the "type X back to confirm" destructive-
+  action prompt shared by `create-proxmox-resource.sh` and
+  `sync-host-keys.sh`.
+- `sync-host-keys-edit-sops.py` — the `.sops.yaml` anchor/key_groups editor
+  `sync-host-keys.sh` shells out to (see that script for why: precise,
+  idempotent YAML edits are impractical in bash).
+
+### Top level
+
 - `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
   default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
   cross-script config here instead of duplicating it per-script.
@@ -106,27 +200,6 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
   reference `variables.nix` (confirmed empirically — `nix flake metadata`
   errors on it), so this is the closest equivalent to a single source of
   truth for the tracked release.
-- `scripts/rotate-admin-key.sh <backup-admin-key> [--new-key-file <path>]
-  [--dry-run]` — rotates `.sops.yaml`'s `&admin` age key: decrypts with a
-  backed-up copy of the key currently trusted as `&admin` (verified by
-  deriving its public key and comparing, not taken on faith), replaces the
-  `&admin` line with a new key already present in the environment
-  (defaults to wherever sops/age itself would look), and runs
-  `sops updatekeys` on every `secrets/*.yaml`. One-way: the old key can no
-  longer decrypt anything re-encrypted this way. This is the automation
-  for the manual steps `sync-host-keys.sh`/`create-proxmox-resource.sh`
-  print when they bootstrap a brand-new, not-yet-trusted key on a machine
-  with no prior admin access.
-- `scripts/backup-admin-key.sh <dest-path> [--key-file <path>] [--force]
-  [--dry-run]` — copies the local sops age key (source resolution matches
-  sops/age itself: `$SOPS_AGE_KEY` inline, then `--key-file`, then
-  `$SOPS_AGE_KEY_FILE`, then the XDG default) to an arbitrary destination
-  path with `0600` permissions, validating it's a real age identity and
-  round-tripping the public key before and after the write. Refuses to
-  overwrite an existing `<dest-path>` without `--force`. Purely a local
-  filesystem copy — never touches `.sops.yaml`/`secrets/*.yaml` or the
-  repo at all. The resulting file is exactly what `rotate-admin-key.sh`
-  expects as its backup-key argument.
 
 `sync-host-keys.sh`, `create-proxmox-resource.sh`, and
 `rotate-admin-key.sh` genuinely mutate real state when run for real (not

README.md

@@ -34,7 +34,7 @@ anywhere in this repo — that's live infrastructure state, not something a
 committed file can keep accurate, and it changes independently of the code.
 Check the Proxmox node itself, or `/etc/flake-target` on a running host (see
 below), if you need to know what's really out there right now.
-`scripts/create-proxmox-resource.sh`'s duplicate-host guard works the same
+`scripts/proxmox/create-proxmox-resource.sh`'s duplicate-host guard works the same
 way: it checks the Proxmox node directly rather than any file here.
 
 Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
@@ -114,7 +114,7 @@ Three different paths depending on target, none of them involving a manual
   disk image and attached to a new VM with no install step — see
   `docs/proxmox-images.md`.
 
-`scripts/create-proxmox-resource.sh --type lxc|vm --host <name>` automates
+`scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --host <name>` automates
 either of the last two end to end (host-key registration, building the
 image directly on the Proxmox node itself, `pct create`/`qm create`), with
 `--dry-run` and a guard against duplicating an already-deployed host's

docs/auto-installer.md

@@ -73,7 +73,7 @@ booting one:
 
 First boot runs `boot.postBootCommands` (registers the Nix store DB and
 system profile) — there's no separate activation step to run yourself.
-`scripts/create-proxmox-resource.sh --type lxc --host <name>` automates all
+`scripts/proxmox/create-proxmox-resource.sh --type lxc --host <name>` automates all
 of this (host-key handling, building the tarball directly on the Proxmox
 node itself, `pct create` with the flags above) — see its `--help`.
 
@@ -102,7 +102,7 @@ groups required, got 0`, and *every* secret (including this host's own
 login) permanently fails to decrypt, silently — no error in the boot log
 at all, since the activation step that would install secrets only runs on
 a from-scratch first activation and skips silently once `/run/current-system`
-already exists. `scripts/create-proxmox-resource.sh` always builds with
+already exists. `scripts/proxmox/create-proxmox-resource.sh` always builds with
 `NIXOS_HOST_KEYS_DIR` set for this reason.
 
 ## Layout
@@ -116,10 +116,10 @@ already exists. `scripts/create-proxmox-resource.sh` always builds with
   `docs/pxe-boot.md`).
 - `modules/installer/host-keys.nix` — optionally bakes pre-generated SSH
   host keys into the image; see "Host keys" below.
-- `scripts/sync-host-keys.sh` — admin-workstation tool that generates,
+- `scripts/secrets/sync-host-keys.sh` — admin-workstation tool that generates,
   registers, and (via `--remove`/`--regenerate-all-keys`) retires host
   keys; see "Creating a New Machine" below.
-- `scripts/prepare-host-key.sh` — narrower predecessor: generates a single
+- `scripts/secrets/prepare-host-key.sh` — narrower predecessor: generates a single
   key by an arbitrary name without touching `.sops.yaml`. Still useful for
   pre-generating a key *before* its flake target exists (`sync-host-keys.sh`
   can only act on targets `nixosConfigurations` already has); otherwise
@@ -238,7 +238,7 @@ GitHub token behind sops-nix for all of them).
 2. **On your admin workstation, generate and register its host key:**
 
    ```sh
-   ./scripts/sync-host-keys.sh <flake-target>
+   ./scripts/secrets/sync-host-keys.sh <flake-target>
    ```
 
    This generates `host-keys/<flake-target>_ssh_host_ed25519_key(.pub)`,
@@ -250,7 +250,7 @@ GitHub token behind sops-nix for all of them).
 
    Doing this for every host that needs one at once — after adding several
    new targets, or just to catch up any that were missed — is
-   `./scripts/sync-host-keys.sh --all`. See `scripts/sync-host-keys.sh --help`
+   `./scripts/secrets/sync-host-keys.sh --all`. See `scripts/secrets/sync-host-keys.sh --help`
    for its other modes (`--remove`, `--regenerate-all-keys`).
 
 3. **Commit and push.** The flake build the installer uses has to see the

docs/nix-cache.md

@@ -46,18 +46,26 @@ the new key up automatically on next activation — no more manual
 
 ## Remote builder SSH keys
 
-On each client, install the private key used to authenticate as `nixremote`:
+Each client authenticates as `nixremote` using its **own default root SSH
+identity** (`/root/.ssh/id_ed25519`) — not a separately-named or shared
+keypair. If a client doesn't have one yet:
 
 ```bash
-sudo install -d -m 0700 /root/.ssh
+sudo ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
-sudo install -m 0600 ./nixremote /root/.ssh/nixremote
-sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
 ```
 
-On `nix-cache`, install the matching public key used by `nixremote` authorized keys.
+Then add its `.pub` contents as a new entry in `vars.remoteBuilderAuthorizedKeys`
+(`variables.nix`) and rebuild `nix-cache` to pick it up (that list is
+declarative — an imperative `ssh-copy-id nixremote@nix-cache` won't stick;
+it gets overwritten on every rebuild). Verify with:
 
-The committed `nixremote` authorized keys are public SSH keys only. Keep the
+```bash
-matching private keys on client hosts and out of the repository.
+sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
+```
+
+The committed `remoteBuilderAuthorizedKeys` entries are public SSH keys
+only. Keep the matching private keys on client hosts and out of the
+repository.
 
 nix-cache's own SSH *host* key is trusted declaratively via
 `programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
@@ -76,8 +84,8 @@ After deployment:
 curl http://nix-cache/nix-cache-info
 nix store ping --store http://nix-cache
 nix show-config | grep -E 'substituters|trusted-public-keys|builders-use-substitutes'
-sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
+sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
-nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/nixremote 4 2 big-parallel,kvm,nixos-test,benchmark' -L
+nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/id_ed25519 4 2 big-parallel,kvm,nixos-test,benchmark' -L
 nix path-info -r nixpkgs#hello
 curl -I "http://nix-cache/$(basename "$(nix path-info nixpkgs#hello)").narinfo"
 ```

docs/proxmox-images.md

@@ -7,7 +7,7 @@ config (`modules/disko/proxmox.nix`) already used to format a real disk on
 install, so there's nothing host-specific to write; it's available for every
 `proxmox-*` target automatically.
 
-`scripts/create-proxmox-resource.sh --type vm --host <name>` automates the
+`scripts/proxmox/create-proxmox-resource.sh --type vm --host <name>` automates the
 whole walkthrough below (and the equivalent LXC one) end to end, including
 host-key handling and building the image directly on the Proxmox node
 itself (no local build, no image transfer) — see its `--help`. The steps
@@ -51,7 +51,7 @@ sudo ./result \
   --build-memory 2048
 ```
 
-Generate the key first with `scripts/sync-host-keys.sh <hostname>`, same
+Generate the key first with `scripts/secrets/sync-host-keys.sh <hostname>`, same
 as any other host — see `docs/auto-installer.md` for the full walkthrough
 (it registers the new key in `.sops.yaml` and re-encrypts the affected
 `secrets/*.yaml` files too, no manual editing needed).

modules/installer/common.nix

@@ -130,7 +130,7 @@
         # at *activation* time, which runs before systemd would otherwise
         # generate one on first boot. Without pre-seeding it here, secrets
         # (including the login password) fail to decrypt on first boot.
-        # Generate the key with scripts/prepare-host-key.sh first.
+        # Generate the key with scripts/secrets/prepare-host-key.sh first.
         #
         # Two places a key can come from, checked in order:
         #   /etc/host-keys  — baked into this image at build time (see
@@ -150,7 +150,7 @@
         else
             echo "WARNING: no SSH host key found for ''${choice} (checked /etc/host-keys and /root/host-keys)"
             echo "sops-nix secrets (including the login password) will NOT decrypt on first boot."
-            echo "Run scripts/prepare-host-key.sh for host ''${choice} on your admin workstation first,"
+            echo "Run scripts/secrets/prepare-host-key.sh for host ''${choice} on your admin workstation first,"
             echo "then either rebuild this image with NIXOS_HOST_KEYS_DIR set, or scp the result to"
             echo "/root/host-keys/ on this machine."
             read -rp "Continue without a pre-seeded key anyway? (y/N): " skip_key

modules/nix-cache/remote-builder-client.nix

@@ -1,10 +1,14 @@
 { pkgs, vars, ... }:
 
 {
-  # Install the remote builder key on each client host (do not commit private keys):
+  # Authenticate as nixremote using the client host's own default root SSH
-  #   sudo install -d -m 0700 /root/.ssh
+  # identity (/root/.ssh/id_ed25519) rather than a separately-named key --
-  #   sudo install -m 0600 ./nixremote /root/.ssh/nixremote
+  # matches vars.remoteBuilderAuthorizedKeys, which already authorizes
-  #   sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
+  # each host's own default key (one entry per host, not a shared
+  # dedicated keypair). If this host doesn't have one yet:
+  #   sudo -u root ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
+  #   # then add its .pub to vars.remoteBuilderAuthorizedKeys and rebuild nix-cache
+  #   sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
   # Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
   # can connect the first time without a manual ssh-keyscan/known_hosts
   # step on every new client.
@@ -20,7 +24,7 @@
       {
         hostName = vars.nixCacheHost;
         sshUser = vars.remoteBuilderUser;
-        sshKey = "/root/.ssh/${vars.remoteBuilderUser}";
+        sshKey = "/root/.ssh/id_ed25519";
         inherit (pkgs.stdenv.hostPlatform) system;
         maxJobs = 4;
         speedFactor = 2;

modules/platforms/lxc.nix

@@ -14,7 +14,7 @@ let
   # Without this, config.system.build.tarball's built-in system just
   # generates a fresh host key at first boot like any other host would --
   # but sops-nix derives its decryption key from *this* file, and
-  # .sops.yaml only trusts whatever key scripts/sync-host-keys.sh already
+  # .sops.yaml only trusts whatever key scripts/secrets/sync-host-keys.sh already
   # registered for this exact target name. A freshly-generated key can
   # never match that, so every secret (including this host's own login)
   # permanently fails to decrypt. Confirmed live: sops-install-secrets
@@ -26,7 +26,7 @@ let
   hostKeysDir = /. + hostKeysDirStr;
 
   # flakeTarget ("${platform}-${buildType}") comes in via specialArgs from
-  # flake.nix's mkTarget -- exactly the name scripts/sync-host-keys.sh
+  # flake.nix's mkTarget -- exactly the name scripts/secrets/sync-host-keys.sh
   # registers keys under. Deliberately not read back from
   # config.environment.etc."flake-target" (which is set to the same value)
   # -- this module also *contributes* to environment.etc below, and a

scripts/codex-maintenance.sh

@@ -41,6 +41,20 @@ echo
 echo "Running statix lint..."
 nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check .
 
+echo
+echo "Checking nix-cache host key for drift..."
+if bash "${script_dir}/secrets/sync-nix-cache-host-key.sh" --check; then
+  :
+else
+  drift_status=$?
+  if [[ "$drift_status" -eq 2 ]]; then
+    echo "nix-cache unreachable from here -- skipping host-key drift check."
+  else
+    echo "WARNING: nix-cache's host key has drifted from variables.nix (see above)." >&2
+    echo "  Run 'bash scripts/secrets/sync-nix-cache-host-key.sh' to fix." >&2
+  fi
+fi
+
 echo
 echo "Evaluating host toplevel derivations..."
 for host in $hosts; do

scripts/env.sh

@@ -3,10 +3,10 @@
 # second copy of these values in every script:
 #   source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/env.sh"
 # Every variable can still be overridden per-invocation via the
-# environment (e.g. PROXMOX_STORAGE=tank-nvme ./scripts/create-proxmox-resource.sh ...)
+# environment (e.g. PROXMOX_STORAGE=tank-nvme ./scripts/proxmox/create-proxmox-resource.sh ...)
 # since each one only sets a default if unset.
 
-# SSH-reachable Proxmox node that scripts/create-proxmox-resource.sh runs
+# SSH-reachable Proxmox node that scripts/proxmox/create-proxmox-resource.sh runs
 # pct/qm on. Matches the Proxmox web UI hostname already used in
 # hosts/nixos/home.nix's desktop shortcuts (pve.<homeDomain> from
 # variables.nix) -- change this if that's not actually reachable over SSH,
@@ -15,7 +15,7 @@
 : "${PROXMOX_SSH_USER:=root}"
 
 # Where this flake repo lives on the Proxmox node itself.
-# scripts/create-proxmox-resource.sh builds images directly on the node
+# scripts/proxmox/create-proxmox-resource.sh builds images directly on the node
 # instead of transferring them over the network -- it clones the repo here
 # (from this checkout's own `origin` remote) the first time it doesn't
 # find it, installing build tooling via scripts/codex-setup.sh, then

scripts/lib/confirm.sh

@@ -0,0 +1,21 @@
+#!/usr/bin/env bash
+# Shared "type X to confirm" prompt for scripts/proxmox/create-proxmox-resource.sh
+# (--modify, and replacing an existing --allow-duplicate-host resource) and
+# scripts/secrets/sync-host-keys.sh (--regenerate-all-keys) -- three destructive
+# confirmations that all work the same way (echo the expected value back
+# exactly), kept in one place so the prompt/comparison logic can't drift.
+# Source alongside env.sh:
+#   source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/confirm.sh"
+#
+# Deliberately does NOT print anything on mismatch or decide exit-vs-return
+# -- callers vary on both (a top-level script exits, a subcommand function
+# returns; wording differs too), so that stays at the call site.
+
+# confirm_typed <expected> <prompt>
+# Prints <prompt> via `read -rp`, then reports (via exit status) whether the
+# typed input matched <expected> exactly.
+confirm_typed() {
+  local expected="$1" prompt="$2" input
+  read -rp "$prompt" input
+  [[ "$input" == "$expected" ]]
+}

scripts/lib/sops-age.sh

@@ -0,0 +1,52 @@
+#!/usr/bin/env bash
+# Shared sops/age helpers for scripts/secrets/backup-admin-key.sh,
+# scripts/secrets/rotate-admin-key.sh, and scripts/secrets/sync-host-keys.sh -- all three
+# derive an age public key from a private identity file the same way, two
+# of them resolve the same sops/age default key-file path, and two of them
+# run `sops updatekeys` the same way. Kept in one place so they can't drift
+# apart. Source alongside env.sh:
+#   source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/sops-age.sh"
+#
+# Uses NIX_OPTS (an array of extra `nix-shell` options -- see env.sh's
+# nix_extra_opts) if the caller has already set it, same convention as
+# lib/ssh-host-keys.sh. Falls back to no extra options if the caller never
+# sourced env.sh.
+if ! declare -p NIX_OPTS >/dev/null 2>&1; then
+  declare -a NIX_OPTS=()
+fi
+
+# sops/age's own default identity-file resolution order, minus $SOPS_AGE_KEY
+# itself (an inline identity, not a path -- callers that accept it check it
+# separately, before falling back to this).
+: "${DEFAULT_SOPS_AGE_KEY_FILE:=${SOPS_AGE_KEY_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/sops/age/keys.txt}}"
+
+# age_pubkey_from_identity_file <identity-file>
+# Prints the age public key for a private identity file (age-keygen -y).
+age_pubkey_from_identity_file() {
+  local identity_file="$1"
+  nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '${identity_file}'"
+}
+
+# sops_yaml_admin_pubkey <sops-yaml-path>
+# Prints .sops.yaml's current &admin age public key, or empty (not an error
+# under set -e) if no such anchor line exists -- callers that need to treat
+# "missing" as fatal check for an empty result themselves.
+sops_yaml_admin_pubkey() {
+  local sops_yaml="$1"
+  grep -E '^  - &admin age1' "$sops_yaml" 2>/dev/null | awk '{print $NF}' || true
+}
+
+# sops_updatekeys <secrets-file> [key-file]
+# Re-encrypts <secrets-file> for .sops.yaml's current recipient set. If
+# <key-file> is given, decrypts with that identity (SOPS_AGE_KEY_FILE)
+# instead of whatever's ambient -- needed when the ambient default key
+# doesn't match yet (e.g. mid-rotation, decrypting with the outgoing key).
+sops_updatekeys() {
+  local secrets_file="$1" key_file="${2:-}"
+  if [[ -n "$key_file" ]]; then
+    SOPS_AGE_KEY_FILE="$key_file" nix-shell "${NIX_OPTS[@]}" -p sops --run \
+      "sops updatekeys --yes '${secrets_file}'"
+  else
+    nix-shell "${NIX_OPTS[@]}" -p sops --run "sops updatekeys --yes '${secrets_file}'"
+  fi
+}

scripts/lib/ssh-host-keys.sh

@@ -1,6 +1,6 @@
 #!/usr/bin/env bash
-# Shared SSH-host-key / age-conversion helpers for scripts/sync-host-keys.sh
+# Shared SSH-host-key / age-conversion helpers for scripts/secrets/sync-host-keys.sh
-# and scripts/prepare-host-key.sh -- both generate the same kind of key
+# and scripts/secrets/prepare-host-key.sh -- both generate the same kind of key
 # (ed25519, no passphrase, the sops-nix age-derivation input) and convert it
 # to an age recipient the same way; kept in one place so the two can't
 # drift apart.

scripts/proxmox/configure-nix-cache-client.sh

@@ -0,0 +1,182 @@
+#!/usr/bin/env bash
+# Points a non-NixOS Debian machine's Nix install at nix-cache: adds it as
+# a substituter (with cache.nixos.org kept as fallback) and, once the
+# remote-builder private key is installed, as a distributed-build machine
+# too.
+#
+# This is the non-NixOS equivalent of modules/nix-cache/client.nix +
+# modules/nix-cache/remote-builder-client.nix -- those two only apply to
+# hosts built from this flake. A plain Debian box with Nix installed
+# (single- or multi-user install, nix-daemon running) has no NixOS module
+# system to pick that config up, so this edits /etc/nix/nix.conf by hand
+# instead. Run this ON the target Debian machine, as root.
+#
+# The values below mirror variables.nix / modules/nix-cache/client.nix in
+# this repo -- update both if nix-cache is ever rebuilt with a new host
+# key or the cache signing key is rotated (see docs/nix-cache.md).
+#
+# REMOTE_BUILDER_KEY defaults to this machine's own default root SSH
+# identity (matches modules/nix-cache/remote-builder-client.nix's
+# convention for real NixOS clients: authenticate as nixremote with the
+# host's own default key, added individually to
+# vars.remoteBuilderAuthorizedKeys, rather than a separately-named or
+# shared keypair) -- generate one with
+# `ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519` if this machine
+# doesn't have one yet, then add its .pub to vars.remoteBuilderAuthorizedKeys
+# and rebuild nix-cache.
+#
+# Usage:
+#   sudo ./configure-nix-cache-client.sh [--dry-run] [--no-remote-builder] [--no-restart]
+#
+# Env overrides (defaults match variables.nix):
+#   NIX_CACHE_HOST, NIX_CACHE_HOST_KEY, REMOTE_BUILDER_USER, REMOTE_BUILDER_KEY
+
+set -euo pipefail
+
+: "${NIX_CACHE_HOST:=nix-cache}"
+: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache}"
+: "${REMOTE_BUILDER_USER:=nixremote}"
+: "${REMOTE_BUILDER_KEY:=/root/.ssh/id_ed25519}"
+
+CACHE_PUB_KEY="cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4="
+FALLBACK_URL="https://cache.nixos.org/"
+FALLBACK_PUB_KEY="cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
+
+NIX_CONF="/etc/nix/nix.conf"
+KNOWN_HOSTS="/etc/ssh/ssh_known_hosts"
+MARKER_BEGIN="# BEGIN nix-cache client config (configure-nix-cache-client.sh)"
+MARKER_END="# END nix-cache client config"
+
+dry_run=0
+with_remote_builder=1
+restart_daemon=1
+
+for arg in "$@"; do
+  case "$arg" in
+    --dry-run) dry_run=1 ;;
+    --no-remote-builder) with_remote_builder=0 ;;
+    --no-restart) restart_daemon=0 ;;
+    -h|--help)
+      sed -n '2,20p' "$0"
+      exit 0
+      ;;
+    *)
+      echo "ERROR: unknown argument: $arg" >&2
+      exit 1
+      ;;
+  esac
+done
+
+if [[ "$dry_run" -eq 0 && "$EUID" -ne 0 ]]; then
+  echo "ERROR: must run as root (writes $NIX_CONF and, unless --no-remote-builder, $KNOWN_HOSTS)." >&2
+  exit 1
+fi
+
+if ! command -v nix >/dev/null 2>&1; then
+  echo "ERROR: no 'nix' binary on PATH -- install the Nix package manager first." >&2
+  exit 1
+fi
+
+if [[ ! -f "$NIX_CONF" ]]; then
+  echo "ERROR: $NIX_CONF not found -- expected an existing multi-user Nix install." >&2
+  exit 1
+fi
+
+builder_line=""
+if [[ "$with_remote_builder" -eq 1 ]]; then
+  if [[ -f "$REMOTE_BUILDER_KEY" ]]; then
+    case "$(uname -m)" in
+      x86_64) nix_system="x86_64-linux" ;;
+      aarch64) nix_system="aarch64-linux" ;;
+      *)
+        echo "WARNING: unrecognized architecture '$(uname -m)' -- skipping remote builder, keeping substituter config." >&2
+        with_remote_builder=0
+        ;;
+    esac
+    if [[ "$with_remote_builder" -eq 1 ]]; then
+      builder_line="builders = ssh://${REMOTE_BUILDER_USER}@${NIX_CACHE_HOST} ${nix_system} ${REMOTE_BUILDER_KEY} 4 2 big-parallel,kvm,nixos-test,benchmark"
+    fi
+  else
+    echo "WARNING: $REMOTE_BUILDER_KEY not found -- skipping remote builder config (substituter still configured)." >&2
+    echo "  See docs/nix-cache.md 'Remote builder SSH keys' for how to install it, then re-run this script." >&2
+    with_remote_builder=0
+  fi
+fi
+
+block="$(cat <<EOF
+$MARKER_BEGIN
+extra-substituters = http://${NIX_CACHE_HOST} ${FALLBACK_URL}
+extra-trusted-public-keys = ${CACHE_PUB_KEY} ${FALLBACK_PUB_KEY}
+EOF
+)"
+if [[ "$with_remote_builder" -eq 1 ]]; then
+  block="${block}
+builders-use-substitutes = true
+${builder_line}"
+fi
+block="${block}
+$MARKER_END"
+
+echo "== nix.conf block to install =="
+echo "$block"
+echo "================================"
+
+if [[ "$dry_run" -eq 1 ]]; then
+  echo "(--dry-run: not writing $NIX_CONF)"
+else
+  tmp_conf="$(mktemp)"
+  trap 'rm -f "$tmp_conf"' EXIT
+
+  if grep -qF "$MARKER_BEGIN" "$NIX_CONF"; then
+    awk -v begin="$MARKER_BEGIN" -v end="$MARKER_END" -v block="$block" '
+      $0 == begin { print block; skip = 1; next }
+      $0 == end   { skip = 0; next }
+      skip        { next }
+      { print }
+    ' "$NIX_CONF" > "$tmp_conf"
+  else
+    cp "$NIX_CONF" "$tmp_conf"
+    printf '\n%s\n' "$block" >> "$tmp_conf"
+  fi
+
+  cp "$NIX_CONF" "${NIX_CONF}.bak.$(date +%Y%m%d%H%M%S)"
+  install -m 0644 "$tmp_conf" "$NIX_CONF"
+  echo "Updated $NIX_CONF (backup saved alongside it)."
+fi
+
+if [[ "$with_remote_builder" -eq 1 ]]; then
+  known_hosts_line="${NIX_CACHE_HOST} ${NIX_CACHE_HOST_KEY}"
+  if [[ "$dry_run" -eq 1 ]]; then
+    echo "(--dry-run: would ensure this line is present in $KNOWN_HOSTS)"
+    echo "  $known_hosts_line"
+  else
+    mkdir -p "$(dirname "$KNOWN_HOSTS")"
+    touch "$KNOWN_HOSTS"
+    if ! grep -qF "$known_hosts_line" "$KNOWN_HOSTS" 2>/dev/null; then
+      echo "$known_hosts_line" >> "$KNOWN_HOSTS"
+      echo "Added nix-cache's SSH host key to $KNOWN_HOSTS."
+    fi
+  fi
+fi
+
+if [[ "$dry_run" -eq 0 && "$restart_daemon" -eq 1 ]]; then
+  if command -v systemctl >/dev/null 2>&1 && systemctl is-active --quiet nix-daemon 2>/dev/null; then
+    systemctl restart nix-daemon
+    echo "Restarted nix-daemon to pick up the new config."
+  else
+    echo "nix-daemon not managed by systemd (or not running) -- restart it manually to pick up the new config."
+  fi
+fi
+
+cat <<EOF
+
+Done. Verify with:
+  curl http://${NIX_CACHE_HOST}/nix-cache-info
+  nix show-config | grep -E 'substituters|trusted-public-keys|builders'
+EOF
+if [[ "$with_remote_builder" -eq 1 ]]; then
+  cat <<EOF
+  ssh -i ${REMOTE_BUILDER_KEY} ${REMOTE_BUILDER_USER}@${NIX_CACHE_HOST} nix-store --version
+  nix build nixpkgs#hello -L
+EOF
+fi

scripts/proxmox/create-proxmox-resource.sh

@@ -13,9 +13,9 @@
 # alone wouldn't carry it) before building.
 #
 # Usage:
-#   scripts/create-proxmox-resource.sh --type lxc|vm --host <name> [options]
+#   scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --host <name> [options]
-#   scripts/create-proxmox-resource.sh --type lxc|vm --list
+#   scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --list
-#   scripts/create-proxmox-resource.sh --modify --vmid <n> [--cores N] [--memory MB] [--grow-disk GB]
+#   scripts/proxmox/create-proxmox-resource.sh --modify --vmid <n> [--cores N] [--memory MB] [--grow-disk GB]
 #
 # SAFETY:
 #   - The default (create) mode only ever creates a NEW resource -- it
@@ -41,13 +41,15 @@
 # See --help for the full option list.
 set -euo pipefail
 
-repo_root="$(cd "$(dirname "$0")/.." && pwd)"
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
-# shellcheck source=env.sh
+# shellcheck source=../env.sh
 source "${repo_root}/scripts/env.sh"
-# shellcheck source=lib/nix-eval.sh
+# shellcheck source=../lib/nix-eval.sh
 source "${repo_root}/scripts/lib/nix-eval.sh"
+# shellcheck source=../lib/confirm.sh
+source "${repo_root}/scripts/lib/confirm.sh"
 
-sync_keys="${repo_root}/scripts/sync-host-keys.sh"
+sync_keys="${repo_root}/scripts/secrets/sync-host-keys.sh"
 
 usage() {
   cat <<EOF
@@ -230,8 +232,7 @@ cmd_modify() {
   fi
 
   echo
-  read -rp "Type the VMID (${vmid}) to confirm these changes: " confirm
+  if ! confirm_typed "$vmid" "Type the VMID (${vmid}) to confirm these changes: "; then
-  if [[ "$confirm" != "$vmid" ]]; then
     echo "Cancelled -- input didn't match ${vmid}."
     exit 1
   fi
@@ -429,8 +430,7 @@ REMOTE_SCRIPT
       echo "  - ${kind} VMID ${id} (${n})"
     done
     echo
-    read -rp "Type the hostname (${host}) to confirm destroying the above and replacing it: " confirm
+    if ! confirm_typed "$host" "Type the hostname (${host}) to confirm destroying the above and replacing it: "; then
-    if [[ "$confirm" != "$host" ]]; then
       echo "Cancelled -- input didn't match ${host}." >&2
       exit 1
     fi
@@ -524,7 +524,7 @@ ensure_remote_repo() {
   echo
   echo "==> Ensuring ${remote_repo_dir} exists and is current on ${node}..."
   if [[ "$dry_run" -eq 1 ]]; then
-    echo "[dry-run] would ensure ${remote_repo_dir} exists on ${node} (clone if missing, git pull if present), and would verify/bootstrap build tooling there (scripts/codex-setup.sh) if \`nix\` isn't already on PATH"
+    echo "[dry-run] would ensure ${remote_repo_dir} exists on ${node} (clone if missing, git pull if present), and would verify/bootstrap build tooling there (scripts/codex-setup.sh) if \`nix\` isn't already on PATH -- and if that bootstrap actually ran, would also configure ${node} as a nix-cache client (scripts/proxmox/configure-nix-cache-client.sh)"
     return
   fi
 
@@ -571,6 +571,21 @@ ensure_remote_repo() {
   else
     echo "==> Bootstrapping build tooling on ${node} (scripts/codex-setup.sh)..."
     ssh "$ssh_target" "cd '${remote_repo_dir}' && bash scripts/codex-setup.sh"
+
+    # Only on this first-time bootstrap, not every run -- a node that
+    # already has tooling either already went through this once, or had
+    # it configured some other way, and re-running is harmless but
+    # pointless. Non-fatal: this only makes the node's own builds faster
+    # (substitute from nix-cache instead of building from source) and
+    # offloadable to it as a remote builder -- worth trying, not worth
+    # aborting the image build over if nix-cache happens to be down right
+    # now. Needs ensure_nix_profile first, same as the tooling_check_cmd
+    # above -- ssh's non-interactive command execution won't have picked
+    # up a freshly single-user-installed `nix` otherwise.
+    echo "==> Configuring ${node} as a nix-cache substituter/remote-builder client..."
+    if ! ssh "$ssh_target" "cd '${remote_repo_dir}' && . scripts/lib/nix-bootstrap.sh && ensure_nix_profile && bash scripts/proxmox/configure-nix-cache-client.sh"; then
+      echo "WARNING: configure-nix-cache-client.sh failed on ${node} -- continuing without it (${node} will build from source / against cache.nixos.org only)." >&2
+    fi
   fi
 }
 

scripts/secrets/backup-admin-key.sh

@@ -6,7 +6,7 @@
 # copy is ever lost, or to run either script from a different machine.
 #
 # Usage:
-#   scripts/backup-admin-key.sh <dest-path> [--key-file <path>] [--force] [--dry-run]
+#   scripts/secrets/backup-admin-key.sh <dest-path> [--key-file <path>] [--force] [--dry-run]
 #
 # Source key resolution matches sops/age's own default order:
 #   $SOPS_AGE_KEY (inline identity text) if set, else
@@ -15,11 +15,13 @@
 #   ${XDG_CONFIG_HOME:-$HOME/.config}/sops/age/keys.txt
 set -euo pipefail
 
-repo_root="$(cd "$(dirname "$0")/.." && pwd)"
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
 sops_yaml="${repo_root}/.sops.yaml"
 
-# shellcheck source=env.sh
+# shellcheck source=../env.sh
 source "${repo_root}/scripts/env.sh"
+# shellcheck source=../lib/sops-age.sh
+source "${repo_root}/scripts/lib/sops-age.sh"
 
 # Pin cwd for the same reason rotate-admin-key.sh does: age/sops calls
 # below should never depend on wherever the caller's shell happened to be.
@@ -43,7 +45,7 @@ EOF
 
 dry_run=0
 force=0
-key_file="${SOPS_AGE_KEY_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/sops/age/keys.txt}"
+key_file="$DEFAULT_SOPS_AGE_KEY_FILE"
 args=()
 
 while [[ $# -gt 0 ]]; do
@@ -103,13 +105,13 @@ scratch="$(mktemp)"
 trap 'rm -f "$scratch"' EXIT
 ( umask 077; printf '%s\n' "$src_content" > "$scratch" )
 
-src_pub="$(nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '$scratch'")" || {
+src_pub="$(age_pubkey_from_identity_file "$scratch")" || {
   echo "ERROR: source doesn't look like a valid age identity (age-keygen -y failed)." >&2
   exit 1
 }
 echo "    public key: ${src_pub}"
 
-current_admin_pub="$(grep -E '^  - &admin age1' "$sops_yaml" 2>/dev/null | awk '{print $NF}' || true)"
+current_admin_pub="$(sops_yaml_admin_pubkey "$sops_yaml")"
 if [[ -n "$current_admin_pub" && "$current_admin_pub" != "$src_pub" ]]; then
   echo "NOTE: this key does not match .sops.yaml's current &admin entry (${current_admin_pub})."
   echo "      Backing it up anyway -- this script doesn't require it to be the admin key."
@@ -131,7 +133,7 @@ fi
 mkdir -p "$(dirname "$dest")"
 install -m 600 "$scratch" "$dest"
 
-dest_pub="$(nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '$dest'")"
+dest_pub="$(age_pubkey_from_identity_file "$dest")"
 if [[ "$dest_pub" != "$src_pub" ]]; then
   echo "ERROR: ${dest} was written but its public key doesn't match the source -- investigate before relying on this backup." >&2
   exit 1
@@ -144,5 +146,5 @@ Done. Backed up to: ${dest}
 
 This is a private key -- store it somewhere offline/secure, not in this
 repo or anywhere it'd get committed. Restore it with:
-  scripts/rotate-admin-key.sh ${dest}
+  scripts/secrets/rotate-admin-key.sh ${dest}
 EOF

scripts/secrets/prepare-host-key.sh

@@ -2,7 +2,7 @@
 # Generates a new machine's SSH host key by an arbitrary name, before it
 # necessarily has a flake target yet -- prints the .sops.yaml snippet to
 # add by hand. For any host that already has a flake target,
-# scripts/sync-host-keys.sh <target> does this same job plus the
+# scripts/secrets/sync-host-keys.sh <target> does this same job plus the
 # .sops.yaml/key_groups registration and re-encryption automatically; use
 # this script only to pre-generate a key ahead of adding the flake target
 # itself.
@@ -22,13 +22,13 @@
 # new machine.
 set -euo pipefail
 
-repo_root="$(cd "$(dirname "$0")/.." && pwd)"
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
-# shellcheck source=env.sh
+# shellcheck source=../env.sh
 source "${repo_root}/scripts/env.sh"
-# shellcheck source=lib/ssh-host-keys.sh
+# shellcheck source=../lib/ssh-host-keys.sh
 source "${repo_root}/scripts/lib/ssh-host-keys.sh"
 
-hostname="${1:?usage: scripts/prepare-host-key.sh <hostname>}"
+hostname="${1:?usage: scripts/secrets/prepare-host-key.sh <hostname>}"
 sops_yaml="${repo_root}/.sops.yaml"
 
 if [[ ! -f "$sops_yaml" ]]; then

scripts/secrets/rotate-admin-key.sh

@@ -10,7 +10,7 @@
 # sync-host-keys.sh print when they bootstrap a brand-new, not-yet-trusted
 # age key on a machine that's never had admin access before:
 #
-#   scripts/rotate-admin-key.sh /path/to/backed-up/admin/keys.txt
+#   scripts/secrets/rotate-admin-key.sh /path/to/backed-up/admin/keys.txt
 #
 # The backup key's *public* key must match .sops.yaml's current &admin
 # entry -- this script verifies that by deriving it, it doesn't just trust
@@ -19,11 +19,13 @@
 # the common case is just pointing this at the restored backup.
 set -euo pipefail
 
-repo_root="$(cd "$(dirname "$0")/.." && pwd)"
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
 sops_yaml="${repo_root}/.sops.yaml"
 
-# shellcheck source=env.sh
+# shellcheck source=../env.sh
 source "${repo_root}/scripts/env.sh"
+# shellcheck source=../lib/sops-age.sh
+source "${repo_root}/scripts/lib/sops-age.sh"
 
 # sops resolves .sops.yaml by walking up from the process's cwd, not from
 # the target file's own path -- if this script were invoked from somewhere
@@ -53,7 +55,7 @@ EOF
 }
 
 dry_run=0
-new_key_file="${SOPS_AGE_KEY_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/sops/age/keys.txt}"
+new_key_file="$DEFAULT_SOPS_AGE_KEY_FILE"
 args=()
 
 while [[ $# -gt 0 ]]; do
@@ -93,13 +95,9 @@ backup_key="${args[0]}"
 
 nix_extra_opts
 
-age_pub() {
-  nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '$1'"
-}
-
 echo "==> Deriving public keys..."
-old_pub="$(age_pub "$backup_key")"
+old_pub="$(age_pubkey_from_identity_file "$backup_key")"
-new_pub="$(age_pub "$new_key_file")"
+new_pub="$(age_pubkey_from_identity_file "$new_key_file")"
 echo "    backup (old admin) key: ${old_pub}"
 echo "    new admin key:          ${new_pub}"
 
@@ -108,12 +106,11 @@ if [[ "$old_pub" == "$new_pub" ]]; then
   exit 1
 fi
 
-current_admin_line="$(grep -E '^  - &admin age1' "$sops_yaml" || true)"
+current_admin_pub="$(sops_yaml_admin_pubkey "$sops_yaml")"
-if [[ -z "$current_admin_line" ]]; then
+if [[ -z "$current_admin_pub" ]]; then
   echo "ERROR: couldn't find a '&admin age1...' line in ${sops_yaml}." >&2
   exit 1
 fi
-current_admin_pub="$(awk '{print $NF}' <<<"$current_admin_line")"
 
 if [[ "$current_admin_pub" != "$old_pub" ]]; then
   echo "ERROR: ${backup_key} doesn't match the current &admin key in .sops.yaml." >&2
@@ -129,9 +126,17 @@ if [[ "${#secrets_files[@]}" -eq 0 ]]; then
   exit 1
 fi
 
+# sops_can_decrypt <key-file> <secrets-file>: used both to confirm the
+# backup key still works before touching anything, and again after
+# rotation to confirm the new key does too.
+sops_can_decrypt() {
+  local key_file="$1" secrets_file="$2"
+  SOPS_AGE_KEY_FILE="$key_file" nix-shell "${NIX_OPTS[@]}" -p sops --run \
+    "sops -d '${secrets_file}'" >/dev/null
+}
+
 echo "==> Confirming the backup key can actually decrypt..."
-if ! SOPS_AGE_KEY_FILE="$backup_key" nix-shell "${NIX_OPTS[@]}" -p sops --run \
+if ! sops_can_decrypt "$backup_key" "${secrets_files[0]}"; then
-  "sops -d '${secrets_files[0]}'" >/dev/null; then
   echo "ERROR: backup key failed to decrypt $(basename "${secrets_files[0]}") -- aborting." >&2
   exit 1
 fi
@@ -162,14 +167,12 @@ echo "    Updated."
 echo "==> Re-encrypting secrets/*.yaml for the new recipient set..."
 for f in "${secrets_files[@]}"; do
   echo "==> $(basename "$f")"
-  SOPS_AGE_KEY_FILE="$backup_key" nix-shell "${NIX_OPTS[@]}" -p sops --run \
+  sops_updatekeys "$f" "$backup_key"
-    "sops updatekeys --yes '${f}'"
 done
 
 echo "==> Verifying the new key can decrypt everything..."
 for f in "${secrets_files[@]}"; do
-  if ! SOPS_AGE_KEY_FILE="$new_key_file" nix-shell "${NIX_OPTS[@]}" -p sops --run \
+  if ! sops_can_decrypt "$new_key_file" "$f"; then
-    "sops -d '${f}'" >/dev/null; then
     echo "ERROR: new key failed to decrypt $(basename "$f") after rotation -- investigate before committing." >&2
     exit 1
   fi

scripts/secrets/sync-host-keys.sh

@@ -24,17 +24,21 @@
 # ever touches keys it itself manages.
 set -euo pipefail
 
-repo_root="$(cd "$(dirname "$0")/.." && pwd)"
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
 sops_yaml="${repo_root}/.sops.yaml"
 keydir="${repo_root}/host-keys"
 editor="${repo_root}/scripts/lib/sync-host-keys-edit-sops.py"
 
-# shellcheck source=env.sh
+# shellcheck source=../env.sh
 source "${repo_root}/scripts/env.sh"
-# shellcheck source=lib/nix-eval.sh
+# shellcheck source=../lib/nix-eval.sh
 source "${repo_root}/scripts/lib/nix-eval.sh"
-# shellcheck source=lib/ssh-host-keys.sh
+# shellcheck source=../lib/ssh-host-keys.sh
 source "${repo_root}/scripts/lib/ssh-host-keys.sh"
+# shellcheck source=../lib/sops-age.sh
+source "${repo_root}/scripts/lib/sops-age.sh"
+# shellcheck source=../lib/confirm.sh
+source "${repo_root}/scripts/lib/confirm.sh"
 
 mkdir -p "$keydir"
 
@@ -78,7 +82,7 @@ ensure_admin_decrypt_key() {
     return
   fi
 
-  local key_file="${SOPS_AGE_KEY_FILE:-${XDG_CONFIG_HOME:-$HOME/.config}/sops/age/keys.txt}"
+  local key_file="$DEFAULT_SOPS_AGE_KEY_FILE"
 
   if [[ -s "$key_file" ]]; then
     echo "Found existing sops age key at ${key_file}."
@@ -97,7 +101,7 @@ ensure_admin_decrypt_key() {
   mkdir -p "$(dirname "$key_file")"
   nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -o '${key_file}'" 2>&1 | grep -v "^Public key:" || true
   local new_pub
-  new_pub="$(nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '${key_file}'")"
+  new_pub="$(age_pubkey_from_identity_file "$key_file")"
 
   cat <<EOF
 
@@ -239,7 +243,7 @@ apply_edit_plan() {
       while IFS= read -r basename; do
         [[ -z "$basename" ]] && continue
         echo "==> secrets/${basename}"
-        nix-shell "${NIX_OPTS[@]}" -p sops --run "sops updatekeys --yes '${repo_root}/secrets/${basename}'"
+        sops_updatekeys "${repo_root}/secrets/${basename}"
       done <<<"$changed"
     fi
   fi
@@ -359,8 +363,7 @@ cmd_regenerate_all() {
   echo "image/tarball before it can decrypt secrets again."
 
   if [[ "$dry_run" -ne 1 ]]; then
-    read -rp "Type REGENERATE to confirm: " confirm
+    if ! confirm_typed "REGENERATE" "Type REGENERATE to confirm: "; then
-    if [[ "$confirm" != "REGENERATE" ]]; then
       echo "Cancelled."
       return
     fi

scripts/secrets/sync-nix-cache-host-key.sh

@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+# Detects and fixes drift between the ed25519 SSH host key nix-cache is
+# actually serving right now and vars.nixCacheHostKey (variables.nix) --
+# the value modules/nix-cache/remote-builder-client.nix bakes into every
+# client's declarative programs.ssh.knownHosts, and
+# scripts/proxmox/configure-nix-cache-client.sh hardcodes as its own
+# default for non-NixOS clients.
+#
+# This value has no automatic source of truth: nix-cache's host key is
+# generated once (first boot / container recreate) and never touches this
+# repo again unless someone remembers to update it by hand afterwards. It
+# drifted silently once already -- confirmed live: variables.nix recorded
+# a key that no longer matched what nix-cache actually presented, which
+# would fail every real client's SSH host-key verification for
+# distributed builds without ever producing an obvious error pointing
+# back here (a client just sees "Host key verification failed" against
+# *some* key, with no hint that the trusted value itself was stale).
+#
+# codex-maintenance.sh runs this in --check mode on every invocation so
+# that drift surfaces as a warning instead of a future debugging session.
+#
+# Usage:
+#   scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run] [--host <name>]
+#
+#   --check     Only report drift (exit 1 if found, 2 if nix-cache is
+#               unreachable); never writes. For CI/maintenance use.
+#   --dry-run   Show what would change; never writes.
+#   --host      Override the hostname to scan (default: variables.nix's
+#               nixCacheHost / env.sh's NIX_CACHE_HOST).
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
+variables_nix="${repo_root}/variables.nix"
+client_script="${repo_root}/scripts/proxmox/configure-nix-cache-client.sh"
+
+# shellcheck source=../env.sh
+source "${repo_root}/scripts/env.sh"
+
+check_only=0
+dry_run=0
+host="${NIX_CACHE_HOST}"
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --check) check_only=1; shift ;;
+    --dry-run) dry_run=1; shift ;;
+    --host)
+      host="${2:?--host requires a hostname}"
+      shift 2
+      ;;
+    -h|--help)
+      sed -n '2,23p' "$0"
+      exit 0
+      ;;
+    *)
+      echo "ERROR: unknown argument: $1" >&2
+      exit 1
+      ;;
+  esac
+done
+
+current_value="$(grep -oE 'nixCacheHostKey = "[^"]+"' "$variables_nix" | sed -E 's/nixCacheHostKey = "(.*)"/\1/')"
+if [[ -z "$current_value" ]]; then
+  echo "ERROR: couldn't find nixCacheHostKey in $variables_nix" >&2
+  exit 1
+fi
+current_type_blob="$(awk '{print $1, $2}' <<<"$current_value")"
+current_label="$(awk '{print $3}' <<<"$current_value")"
+
+echo "Scanning ${host} for its current ed25519 SSH host key..."
+nix_extra_opts
+scanned="$(nix-shell "${NIX_OPTS[@]}" -p openssh --run "ssh-keyscan -t ed25519 -T 5 '${host}'" 2>/dev/null | grep -v '^#' | head -1 || true)"
+if [[ -z "$scanned" ]]; then
+  echo "ERROR: couldn't reach ${host} (or got no ed25519 host key back) via ssh-keyscan." >&2
+  exit 2
+fi
+scanned_type_blob="$(awk '{print $2, $3}' <<<"$scanned")"
+
+if [[ "$current_type_blob" == "$scanned_type_blob" ]]; then
+  echo "Up to date: ${host}'s host key matches variables.nix's nixCacheHostKey."
+  exit 0
+fi
+
+echo "DRIFT DETECTED:"
+echo "  variables.nix has: $current_type_blob"
+echo "  ${host} is now:    $scanned_type_blob"
+
+if [[ "$check_only" -eq 1 ]]; then
+  echo
+  echo "Run 'scripts/secrets/sync-nix-cache-host-key.sh' (no flags) to fix." >&2
+  exit 1
+fi
+
+new_value="${scanned_type_blob} ${current_label}"
+
+if [[ "$dry_run" -eq 1 ]]; then
+  echo "(--dry-run: would update variables.nix and ${client_script##*/} to:)"
+  echo "  $new_value"
+  exit 0
+fi
+
+sed -i "s|nixCacheHostKey = \"[^\"]*\"|nixCacheHostKey = \"${new_value}\"|" "$variables_nix"
+sed -i "s|NIX_CACHE_HOST_KEY:=[^}]*}|NIX_CACHE_HOST_KEY:=${new_value}}|" "$client_script"
+
+echo "Updated variables.nix and ${client_script##*/} to:"
+echo "  $new_value"
+echo
+echo "This only takes effect on already-deployed NixOS clients after their"
+echo "next rebuild (programs.ssh.knownHosts is declarative). Review with"
+echo "'git diff', then run 'bash scripts/codex-maintenance.sh' before committing."

secrets/nix-cache.yaml

@@ -4,31 +4,31 @@ sops:
     age:
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmOHlPcGZvN3o5aEZCcFdz
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBPWE1HTUhiSUp5ZEUwWEpI
-            WlYydFdoQ0ZHbElMdTRkZWljMmoxTHZqNjJZCnE3cmgzKzZya3FROEczbHVveDND
+            bGpkZlBIMUo5ZlYrQ09SN3Q1a0ZkQ0ZnOEhVCnJPNEZQenVWWGZiODlzQzNEc1Zq
-            VnEzRW12cnRKdzhkZm5uTXpkTEtrNUEKLS0tIHFDbkc4Mk4wVlM2R29zZXYwR2Ri
+            c3l4OWZJTElJc2Y2UE15OGtEUzhyY1EKLS0tIHNJUStyWnlQWjZBbEZjQ3UwdUpz
-            ODBML1p4eUZiZldYUERQTUhTU1llV0UKxjvH6zbW6wKghzR1o34CyKPEa2FqZmo0
+            ZndoUDR6bisrNGJCUHk3TGI4bTZaMFUK87fFsm9ne9s+PK2pcwtrDjqyGBss2r2E
-            PxgqyuXkIwas9soXVAkScx7ElaV09Fjaj+mDrKwi4a+DwdoSP7czyA==
+            8lhqoeiKZ2j96z8kP/7ChzovwTCmqdcmAQuyNQD+ZAFijseipSvfbQ==
             -----END AGE ENCRYPTED FILE-----
           recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0VDBXYUZNbnpEZHlqV05C
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUYi9SRFFGV3Z6cFd2Znk5
-            WWxQZjVYcWlZeXlGc0RiaUpERzR0T21CWWtBCkM0dW5kYlFNK1RGSzRudFZ2Z0t3
+            b2FLbWtzTllJMDBUaGk0NTViOTNBa2hQclVZClZHKzNhbGVjQUJhWkFWdTFBMG5a
-            aVBreElGM1BIT0RzQkxTVTA3S2NhQncKLS0tIERyTU8zWWRTRm95SmRZQ1BhalVV
+            cUFJdUdyVG5HQXJRRnJId3hqRTN2cXMKLS0tIEFMRjh3WE1ON0U2TTNTZ3hxMTR4
-            SVdBajN1V1BuM2s4K216S3c0VHczNlkKM5jvsSEfCBA5uZRjBJNbM91lLRQkj+jK
+            ZGRlemlIbDZKeExmVHROc3Eyak5DdzQKaLwIVDi6BN4cxpVxJoqTYvJETPOp4thc
-            rM5uSfGLTvSjPgXIMIq03OXxH1CE7GoKxAPwFrJdAFMMQcutIethhw==
+            l9uVMvIGuEsEZgDsvShw1dYLljd+uGy/A+dXbcxIUCP/mmPkwmd1Pw==
             -----END AGE ENCRYPTED FILE-----
           recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvbk9tRE9jYkpxNThQeElH
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArOWovSW9DeFpxL0VDUDQ3
-            dWpKYThDb1ZkUWtMWlRxc2tYbjhBbEFEcWtVCkNiK0NBRko2TkJQdUVtdGtHNGcv
+            SHUwTzJVZUtPV01ZRkdCUXZGL2lTRCtCNFNnCjBSNExqRW5mTEN5SFVucHJHSzZt
-            andSaUlKSFA4THRyZXNTYmI0Yk5WbmsKLS0tIHlQTUtpWDdPeTVZL0M1RElRdFNk
+            cDlNc3BjY3M1c1k1Z2tkVEg4R1pacGsKLS0tIEFWbHNKZW0vbVh1Y2VhQW93OUwx
-            QWlGdFo5NkZWSmY0YXdpNzNUQnlsK3MKtzC0bM7Ek+K73nMranOA1Mc98RUnYnq1
+            MWV0eW9sOXdQd0l2ZjlWOEVVc1dwcTgK2s4p9xoNkawH2OkGsl80bNIo3ad5vn4W
-            hAt0QEFKWK4QVKubaN/rG3AzE0U7qPKWHTzoxgnAiL3WyV9teLW+iA==
+            Z2w+jwppSoUmbQnD3WFbLmSSxmuobmU8HILwElv6SZu+KE3aspF6XA==
             -----END AGE ENCRYPTED FILE-----
-          recipient: age164px2a8e48ptsf9ngtan38aa6jls4jdl26mzrgzf6sn3vcvt49hqjrgr8w
+          recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
     lastmodified: "2026-07-19T23:30:21Z"
     mac: ENC[AES256_GCM,data:kLGE2xawQT7mx+sfw68hmGk5nCEGiEjZrqTEl9B1dtQmTrMwmoVr/1RISi4LfJrwxy31mDgff4lcIL4wIJuM373uk3X8j4RNyYQNTfKEkORT6r8NHeepNs267O77pKGd7OmcM4MT/BqOnB8ELS7Wlf2ect7CAlvUUVyc8icxgZE=,iv:EYLDsHYHZ1XOQXafOTqHHWpk/OBNq/R6IJnOBYV33E4=,tag:thxrCPC5oGvDjhK7Dz87YA==,type:str]
     unencrypted_suffix: _unencrypted

variables.nix

@@ -26,7 +26,7 @@
   # fresh client that has never manually ssh'd to nix-cache before. Update
   # this if nix-cache's host key is ever rotated or the host is rebuilt
   # from scratch.
-  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
+  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache";
 
   # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
   # host (modules/nix-cache/server.nix) — one per client host that's allowed
@@ -38,6 +38,7 @@
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGtOWOCS+ImHc7NehguoyD7PbonGosKMZqc9+QR3v/h root@nixos"
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHxXTQxFnArK5HXG7czeoybZebCGfxpUdusJkPn+BCSp root@server"
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEj26SL/emsVjW2YhRucJVp2kTz8WgcEQgjBEBLRikk root@claude"
   ];
 
   # Admin SSH public key, authorized on the primary user of every host and