Fix nix-cache remote-builder trust: stale host key + wrong sshKey path

variables.nix's nixCacheHostKey no longer matched nix-cache's actual SSH
host key (confirmed via ssh-keyscan against the live container), so every
declaratively-configured client's programs.ssh.knownHosts trusted the
wrong key -- distributed builds would fail host-key verification. Also,
modules/nix-cache/remote-builder-client.nix hardcoded sshKey to
/root/.ssh/nixremote, but the `server` host only has its own default
/root/.ssh/id_ed25519 installed (confirmed live via qm guest-agent) --
that file was never even present, so the build machine config pointed at
nothing. Standardize on each client's own default identity, matching the
per-host-key pattern vars.remoteBuilderAuthorizedKeys already uses instead
of a shared/differently-named keypair, and add
scripts/secrets/sync-nix-cache-host-key.sh (wired into
codex-maintenance.sh's --check) so the host-key drift doesn't silently
recur next time nix-cache is rebuilt or recreated.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V7yVH71vGrDVzovh9UaMu8

.gitea/workflows/check-nixos.yml

@@ -13,17 +13,9 @@ jobs:
     steps:
       - name: Check out repository
         uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
 
       - name: Install Nix
         uses: DeterminateSystems/nix-installer-action@v19
 
-      # Scoped to files changed since the PR base / previous push -- see
+      - name: Run maintenance checks (secrets, fmt, lint, eval)
-      # scripts/codex-maintenance.sh. CI never passes --full-check: that
-      # full sweep is for local/manual use, since it's slow enough to time
-      # out this runner.
-      - name: Run maintenance checks (secrets, fmt, lint, eval -- changed files only)
-        env:
-          MAINT_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
         run: bash scripts/codex-maintenance.sh

.github/workflows/check-nixos.yml

@@ -13,17 +13,9 @@ jobs:
     steps:
       - name: Check out repository
         uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
 
       - name: Install Nix
         uses: DeterminateSystems/nix-installer-action@v19
 
-      # Scoped to files changed since the PR base / previous push -- see
+      - name: Run maintenance checks (secrets, fmt, lint, eval)
-      # scripts/codex-maintenance.sh. CI never passes --full-check: that
-      # full sweep is for local/manual use, since it's slow enough to time
-      # out this runner.
-      - name: Run maintenance checks (secrets, fmt, lint, eval -- changed files only)
-        env:
-          MAINT_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
         run: bash scripts/codex-maintenance.sh

AGENTS.md

@@ -35,14 +35,9 @@ Use these commands when validating changes:
 ```bash
 bash scripts/codex-setup.sh
 bash scripts/codex-maintenance.sh
+bash scripts/codex-maintenance.sh dry-run
 ```
 
-With no flags, `codex-maintenance.sh` scopes fmt-check/statix/eval to files
-changed against a base ref — this is what CI runs on every push/PR. For the
-full sweep (every host, every package — slow; CI never runs this), use
-`bash scripts/codex-maintenance.sh --full-check` (add `--dry-run` for build
-planning on top of whichever scope is active).
-
 Host evaluation is safe when limited to drvPath checks:
 
 ```bash

CLAUDE.md

@@ -37,22 +37,11 @@ machines when deployed.
 # One-time environment bootstrap (installs Nix if missing, prints hosts)
 bash scripts/codex-setup.sh
 
-# Changed-files-only validation: secret grep (whole repo), nixpkgs-fmt --check
+# Full validation: secret grep, nixpkgs-fmt --check, statix lint, eval all hosts
-# and statix on changed *.nix files, eval of the hosts/packages those changes
-# can affect. This is what CI runs on every push/PR.
 bash scripts/codex-maintenance.sh
 
-# Full sweep: nixpkgs-fmt --check/statix over the whole tree, eval every host
+# Same, plus a dry-run build (no result symlink) of every host's toplevel
-# and package. Slow (minutes) -- CI never runs this; use it locally before a
+bash scripts/codex-maintenance.sh dry-run
-# release or after touching modules/common/*, flake.nix, or variables.nix for
-# extra confidence beyond the automatic full-fallback those paths already
-# trigger in the default mode (see below).
-bash scripts/codex-maintenance.sh --full-check
-
-# Either mode, plus a dry-run build (no result symlink) of every host/package
-# in whichever scope is active
-bash scripts/codex-maintenance.sh --dry-run
-bash scripts/codex-maintenance.sh --full-check --dry-run
 
 # List the hosts the flake currently exposes
 nix eval --json .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]'
@@ -69,28 +58,18 @@ maintenance script pulls them via `nix run github:NixOS/nixpkgs/nixos-25.11#<too
 There is no test suite — "correctness" here means the flake evaluates and
 `nixpkgs-fmt`/`statix` are clean.
 
-With no flags, `codex-maintenance.sh` diffs against a base ref (env
+**In an interactive agent session**, prefer targeted checks over full-repo
-`MAINT_BASE_SHA`, else the PR base SHA in CI, else `HEAD^` locally) and scopes
+sweeps: after editing one or two hosts/modules, evaluate just the
-fmt-check/statix to the changed `*.nix` files and eval to the hosts/packages
+`nixosConfigurations.<host>` you touched (plus any `config.system.build.tarball`
-those changes can affect — a `hosts/<name>/host.nix` edit only evals that
+/`diskoImagesScript`/package output affected) rather than looping over every
-host's targets, a `modules/platforms/<platform>.nix` edit only evals that
+host — `codex-maintenance.sh` evaluates every `nixosConfigurations` host plus
-platform's hosts, and so on. A change to `flake.nix`, `flake.lock`,
+every package/tarball/image variant and is slow to run after each small
-`variables.nix`, `modules/common/*`, or any other `modules/*.nix` file outside
+change. Reserve a full
-`platforms/`/`build-types/` (whose blast radius isn't safely inferable from
+`codex-maintenance.sh` run for changes that plausibly affect every host
-the path alone) falls back to evaluating every host and package, same as
+(`modules/common/*`, `flake.nix`, `variables.nix`) or as a final check before
-`--full-check` would, just without the whole-tree fmt/statix sweep. This
+committing. This is a session-workflow preference only — it does not apply to
-exists because the whole-tree sweep is what was timing out CI; **CI always
+CI, which should keep running the full script on every push/PR regardless of
-runs the plain, no-flag form and never passes `--full-check`.**
+diff size; that's the point of it.
-
-The default mode's diff is against the working tree (uncommitted and staged
-edits included, not just committed ones), so it's already the right tool for
-an interactive session too: after editing one or two hosts/modules, plain
-`bash scripts/codex-maintenance.sh` naturally scopes to just what you
-touched. Reserve `--full-check` for changes that plausibly affect every host
-(`modules/common/*`, `flake.nix`, `variables.nix` — though the default mode
-already falls back to evaluating everything for those paths, `--full-check`
-additionally re-checks fmt/statix over the whole tree) or as a final check
-before committing.
 
 ## Scripts
 
@@ -137,6 +116,22 @@ lift the shared part into `scripts/lib/` instead of copying it.
   Purely a local filesystem copy — never touches `.sops.yaml`/
   `secrets/*.yaml` or the repo at all. The resulting file is exactly what
   `rotate-admin-key.sh` expects as its backup-key argument.
+- `scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run]
+  [--host <name>]` — detects drift between the ed25519 SSH host key
+  nix-cache is actually serving right now (via `ssh-keyscan`) and
+  `vars.nixCacheHostKey` (`variables.nix`), the value
+  `modules/nix-cache/remote-builder-client.nix` bakes into every real
+  client's declarative `programs.ssh.knownHosts` and
+  `configure-nix-cache-client.sh` hardcodes as its own default for
+  non-NixOS clients. That value has no automatic source of truth — it's
+  set once from whatever nix-cache's host key happened to be at the time,
+  and silently goes stale if the host is ever rebuilt/recreated with a new
+  key, breaking every client's distributed-build SSH trust with no error
+  that points back here. `--check` (used by `codex-maintenance.sh`, which
+  treats an unreachable nix-cache — e.g. from a non-LAN CI runner — as a
+  silent skip rather than a failure) only reports drift; the no-flags form
+  updates both files in place. Declarative clients still need a rebuild to
+  pick up the fix.
 
 ### `scripts/proxmox/`
 

README.md

@@ -74,19 +74,10 @@ Safe validation commands for Codex and local review:
 
 ```bash
 bash scripts/codex-setup.sh
+bash scripts/codex-maintenance.sh dry-run
 bash scripts/codex-maintenance.sh
 ```
 
-`codex-maintenance.sh` with no flags (what CI runs on every push/PR) scopes
-fmt-check/statix/eval to files changed against a base ref — fast, but only
-as thorough as the diff. For the full sweep (every host, every package,
-fmt-check and statix over the whole tree — slow, CI never runs this):
-
-```bash
-bash scripts/codex-maintenance.sh --full-check
-bash scripts/codex-maintenance.sh --full-check --dry-run
-```
-
 For individual host evaluation:
 
 ```bash

docs/flake-lock-automation.md

@@ -8,14 +8,9 @@ and to verify that declared NixOS hosts still evaluate after dependency updates.
 - A scheduled workflow runs `nix flake update` once per week.
 - On GitHub, any resulting `flake.lock` change is proposed through a pull request.
 - On Gitea, the workflow can commit and push `flake.lock` directly when PR automation is not configured.
-- A separate CI workflow runs `scripts/codex-maintenance.sh` before merge.
+- A separate CI workflow evaluates every configured host before merge, listed
-  Its default mode scopes eval to the hosts/packages a change can affect,
+  dynamically via `nix eval --json .#nixosConfigurations --apply builtins.attrNames`
-  determined from a git diff against the PR base — but a `flake.lock` change
+  rather than hand-enumerated, so it can't drift as `<platform>-<buildtype>`
-  is treated as repo-wide and always falls back to evaluating every host, so
-  a lock-file update PR still gets full coverage. Hosts are still listed
-  dynamically via
-  `nix eval --json .#nixosConfigurations --apply builtins.attrNames` rather
-  than hand-enumerated, so that fallback can't drift as `<platform>-<buildtype>`
   targets are added or removed. See `README.md` for the current target list.
 
 ## Why hosts should stop using `--upgrade-all`

docs/nix-cache.md

@@ -46,18 +46,26 @@ the new key up automatically on next activation — no more manual
 
 ## Remote builder SSH keys
 
-On each client, install the private key used to authenticate as `nixremote`:
+Each client authenticates as `nixremote` using its **own default root SSH
+identity** (`/root/.ssh/id_ed25519`) — not a separately-named or shared
+keypair. If a client doesn't have one yet:
 
 ```bash
-sudo install -d -m 0700 /root/.ssh
+sudo ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
-sudo install -m 0600 ./nixremote /root/.ssh/nixremote
-sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
 ```
 
-On `nix-cache`, install the matching public key used by `nixremote` authorized keys.
+Then add its `.pub` contents as a new entry in `vars.remoteBuilderAuthorizedKeys`
+(`variables.nix`) and rebuild `nix-cache` to pick it up (that list is
+declarative — an imperative `ssh-copy-id nixremote@nix-cache` won't stick;
+it gets overwritten on every rebuild). Verify with:
 
-The committed `nixremote` authorized keys are public SSH keys only. Keep the
+```bash
-matching private keys on client hosts and out of the repository.
+sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
+```
+
+The committed `remoteBuilderAuthorizedKeys` entries are public SSH keys
+only. Keep the matching private keys on client hosts and out of the
+repository.
 
 nix-cache's own SSH *host* key is trusted declaratively via
 `programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
@@ -76,8 +84,8 @@ After deployment:
 curl http://nix-cache/nix-cache-info
 nix store ping --store http://nix-cache
 nix show-config | grep -E 'substituters|trusted-public-keys|builders-use-substitutes'
-sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
+sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
-nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/nixremote 4 2 big-parallel,kvm,nixos-test,benchmark' -L
+nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/id_ed25519 4 2 big-parallel,kvm,nixos-test,benchmark' -L
 nix path-info -r nixpkgs#hello
 curl -I "http://nix-cache/$(basename "$(nix path-info nixpkgs#hello)").narinfo"
 ```

modules/nix-cache/remote-builder-client.nix

@@ -1,10 +1,14 @@
 { pkgs, vars, ... }:
 
 {
-  # Install the remote builder key on each client host (do not commit private keys):
+  # Authenticate as nixremote using the client host's own default root SSH
-  #   sudo install -d -m 0700 /root/.ssh
+  # identity (/root/.ssh/id_ed25519) rather than a separately-named key --
-  #   sudo install -m 0600 ./nixremote /root/.ssh/nixremote
+  # matches vars.remoteBuilderAuthorizedKeys, which already authorizes
-  #   sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
+  # each host's own default key (one entry per host, not a shared
+  # dedicated keypair). If this host doesn't have one yet:
+  #   sudo -u root ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
+  #   # then add its .pub to vars.remoteBuilderAuthorizedKeys and rebuild nix-cache
+  #   sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
   # Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
   # can connect the first time without a manual ssh-keyscan/known_hosts
   # step on every new client.
@@ -20,7 +24,7 @@
       {
         hostName = vars.nixCacheHost;
         sshUser = vars.remoteBuilderUser;
-        sshKey = "/root/.ssh/${vars.remoteBuilderUser}";
+        sshKey = "/root/.ssh/id_ed25519";
         inherit (pkgs.stdenv.hostPlatform) system;
         maxJobs = 4;
         speedFactor = 2;

scripts/bump-nixpkgs-release.sh

@@ -129,6 +129,5 @@ echo "flake.lock still points at the old input revisions until refreshed. Either
 echo "  nix flake update nixpkgs home-manager   # just these two inputs"
 echo "  nix flake update                        # everything — see docs/flake-lock-automation.md"
 echo
-echo "Then run 'bash scripts/codex-maintenance.sh --full-check --dry-run' before"
+echo "Then run 'bash scripts/codex-maintenance.sh dry-run' before committing —"
-echo "committing — a channel bump can shift option defaults across every host,"
+echo "a channel bump can shift option defaults across every host."
-echo "and only --dry-run actually builds anything to catch that."

scripts/codex-maintenance.sh

@@ -1,21 +1,4 @@
 #!/usr/bin/env bash
-# Validation entry point for CI and local/agent review.
-#
-# Default mode (what CI runs on every push/PR): fmt-check, statix, and eval
-# are scoped to files that actually changed against a base ref, plus
-# whichever hosts/packages those changes can affect. This exists because
-# the unscoped sweep below is slow enough to time out CI runners -- see
-# --full-check.
-#
-# --full-check: the historical full sweep (every host, every package,
-# fmt --check ./statix check . over the whole tree). Slow -- minutes, not
-# seconds. CI never passes this; run it locally before a release or after
-# touching modules/common/*, flake.nix, or variables.nix if you want extra
-# confidence beyond what the changed-files scope already covers for those
-# paths (see below).
-#
-# --dry-run: adds `nix build --dry-run --no-link` for whatever scope is
-# active (changed-files scope by default, full scope under --full-check).
 set -euo pipefail
 
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -24,39 +7,7 @@ source "${script_dir}/lib/nix-bootstrap.sh"
 # shellcheck source=lib/nix-eval.sh
 source "${script_dir}/lib/nix-eval.sh"
 
-repo_root="$(cd "${script_dir}/.." && pwd)"
+MODE="${1:-validate}"
-cd "$repo_root"
-
-full_check=false
-dry_run=false
-
-usage() {
-  cat <<'EOF'
-Usage: scripts/codex-maintenance.sh [--full-check] [--dry-run]
-
-  --full-check  Run the full sweep: fmt-check and statix over the whole
-                repo, eval every host and package. Slow. Never run by CI.
-  --dry-run     Additionally run `nix build --dry-run --no-link` for
-                whatever scope is active.
-
-With neither flag (the CI default), fmt-check/statix/eval are scoped to
-files changed against a base ref (env MAINT_BASE_SHA, else the PR base,
-else HEAD^), plus the hosts/packages those changes can affect.
-EOF
-}
-
-for arg in "$@"; do
-  case "$arg" in
-    --full-check) full_check=true ;;
-    --dry-run) dry_run=true ;;
-    -h|--help) usage; exit 0 ;;
-    *)
-      echo "Unknown argument: $arg" >&2
-      usage >&2
-      exit 1
-      ;;
-  esac
-done
 
 ensure_nix_profile
 
@@ -65,6 +16,12 @@ if ! command -v nix >/dev/null 2>&1; then
   exit 127
 fi
 
+hosts="$(list_flake_targets .)"
+
+echo "Hosts:"
+echo "$hosts"
+
+echo
 echo "Checking for obvious committed secrets..."
 if grep -RInE 'github_pat_|ghp_|access-tokens|hashedPassword[[:space:]]*=' \
   --exclude-dir=.git \
@@ -76,164 +33,31 @@ else
   echo "No obvious token patterns found."
 fi
 
-mapfile -t all_hosts < <(list_flake_targets .)
-mapfile -t all_packages < <(nix eval --json "${NIX_EVAL_FLAGS[@]}" .#packages.x86_64-linux --apply builtins.attrNames | jq -r '.[]')
-
-# host_targets_for_dir <hosts-subdir-name>
-# Prints the nixosConfigurations target names whose hostPath is
-# ./hosts/<dir>/host.nix, derived straight from flake.nix's generatedTargets
-# (one mkTarget { ... } call per line) rather than a hand-maintained table,
-# so it can't drift the way a copied mapping would.
-host_targets_for_dir() {
-  local dir="$1"
-  grep -oE '^[[:space:]]*[A-Za-z0-9_-]+ = mkTarget \{[^}]*hostPath = \./hosts/'"${dir}"'/host\.nix;[^}]*\};' flake.nix \
-    | sed -E 's/^[[:space:]]*([A-Za-z0-9_-]+) = mkTarget.*/\1/' \
-    || true
-}
-
-declare -a changed_files=()
-scope_desc="full repo"
-
-if ! $full_check; then
-  resolve_base_ref() {
-    if [[ -n "${MAINT_BASE_SHA:-}" ]] && git cat-file -e "${MAINT_BASE_SHA}^{commit}" 2>/dev/null; then
-      echo "$MAINT_BASE_SHA"
-      return
-    fi
-    if git rev-parse --verify -q HEAD^ >/dev/null 2>&1; then
-      echo "HEAD^"
-      return
-    fi
-    git hash-object -t tree /dev/null
-  }
-
-  base_ref="$(resolve_base_ref)"
-  echo
-  echo "Changed-files scope: diffing against ${base_ref}"
-  mapfile -t changed_files < <(git diff --name-only --diff-filter=ACMR "$base_ref" -- . | sort -u)
-
-  if [[ ${#changed_files[@]} -eq 0 ]]; then
-    echo "No changed files detected."
-  else
-    printf '  %s\n' "${changed_files[@]}"
-  fi
-  scope_desc="changed files only (base: ${base_ref})"
-fi
-
-# Whole-tree fmt/lint always run under --full-check; otherwise scoped below.
-declare -a changed_nix_files=()
-for f in "${changed_files[@]:-}"; do
-  [[ "$f" == *.nix && -f "$f" ]] && changed_nix_files+=("$f")
-done
-
 echo
 echo "Checking Nix formatting with nixpkgs-fmt..."
-if $full_check; then
+nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check .
-  nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check .
-elif [[ ${#changed_nix_files[@]} -gt 0 ]]; then
-  nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check "${changed_nix_files[@]}"
-else
-  echo "No changed .nix files; skipping."
-fi
 
 echo
 echo "Running statix lint..."
-if $full_check; then
+nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check .
-  nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check .
-elif [[ ${#changed_nix_files[@]} -gt 0 ]]; then
-  for f in "${changed_nix_files[@]}"; do
-    nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check "$f"
-  done
-else
-  echo "No changed .nix files; skipping."
-fi
-
-# Figure out which hosts/packages this run needs to eval (and, under
-# --dry-run, build). full_check always means "everything"; otherwise a
-# change to flake.nix/flake.lock/variables.nix/modules/common/* (repo-wide
-# inputs) or to any other modules/*.nix outside platforms//build-types
-# (whose blast radius isn't safely inferable from the path alone -- see
-# CLAUDE.md's "Grep modules/build-types/*.nix for each build type's imports
-# list") also falls back to everything, on the same reasoning CLAUDE.md
-# already gives interactive sessions for when to run the full sweep.
-# Anything more targeted -- a host.nix, a platform module, a build-type
-# module -- narrows to just the hosts it can affect.
-declare -A affected_hosts=()
-eval_packages=false
-
-if $full_check; then
-  for h in "${all_hosts[@]}"; do affected_hosts[$h]=1; done
-  eval_packages=true
-else
-  full_fallback=false
-  for f in "${changed_files[@]:-}"; do
-    case "$f" in
-      flake.nix|flake.lock|variables.nix|modules/common/*)
-        full_fallback=true
-        ;;
-    esac
-  done
-
-  if ! $full_fallback; then
-    for f in "${changed_files[@]:-}"; do
-      case "$f" in
-        hosts/*/*)
-          hostdir="${f#hosts/}"
-          hostdir="${hostdir%%/*}"
-          while IFS= read -r t; do
-            [[ -n "$t" ]] && affected_hosts[$t]=1
-          done < <(host_targets_for_dir "$hostdir")
-          ;;
-        modules/platforms/*.nix)
-          platform="$(basename "$f" .nix)"
-          for h in "${all_hosts[@]}"; do
-            [[ "$h" == "${platform}-"* ]] && affected_hosts[$h]=1
-          done
-          ;;
-        modules/build-types/*.nix)
-          buildtype="$(basename "$f" .nix)"
-          for h in "${all_hosts[@]}"; do
-            [[ "$h" == *"-${buildtype}" ]] && affected_hosts[$h]=1
-          done
-          ;;
-        modules/installer/*)
-          # iso.nix (imported by both the "installer" nixosConfigurations
-          # target and netbootSystem, which backs packages.pxe) pulls in
-          # common.nix, so a common.nix change reaches all three.
-          affected_hosts[installer]=1
-          eval_packages=true
-          ;;
-        modules/pxe-boot/*)
-          # stage-installer-artifacts.nix is imported by
-          # modules/build-types/pxe-boot.nix only -- same blast radius as a
-          # build-types/*.nix change, not a packages one.
-          for h in "${all_hosts[@]}"; do
-            [[ "$h" == *"-pxe-boot" ]] && affected_hosts[$h]=1
-          done
-          ;;
-        modules/*)
-          full_fallback=true
-          ;;
-      esac
-    done
-  fi
-
-  if $full_fallback; then
-    echo
-    echo "Changed files affect shared config; falling back to evaluating every host/package."
-    for h in "${all_hosts[@]}"; do affected_hosts[$h]=1; done
-    eval_packages=true
-  fi
-fi
-
-mapfile -t hosts < <(for h in "${!affected_hosts[@]}"; do echo "$h"; done | sort)
 
 echo
-if [[ ${#hosts[@]} -eq 0 ]]; then
+echo "Checking nix-cache host key for drift..."
-  echo "No hosts affected by changed files; skipping host eval."
+if bash "${script_dir}/secrets/sync-nix-cache-host-key.sh" --check; then
+  :
 else
-  echo "Evaluating host toplevel derivations (${scope_desc})..."
+  drift_status=$?
-  for host in "${hosts[@]}"; do
+  if [[ "$drift_status" -eq 2 ]]; then
+    echo "nix-cache unreachable from here -- skipping host-key drift check."
+  else
+    echo "WARNING: nix-cache's host key has drifted from variables.nix (see above)." >&2
+    echo "  Run 'bash scripts/secrets/sync-nix-cache-host-key.sh' to fix." >&2
+  fi
+fi
+
+echo
+echo "Evaluating host toplevel derivations..."
+for host in $hosts; do
   echo "==> $host"
   nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
 
@@ -252,24 +76,20 @@ else
       nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript.drvPath"
       ;;
   esac
-  done
+done
-fi
 
 echo
-if ! $eval_packages; then
+echo "Evaluating buildable packages..."
-  echo "No packages affected by changed files; skipping package eval."
+packages="$(nix eval --json "${NIX_EVAL_FLAGS[@]}" .#packages.x86_64-linux --apply builtins.attrNames | jq -r '.[]')"
-else
+for pkg in $packages; do
-  echo "Evaluating buildable packages..."
-  for pkg in "${all_packages[@]}"; do
   echo "==> packages.x86_64-linux.${pkg}"
   nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
-  done
+done
-fi
 
-if $dry_run; then
+if [[ "$MODE" == "dry-run" ]]; then
   echo
-  echo "Running dry-run builds for the active scope. This will not create result symlinks."
+  echo "Running dry-run builds for all hosts. This will not create result symlinks."
-  for host in "${hosts[@]:-}"; do
+  for host in $hosts; do
     echo "==> Dry-run build: $host"
     nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel"
 
@@ -285,14 +105,12 @@ if $dry_run; then
     esac
   done
 
-  if $eval_packages; then
   echo
-    echo "Running dry-run builds for packages."
+  echo "Running dry-run builds for all packages."
-    for pkg in "${all_packages[@]}"; do
+  for pkg in $packages; do
     echo "==> Dry-run build: packages.x86_64-linux.${pkg}"
     nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
   done
-  fi
 fi
 
 echo

scripts/codex-setup.sh

@@ -82,6 +82,13 @@ if ! command -v jq >/dev/null 2>&1; then
 fi
 
 echo "Available NixOS hosts:"
-list_flake_targets .
+hosts="$(list_flake_targets .)"
+echo "$hosts"
 
-echo "Codex setup complete. Run bash scripts/codex-maintenance.sh to validate changes."
+echo "Evaluating all host toplevel derivations..."
+for host in $hosts; do
+  echo "==> Evaluating $host"
+  nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
+done
+
+echo "Codex setup complete."

scripts/proxmox/configure-nix-cache-client.sh

@@ -15,6 +15,16 @@
 # this repo -- update both if nix-cache is ever rebuilt with a new host
 # key or the cache signing key is rotated (see docs/nix-cache.md).
 #
+# REMOTE_BUILDER_KEY defaults to this machine's own default root SSH
+# identity (matches modules/nix-cache/remote-builder-client.nix's
+# convention for real NixOS clients: authenticate as nixremote with the
+# host's own default key, added individually to
+# vars.remoteBuilderAuthorizedKeys, rather than a separately-named or
+# shared keypair) -- generate one with
+# `ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519` if this machine
+# doesn't have one yet, then add its .pub to vars.remoteBuilderAuthorizedKeys
+# and rebuild nix-cache.
+#
 # Usage:
 #   sudo ./configure-nix-cache-client.sh [--dry-run] [--no-remote-builder] [--no-restart]
 #
@@ -24,9 +34,9 @@
 set -euo pipefail
 
 : "${NIX_CACHE_HOST:=nix-cache}"
-: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache}"
+: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache}"
 : "${REMOTE_BUILDER_USER:=nixremote}"
-: "${REMOTE_BUILDER_KEY:=/root/.ssh/nixremote}"
+: "${REMOTE_BUILDER_KEY:=/root/.ssh/id_ed25519}"
 
 CACHE_PUB_KEY="cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4="
 FALLBACK_URL="https://cache.nixos.org/"

scripts/secrets/sync-nix-cache-host-key.sh

@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+# Detects and fixes drift between the ed25519 SSH host key nix-cache is
+# actually serving right now and vars.nixCacheHostKey (variables.nix) --
+# the value modules/nix-cache/remote-builder-client.nix bakes into every
+# client's declarative programs.ssh.knownHosts, and
+# scripts/proxmox/configure-nix-cache-client.sh hardcodes as its own
+# default for non-NixOS clients.
+#
+# This value has no automatic source of truth: nix-cache's host key is
+# generated once (first boot / container recreate) and never touches this
+# repo again unless someone remembers to update it by hand afterwards. It
+# drifted silently once already -- confirmed live: variables.nix recorded
+# a key that no longer matched what nix-cache actually presented, which
+# would fail every real client's SSH host-key verification for
+# distributed builds without ever producing an obvious error pointing
+# back here (a client just sees "Host key verification failed" against
+# *some* key, with no hint that the trusted value itself was stale).
+#
+# codex-maintenance.sh runs this in --check mode on every invocation so
+# that drift surfaces as a warning instead of a future debugging session.
+#
+# Usage:
+#   scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run] [--host <name>]
+#
+#   --check     Only report drift (exit 1 if found, 2 if nix-cache is
+#               unreachable); never writes. For CI/maintenance use.
+#   --dry-run   Show what would change; never writes.
+#   --host      Override the hostname to scan (default: variables.nix's
+#               nixCacheHost / env.sh's NIX_CACHE_HOST).
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
+variables_nix="${repo_root}/variables.nix"
+client_script="${repo_root}/scripts/proxmox/configure-nix-cache-client.sh"
+
+# shellcheck source=../env.sh
+source "${repo_root}/scripts/env.sh"
+
+check_only=0
+dry_run=0
+host="${NIX_CACHE_HOST}"
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --check) check_only=1; shift ;;
+    --dry-run) dry_run=1; shift ;;
+    --host)
+      host="${2:?--host requires a hostname}"
+      shift 2
+      ;;
+    -h|--help)
+      sed -n '2,23p' "$0"
+      exit 0
+      ;;
+    *)
+      echo "ERROR: unknown argument: $1" >&2
+      exit 1
+      ;;
+  esac
+done
+
+current_value="$(grep -oE 'nixCacheHostKey = "[^"]+"' "$variables_nix" | sed -E 's/nixCacheHostKey = "(.*)"/\1/')"
+if [[ -z "$current_value" ]]; then
+  echo "ERROR: couldn't find nixCacheHostKey in $variables_nix" >&2
+  exit 1
+fi
+current_type_blob="$(awk '{print $1, $2}' <<<"$current_value")"
+current_label="$(awk '{print $3}' <<<"$current_value")"
+
+echo "Scanning ${host} for its current ed25519 SSH host key..."
+nix_extra_opts
+scanned="$(nix-shell "${NIX_OPTS[@]}" -p openssh --run "ssh-keyscan -t ed25519 -T 5 '${host}'" 2>/dev/null | grep -v '^#' | head -1 || true)"
+if [[ -z "$scanned" ]]; then
+  echo "ERROR: couldn't reach ${host} (or got no ed25519 host key back) via ssh-keyscan." >&2
+  exit 2
+fi
+scanned_type_blob="$(awk '{print $2, $3}' <<<"$scanned")"
+
+if [[ "$current_type_blob" == "$scanned_type_blob" ]]; then
+  echo "Up to date: ${host}'s host key matches variables.nix's nixCacheHostKey."
+  exit 0
+fi
+
+echo "DRIFT DETECTED:"
+echo "  variables.nix has: $current_type_blob"
+echo "  ${host} is now:    $scanned_type_blob"
+
+if [[ "$check_only" -eq 1 ]]; then
+  echo
+  echo "Run 'scripts/secrets/sync-nix-cache-host-key.sh' (no flags) to fix." >&2
+  exit 1
+fi
+
+new_value="${scanned_type_blob} ${current_label}"
+
+if [[ "$dry_run" -eq 1 ]]; then
+  echo "(--dry-run: would update variables.nix and ${client_script##*/} to:)"
+  echo "  $new_value"
+  exit 0
+fi
+
+sed -i "s|nixCacheHostKey = \"[^\"]*\"|nixCacheHostKey = \"${new_value}\"|" "$variables_nix"
+sed -i "s|NIX_CACHE_HOST_KEY:=[^}]*}|NIX_CACHE_HOST_KEY:=${new_value}}|" "$client_script"
+
+echo "Updated variables.nix and ${client_script##*/} to:"
+echo "  $new_value"
+echo
+echo "This only takes effect on already-deployed NixOS clients after their"
+echo "next rebuild (programs.ssh.knownHosts is declarative). Review with"
+echo "'git diff', then run 'bash scripts/codex-maintenance.sh' before committing."

variables.nix

@@ -26,7 +26,7 @@
   # fresh client that has never manually ssh'd to nix-cache before. Update
   # this if nix-cache's host key is ever rotated or the host is rebuilt
   # from scratch.
-  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
+  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache";
 
   # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
   # host (modules/nix-cache/server.nix) — one per client host that's allowed