Re-encrypt secrets/nix-cache.yaml for current lxc-nix-cache key

.sops.yaml's &lxc-nix-cache anchor was rotated (commit b3c8145) after
today's LXC rebuild, but secrets/nix-cache.yaml was never re-run through
sops updatekeys -- it stayed encrypted for the pre-rotation key. Per
docs/nix-cache.md and docs/auto-installer.md, sops-nix decrypt failures
are silent: cache-priv-key never materialized, nix-serve had nothing to
bind its secretKeyFile to and failed to start, and nginx proxy_pass had
no upstream -- the 502 Bad Gateway seen from clients.

Confirmed the new key matches the currently-deployed host by deriving
the age identity from host-keys/lxc-nix-cache_ssh_host_ed25519_key.pub
via ssh-to-age and comparing to .sops.yaml.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CLAUDE.md

@@ -97,10 +97,29 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
   Refuses to create a target whose host identity already exists live on
   the node (checked directly via `qm`/`pct`, not any file in this repo)
   unless `--allow-duplicate-host` is passed. `--dry-run` throughout both
-  modes.
+  modes. The first time it has to bootstrap build tooling on a node (i.e.
+  `nix` wasn't already on its `PATH`), it also runs
+  `scripts/configure-nix-cache-client.sh` there (non-fatally — a failure
+  just falls back to building from source / `cache.nixos.org`) so the
+  node substitutes from and can offload builds to nix-cache on every
+  subsequent run, not just this one.
 - `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
   default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
   cross-script config here instead of duplicating it per-script.
+- `scripts/configure-nix-cache-client.sh [--dry-run] [--no-remote-builder]
+  [--no-restart]` — the non-NixOS equivalent of
+  `modules/nix-cache/client.nix`/`remote-builder-client.nix`, for a plain
+  Debian machine with the Nix package manager (not NixOS) already
+  installed: run as root *on that machine* to add nix-cache as a
+  substituter in `/etc/nix/nix.conf` (`https://cache.nixos.org/` kept as
+  fallback) via `extra-substituters`/`extra-trusted-public-keys` so it
+  layers on top of whatever's already there instead of clobbering it, and,
+  if `/root/.ssh/nixremote` is already present (see docs/nix-cache.md
+  "Remote builder SSH keys"), configures it as a distributed-build
+  machine too and trusts nix-cache's SSH host key in
+  `/etc/ssh/ssh_known_hosts`. Idempotent (re-running replaces its own
+  marked block rather than duplicating it); restarts `nix-daemon` by
+  default so the change takes effect immediately.
 - `scripts/bump-nixpkgs-release.sh` — bumps `flake.nix`'s `nixpkgs.url`/
   `home-manager.url` in place. Exists because flake input URLs can't
   reference `variables.nix` (confirmed empirically — `nix flake metadata`

scripts/configure-nix-cache-client.sh

@@ -0,0 +1,172 @@
+#!/usr/bin/env bash
+# Points a non-NixOS Debian machine's Nix install at nix-cache: adds it as
+# a substituter (with cache.nixos.org kept as fallback) and, once the
+# remote-builder private key is installed, as a distributed-build machine
+# too.
+#
+# This is the non-NixOS equivalent of modules/nix-cache/client.nix +
+# modules/nix-cache/remote-builder-client.nix -- those two only apply to
+# hosts built from this flake. A plain Debian box with Nix installed
+# (single- or multi-user install, nix-daemon running) has no NixOS module
+# system to pick that config up, so this edits /etc/nix/nix.conf by hand
+# instead. Run this ON the target Debian machine, as root.
+#
+# The values below mirror variables.nix / modules/nix-cache/client.nix in
+# this repo -- update both if nix-cache is ever rebuilt with a new host
+# key or the cache signing key is rotated (see docs/nix-cache.md).
+#
+# Usage:
+#   sudo ./configure-nix-cache-client.sh [--dry-run] [--no-remote-builder] [--no-restart]
+#
+# Env overrides (defaults match variables.nix):
+#   NIX_CACHE_HOST, NIX_CACHE_HOST_KEY, REMOTE_BUILDER_USER, REMOTE_BUILDER_KEY
+
+set -euo pipefail
+
+: "${NIX_CACHE_HOST:=nix-cache}"
+: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache}"
+: "${REMOTE_BUILDER_USER:=nixremote}"
+: "${REMOTE_BUILDER_KEY:=/root/.ssh/nixremote}"
+
+CACHE_PUB_KEY="cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4="
+FALLBACK_URL="https://cache.nixos.org/"
+FALLBACK_PUB_KEY="cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="
+
+NIX_CONF="/etc/nix/nix.conf"
+KNOWN_HOSTS="/etc/ssh/ssh_known_hosts"
+MARKER_BEGIN="# BEGIN nix-cache client config (configure-nix-cache-client.sh)"
+MARKER_END="# END nix-cache client config"
+
+dry_run=0
+with_remote_builder=1
+restart_daemon=1
+
+for arg in "$@"; do
+  case "$arg" in
+    --dry-run) dry_run=1 ;;
+    --no-remote-builder) with_remote_builder=0 ;;
+    --no-restart) restart_daemon=0 ;;
+    -h|--help)
+      sed -n '2,20p' "$0"
+      exit 0
+      ;;
+    *)
+      echo "ERROR: unknown argument: $arg" >&2
+      exit 1
+      ;;
+  esac
+done
+
+if [[ "$dry_run" -eq 0 && "$EUID" -ne 0 ]]; then
+  echo "ERROR: must run as root (writes $NIX_CONF and, unless --no-remote-builder, $KNOWN_HOSTS)." >&2
+  exit 1
+fi
+
+if ! command -v nix >/dev/null 2>&1; then
+  echo "ERROR: no 'nix' binary on PATH -- install the Nix package manager first." >&2
+  exit 1
+fi
+
+if [[ ! -f "$NIX_CONF" ]]; then
+  echo "ERROR: $NIX_CONF not found -- expected an existing multi-user Nix install." >&2
+  exit 1
+fi
+
+builder_line=""
+if [[ "$with_remote_builder" -eq 1 ]]; then
+  if [[ -f "$REMOTE_BUILDER_KEY" ]]; then
+    case "$(uname -m)" in
+      x86_64) nix_system="x86_64-linux" ;;
+      aarch64) nix_system="aarch64-linux" ;;
+      *)
+        echo "WARNING: unrecognized architecture '$(uname -m)' -- skipping remote builder, keeping substituter config." >&2
+        with_remote_builder=0
+        ;;
+    esac
+    if [[ "$with_remote_builder" -eq 1 ]]; then
+      builder_line="builders = ssh://${REMOTE_BUILDER_USER}@${NIX_CACHE_HOST} ${nix_system} ${REMOTE_BUILDER_KEY} 4 2 big-parallel,kvm,nixos-test,benchmark"
+    fi
+  else
+    echo "WARNING: $REMOTE_BUILDER_KEY not found -- skipping remote builder config (substituter still configured)." >&2
+    echo "  See docs/nix-cache.md 'Remote builder SSH keys' for how to install it, then re-run this script." >&2
+    with_remote_builder=0
+  fi
+fi
+
+block="$(cat <<EOF
+$MARKER_BEGIN
+extra-substituters = http://${NIX_CACHE_HOST} ${FALLBACK_URL}
+extra-trusted-public-keys = ${CACHE_PUB_KEY} ${FALLBACK_PUB_KEY}
+EOF
+)"
+if [[ "$with_remote_builder" -eq 1 ]]; then
+  block="${block}
+builders-use-substitutes = true
+${builder_line}"
+fi
+block="${block}
+$MARKER_END"
+
+echo "== nix.conf block to install =="
+echo "$block"
+echo "================================"
+
+if [[ "$dry_run" -eq 1 ]]; then
+  echo "(--dry-run: not writing $NIX_CONF)"
+else
+  tmp_conf="$(mktemp)"
+  trap 'rm -f "$tmp_conf"' EXIT
+
+  if grep -qF "$MARKER_BEGIN" "$NIX_CONF"; then
+    awk -v begin="$MARKER_BEGIN" -v end="$MARKER_END" -v block="$block" '
+      $0 == begin { print block; skip = 1; next }
+      $0 == end   { skip = 0; next }
+      skip        { next }
+      { print }
+    ' "$NIX_CONF" > "$tmp_conf"
+  else
+    cp "$NIX_CONF" "$tmp_conf"
+    printf '\n%s\n' "$block" >> "$tmp_conf"
+  fi
+
+  cp "$NIX_CONF" "${NIX_CONF}.bak.$(date +%Y%m%d%H%M%S)"
+  install -m 0644 "$tmp_conf" "$NIX_CONF"
+  echo "Updated $NIX_CONF (backup saved alongside it)."
+fi
+
+if [[ "$with_remote_builder" -eq 1 ]]; then
+  known_hosts_line="${NIX_CACHE_HOST} ${NIX_CACHE_HOST_KEY}"
+  if [[ "$dry_run" -eq 1 ]]; then
+    echo "(--dry-run: would ensure this line is present in $KNOWN_HOSTS)"
+    echo "  $known_hosts_line"
+  else
+    mkdir -p "$(dirname "$KNOWN_HOSTS")"
+    touch "$KNOWN_HOSTS"
+    if ! grep -qF "$known_hosts_line" "$KNOWN_HOSTS" 2>/dev/null; then
+      echo "$known_hosts_line" >> "$KNOWN_HOSTS"
+      echo "Added nix-cache's SSH host key to $KNOWN_HOSTS."
+    fi
+  fi
+fi
+
+if [[ "$dry_run" -eq 0 && "$restart_daemon" -eq 1 ]]; then
+  if command -v systemctl >/dev/null 2>&1 && systemctl is-active --quiet nix-daemon 2>/dev/null; then
+    systemctl restart nix-daemon
+    echo "Restarted nix-daemon to pick up the new config."
+  else
+    echo "nix-daemon not managed by systemd (or not running) -- restart it manually to pick up the new config."
+  fi
+fi
+
+cat <<EOF
+
+Done. Verify with:
+  curl http://${NIX_CACHE_HOST}/nix-cache-info
+  nix show-config | grep -E 'substituters|trusted-public-keys|builders'
+EOF
+if [[ "$with_remote_builder" -eq 1 ]]; then
+  cat <<EOF
+  ssh -i ${REMOTE_BUILDER_KEY} ${REMOTE_BUILDER_USER}@${NIX_CACHE_HOST} nix-store --version
+  nix build nixpkgs#hello -L
+EOF
+fi

scripts/create-proxmox-resource.sh

@@ -524,7 +524,7 @@ ensure_remote_repo() {
   echo
   echo "==> Ensuring ${remote_repo_dir} exists and is current on ${node}..."
   if [[ "$dry_run" -eq 1 ]]; then
-    echo "[dry-run] would ensure ${remote_repo_dir} exists on ${node} (clone if missing, git pull if present), and would verify/bootstrap build tooling there (scripts/codex-setup.sh) if \`nix\` isn't already on PATH"
+    echo "[dry-run] would ensure ${remote_repo_dir} exists on ${node} (clone if missing, git pull if present), and would verify/bootstrap build tooling there (scripts/codex-setup.sh) if \`nix\` isn't already on PATH -- and if that bootstrap actually ran, would also configure ${node} as a nix-cache client (scripts/configure-nix-cache-client.sh)"
     return
   fi
 
@@ -571,6 +571,21 @@ ensure_remote_repo() {
   else
     echo "==> Bootstrapping build tooling on ${node} (scripts/codex-setup.sh)..."
     ssh "$ssh_target" "cd '${remote_repo_dir}' && bash scripts/codex-setup.sh"
+
+    # Only on this first-time bootstrap, not every run -- a node that
+    # already has tooling either already went through this once, or had
+    # it configured some other way, and re-running is harmless but
+    # pointless. Non-fatal: this only makes the node's own builds faster
+    # (substitute from nix-cache instead of building from source) and
+    # offloadable to it as a remote builder -- worth trying, not worth
+    # aborting the image build over if nix-cache happens to be down right
+    # now. Needs ensure_nix_profile first, same as the tooling_check_cmd
+    # above -- ssh's non-interactive command execution won't have picked
+    # up a freshly single-user-installed `nix` otherwise.
+    echo "==> Configuring ${node} as a nix-cache substituter/remote-builder client..."
+    if ! ssh "$ssh_target" "cd '${remote_repo_dir}' && . scripts/lib/nix-bootstrap.sh && ensure_nix_profile && bash scripts/configure-nix-cache-client.sh"; then
+      echo "WARNING: configure-nix-cache-client.sh failed on ${node} -- continuing without it (${node} will build from source / against cache.nixos.org only)." >&2
+    fi
   fi
 }
 

secrets/nix-cache.yaml

@@ -4,31 +4,31 @@ sops:
     age:
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmOHlPcGZvN3o5aEZCcFdz
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBPWE1HTUhiSUp5ZEUwWEpI
-            WlYydFdoQ0ZHbElMdTRkZWljMmoxTHZqNjJZCnE3cmgzKzZya3FROEczbHVveDND
+            bGpkZlBIMUo5ZlYrQ09SN3Q1a0ZkQ0ZnOEhVCnJPNEZQenVWWGZiODlzQzNEc1Zq
-            VnEzRW12cnRKdzhkZm5uTXpkTEtrNUEKLS0tIHFDbkc4Mk4wVlM2R29zZXYwR2Ri
+            c3l4OWZJTElJc2Y2UE15OGtEUzhyY1EKLS0tIHNJUStyWnlQWjZBbEZjQ3UwdUpz
-            ODBML1p4eUZiZldYUERQTUhTU1llV0UKxjvH6zbW6wKghzR1o34CyKPEa2FqZmo0
+            ZndoUDR6bisrNGJCUHk3TGI4bTZaMFUK87fFsm9ne9s+PK2pcwtrDjqyGBss2r2E
-            PxgqyuXkIwas9soXVAkScx7ElaV09Fjaj+mDrKwi4a+DwdoSP7czyA==
+            8lhqoeiKZ2j96z8kP/7ChzovwTCmqdcmAQuyNQD+ZAFijseipSvfbQ==
             -----END AGE ENCRYPTED FILE-----
           recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0VDBXYUZNbnpEZHlqV05C
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUYi9SRFFGV3Z6cFd2Znk5
-            WWxQZjVYcWlZeXlGc0RiaUpERzR0T21CWWtBCkM0dW5kYlFNK1RGSzRudFZ2Z0t3
+            b2FLbWtzTllJMDBUaGk0NTViOTNBa2hQclVZClZHKzNhbGVjQUJhWkFWdTFBMG5a
-            aVBreElGM1BIT0RzQkxTVTA3S2NhQncKLS0tIERyTU8zWWRTRm95SmRZQ1BhalVV
+            cUFJdUdyVG5HQXJRRnJId3hqRTN2cXMKLS0tIEFMRjh3WE1ON0U2TTNTZ3hxMTR4
-            SVdBajN1V1BuM2s4K216S3c0VHczNlkKM5jvsSEfCBA5uZRjBJNbM91lLRQkj+jK
+            ZGRlemlIbDZKeExmVHROc3Eyak5DdzQKaLwIVDi6BN4cxpVxJoqTYvJETPOp4thc
-            rM5uSfGLTvSjPgXIMIq03OXxH1CE7GoKxAPwFrJdAFMMQcutIethhw==
+            l9uVMvIGuEsEZgDsvShw1dYLljd+uGy/A+dXbcxIUCP/mmPkwmd1Pw==
             -----END AGE ENCRYPTED FILE-----
           recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvbk9tRE9jYkpxNThQeElH
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArOWovSW9DeFpxL0VDUDQ3
-            dWpKYThDb1ZkUWtMWlRxc2tYbjhBbEFEcWtVCkNiK0NBRko2TkJQdUVtdGtHNGcv
+            SHUwTzJVZUtPV01ZRkdCUXZGL2lTRCtCNFNnCjBSNExqRW5mTEN5SFVucHJHSzZt
-            andSaUlKSFA4THRyZXNTYmI0Yk5WbmsKLS0tIHlQTUtpWDdPeTVZL0M1RElRdFNk
+            cDlNc3BjY3M1c1k1Z2tkVEg4R1pacGsKLS0tIEFWbHNKZW0vbVh1Y2VhQW93OUwx
-            QWlGdFo5NkZWSmY0YXdpNzNUQnlsK3MKtzC0bM7Ek+K73nMranOA1Mc98RUnYnq1
+            MWV0eW9sOXdQd0l2ZjlWOEVVc1dwcTgK2s4p9xoNkawH2OkGsl80bNIo3ad5vn4W
-            hAt0QEFKWK4QVKubaN/rG3AzE0U7qPKWHTzoxgnAiL3WyV9teLW+iA==
+            Z2w+jwppSoUmbQnD3WFbLmSSxmuobmU8HILwElv6SZu+KE3aspF6XA==
             -----END AGE ENCRYPTED FILE-----
-          recipient: age164px2a8e48ptsf9ngtan38aa6jls4jdl26mzrgzf6sn3vcvt49hqjrgr8w
+          recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
     lastmodified: "2026-07-19T23:30:21Z"
     mac: ENC[AES256_GCM,data:kLGE2xawQT7mx+sfw68hmGk5nCEGiEjZrqTEl9B1dtQmTrMwmoVr/1RISi4LfJrwxy31mDgff4lcIL4wIJuM373uk3X8j4RNyYQNTfKEkORT6r8NHeepNs267O77pKGd7OmcM4MT/BqOnB8ELS7Wlf2ect7CAlvUUVyc8icxgZE=,iv:EYLDsHYHZ1XOQXafOTqHHWpk/OBNq/R6IJnOBYV33E4=,tag:thxrCPC5oGvDjhK7Dz87YA==,type:str]
     unencrypted_suffix: _unencrypted