Make lxc-docker a privileged container: unprivileged can't NFS-mount at all

The kernel's NFS client filesystem doesn't set FS_USERNS_MOUNT, so mounting
NFS from inside any non-init user namespace -- exactly what an unprivileged
LXC container's UID-mapped root runs in -- is rejected at the VFS layer
with EPERM, regardless of Proxmox's mount=nfs;nfs4 container feature (which
only patches the AppArmor layer). Confirmed live on the redeployed lxc-docker
container: TCP to the NFS server's port 2049 succeeds, the server's export
table matches the container's IP, and mount.nfs: Operation not permitted
still fires immediately with no corresponding denial anywhere in the
server's own logs -- a kernel-level rejection that no amount of DNS/
automount/export tweaking (this branch's earlier commits) could ever fix.

modules/platforms/lxc.nix now keys proxmoxLXC.privileged off hostName
("docker" -> true) rather than a blanket false, since build-types/docker.nix
is also composed for linode-docker/proxmox-docker, which don't import
proxmox-lxc.nix at all -- setting this option there would break their eval.
create-proxmox-resource.sh reads the value back via a new
flake_target_lxc_privileged helper instead of hardcoding --unprivileged 1,
so the two stay in sync automatically for every lxc-* target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE

.sops.yaml

@@ -3,11 +3,12 @@ keys:
   - &docker age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e
   - &server age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
   - &nix-cache age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
-  - &lxc-minimal age1qz9d4ka4xgexujyd247s7lp737sulp5fhxl5d65fj2ykvc4j4edqrsdks8
   - &nix-minimal age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu
   - &proxmox-minimal age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg
   - &lxc-nix-cache age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
   - &lxc-docker age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs
+  - &lxc-minimal age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep
+  - &lxc-pxe-boot age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp
 
 creation_rules:
   # Shared across every currently-deployed host: root/nixos password hash,
@@ -26,6 +27,7 @@ creation_rules:
           - *lxc-nix-cache
           - *proxmox-minimal
           - *lxc-docker
+          - *lxc-pxe-boot
 
   - path_regex: secrets/nix-cache\.yaml$
     key_groups:

CLAUDE.md

@@ -137,6 +137,22 @@ lift the shared part into `scripts/lib/` instead of copying it.
   Purely a local filesystem copy — never touches `.sops.yaml`/
   `secrets/*.yaml` or the repo at all. The resulting file is exactly what
   `rotate-admin-key.sh` expects as its backup-key argument.
+- `scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run]
+  [--host <name>]` — detects drift between the ed25519 SSH host key
+  nix-cache is actually serving right now (via `ssh-keyscan`) and
+  `vars.nixCacheHostKey` (`variables.nix`), the value
+  `modules/nix-cache/remote-builder-client.nix` bakes into every real
+  client's declarative `programs.ssh.knownHosts` and
+  `configure-nix-cache-client.sh` hardcodes as its own default for
+  non-NixOS clients. That value has no automatic source of truth — it's
+  set once from whatever nix-cache's host key happened to be at the time,
+  and silently goes stale if the host is ever rebuilt/recreated with a new
+  key, breaking every client's distributed-build SSH trust with no error
+  that points back here. `--check` (used by `codex-maintenance.sh`, which
+  treats an unreachable nix-cache — e.g. from a non-LAN CI runner — as a
+  silent skip rather than a failure) only reports drift; the no-flags form
+  updates both files in place. Declarative clients still need a rebuild to
+  pick up the fix.
 
 ### `scripts/proxmox/`
 

docs/nix-cache.md

@@ -46,18 +46,26 @@ the new key up automatically on next activation — no more manual
 
 ## Remote builder SSH keys
 
-On each client, install the private key used to authenticate as `nixremote`:
+Each client authenticates as `nixremote` using its **own default root SSH
+identity** (`/root/.ssh/id_ed25519`) — not a separately-named or shared
+keypair. If a client doesn't have one yet:
 
 ```bash
-sudo install -d -m 0700 /root/.ssh
-sudo install -m 0600 ./nixremote /root/.ssh/nixremote
-sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
+sudo ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
 ```
 
-On `nix-cache`, install the matching public key used by `nixremote` authorized keys.
+Then add its `.pub` contents as a new entry in `vars.remoteBuilderAuthorizedKeys`
+(`variables.nix`) and rebuild `nix-cache` to pick it up (that list is
+declarative — an imperative `ssh-copy-id nixremote@nix-cache` won't stick;
+it gets overwritten on every rebuild). Verify with:
 
-The committed `nixremote` authorized keys are public SSH keys only. Keep the
-matching private keys on client hosts and out of the repository.
+```bash
+sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
+```
+
+The committed `remoteBuilderAuthorizedKeys` entries are public SSH keys
+only. Keep the matching private keys on client hosts and out of the
+repository.
 
 nix-cache's own SSH *host* key is trusted declaratively via
 `programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
@@ -76,8 +84,8 @@ After deployment:
 curl http://nix-cache/nix-cache-info
 nix store ping --store http://nix-cache
 nix show-config | grep -E 'substituters|trusted-public-keys|builders-use-substitutes'
-sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
-nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/nixremote 4 2 big-parallel,kvm,nixos-test,benchmark' -L
+sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
+nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/id_ed25519 4 2 big-parallel,kvm,nixos-test,benchmark' -L
 nix path-info -r nixpkgs#hello
 curl -I "http://nix-cache/$(basename "$(nix path-info nixpkgs#hello)").narinfo"
 ```

modules/docker/mount-data.nix

@@ -9,11 +9,25 @@ let
   # (the VM platforms rely on automount itself to get that same
   # non-blocking behavior, so they don't need `nofail` too).
   automountOpts = if config.boot.isContainer then [ "nofail" ] else [ "x-systemd.automount" ];
+
+  # A bare hostname here never resolves reliably: systemd-resolved only
+  # ever tries LLMNR for single-label names (never DNS, regardless of any
+  # configured search domain), and a *global* search domain (the first fix
+  # attempted here) backfires worse -- confirmed live on lxc-docker, adding
+  # `networking.search` made systemd-resolved prioritize its domain-matched
+  # but server-less global scope over eth0's correctly-configured one for
+  # every "*.sweet.home" query, silently sending them to public fallback
+  # DNS instead. `resolvectl query --interface=eth0 server.sweet.home`
+  # resolved fine throughout, proving the LAN DNS server was never the
+  # problem -- only the ambient, unqualified device string was. Using the
+  # FQDN directly sidesteps all of that, matching the pattern
+  # ../raspi/mount-data.nix already uses for the same reason.
+  nfsServer = "${vars.nfsServerHost}.${vars.homeDomain}";
 in
 {
   fileSystems = {
     ${vars.nfsShares.dockerConfig.mountpoint} = {
-      device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath}";
+      device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath}";
       fsType = "nfs";
 
       options = [
@@ -24,7 +38,7 @@ in
     };
 
     ${vars.nfsShares.dockerDatabases.mountpoint} = {
-      device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath}";
+      device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath}";
       fsType = "nfs";
 
       options = [
@@ -35,7 +49,7 @@ in
     };
 
     ${vars.nfsShares.dockerVolumes.mountpoint} = {
-      device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
+      device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
       fsType = "nfs";
 
       options = [
@@ -46,7 +60,7 @@ in
     };
 
     ${vars.nfsShares.nextcloudData.mountpoint} = {
-      device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath}";
+      device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath}";
       fsType = "nfs";
 
       options = [
@@ -57,7 +71,7 @@ in
     };
 
     ${vars.nfsShares.raspiVolumes.mountpoint} = {
-      device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath}";
+      device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath}";
       fsType = "nfs";
 
       options = [

modules/nix-cache/remote-builder-client.nix

@@ -1,10 +1,14 @@
 { pkgs, vars, ... }:
 
 {
-  # Install the remote builder key on each client host (do not commit private keys):
-  #   sudo install -d -m 0700 /root/.ssh
-  #   sudo install -m 0600 ./nixremote /root/.ssh/nixremote
-  #   sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
+  # Authenticate as nixremote using the client host's own default root SSH
+  # identity (/root/.ssh/id_ed25519) rather than a separately-named key --
+  # matches vars.remoteBuilderAuthorizedKeys, which already authorizes
+  # each host's own default key (one entry per host, not a shared
+  # dedicated keypair). If this host doesn't have one yet:
+  #   sudo -u root ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
+  #   # then add its .pub to vars.remoteBuilderAuthorizedKeys and rebuild nix-cache
+  #   sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
   # Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
   # can connect the first time without a manual ssh-keyscan/known_hosts
   # step on every new client.
@@ -20,7 +24,7 @@
       {
         hostName = vars.nixCacheHost;
         sshUser = vars.remoteBuilderUser;
-        sshKey = "/root/.ssh/${vars.remoteBuilderUser}";
+        sshKey = "/root/.ssh/id_ed25519";
         inherit (pkgs.stdenv.hostPlatform) system;
         maxJobs = 4;
         speedFactor = 2;

modules/platforms/lxc.nix

@@ -1,4 +1,4 @@
-{ lib, modulesPath, flakeTarget, ... }:
+{ config, lib, modulesPath, flakeTarget, ... }:
 
 let
   # Bakes this exact flake target's pre-generated SSH host key straight
@@ -58,8 +58,28 @@ in
     # host.nix declares each host's real hostname (networking.hostName);
     # keep that instead of letting Proxmox's ambient container config win.
     manageHostName = true;
-    # Unprivileged matches how these containers are actually created.
-    privileged = false;
+    # Unprivileged by default -- matches how these containers are actually
+    # created (scripts/proxmox/create-proxmox-resource.sh reads this value
+    # back to decide `pct create`'s --unprivileged flag, so the two stay
+    # in sync).
+    #
+    # lxc-docker is the one exception: the kernel's NFS client doesn't set
+    # FS_USERNS_MOUNT, so mounting NFS from inside *any* non-init user
+    # namespace -- which is exactly what an unprivileged container's
+    # UID-mapped root runs in -- is rejected at the VFS layer with EPERM,
+    # no matter what Proxmox's own `mount=nfs;nfs4` container feature
+    # allows at the AppArmor layer (confirmed live: TCP to the NFS server
+    # succeeds, the server's export table matches the container's IP, and
+    # `mount.nfs: Operation not permitted` still fires immediately with no
+    # corresponding denial anywhere in the server's logs -- a kernel-level
+    # rejection, not a network or export-permission one). Keying off
+    # hostName rather than something docker-build-type-specific because
+    # modules/build-types/docker.nix is also composed for linode-docker/
+    # proxmox-docker, which don't import proxmox-lxc.nix at all --setting
+    # this option there would break their eval with "option does not
+    # exist" regardless of any mkIf guard, since mkIf only makes a value
+    # conditional, not whether the option needs to exist somewhere.
+    privileged = config.networking.hostName == "docker";
   };
 
   boot.loader = {

scripts/codex-maintenance.sh

@@ -228,6 +228,20 @@ fi
 
 mapfile -t hosts < <(for h in "${!affected_hosts[@]}"; do echo "$h"; done | sort)
 
+echo
+echo "Checking nix-cache host key for drift..."
+if bash "${script_dir}/secrets/sync-nix-cache-host-key.sh" --check; then
+  :
+else
+  drift_status=$?
+  if [[ "$drift_status" -eq 2 ]]; then
+    echo "nix-cache unreachable from here -- skipping host-key drift check."
+  else
+    echo "WARNING: nix-cache's host key has drifted from variables.nix (see above)." >&2
+    echo "  Run 'bash scripts/secrets/sync-nix-cache-host-key.sh' to fix." >&2
+  fi
+fi
+
 echo
 if [[ ${#hosts[@]} -eq 0 ]]; then
   echo "No hosts affected by changed files; skipping host eval."

scripts/lib/nix-eval.sh

@@ -34,3 +34,21 @@ flake_target_hostname() {
   nix eval --raw "${NIX_EVAL_FLAGS[@]}" \
     "${flake_ref}#nixosConfigurations.${target}.config.networking.hostName" 2>/dev/null
 }
+
+# flake_target_lxc_privileged <flake_ref> <target>
+# Prints "true" or "false" for one lxc-* target's config.proxmoxLXC.privileged
+# (modules/platforms/lxc.nix is the single source of truth -- e.g.
+# lxc-docker sets this true so it can NFS-mount; every other lxc-* host
+# stays unprivileged). Only meaningful for lxc-* targets -- the option
+# doesn't exist for linode-*/proxmox-* (nixpkgs' proxmox-lxc.nix, which
+# declares it, is only ever imported by modules/platforms/lxc.nix). Empty
+# (not an error under set -e) if the eval fails.
+flake_target_lxc_privileged() {
+  local flake_ref="$1" target="$2"
+  # Not --raw: the option is a Nix boolean, and --raw can only coerce
+  # strings ("cannot coerce a Boolean to a string"). Plain `nix eval`
+  # prints a bare `true`/`false` for a boolean, which is exactly the
+  # string this needs.
+  nix eval "${NIX_EVAL_FLAGS[@]}" \
+    "${flake_ref}#nixosConfigurations.${target}.config.proxmoxLXC.privileged" 2>/dev/null
+}

scripts/proxmox/configure-nix-cache-client.sh

@@ -15,6 +15,16 @@
 # this repo -- update both if nix-cache is ever rebuilt with a new host
 # key or the cache signing key is rotated (see docs/nix-cache.md).
 #
+# REMOTE_BUILDER_KEY defaults to this machine's own default root SSH
+# identity (matches modules/nix-cache/remote-builder-client.nix's
+# convention for real NixOS clients: authenticate as nixremote with the
+# host's own default key, added individually to
+# vars.remoteBuilderAuthorizedKeys, rather than a separately-named or
+# shared keypair) -- generate one with
+# `ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519` if this machine
+# doesn't have one yet, then add its .pub to vars.remoteBuilderAuthorizedKeys
+# and rebuild nix-cache.
+#
 # Usage:
 #   sudo ./configure-nix-cache-client.sh [--dry-run] [--no-remote-builder] [--no-restart]
 #
@@ -24,9 +34,9 @@
 set -euo pipefail
 
 : "${NIX_CACHE_HOST:=nix-cache}"
-: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache}"
+: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache}"
 : "${REMOTE_BUILDER_USER:=nixremote}"
-: "${REMOTE_BUILDER_KEY:=/root/.ssh/nixremote}"
+: "${REMOTE_BUILDER_KEY:=/root/.ssh/id_ed25519}"
 
 CACHE_PUB_KEY="cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4="
 FALLBACK_URL="https://cache.nixos.org/"

scripts/proxmox/create-proxmox-resource.sh

@@ -766,13 +766,22 @@ if [[ "$type" == "lxc" ]]; then
   # 512M default otherwise (confirmed live: --memory 2048 left swap at
   # 512). Default to matching whatever --memory resolved to above.
   local_swap="${swap:-$memory}"
-  # --unprivileged 1: modules/platforms/lxc.nix sets proxmoxLXC.privileged
-  # = false, so the NixOS config inside the image assumes it's running as
-  # an unprivileged container (cgroup/capability/mount expectations baked
-  # in at boot). `pct create`'s own CLI default for this flag is
-  # privileged (unlike the web UI, which defaults its checkbox the other
-  # way) -- leaving it unset creates a privileged container running a
-  # NixOS config that assumes unprivileged, a real mismatch.
+  # --unprivileged: read back from modules/platforms/lxc.nix's own
+  # proxmoxLXC.privileged (via flake_target_lxc_privileged) rather than
+  # hardcoded, since that's no longer the same for every lxc-* target --
+  # lxc-docker sets it true so the container's NFS mounts work at all (the
+  # kernel's NFS client can't mount from inside any unprivileged
+  # container's user namespace, no matter what AppArmor allows -- see that
+  # option's own comment). The NixOS config inside the image bakes in
+  # cgroup/capability/mount expectations matching whichever value it was
+  # built with, so this must stay in sync with it -- `pct create`'s own
+  # CLI default for this flag is privileged (unlike the web UI, which
+  # defaults its checkbox the other way), so leaving it unset would create
+  # a privileged container running a NixOS config that assumes
+  # unprivileged for every target except lxc-docker, a real mismatch.
+  privileged_eval="$(flake_target_lxc_privileged "$repo_root" "$flake_target")"
+  unprivileged_flag=1
+  [[ "$privileged_eval" == "true" ]] && unprivileged_flag=0
   #
   # --features nesting=1,keyctl=1: required for a modern (v247+) systemd
   # guest to actually boot unprivileged -- confirmed live: without this,
@@ -790,7 +799,7 @@ if [[ "$type" == "lxc" ]]; then
   # hands the whole string to `ssh` as a single command for the *remote*
   # shell to parse -- unquoted, that `;` would be read as a remote
   # command separator and silently truncate this into two commands.
-  create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --unprivileged 1 --features '${PROXMOX_DEFAULT_LXC_FEATURES}' --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --swap ${local_swap} --net0 name=eth0,bridge=${bridge},ip=dhcp"
+  create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --unprivileged ${unprivileged_flag} --features '${PROXMOX_DEFAULT_LXC_FEATURES}' --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --swap ${local_swap} --net0 name=eth0,bridge=${bridge},ip=dhcp"
   remote "$create_cmd"
   remote "pct start ${vmid}"
 else

scripts/secrets/sync-nix-cache-host-key.sh

@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+# Detects and fixes drift between the ed25519 SSH host key nix-cache is
+# actually serving right now and vars.nixCacheHostKey (variables.nix) --
+# the value modules/nix-cache/remote-builder-client.nix bakes into every
+# client's declarative programs.ssh.knownHosts, and
+# scripts/proxmox/configure-nix-cache-client.sh hardcodes as its own
+# default for non-NixOS clients.
+#
+# This value has no automatic source of truth: nix-cache's host key is
+# generated once (first boot / container recreate) and never touches this
+# repo again unless someone remembers to update it by hand afterwards. It
+# drifted silently once already -- confirmed live: variables.nix recorded
+# a key that no longer matched what nix-cache actually presented, which
+# would fail every real client's SSH host-key verification for
+# distributed builds without ever producing an obvious error pointing
+# back here (a client just sees "Host key verification failed" against
+# *some* key, with no hint that the trusted value itself was stale).
+#
+# codex-maintenance.sh runs this in --check mode on every invocation so
+# that drift surfaces as a warning instead of a future debugging session.
+#
+# Usage:
+#   scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run] [--host <name>]
+#
+#   --check     Only report drift (exit 1 if found, 2 if nix-cache is
+#               unreachable); never writes. For CI/maintenance use.
+#   --dry-run   Show what would change; never writes.
+#   --host      Override the hostname to scan (default: variables.nix's
+#               nixCacheHost / env.sh's NIX_CACHE_HOST).
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
+variables_nix="${repo_root}/variables.nix"
+client_script="${repo_root}/scripts/proxmox/configure-nix-cache-client.sh"
+
+# shellcheck source=../env.sh
+source "${repo_root}/scripts/env.sh"
+
+check_only=0
+dry_run=0
+host="${NIX_CACHE_HOST}"
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --check) check_only=1; shift ;;
+    --dry-run) dry_run=1; shift ;;
+    --host)
+      host="${2:?--host requires a hostname}"
+      shift 2
+      ;;
+    -h|--help)
+      sed -n '2,23p' "$0"
+      exit 0
+      ;;
+    *)
+      echo "ERROR: unknown argument: $1" >&2
+      exit 1
+      ;;
+  esac
+done
+
+current_value="$(grep -oE 'nixCacheHostKey = "[^"]+"' "$variables_nix" | sed -E 's/nixCacheHostKey = "(.*)"/\1/')"
+if [[ -z "$current_value" ]]; then
+  echo "ERROR: couldn't find nixCacheHostKey in $variables_nix" >&2
+  exit 1
+fi
+current_type_blob="$(awk '{print $1, $2}' <<<"$current_value")"
+current_label="$(awk '{print $3}' <<<"$current_value")"
+
+echo "Scanning ${host} for its current ed25519 SSH host key..."
+nix_extra_opts
+scanned="$(nix-shell "${NIX_OPTS[@]}" -p openssh --run "ssh-keyscan -t ed25519 -T 5 '${host}'" 2>/dev/null | grep -v '^#' | head -1 || true)"
+if [[ -z "$scanned" ]]; then
+  echo "ERROR: couldn't reach ${host} (or got no ed25519 host key back) via ssh-keyscan." >&2
+  exit 2
+fi
+scanned_type_blob="$(awk '{print $2, $3}' <<<"$scanned")"
+
+if [[ "$current_type_blob" == "$scanned_type_blob" ]]; then
+  echo "Up to date: ${host}'s host key matches variables.nix's nixCacheHostKey."
+  exit 0
+fi
+
+echo "DRIFT DETECTED:"
+echo "  variables.nix has: $current_type_blob"
+echo "  ${host} is now:    $scanned_type_blob"
+
+if [[ "$check_only" -eq 1 ]]; then
+  echo
+  echo "Run 'scripts/secrets/sync-nix-cache-host-key.sh' (no flags) to fix." >&2
+  exit 1
+fi
+
+new_value="${scanned_type_blob} ${current_label}"
+
+if [[ "$dry_run" -eq 1 ]]; then
+  echo "(--dry-run: would update variables.nix and ${client_script##*/} to:)"
+  echo "  $new_value"
+  exit 0
+fi
+
+sed -i "s|nixCacheHostKey = \"[^\"]*\"|nixCacheHostKey = \"${new_value}\"|" "$variables_nix"
+sed -i "s|NIX_CACHE_HOST_KEY:=[^}]*}|NIX_CACHE_HOST_KEY:=${new_value}}|" "$client_script"
+
+echo "Updated variables.nix and ${client_script##*/} to:"
+echo "  $new_value"
+echo
+echo "This only takes effect on already-deployed NixOS clients after their"
+echo "next rebuild (programs.ssh.knownHosts is declarative). Review with"
+echo "'git diff', then run 'bash scripts/codex-maintenance.sh' before committing."

secrets/common.yaml

@@ -5,85 +5,94 @@ sops:
     age:
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaMUpxYmUzaVY4d1pRY3g3
-            empaVUJSN0JaSlNQQlJtZENwc0VMYVhJSkFvCmJQMkpSd3RoYzlKQjBSZWErQzR2
-            aG5SVndOQWpSRTBDSWJVQkg0c0hiNFUKLS0tIHhiaEdpY3gwZkpCcHl2TW5CNThn
-            aVVUQy9Qd0trb0RNdUpVTXdrdGlrTmcK1uphQAyDV+Gk5+K1YOqw1Z8ynGP5sAPF
-            q5icujja/SGexX18hPYXbkyUtOrBYjW62gCuGJinSBPROoFUJbiP7g==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWcWtCck9HajZXTW8xTGFR
+            UnplbDZ1SWcxNnpDYjRyVG5QeGw0YXRTUVJFCjV3anBuVWNnNHFJa2tXeEdZR0ly
+            eXVtRXM3T3BHbXRSa2lwWVBDS0pYTTgKLS0tIEhDY280OWVwNEphU1FJRGRSbzFH
+            LzdQVVlmRDl0em82ZVVWNkdFSFZ5VXMKhKnOOigvGhy+GzcGjJBzfHuoiRKbgVVA
+            S/zM+fDQdS1qlg3Bp/GofCpeKpmERe/SLMlib5LgCUCWyxB+2IPPKw==
             -----END AGE ENCRYPTED FILE-----
           recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKOWZxcFlhRHhKelhDS29M
-            NGRlTEhJUGUzcFV6QVFCdTNkdEN3MlljWnl3CnI3LzVxSHBwSi9TYlRUcFdyN2py
-            cVZBV0Q3Z3FhRXA0T1NFQzd2R00yeEUKLS0tIHkxL2dHV3RkVFYxZTJNTVZvVFIr
-            TXpJZzdnYlpJaXBmcjdWWUtxNkc4dWMKVsImJiavzUzSFn78pciNJPHaS3KWqJer
-            VkxF6kF3tl2HmW46eVXtsQowdu+zVR9HS35i/HvQ1r1TyP6qT5ofcQ==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBbDhMaUNNMk9MY2FPUm51
+            MVYzUEdoKytEVDRSU2RESGFCamkrVXpITmhJCkl4VVJmM1NBbUE0cCtDWEgyWTNo
+            STJ2cEZZRStJZkY3YTM1TCtiMnpXemcKLS0tIEpkODdLTlAxaFkxbk4ycW1lanVh
+            TnVPR2xwVG9aTHhPYm01SG44ajJpeUEKNObIucwjlj3Y7YqhJwQGdeV1wFAcANmn
+            564nMYDfh/KvN86r5yhwwYzkJHV2ejlh/30lxAPG5Z9gdtgnFRJrVw==
             -----END AGE ENCRYPTED FILE-----
           recipient: age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyWjdSanRRaEI4VHFZVkdw
-            NkxMODRuakFJTFNoTHZIRHptZFNxaGZ3K3lnCjJGR1FJalRsYmtLcHFxZW8yNU5W
-            MzBjWmltbzI3MTByUjB6djlEdllHNXMKLS0tIEhEOVhLVjZkRi9vUEtDWUhxT0NX
-            eUZ3bGdEaHdGbXZIYkowYUZuTWJ3d2MKMwgxOqlMH7GfLlseD4J277Dcg0KCD3d5
-            jwJDb82kRWoBMicTJZXoq/5oe5blJNa7dWYoqgkYxMA2O+0igT+I6g==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSTzNpdzZ3U05zKzNnSXkz
+            Z1NmNURPbXlnelNIQTVUYXBHc002VmlLU2xrCm93TUwxaHgyWWdGVVdOQlFjTFAw
+            aWNHeEFFRjVMRnk2TUpxY1lUQlFzVGMKLS0tIHBTbGhmcVhxa2h0M21qcElnRGp1
+            RDBNSnRhUWExcnh0TFlaNzZ4MXBQYkEKrpCjW26PsgwM8AqS3d+EjBkpNlUcUxlQ
+            a4GlRW+FDTKrVzvfnZRhILKU/ekJQP5nGyjLS9QdWloDxiANbUjddA==
             -----END AGE ENCRYPTED FILE-----
           recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsZzdRMm9zdlJHckFoREVs
-            UkVXL3dCeWRuN2NqcnVsOWVNT1FLUHY3andrCmNFY0ZCby9SdnUyYlV6R05YNDB0
-            em43SjRCNVhYbmpRZUVWT0cxQlhGdmsKLS0tIDZHazNyck5VNHBuNVM5bmRZUlpR
-            QzBSNy84VDdLVkZZbnNlUFYydXlreEEKYZaR2b7tyRAhPdP+ytpP0veUTi7pY9Nw
-            pK0h4hcegLNYJL1AfOYwFQoW7vb256GdmwdcuuBl1YBGXWGraaBnZg==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGc0UwZjFJdUZXaGpjS09Q
+            Nk9jNElBYlVjWVFCeHFObm9ScVRrMVQrNUg0CnpGWkE3U0xSS0hHcVlnOERRY0Rx
+            Y3pKeU51QmovdUt2VkcyajZpWDk2ZDAKLS0tIHA0VmZqc0xQS0RvbU4zNnRhL2VK
+            VWFhWExzb2M3ZjgrOERzVlN6S3c5SUUKa2QQz0S9i9IPCZnwDGqLsnCZhdsC3Dwa
+            gND6ExDrXuvDK5HgZjixCzpYOFC/JloJPOONH8gkc3hCany6N9rCpw==
             -----END AGE ENCRYPTED FILE-----
           recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFSnZiOGpsL1BESWVEdUxx
-            dXc5LzBHTlNodGJ1S3hOeHd5Wmgxb3BPSHl3CnBEWDNTc3NDdzN3RHQxUlNLOEs5
-            RU1SS2tVT29XbDJCWVliWnVkekN4L1kKLS0tIGFqY2pNK2h2S2ZWMndrRVN6eUdN
-            M0NLY2EvTTVIYlZvdk9XY0NwNE14ZlEKcOwgCK3g56kId/4tEt+2iCcylisn+Fca
-            5VWamsXdbwxbxmCEEUbgN3aHrdVz3mV4+8FZqA34yXz8pc45/PUcFg==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0Y3VjL1dYVDU0eXdCKzJH
+            VG00eFZoK2ZnYUZIaWhnLzVlN01XTFBMRGlRCmNBSXJoVHVxd1o3bitjSzNsMjZZ
+            ZU9pSFN0UG00ZUdDY0ZWZ2t0YlVLbTgKLS0tIHA1bUNBT0V0bTBWcmY2UVhkYW1J
+            dmExckdTUVlCWWdpaG0vUDl1aG94d00KdbkEFjCfdkdRABYXLfK/kwwuQubOK51q
+            w5SIKfOkyGX3WATarPWNQAxmNknozQECub1tCs8G0l7eCUjfpLuXdw==
             -----END AGE ENCRYPTED FILE-----
-          recipient: age1qz9d4ka4xgexujyd247s7lp737sulp5fhxl5d65fj2ykvc4j4edqrsdks8
+          recipient: age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJU1cydFFuRnpCUU5mbjFu
-            TTIxREkxZ1hMalVtNjNjcHVaUXpPNWdib2tzCkRScWRTWHhxN3lhckI4UFpMSFRa
-            RGxWcldlWnd5SVJFNWxQWjB3R0pMb3MKLS0tIGdjLzlhTHgyT2hjTmRTK1RiZmxS
-            OWRWYklBWkoyUG8raFNzd3JlRUtEUEUKw+NdAp1Mz0dEOUGPbqCV8y7029I3Kye0
-            keU2T29JGCN1D6x//1NcMUYSaFZKgv0ZVSjVUCl1EnmlJ0nBdbaDXA==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0L0tmV0dldkczS0lSRnR2
+            cE5YaTR3eTc0QTAwSmNZKzVVeC93a1dGT3pBCkFYOWZSekxvMi9JclVKeU5CRk9N
+            SVVXQittb3AvU1FEUFYzQ1VVVEgrWFEKLS0tIGRxVFMxQjBKTWZlakxmWXBXRCt1
+            RTI2amIxYXdsMGdEVmliUXpaUU44dzgK9l6o/NozRvk8cViqgbDetifjZuoRXf5f
+            qQUVc6Qg97RZN2TIczN3YEKYXhzk7A1ihD4yds7FBgzMuvYQWH983g==
             -----END AGE ENCRYPTED FILE-----
           recipient: age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnNFlzYVBqaGFDVE9xbmt5
-            OVd4ZlZRdEVJd0JFdkRIYVA5ZUEwYjJjWVJjCnZzTExNbDlYbGN6c1dYU2ZlNEVs
-            ZUo4MXdpc0tzbDI0MHl5eXc3d09VRDQKLS0tIGs4SGMya1RZQ0hpa2NtQk9Wa29C
-            bmxIYWk1VThsaFhxRDlOOGhGQ2tYamsKs+PcOiaeNFujCWwZBr+nq1MzrGohl+ch
-            TASI7eNsiHmSVRzSMHv4mX+8yXDHIRgHbFuUty2gdgGFLRjYXe4gDA==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkdkpXZnJNN1ZuU0MybWhG
+            WWE3TVNFOURzR25wcTZ6MjIvNG4wSmUrbUhzCnM2aXRHV1h2ZmRVV2xpNVU0WkRy
+            bmFMcHhtRW9FTlRlbklNODErQkZHRWMKLS0tIGwvcXVOVlE4cHNUeUVCV0NLd1FC
+            QUgrWmVXRHE1d05KaEtuUHdISytvam8KgjQaJeQvOUhcPyrfSPhniRBDahaINlkj
+            d9cXhwSVvsB8L9SN6U4mPK/6s8FhWgkwFBjUV4rCMGVr2ngjMZ8BWg==
             -----END AGE ENCRYPTED FILE-----
           recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkcmVQYXV1NzRrQ2IxTElt
-            QUJRMkhiUUJqRDdwQW03K0lCTEtkSTU5UDJrCjh6SldpQkF1NzIybTJTdnBlOHdQ
-            VlREMy9hM0ZaL21Pa0VPUzcvb0pQQjQKLS0tIFcrK0I3Z3Nhem45eHpZYmdWeit6
-            eXFtRHVxODhaNTVDdVR1a2tOb1N1U1EKZBdLHq8PRMKX2ndFr3AxFVAZRyvhFa1u
-            72R5tordo4IR3HSxG3Z06rokOITd+KKhaQ8NEWEC8qioAAMxEC9QWw==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBNdGUyZnBxemhEQ2MySkhB
+            RDNtM01pVWxEb2hpVllYOTFWTHdNVWxhUzFNCkZTNlZCME8vYnowSVpKN0dvOWk1
+            QjE1K25QWkNCd2RKenRMR2l5empMclEKLS0tIEtCb3FnREozbWV0RXNQcEtqV21r
+            Y3YxQlJsVml6dm9LMlZ5VDIzNGVTZFEK5Usyy2tYoExD9X9Zw1tof56qA8bPVJAQ
+            zFvHVFQsmou/qH63UPbuJ4xkNf+j8payFLb1iZq0UhgWv0gVlu1q1w==
             -----END AGE ENCRYPTED FILE-----
           recipient: age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg
         - enc: |
             -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0WjJPbXhXNTNyQUZzMmxS
-            Rkd3Yk5Ga3pPRXFtMFpXcmpBZ3RJYnJlaHhjClRDbFBHRVZ0SDdrNFdVWmUwalpT
-            WUt5a2RUTnQ4VFBGMlowR3JycDM3aXMKLS0tIERSeFd4cGdSbzRpSmpSQnB1Yng5
-            OHBma3lYeWEwQ2RvelpzZHRkN1JPWFEKLXkJfEkH3lo5Z4mj8PZXTfAfZn6B46To
-            U0G54tUh0U1EeMnI5ZPRGwDxi5K0eD68HjeE9MBvFBysOChP/ANDPQ==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXb2FWU3JhN293V3lYWTh1
+            ZGY4bFRjc2srODdVUk5BK0NqVlRVRXZrcVU0Cm9vYlJkblZ0KzVMVmN1MmVkUlQ0
+            WnhKeXUwT3B6eUt1L0NLQTl3aFJ4WGMKLS0tIFY4M1k3VkR6ZUtUOEI1N1VhMStv
+            dXBoTVdGeXdIT3YvUmo1MXhld1VXWkkKEZkOaVkAxRL85ubMWOQuq2xreU8R1EGx
+            28y71UsnqCSOTRj+fqHgZMo6pqsSHC5I7e+/BsSpshmbJbe86glhuw==
             -----END AGE ENCRYPTED FILE-----
           recipient: age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhVE4raWVUTmVvOE1lV0Vz
+            d2hvVWlBcWM0b0VOVFEyN0F1WVZBWkVUcGlVCmRBdWV2WmRDMHFtVGwvbk5uZXBi
+            cFB4UjJpQkNlckhXaUdsd3B6UEV0SjAKLS0tICtLUDVNWEREbVVxNEYrMzlXdHFs
+            RDdENE9FbDJXQkNOM1BhSGhGRHgvL0kKovOmMfELUnHFeAL+kqnmD8quZdwLhuUN
+            fDetQQWrz0lQFH3CQ5fGLw9PelKcywifJV2a0zQB+U7aSIfuhfVThQ==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp
     lastmodified: "2026-07-19T02:30:40Z"
     mac: ENC[AES256_GCM,data:UiL3VMDF6rq4Nr87KspcDx434q3tfNXeb5pwH2O+4ssNQ6xzcYDdzXBnhAY3zLBsqPMKrvHBd4Ot/gEMcq3FMIVe7Q6p9yWKpep66KZ/yWEhAlwIVhD79Oj8VS+1CHKjf25zpRdhZorp04oeFQQd9VfjJB4EE/Q1aVbwTGlpIic=,iv:i/0conaFgFia+wzNTdUL6tlSTw35HTK3Ap1Sr5RGHf8=,tag:ULbz5FllShA/JjlSRdxA0g==,type:str]
     unencrypted_suffix: _unencrypted

variables.nix

@@ -26,7 +26,7 @@
   # fresh client that has never manually ssh'd to nix-cache before. Update
   # this if nix-cache's host key is ever rotated or the host is rebuilt
   # from scratch.
-  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
+  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache";
 
   # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
   # host (modules/nix-cache/server.nix) — one per client host that's allowed