Fix PXE netboot installer hanging at boot (ISO/netboot module conflict)

The netboot build composed ./modules/installer/iso.nix (which pulls in nixpkgs' installation-cd-minimal.nix) together with nixpkgs' netboot-minimal.nix. Both installation-cd-base.nix and netboot.nix set fileSystems."/" via the identical lib.mkImageMediaOverride (mkOverride 60) priority - genuinely conflicting root-filesystem strategies (ISO-by-label vs. netboot-tmpfs) at the same priority, and the ISO one was winning. Every netboot boot hung waiting for a device that can never exist outside a real CD/USB: A start job is running for /dev/disk/by-label/nixos-minimal-... Reproduced live: deployed a scratch lxc-pxe-boot on pve-test, pulled its built kernel/initrd, and booted them directly with QEMU to confirm the hang and capture full console output. netboot-minimal.nix's own chain (netboot-base.nix) already imports profiles/installation-device.nix independently, so common.nix's initialHashedPassword override still applies correctly with iso.nix removed from this composition. Rebuilt and re-booted the same way after the fix - full boot to a working shell with SSH up, no hang.
Merge pull request 'Add ad hoc pve1 -> pve-test clone script (vzdump + qmrestore/pct restore)' (#29 ) from worktree-clone-pve1-to-pve-test into main
2026-07-21 22:15:44 +00:00 · 2026-07-21 21:55:25 +00:00 · 2026-07-21 20:22:56 +00:00 · 2026-07-21 20:07:30 +00:00 · 2026-07-21 20:07:14 +00:00 · 2026-07-21 19:57:15 +00:00
27 changed files with 1219 additions and 199 deletions
@@ -13,9 +13,17 @@ jobs:
    steps:
      - name: Check out repository
        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0

      - name: Install Nix
        uses: DeterminateSystems/nix-installer-action@v19

-      - name: Run maintenance checks (secrets, fmt, lint, eval)
+      # Scoped to files changed since the PR base / previous push -- see
+      # scripts/codex-maintenance.sh. CI never passes --full-check: that
+      # full sweep is for local/manual use, since it's slow enough to time
+      # out this runner.
+      - name: Run maintenance checks (secrets, fmt, lint, eval -- changed files only)
+        env:
+          MAINT_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
        run: bash scripts/codex-maintenance.sh
@@ -13,9 +13,17 @@ jobs:
    steps:
      - name: Check out repository
        uses: actions/checkout@v4
+        with:
+          fetch-depth: 0

      - name: Install Nix
        uses: DeterminateSystems/nix-installer-action@v19

-      - name: Run maintenance checks (secrets, fmt, lint, eval)
+      # Scoped to files changed since the PR base / previous push -- see
+      # scripts/codex-maintenance.sh. CI never passes --full-check: that
+      # full sweep is for local/manual use, since it's slow enough to time
+      # out this runner.
+      - name: Run maintenance checks (secrets, fmt, lint, eval -- changed files only)
+        env:
+          MAINT_BASE_SHA: ${{ github.event.pull_request.base.sha || github.event.before }}
        run: bash scripts/codex-maintenance.sh
@@ -3,11 +3,15 @@ keys:
  - &docker age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e
  - &server age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
  - &nix-cache age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
-  - &lxc-minimal age1qz9d4ka4xgexujyd247s7lp737sulp5fhxl5d65fj2ykvc4j4edqrsdks8
  - &nix-minimal age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu
  - &proxmox-minimal age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg
  - &lxc-nix-cache age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
  - &lxc-docker age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs
+  - &lxc-minimal age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep
+  - &lxc-pxe-boot age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp
+  - &lxc-gui age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05
+  - &proxmox-server age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
+  - &vm-server age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh

 creation_rules:
  # Shared across every currently-deployed host: root/nixos password hash,
@@ -26,6 +30,10 @@ creation_rules:
          - *lxc-nix-cache
          - *proxmox-minimal
          - *lxc-docker
+          - *lxc-pxe-boot
+          - *lxc-gui
+          - *proxmox-server
+          - *vm-server

  - path_regex: secrets/nix-cache\.yaml$
    key_groups:
@@ -39,6 +47,8 @@ creation_rules:
      - age:
          - *admin
          - *server
+          - *proxmox-server
+          - *vm-server

  - path_regex: secrets/docker\.yaml$
    key_groups:
@@ -35,9 +35,14 @@ Use these commands when validating changes:
 ```bash
 bash scripts/codex-setup.sh
 bash scripts/codex-maintenance.sh
-bash scripts/codex-maintenance.sh dry-run
 ```

+With no flags, `codex-maintenance.sh` scopes fmt-check/statix/eval to files
+changed against a base ref — this is what CI runs on every push/PR. For the
+full sweep (every host, every package — slow; CI never runs this), use
+`bash scripts/codex-maintenance.sh --full-check` (add `--dry-run` for build
+planning on top of whichever scope is active).
+
 Host evaluation is safe when limited to drvPath checks:

 ```bash
@@ -0,0 +1,150 @@
+# Flake End-to-End Audit Report
+
+**Date:** 2026-07-21
+**Scope:** Full static lint/eval sweep + live build/deploy/interrogate/destroy testing of every `lxc-*` and `proxmox-*` flake target against `pve.sweet.home`, plus an audit of the operator's ability to manage the flake/secrets tooling.
+**Branch:** `worktree-flake-e2e-audit` (this session's isolated worktree)
+
+## Executive Summary
+
+The flake itself is in good shape: `nixpkgs-fmt`, `statix`, and a full eval + dry-run build of every host and package are all clean. Every `lxc-*`/`proxmox-*` target's NixOS configuration builds successfully — no target has a broken derivation graph.
+
+The issues found are **operational, not code-level**:
+
+1. **pve.sweet.home is critically low on disk space** (91-95% full during this session) and cannot currently build the two largest closures (`gui`, `pxe-boot`) to completion — this actively blocks deploying/redeploying those hosts via the documented workflow.
+2. **A real, reproducible secrets-decryption failure** was caught live: a stale cached container image (built before a same-day sops-key fix) boots with sshd never starting and every secret failing to decrypt. This is a **general hazard in `create-proxmox-resource.sh`'s "reuse the cached image if present" default**, not a one-off.
+3. **sops key/anchor drift**: `proxmox-minimal` has a `.sops.yaml` recipient anchor with no corresponding private key anywhere in this environment; several `lxc-*`/`proxmox-*` targets have no sops registration at all yet.
+4. One concrete script bug was found and **fixed in this session**: `create-proxmox-resource.sh` never enabled the QEMU guest agent channel on VMs it creates, despite the guest OS already running it.
+5. A management-surface audit (of the operator's ability to run this repo day to day) found 5 process gaps, detailed below.
+
+Nothing here required or received a `nixos-rebuild switch/boot/test`, `nixos-install`, or any disk-formatting command — all validation was `nix build`/`nix eval`, plus disposable `pct`/`qm` create-then-destroy cycles via the repo's own `create-proxmox-resource.sh`.
+
+---
+
+## 1. Static Analysis Results — all clean
+
+`bash scripts/codex-maintenance.sh --full-check --dry-run` (whole-tree sweep, not just changed files):
+
+| Check | Result |
+|---|---|
+| Secret grep | Clean — only the documented exceptions (installer's own hashed passwords, `access-tokens` comment references) |
+| `nixpkgs-fmt --check` | 0/53 files would be reformatted |
+| `statix` | No lint warnings |
+| nix-cache host key drift check | Up to date |
+| Full eval of every host's `system.build.toplevel` | All 19 `nixosConfigurations` targets evaluate cleanly |
+| Dry-run build of every host + package | All succeed, no derivation errors |
+
+No drift, no formatting issues, no lint findings anywhere in the tree.
+
+---
+
+## 2. Per-Target Test Results
+
+Legend: **LIVE** = built on pve, `pct`/`qm` create → interrogated → destroyed. **BUILD-ONLY** = `nix build` validated the config (mostly `.config.system.build.toplevel`, occasionally `.tarball`), no resource created on pve.
+
+| Target | Test type | Result | Notes |
+|---|---|---|---|
+| `lxc-docker` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT105 is already running this identity in production; `--allow-duplicate-host` would have destroyed it. |
+| `lxc-minimal` | **LIVE** | ✅ PASS (after retry) | First attempt reused a stale cached tarball predating a same-day sops-key commit → activation failed, sshd never started (see Finding #2). Redeployed with `--force-rebuild`: clean boot, `systemctl is-system-running` = `running`, secrets decrypted, sshd listening, users correct. |
+| `lxc-nix-cache` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — CT101 is already running this identity. First local build attempt appeared to hang on a remote-builder handoff to nix-cache; killed and retried with `--builders ""` (local-only), succeeded. |
+| `lxc-gui` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Registered a fresh sops key (no prior registration existed), built successfully through the full NixOS system closure, then **failed packaging the tarball**: `No space left on device` on pve's root filesystem. Not a flake defect. |
+| `lxc-pxe-boot` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Same failure as `lxc-gui` — this target additionally builds a full nested installer/netboot image (`stage-installer-artifacts.nix`), making it similarly large. Failed with the same `No space left on device` error, immediately after the gui attempt had already consumed pve's remaining headroom. |
+| `lxc-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; live deploy also would have hit `boot.zfs.extraPools` trying to import a real ZFS pool that doesn't exist in an isolated test container — an expected limitation of testing this build type outside its real hardware, not a bug. |
+| `lxc-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
+| `lxc-tor-relay` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT106 already holds this identity in production. |
+| `proxmox-docker` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — both CT105 *and* VM103 already hold `docker` identities. Combined `toplevel` + `diskoImagesScript` build crashed with a **Nix-internal assertion failure** (`worker.cc:360`) under this session's memory pressure (see Finding #6) — not a flake bug. Retried with `toplevel` alone: clean. |
+| `proxmox-minimal` | **LIVE (attempted)** | ⚠️ BLOCKED by key drift → BUILD-ONLY | `.sops.yaml` has a registered `&proxmox-minimal` anchor but **no corresponding private key exists anywhere in this environment** — the script correctly refused to generate a mismatched replacement. Fell back to `toplevel` build: ✅ PASS. |
+| `proxmox-nix-cache` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
+| `proxmox-gui` | BUILD-ONLY | ⚠️ Killed after ~40min (resource-limited) | This session's local build machine has only 2GB RAM; swap filled completely (2.0/2.0GB) and the build stalled, so it was killed rather than risk destabilizing the session further. **Not a flake defect** — the equivalent `gui` NixOS configuration already proved fully buildable during the `lxc-gui` live attempt above (it built the entire system closure successfully and only failed at the pve-side tarball-packaging step due to disk space, not the config). |
+| `proxmox-pxe-boot` | BUILD-ONLY | ⚠️ Killed after ~35min (resource-limited) | Was deep into building the nested installer's kernel initrd (this build type bundles a full netboot installer image via `stage-installer-artifacts.nix`) when killed to keep the audit moving. **Not a flake defect** — this target's own module logic was already effectively validated via the earlier *live* pve deploy attempt (`lxc-pxe-boot` above), which built the complete image and only failed at the final tarball-packaging step due to pve's disk space (Finding 1). |
+| `proxmox-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; same ZFS-pool caveat as `lxc-server` would apply to a live deploy. |
+| `proxmox-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
+
+**Not tested at all:** `linode-*` targets (not deployable to Proxmox) and `installer` (not a normal host) — both were still covered by the static eval/dry-run-build sweep above.
+
+---
+
+## 3. Findings, Ranked by Severity
+
+### Finding 1 — pve.sweet.home is critically low on disk space (blocks real deployments)
+
+At session start: `/dev/mapper/pve-root` was **95% full, 5.3GB free** (of 94GB). After two failed large builds it recovered slightly to **91% full, 8.2GB free** (nix cleans up its own failed-build scratch space). `/nix/store` alone is 26GB; `nix-store --gc --print-dead` reports **zero** reclaimable garbage — everything currently in the store is a live GC root, so `nix-collect-garbage` won't help without first removing old roots.
+
+**Why it matters:** `create-proxmox-resource.sh` builds every VM/CT image **directly on pve**, not on a build machine and transferred over. With <10GB headroom, any closure approaching a few GB (the `gui` build type: full Cinnamon desktop + Firefox + LibreOffice + GIMP + VS Code + xrdp; the `pxe-boot` build type: nginx/atftpd *plus* an entire nested installer/netboot image) cannot currently be built there at all. Both `lxc-gui` and `lxc-pxe-boot` failed live with `No space left on device` during this audit.
+
+**Recommended action:** Expand `pve-root`'s LV, or free space by pruning old container templates in `/var/lib/vz/template/cache` (1.5GB) / old backups in `/var/lib/vz/dump` (306MB) / auditing what's pinning 26GB of `/nix/store` as live GC roots (likely `result-*` symlinks — see below). This is real production disk state; **not something this session touched or fixed** — it needs the operator's judgment on what's safe to remove.
+
+**Secondary, smaller finding:** every `create-proxmox-resource.sh` run leaves a `result-<target>` symlink in the node's repo checkout as a permanent GC root (`ls /root/nixos/result-*` on pve showed 3 from this session alone: `lxc-docker`, `lxc-minimal`, `lxc-nix-cache`). These accumulate forever and pin their entire closures in the store. Consider having the script clean up its own `result-*` link after staging the built artifact (or use a temp `--out-link` under `/tmp`), so `nix-collect-garbage` can actually reclaim old build outputs.
+
+### Finding 2 — Stale cached images can silently ship broken secrets (reproduced live)
+
+`create-proxmox-resource.sh`'s default behavior is: if the node already has `<target>.tar.xz`/`.raw` staged, **reuse it** — only `--force-rebuild` forces a fresh build. This session hit exactly the failure mode `docs/auto-installer.md` already warns about: `lxc-minimal`'s cached tarball (built 2026-07-20T15:57Z) predated a same-day sops-key fix commit (2026-07-20T17:49Z, "clean up in ailse 3"). The deployed container booted with:
+
+```
+sops-install-secrets: failed to decrypt '.../common.yaml': Error getting data key: 0 successful groups required, got 0
+Activation script snippet 'setupSecrets' failed (1)
+```
+
+— every secret permanently failed to decrypt, `sshd` never started (though the container otherwise looked "running"). This was **not a code bug**: the currently-committed `secrets/common.yaml` decrypts fine for that host's key when checked independently; the *cached artifact on pve* simply reflected an older commit's ciphertext. Redeploying with `--force-rebuild` fixed it immediately.
+
+**Why it matters:** this is silent and easy to trigger by accident — any operator who redeploys a host without remembering `--force-rebuild` after a secrets change gets a container that looks like it started (`pct start` succeeds, `pct status` = running) but is completely inaccessible.
+
+**Recommended action:** Have `create-proxmox-resource.sh` compare the cached image's build timestamp (or embed the source commit hash in the staged filename) against current HEAD, and warn (or refuse without `--force-rebuild`) if they differ — rather than silently trusting presence alone.
+
+### Finding 3 — sops key/anchor drift
+
+Two concrete instances hit live during this session:
+
+- **`proxmox-minimal`**: `.sops.yaml` already has a registered `&proxmox-minimal` age recipient, but this environment's `host-keys/` directory has no corresponding private key file. `sync-host-keys.sh` correctly refused to generate a replacement (it would silently mismatch whatever's already registered/deployed) — but this means **no environment currently has this host's private key**, unless it exists on some other machine that was never backed up here.
+- **`lxc-gui`**, and by the same logic `lxc-server`/`lxc-tailscale-exit-node`/most `proxmox-*` targets, have **no sops registration at all yet** — expected for undeployed hosts per `docs/auto-installer.md`, but this session's live-testing needed to register `lxc-gui`'s key on the fly, which immediately hit **Finding 3b**: registering a key locally does nothing for pve's build until it's pushed to `origin/main` (pve builds via `git pull`, not from this uncommitted worktree). This is exactly gap #4 the management-surface audit (below) already flagged in the abstract — this session hit it concretely.
+
+**Recommended action:** for `proxmox-minimal`, decide whether to regenerate its key (destroying old-key decrypt access, if anything still holds it) or track down wherever the original private key lives and back it up here. For the general pattern, see the management-surface audit's recommendation to pre-flight-check key registration before building.
+
+### Finding 4 — QEMU guest agent never wired up (found and fixed this session)
+
+`modules/common/configuration.nix:44` sets `services.qemuGuest.enable = true` on every host — the guest-side agent daemon is correctly enabled everywhere. But `scripts/proxmox/create-proxmox-resource.sh`'s `qm create` call never passed `--agent 1`, so **Proxmox never created the virtio-serial channel** the agent needs. Every `proxmox-*` VM this script ever created was silently missing `qm guest exec`/IP-address reporting in the Proxmox UI, despite the guest daemon actually running.
+
+**Status: fixed in this session's worktree** (`scripts/proxmox/create-proxmox-resource.sh`, `qm create` now includes `--agent enabled=1`) — see the diff, included in the PR from this session.
+
+### Finding 5 — Orphaned container on pve (CT102)
+
+`pve.sweet.home` has a stopped LXC container, **VMID 102**, with an essentially empty config (`lock: create` and nothing else — no hostname, no rootfs, no network) — the leftover of a `pct create` that started and never finished. It predates this session (not created by any of this audit's activity) and wasn't touched. **Recommend the operator confirm it's abandoned and remove it** (`pct destroy 102 --purge 1`) — left as-is it may be someone's genuine in-progress work, so it wasn't assumed safe to delete autonomously.
+
+### Finding 6 — Nix-internal crash under memory pressure (tooling, not flake)
+
+Building `proxmox-docker`'s `toplevel` and `diskoImagesScript` together crashed with a Nix-internal assertion failure (`Assertion '!awake.empty()' failed ... worker.cc:360`, a known class of bug in Nix's multi-goal build scheduler) while this session's 2GB-RAM build container was under heavy swap pressure (1.8-2.0/2GB swap in use) from a separate concurrent build. Retrying the same target alone (no concurrency) succeeded cleanly. **Not a flake defect** — purely an artifact of this session's constrained build environment; noted for completeness since it looked alarming in isolation.
+
+### Finding 7 — Management-surface audit: 5 operability gaps
+
+A focused audit of "can the operator actually run this repo day to day" (flake, home-manager, sops, related scripts) found:
+
+1. **No documented recovery path if the `&admin` sops age key is lost without a backup.** `scripts/secrets/backup-admin-key.sh` exists and works but is referenced nowhere in `README.md`/`docs/` — no forcing function ensures a backup was ever taken. `rotate-admin-key.sh` requires the *old* key to re-key; there's no bootstrap-from-nothing path documented (the real fallback — deriving an age identity from any still-live host's own SSH key — isn't written down anywhere).
+2. **home-manager has no standalone iteration path.** It's wired only inside `nixosConfigurations` (`flake.nix`) — no `homeConfigurations` output. The fastest real shortcut (`nix build .#nixosConfigurations.<target>.config.home-manager.users.nixos.home.activationPackage`) isn't documented anywhere, so the practical workflow is a full host rebuild to test one HM tweak.
+3. **Gitea's flake-lock-update workflow pushes straight to `main` with no pre-merge validation.** `.gitea/workflows/update-flake-lock.yml` commits and pushes `nix flake update`'s result directly; `codex-maintenance.sh` only runs *after*, on the resulting push — a genuinely broken lockfile bump lands on `main` before anything catches it. (The GitHub-side workflow is safer — PR-based — but has the opposite gap: nothing alerts if the PR sits unmerged.)
+4. **No pre-flight check that a build target has a registered sops key before building it.** `docs/auto-installer.md` documents the failure mode (silent, total secrets-decrypt failure) but nothing in `create-proxmox-resource.sh` refuses to proceed when it's about to build a target with no `.sops.yaml` anchor — it's on the operator to remember. This session's `lxc-gui` test hit close to this exact gap (needed the key added on the fly, mid-session).
+5. **`vars.remoteBuilderAuthorizedKeys` has the same drift risk as `vars.nixCacheHostKey`, but no checker script.** `sync-nix-cache-host-key.sh --check` guards the latter; the former (and `vars.pxeServerIp`/`vars.pbsIp`) has no equivalent — a rotated/revoked client key just silently stops working with no diagnostic pointing back here.
+
+---
+
+## 4. Action Plan (priority order)
+
+1. **Free up disk space on pve.sweet.home** (or expand `pve-root`). Blocking: `lxc-gui`, `proxmox-gui`, `lxc-pxe-boot`, `proxmox-pxe-boot` cannot currently be built/redeployed on this node at all.
+2. **Decide on `proxmox-minimal`'s orphaned sops key**: locate the original private key and back it up here, or accept regenerating it (breaks decrypt access for whoever/whatever currently holds the old one).
+3. **Merge this session's PR** (see below) to get the `--agent 1` fix and `lxc-gui`'s new sops registration onto `main` — required before `lxc-gui` can be live-redeployed with working secrets.
+4. **Add a staleness guard to `create-proxmox-resource.sh`'s cache-reuse path** (Finding 2) — highest-leverage fix, since it silently produces a broken-but-"running" host.
+5. **Add a pre-flight sops-anchor check to `create-proxmox-resource.sh`** (management-surface gap #4) — same root cause class as #4 above, catch it before building instead of at first boot.
+6. Investigate/clean up **CT102** on pve (Finding 5) — confirm abandoned, then remove.
+7. Document `backup-admin-key.sh` in `README.md`'s Security Notes and add the live-host-key bootstrap-recovery procedure to `docs/` (management-surface gap #1).
+8. Add pre-push validation to the Gitea flake-lock-update workflow (management-surface gap #3).
+9. Lower-priority: document the home-manager `activationPackage` shortcut (gap #2); extend `sync-nix-cache-host-key.sh`'s drift-check pattern to `remoteBuilderAuthorizedKeys` (gap #5).
+10. Follow-up session: finish build-validating `proxmox-gui` and `proxmox-pxe-boot` (both killed here after 35-40min on this session's 2GB-RAM machine — not failures, just unfinished) once pve has headroom (item 1) — ideally from a machine with more RAM. `proxmox-server` and `proxmox-tailscale-exit-node` already passed build-only validation in this session, no follow-up needed.
+
+---
+
+## 5. Uncommitted Changes From This Session
+
+This worktree (`worktree-flake-e2e-audit`) currently has:
+
+- `scripts/proxmox/create-proxmox-resource.sh` — the `--agent enabled=1` fix (Finding 4).
+- `.sops.yaml` / `secrets/common.yaml` — `lxc-gui`'s new age key registered as a recipient (generated live during this session's testing).
+
+Per this session's standard workflow, these will be committed, pushed, and opened as a draft PR rather than pushed to `main` directly — merging it is the operator's call, and is also **prerequisite to live-redeploying `lxc-gui` successfully** (its build will keep hitting the sops-staleness failure from Finding 2 on pve until this registration is on `origin/main`).
@@ -31,17 +31,96 @@ machines when deployed.
  the auto-installer (see `docs/auto-installer.md`). Never commit its
  contents; if `git status` ever shows it as trackable, something is wrong.

+### Two Proxmox nodes: `pve1.sweet.home` (production) and `pve-test.sweet.home` (sandbox)
+
+There are two SSH-reachable Proxmox nodes on the LAN, both defined in
+`scripts/env.sh` (`PVE1_HOST` / `PVE_TEST_HOST`), individually targetable
+via `scripts/proxmox/create-proxmox-resource.sh --node <host>` or by
+overriding `PROXMOX_HOST`. `PROXMOX_HOST` itself still defaults to
+`PVE1_HOST` (production) — that default, and every other script behavior,
+is unchanged from before `pve-test` existed; the only thing new is that
+`pve-test` can now be reached at all. They are **not interchangeable** —
+one is real production infrastructure, the other exists specifically so
+there's somewhere safe to test. The restriction below is a policy for
+Claude specifically, not a change to the tooling's own default or
+anything the operator needs to opt into.
+
+#### `pve1.sweet.home` (production — off-limits to Claude)
+
+A real, live Proxmox node hosting production VMs/containers — not a
+sandbox, and not Claude's to touch by default.
+
+- **Off-limits at all times unless the operator has given explicit,
+  same-session instructions to act on this specific host.** That
+  authorization is scoped to the task it was given for — don't carry it
+  forward to unrelated later work in the same conversation, and never
+  assume it from a previous session.
+- **Read-only for existing state is always fine, authorization or not.**
+  You may SSH in (or use `pvesm`, `qm list`, `pct list`, `qm config`, `pct
+  config`, the Proxmox API, etc.) to inspect the node's config, storage,
+  and any existing VM/container — including ones this repo didn't create.
+- **Never** modify, stop, restart, delete, reconfigure, or create anything
+  on this node (`qm set`, `pct set`, `qm destroy`, `pct destroy`, `qm
+  stop`, `pct stop`, `qm create`, `pct create`, snapshot operations,
+  storage changes, etc.) — including scratch/test resources — without
+  that explicit go-ahead. Use `pve-test.sweet.home` for anything
+  exploratory instead; it exists precisely so `pve1` never has to be the
+  answer to "where do I test this."
+- **This is a Claude-specific policy, not something the scripts enforce.**
+  `scripts/env.sh`/`create-proxmox-resource.sh` default to `pve1` exactly
+  as they did before `pve-test` existed, with no extra flag or prompt
+  required — that's deliberate, so the operator's own existing workflows
+  don't change. Claude, however, must never rely on that default: every
+  Proxmox action Claude takes on its own initiative — not explicitly
+  pointed at `pve1` by the operator this session — targets `pve-test`
+  instead (e.g. `--node "$PVE_TEST_HOST"`, or `PROXMOX_HOST=$PVE_TEST_HOST`).
+  Claude's own default is `pve-test`, full stop, regardless of what the
+  tooling's own unqualified default happens to be.
+
+#### `pve-test.sweet.home` (sandbox — Claude's default target)
+
+A separate Proxmox node set aside for testing. The *tooling's* default is
+still production (`PROXMOX_HOST` → `PVE1_HOST`, see above) — but
+**Claude's own default is this node**: absent an explicit, same-session
+instruction to use `pve1`, every Proxmox action Claude initiates targets
+`pve-test`. Once targeted, it's safe to create, interrogate, and destroy
+resources on without asking first.
+
+- **Test VMs/containers are allowed, but must be torn down.** Create a
+  scratch VM or container here (e.g. via
+  `scripts/proxmox/create-proxmox-resource.sh` or raw `qm`/`pct create`)
+  to validate something. Anything created this way must be destroyed
+  again in the same session, before ending the task — never leave a test
+  resource running. Use a VMID/name that's obviously scratch (and doesn't
+  collide with a real flake target) so it's unambiguous what's safe to
+  remove.
+- **Node-level config is still not yours to change.** Creating/destroying
+  your own scratch guests is fine; Proxmox host config, storage pools, and
+  networking on `pve-test` itself are still the operator's call to make
+  manually, same as on `pve1`.
+
 ## Commands

 ```bash
 # One-time environment bootstrap (installs Nix if missing, prints hosts)
 bash scripts/codex-setup.sh

-# Full validation: secret grep, nixpkgs-fmt --check, statix lint, eval all hosts
+# Changed-files-only validation: secret grep (whole repo), nixpkgs-fmt --check
+# and statix on changed *.nix files, eval of the hosts/packages those changes
+# can affect. This is what CI runs on every push/PR.
 bash scripts/codex-maintenance.sh

-# Same, plus a dry-run build (no result symlink) of every host's toplevel
-bash scripts/codex-maintenance.sh dry-run
+# Full sweep: nixpkgs-fmt --check/statix over the whole tree, eval every host
+# and package. Slow (minutes) -- CI never runs this; use it locally before a
+# release or after touching modules/common/*, flake.nix, or variables.nix for
+# extra confidence beyond the automatic full-fallback those paths already
+# trigger in the default mode (see below).
+bash scripts/codex-maintenance.sh --full-check
+
+# Either mode, plus a dry-run build (no result symlink) of every host/package
+# in whichever scope is active
+bash scripts/codex-maintenance.sh --dry-run
+bash scripts/codex-maintenance.sh --full-check --dry-run

 # List the hosts the flake currently exposes
 nix eval --json .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]'
@@ -58,18 +137,28 @@ maintenance script pulls them via `nix run github:NixOS/nixpkgs/nixos-25.11#<too
 There is no test suite — "correctness" here means the flake evaluates and
 `nixpkgs-fmt`/`statix` are clean.

-**In an interactive agent session**, prefer targeted checks over full-repo
-sweeps: after editing one or two hosts/modules, evaluate just the
-`nixosConfigurations.<host>` you touched (plus any `config.system.build.tarball`
-/`diskoImagesScript`/package output affected) rather than looping over every
-host — `codex-maintenance.sh` evaluates every `nixosConfigurations` host plus
-every package/tarball/image variant and is slow to run after each small
-change. Reserve a full
-`codex-maintenance.sh` run for changes that plausibly affect every host
-(`modules/common/*`, `flake.nix`, `variables.nix`) or as a final check before
-committing. This is a session-workflow preference only — it does not apply to
-CI, which should keep running the full script on every push/PR regardless of
-diff size; that's the point of it.
+With no flags, `codex-maintenance.sh` diffs against a base ref (env
+`MAINT_BASE_SHA`, else the PR base SHA in CI, else `HEAD^` locally) and scopes
+fmt-check/statix to the changed `*.nix` files and eval to the hosts/packages
+those changes can affect — a `hosts/<name>/host.nix` edit only evals that
+host's targets, a `modules/platforms/<platform>.nix` edit only evals that
+platform's hosts, and so on. A change to `flake.nix`, `flake.lock`,
+`variables.nix`, `modules/common/*`, or any other `modules/*.nix` file outside
+`platforms/`/`build-types/` (whose blast radius isn't safely inferable from
+the path alone) falls back to evaluating every host and package, same as
+`--full-check` would, just without the whole-tree fmt/statix sweep. This
+exists because the whole-tree sweep is what was timing out CI; **CI always
+runs the plain, no-flag form and never passes `--full-check`.**
+
+The default mode's diff is against the working tree (uncommitted and staged
+edits included, not just committed ones), so it's already the right tool for
+an interactive session too: after editing one or two hosts/modules, plain
+`bash scripts/codex-maintenance.sh` naturally scopes to just what you
+touched. Reserve `--full-check` for changes that plausibly affect every host
+(`modules/common/*`, `flake.nix`, `variables.nix` — though the default mode
+already falls back to evaluating everything for those paths, `--full-check`
+additionally re-checks fmt/statix over the whole tree) or as a final check
+before committing.

 ## Scripts

@@ -116,6 +205,22 @@ lift the shared part into `scripts/lib/` instead of copying it.
  Purely a local filesystem copy — never touches `.sops.yaml`/
  `secrets/*.yaml` or the repo at all. The resulting file is exactly what
  `rotate-admin-key.sh` expects as its backup-key argument.
+- `scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run]
+  [--host <name>]` — detects drift between the ed25519 SSH host key
+  nix-cache is actually serving right now (via `ssh-keyscan`) and
+  `vars.nixCacheHostKey` (`variables.nix`), the value
+  `modules/nix-cache/remote-builder-client.nix` bakes into every real
+  client's declarative `programs.ssh.knownHosts` and
+  `configure-nix-cache-client.sh` hardcodes as its own default for
+  non-NixOS clients. That value has no automatic source of truth — it's
+  set once from whatever nix-cache's host key happened to be at the time,
+  and silently goes stale if the host is ever rebuilt/recreated with a new
+  key, breaking every client's distributed-build SSH trust with no error
+  that points back here. `--check` (used by `codex-maintenance.sh`, which
+  treats an unreachable nix-cache — e.g. from a non-LAN CI runner — as a
+  silent skip rather than a failure) only reports drift; the no-flags form
+  updates both files in place. Declarative clients still need a rebuild to
+  pick up the fix.

 ### `scripts/proxmox/`

@@ -36,6 +36,11 @@ Check the Proxmox node itself, or `/etc/flake-target` on a running host (see
 below), if you need to know what's really out there right now.
 `scripts/proxmox/create-proxmox-resource.sh`'s duplicate-host guard works the same
 way: it checks the Proxmox node directly rather than any file here.
+Real, production deployments live on `pve1.sweet.home`; there's a second
+node, `pve-test.sweet.home`, set aside purely for scratch/test resources —
+see `scripts/env.sh` (`PVE1_HOST` / `PVE_TEST_HOST`, and the
+`--node`/`PROXMOX_HOST` targeting they feed into) and CLAUDE.md's Proxmox
+section for which is which.

 Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
 (hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay
@@ -74,10 +79,19 @@ Safe validation commands for Codex and local review:

 ```bash
 bash scripts/codex-setup.sh
-bash scripts/codex-maintenance.sh dry-run
 bash scripts/codex-maintenance.sh
 ```

+`codex-maintenance.sh` with no flags (what CI runs on every push/PR) scopes
+fmt-check/statix/eval to files changed against a base ref — fast, but only
+as thorough as the diff. For the full sweep (every host, every package,
+fmt-check and statix over the whole tree — slow, CI never runs this):
+
+```bash
+bash scripts/codex-maintenance.sh --full-check
+bash scripts/codex-maintenance.sh --full-check --dry-run
+```
+
 For individual host evaluation:

 ```bash
@@ -8,9 +8,14 @@ and to verify that declared NixOS hosts still evaluate after dependency updates.
 - A scheduled workflow runs `nix flake update` once per week.
 - On GitHub, any resulting `flake.lock` change is proposed through a pull request.
 - On Gitea, the workflow can commit and push `flake.lock` directly when PR automation is not configured.
- A separate CI workflow evaluates every configured host before merge, listed
-  dynamically via `nix eval --json .#nixosConfigurations --apply builtins.attrNames`
-  rather than hand-enumerated, so it can't drift as `<platform>-<buildtype>`
+- A separate CI workflow runs `scripts/codex-maintenance.sh` before merge.
+  Its default mode scopes eval to the hosts/packages a change can affect,
+  determined from a git diff against the PR base — but a `flake.lock` change
+  is treated as repo-wide and always falls back to evaluating every host, so
+  a lock-file update PR still gets full coverage. Hosts are still listed
+  dynamically via
+  `nix eval --json .#nixosConfigurations --apply builtins.attrNames` rather
+  than hand-enumerated, so that fallback can't drift as `<platform>-<buildtype>`
  targets are added or removed. See `README.md` for the current target list.

 ## Why hosts should stop using `--upgrade-all`
@@ -46,18 +46,26 @@ the new key up automatically on next activation — no more manual

 ## Remote builder SSH keys

-On each client, install the private key used to authenticate as `nixremote`:
+Each client authenticates as `nixremote` using its **own default root SSH
+identity** (`/root/.ssh/id_ed25519`) — not a separately-named or shared
+keypair. If a client doesn't have one yet:

 ```bash
-sudo install -d -m 0700 /root/.ssh
-sudo install -m 0600 ./nixremote /root/.ssh/nixremote
-sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
+sudo ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
 ```

-On `nix-cache`, install the matching public key used by `nixremote` authorized keys.
+Then add its `.pub` contents as a new entry in `vars.remoteBuilderAuthorizedKeys`
+(`variables.nix`) and rebuild `nix-cache` to pick it up (that list is
+declarative — an imperative `ssh-copy-id nixremote@nix-cache` won't stick;
+it gets overwritten on every rebuild). Verify with:

-The committed `nixremote` authorized keys are public SSH keys only. Keep the
-matching private keys on client hosts and out of the repository.
+```bash
+sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
+```
+
+The committed `remoteBuilderAuthorizedKeys` entries are public SSH keys
+only. Keep the matching private keys on client hosts and out of the
+repository.

 nix-cache's own SSH *host* key is trusted declaratively via
 `programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
@@ -76,8 +84,8 @@ After deployment:
 curl http://nix-cache/nix-cache-info
 nix store ping --store http://nix-cache
 nix show-config | grep -E 'substituters|trusted-public-keys|builders-use-substitutes'
-sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
-nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/nixremote 4 2 big-parallel,kvm,nixos-test,benchmark' -L
+sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
+nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/id_ed25519 4 2 big-parallel,kvm,nixos-test,benchmark' -L
 nix path-info -r nixpkgs#hello
 curl -I "http://nix-cache/$(basename "$(nix path-info nixpkgs#hello)").narinfo"
 ```
@@ -119,10 +119,24 @@

      # Same installer environment, built as netboot (kernel + initrd +
      # iPXE script) instead of an ISO — this is what packages.pxe bundles.
+      #
+      # Deliberately imports common.nix directly, NOT ./modules/installer/iso.nix
+      # (which pulls in nixpkgs' installation-cd-minimal.nix) -- confirmed live
+      # that composing the ISO module together with netboot-minimal.nix hangs
+      # every boot waiting for a device that can never exist on a netboot
+      # client ("A start job is running for /dev/disk/by-label/nixos-minimal-...").
+      # Both installation-cd-base.nix and netboot.nix set fileSystems."/" via
+      # the identical lib.mkImageMediaOverride (mkOverride 60) priority --
+      # genuinely conflicting root-filesystem strategies (ISO-by-label vs.
+      # netboot-tmpfs) at the same priority, and the ISO one was winning.
+      # netboot-minimal.nix's own chain (netboot-base.nix) already imports
+      # profiles/installation-device.nix independently, so common.nix's
+      # initialHashedPassword override (which assumes that profile is
+      # present) still applies correctly without iso.nix in the mix.
      netbootSystem = nixpkgs.lib.nixosSystem {
        inherit system;
        modules = [
-          ./modules/installer/iso.nix
+          ./modules/installer/common.nix
          ({ modulesPath, ... }: {
            imports = [
              (modulesPath + "/installer/netboot/netboot-minimal.nix")
@@ -9,11 +9,25 @@ let
  # (the VM platforms rely on automount itself to get that same
  # non-blocking behavior, so they don't need `nofail` too).
  automountOpts = if config.boot.isContainer then [ "nofail" ] else [ "x-systemd.automount" ];
+
+  # A bare hostname here never resolves reliably: systemd-resolved only
+  # ever tries LLMNR for single-label names (never DNS, regardless of any
+  # configured search domain), and a *global* search domain (the first fix
+  # attempted here) backfires worse -- confirmed live on lxc-docker, adding
+  # `networking.search` made systemd-resolved prioritize its domain-matched
+  # but server-less global scope over eth0's correctly-configured one for
+  # every "*.sweet.home" query, silently sending them to public fallback
+  # DNS instead. `resolvectl query --interface=eth0 server.sweet.home`
+  # resolved fine throughout, proving the LAN DNS server was never the
+  # problem -- only the ambient, unqualified device string was. Using the
+  # FQDN directly sidesteps all of that, matching the pattern
+  # ../raspi/mount-data.nix already uses for the same reason.
+  nfsServer = "${vars.nfsServerHost}.${vars.homeDomain}";
 in
 {
  fileSystems = {
    ${vars.nfsShares.dockerConfig.mountpoint} = {
-      device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath}";
+      device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath}";
      fsType = "nfs";

      options = [
@@ -24,7 +38,7 @@ in
    };

    ${vars.nfsShares.dockerDatabases.mountpoint} = {
-      device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath}";
+      device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath}";
      fsType = "nfs";

      options = [
@@ -35,7 +49,7 @@ in
    };

    ${vars.nfsShares.dockerVolumes.mountpoint} = {
-      device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
+      device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
      fsType = "nfs";

      options = [
@@ -46,7 +60,7 @@ in
    };

    ${vars.nfsShares.nextcloudData.mountpoint} = {
-      device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath}";
+      device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath}";
      fsType = "nfs";

      options = [
@@ -57,7 +71,7 @@ in
    };

    ${vars.nfsShares.raspiVolumes.mountpoint} = {
-      device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath}";
+      device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath}";
      fsType = "nfs";

      options = [
@@ -1,10 +1,14 @@
 { pkgs, vars, ... }:

 {
-  # Install the remote builder key on each client host (do not commit private keys):
-  #   sudo install -d -m 0700 /root/.ssh
-  #   sudo install -m 0600 ./nixremote /root/.ssh/nixremote
-  #   sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
+  # Authenticate as nixremote using the client host's own default root SSH
+  # identity (/root/.ssh/id_ed25519) rather than a separately-named key --
+  # matches vars.remoteBuilderAuthorizedKeys, which already authorizes
+  # each host's own default key (one entry per host, not a shared
+  # dedicated keypair). If this host doesn't have one yet:
+  #   sudo -u root ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
+  #   # then add its .pub to vars.remoteBuilderAuthorizedKeys and rebuild nix-cache
+  #   sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
  # Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
  # can connect the first time without a manual ssh-keyscan/known_hosts
  # step on every new client.
@@ -20,7 +24,7 @@
      {
        hostName = vars.nixCacheHost;
        sshUser = vars.remoteBuilderUser;
-        sshKey = "/root/.ssh/${vars.remoteBuilderUser}";
+        sshKey = "/root/.ssh/id_ed25519";
        inherit (pkgs.stdenv.hostPlatform) system;
        maxJobs = 4;
        speedFactor = 2;
@@ -1,4 +1,4 @@
-{ lib, modulesPath, flakeTarget, ... }:
+{ config, lib, modulesPath, flakeTarget, ... }:

 let
  # Bakes this exact flake target's pre-generated SSH host key straight
@@ -58,8 +58,28 @@ in
    # host.nix declares each host's real hostname (networking.hostName);
    # keep that instead of letting Proxmox's ambient container config win.
    manageHostName = true;
-    # Unprivileged matches how these containers are actually created.
-    privileged = false;
+    # Unprivileged by default -- matches how these containers are actually
+    # created (scripts/proxmox/create-proxmox-resource.sh reads this value
+    # back to decide `pct create`'s --unprivileged flag, so the two stay
+    # in sync).
+    #
+    # lxc-docker is the one exception: the kernel's NFS client doesn't set
+    # FS_USERNS_MOUNT, so mounting NFS from inside *any* non-init user
+    # namespace -- which is exactly what an unprivileged container's
+    # UID-mapped root runs in -- is rejected at the VFS layer with EPERM,
+    # no matter what Proxmox's own `mount=nfs;nfs4` container feature
+    # allows at the AppArmor layer (confirmed live: TCP to the NFS server
+    # succeeds, the server's export table matches the container's IP, and
+    # `mount.nfs: Operation not permitted` still fires immediately with no
+    # corresponding denial anywhere in the server's logs -- a kernel-level
+    # rejection, not a network or export-permission one). Keying off
+    # hostName rather than something docker-build-type-specific because
+    # modules/build-types/docker.nix is also composed for linode-docker/
+    # proxmox-docker, which don't import proxmox-lxc.nix at all --setting
+    # this option there would break their eval with "option does not
+    # exist" regardless of any mkIf guard, since mkIf only makes a value
+    # conditional, not whether the option needs to exist somewhere.
+    privileged = config.networking.hostName == "docker";
  };

  boot.loader = {
@@ -129,5 +129,6 @@ echo "flake.lock still points at the old input revisions until refreshed. Either
 echo "  nix flake update nixpkgs home-manager   # just these two inputs"
 echo "  nix flake update                        # everything — see docs/flake-lock-automation.md"
 echo
-echo "Then run 'bash scripts/codex-maintenance.sh dry-run' before committing —"
-echo "a channel bump can shift option defaults across every host."
+echo "Then run 'bash scripts/codex-maintenance.sh --full-check --dry-run' before"
+echo "committing — a channel bump can shift option defaults across every host,"
+echo "and only --dry-run actually builds anything to catch that."
@@ -1,4 +1,21 @@
 #!/usr/bin/env bash
+# Validation entry point for CI and local/agent review.
+#
+# Default mode (what CI runs on every push/PR): fmt-check, statix, and eval
+# are scoped to files that actually changed against a base ref, plus
+# whichever hosts/packages those changes can affect. This exists because
+# the unscoped sweep below is slow enough to time out CI runners -- see
+# --full-check.
+#
+# --full-check: the historical full sweep (every host, every package,
+# fmt --check ./statix check . over the whole tree). Slow -- minutes, not
+# seconds. CI never passes this; run it locally before a release or after
+# touching modules/common/*, flake.nix, or variables.nix if you want extra
+# confidence beyond what the changed-files scope already covers for those
+# paths (see below).
+#
+# --dry-run: adds `nix build --dry-run --no-link` for whatever scope is
+# active (changed-files scope by default, full scope under --full-check).
 set -euo pipefail

 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -7,7 +24,39 @@ source "${script_dir}/lib/nix-bootstrap.sh"
 # shellcheck source=lib/nix-eval.sh
 source "${script_dir}/lib/nix-eval.sh"

-MODE="${1:-validate}"
+repo_root="$(cd "${script_dir}/.." && pwd)"
+cd "$repo_root"
+
+full_check=false
+dry_run=false
+
+usage() {
+  cat <<'EOF'
+Usage: scripts/codex-maintenance.sh [--full-check] [--dry-run]
+
+  --full-check  Run the full sweep: fmt-check and statix over the whole
+                repo, eval every host and package. Slow. Never run by CI.
+  --dry-run     Additionally run `nix build --dry-run --no-link` for
+                whatever scope is active.
+
+With neither flag (the CI default), fmt-check/statix/eval are scoped to
+files changed against a base ref (env MAINT_BASE_SHA, else the PR base,
+else HEAD^), plus the hosts/packages those changes can affect.
+EOF
+}
+
+for arg in "$@"; do
+  case "$arg" in
+    --full-check) full_check=true ;;
+    --dry-run) dry_run=true ;;
+    -h|--help) usage; exit 0 ;;
+    *)
+      echo "Unknown argument: $arg" >&2
+      usage >&2
+      exit 1
+      ;;
+  esac
+done

 ensure_nix_profile

@@ -16,12 +65,6 @@ if ! command -v nix >/dev/null 2>&1; then
  exit 127
 fi

-hosts="$(list_flake_targets .)"
-
-echo "Hosts:"
-echo "$hosts"
-
-echo
 echo "Checking for obvious committed secrets..."
 if grep -RInE 'github_pat_|ghp_|access-tokens|hashedPassword[[:space:]]*=' \
  --exclude-dir=.git \
@@ -33,17 +76,178 @@ else
  echo "No obvious token patterns found."
 fi

+mapfile -t all_hosts < <(list_flake_targets .)
+mapfile -t all_packages < <(nix eval --json "${NIX_EVAL_FLAGS[@]}" .#packages.x86_64-linux --apply builtins.attrNames | jq -r '.[]')
+
+# host_targets_for_dir <hosts-subdir-name>
+# Prints the nixosConfigurations target names whose hostPath is
+# ./hosts/<dir>/host.nix, derived straight from flake.nix's generatedTargets
+# (one mkTarget { ... } call per line) rather than a hand-maintained table,
+# so it can't drift the way a copied mapping would.
+host_targets_for_dir() {
+  local dir="$1"
+  grep -oE '^[[:space:]]*[A-Za-z0-9_-]+ = mkTarget \{[^}]*hostPath = \./hosts/'"${dir}"'/host\.nix;[^}]*\};' flake.nix \
+    | sed -E 's/^[[:space:]]*([A-Za-z0-9_-]+) = mkTarget.*/\1/' \
+    || true
+}
+
+declare -a changed_files=()
+scope_desc="full repo"
+
+if ! $full_check; then
+  resolve_base_ref() {
+    if [[ -n "${MAINT_BASE_SHA:-}" ]] && git cat-file -e "${MAINT_BASE_SHA}^{commit}" 2>/dev/null; then
+      echo "$MAINT_BASE_SHA"
+      return
+    fi
+    if git rev-parse --verify -q HEAD^ >/dev/null 2>&1; then
+      echo "HEAD^"
+      return
+    fi
+    git hash-object -t tree /dev/null
+  }
+
+  base_ref="$(resolve_base_ref)"
+  echo
+  echo "Changed-files scope: diffing against ${base_ref}"
+  mapfile -t changed_files < <(git diff --name-only --diff-filter=ACMR "$base_ref" -- . | sort -u)
+
+  if [[ ${#changed_files[@]} -eq 0 ]]; then
+    echo "No changed files detected."
+  else
+    printf '  %s\n' "${changed_files[@]}"
+  fi
+  scope_desc="changed files only (base: ${base_ref})"
+fi
+
+# Whole-tree fmt/lint always run under --full-check; otherwise scoped below.
+declare -a changed_nix_files=()
+for f in "${changed_files[@]:-}"; do
+  [[ "$f" == *.nix && -f "$f" ]] && changed_nix_files+=("$f")
+done
+
 echo
 echo "Checking Nix formatting with nixpkgs-fmt..."
+if $full_check; then
  nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check .
+elif [[ ${#changed_nix_files[@]} -gt 0 ]]; then
+  nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check "${changed_nix_files[@]}"
+else
+  echo "No changed .nix files; skipping."
+fi

 echo
 echo "Running statix lint..."
+if $full_check; then
  nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check .
+elif [[ ${#changed_nix_files[@]} -gt 0 ]]; then
+  for f in "${changed_nix_files[@]}"; do
+    nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check "$f"
+  done
+else
+  echo "No changed .nix files; skipping."
+fi
+
+# Figure out which hosts/packages this run needs to eval (and, under
+# --dry-run, build). full_check always means "everything"; otherwise a
+# change to flake.nix/flake.lock/variables.nix/modules/common/* (repo-wide
+# inputs) or to any other modules/*.nix outside platforms//build-types
+# (whose blast radius isn't safely inferable from the path alone -- see
+# CLAUDE.md's "Grep modules/build-types/*.nix for each build type's imports
+# list") also falls back to everything, on the same reasoning CLAUDE.md
+# already gives interactive sessions for when to run the full sweep.
+# Anything more targeted -- a host.nix, a platform module, a build-type
+# module -- narrows to just the hosts it can affect.
+declare -A affected_hosts=()
+eval_packages=false
+
+if $full_check; then
+  for h in "${all_hosts[@]}"; do affected_hosts[$h]=1; done
+  eval_packages=true
+else
+  full_fallback=false
+  for f in "${changed_files[@]:-}"; do
+    case "$f" in
+      flake.nix|flake.lock|variables.nix|modules/common/*)
+        full_fallback=true
+        ;;
+    esac
+  done
+
+  if ! $full_fallback; then
+    for f in "${changed_files[@]:-}"; do
+      case "$f" in
+        hosts/*/*)
+          hostdir="${f#hosts/}"
+          hostdir="${hostdir%%/*}"
+          while IFS= read -r t; do
+            [[ -n "$t" ]] && affected_hosts[$t]=1
+          done < <(host_targets_for_dir "$hostdir")
+          ;;
+        modules/platforms/*.nix)
+          platform="$(basename "$f" .nix)"
+          for h in "${all_hosts[@]}"; do
+            [[ "$h" == "${platform}-"* ]] && affected_hosts[$h]=1
+          done
+          ;;
+        modules/build-types/*.nix)
+          buildtype="$(basename "$f" .nix)"
+          for h in "${all_hosts[@]}"; do
+            [[ "$h" == *"-${buildtype}" ]] && affected_hosts[$h]=1
+          done
+          ;;
+        modules/installer/*)
+          # iso.nix (imported by both the "installer" nixosConfigurations
+          # target and netbootSystem, which backs packages.pxe) pulls in
+          # common.nix, so a common.nix change reaches all three.
+          affected_hosts[installer]=1
+          eval_packages=true
+          ;;
+        modules/pxe-boot/*)
+          # stage-installer-artifacts.nix is imported by
+          # modules/build-types/pxe-boot.nix only -- same blast radius as a
+          # build-types/*.nix change, not a packages one.
+          for h in "${all_hosts[@]}"; do
+            [[ "$h" == *"-pxe-boot" ]] && affected_hosts[$h]=1
+          done
+          ;;
+        modules/*)
+          full_fallback=true
+          ;;
+      esac
+    done
+  fi
+
+  if $full_fallback; then
+    echo
+    echo "Changed files affect shared config; falling back to evaluating every host/package."
+    for h in "${all_hosts[@]}"; do affected_hosts[$h]=1; done
+    eval_packages=true
+  fi
+fi
+
+mapfile -t hosts < <(for h in "${!affected_hosts[@]}"; do echo "$h"; done | sort)

 echo
-echo "Evaluating host toplevel derivations..."
-for host in $hosts; do
+echo "Checking nix-cache host key for drift..."
+if bash "${script_dir}/secrets/sync-nix-cache-host-key.sh" --check; then
+  :
+else
+  drift_status=$?
+  if [[ "$drift_status" -eq 2 ]]; then
+    echo "nix-cache unreachable from here -- skipping host-key drift check."
+  else
+    echo "WARNING: nix-cache's host key has drifted from variables.nix (see above)." >&2
+    echo "  Run 'bash scripts/secrets/sync-nix-cache-host-key.sh' to fix." >&2
+  fi
+fi
+
+echo
+if [[ ${#hosts[@]} -eq 0 ]]; then
+  echo "No hosts affected by changed files; skipping host eval."
+else
+  echo "Evaluating host toplevel derivations (${scope_desc})..."
+  for host in "${hosts[@]}"; do
    echo "==> $host"
    nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"

@@ -63,19 +267,23 @@ for host in $hosts; do
        ;;
    esac
  done
+fi

 echo
+if ! $eval_packages; then
+  echo "No packages affected by changed files; skipping package eval."
+else
  echo "Evaluating buildable packages..."
-packages="$(nix eval --json "${NIX_EVAL_FLAGS[@]}" .#packages.x86_64-linux --apply builtins.attrNames | jq -r '.[]')"
-for pkg in $packages; do
+  for pkg in "${all_packages[@]}"; do
    echo "==> packages.x86_64-linux.${pkg}"
    nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
  done
+fi

-if [[ "$MODE" == "dry-run" ]]; then
+if $dry_run; then
  echo
-  echo "Running dry-run builds for all hosts. This will not create result symlinks."
-  for host in $hosts; do
+  echo "Running dry-run builds for the active scope. This will not create result symlinks."
+  for host in "${hosts[@]:-}"; do
    echo "==> Dry-run build: $host"
    nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel"

@@ -91,13 +299,15 @@ if [[ "$MODE" == "dry-run" ]]; then
    esac
  done

+  if $eval_packages; then
    echo
-  echo "Running dry-run builds for all packages."
-  for pkg in $packages; do
+    echo "Running dry-run builds for packages."
+    for pkg in "${all_packages[@]}"; do
      echo "==> Dry-run build: packages.x86_64-linux.${pkg}"
      nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
    done
  fi
+fi

 echo
 echo "Maintenance checks complete."
@@ -82,13 +82,6 @@ if ! command -v jq >/dev/null 2>&1; then
 fi

 echo "Available NixOS hosts:"
-hosts="$(list_flake_targets .)"
-echo "$hosts"
+list_flake_targets .

-echo "Evaluating all host toplevel derivations..."
-for host in $hosts; do
-  echo "==> Evaluating $host"
-  nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
-done
-
-echo "Codex setup complete."
+echo "Codex setup complete. Run bash scripts/codex-maintenance.sh to validate changes."
@@ -6,12 +6,22 @@
 # environment (e.g. PROXMOX_STORAGE=tank-nvme ./scripts/proxmox/create-proxmox-resource.sh ...)
 # since each one only sets a default if unset.

-# SSH-reachable Proxmox node that scripts/proxmox/create-proxmox-resource.sh runs
-# pct/qm on. Matches the Proxmox web UI hostname already used in
-# hosts/nixos/home.nix's desktop shortcuts (pve.<homeDomain> from
-# variables.nix) -- change this if that's not actually reachable over SSH,
-# or if you're targeting a different node in a multi-node cluster.
-: "${PROXMOX_HOST:=pve.sweet.home}"
+# Two SSH-reachable Proxmox nodes exist on the LAN:
+#   - pve1.sweet.home      -- production. Real, live VMs/containers.
+#   - pve-test.sweet.home  -- sandbox/test node, for scratch VMs/containers
+#     that don't belong on production.
+#
+# PROXMOX_HOST is what scripts/proxmox/create-proxmox-resource.sh actually
+# targets by default -- overridable per-invocation with --node <hostname>,
+# or per-variable as usual (e.g. PROXMOX_HOST=$PVE_TEST_HOST). It defaults
+# to production, matching this repo's behavior before pve-test existed --
+# see CLAUDE.md's "Two Proxmox nodes" section for the policy on which
+# situations should target which node (in particular: Claude defaults to
+# pve-test, not this variable's own default, unless explicitly told
+# otherwise).
+: "${PVE1_HOST:=pve1.sweet.home}"
+: "${PVE_TEST_HOST:=pve-test.sweet.home}"
+: "${PROXMOX_HOST:=$PVE1_HOST}"
 : "${PROXMOX_SSH_USER:=root}"

 # Where this flake repo lives on the Proxmox node itself.
@@ -25,7 +35,7 @@
 # Storage pool names -- Proxmox's own stock-install defaults, but this
 # varies a lot by setup (ZFS pool name, custom LVM-thin volume, etc.).
 # Verify with `pvesm status` on the node and correct these if wrong.
-: "${PROXMOX_STORAGE:=local-lvm}"     # VM disks / CT rootfs
+: "${PROXMOX_STORAGE:=local-zfs}"     # VM disks / CT rootfs
 : "${PROXMOX_ISO_STORAGE:=local}"     # uploaded images/ISOs/CT templates

 : "${PROXMOX_BRIDGE:=vmbr0}"
@@ -63,10 +73,10 @@
 # nothing here forces a mount to happen.
 : "${PROXMOX_DEFAULT_LXC_FEATURES:=nesting=1,keyctl=1,mount=nfs;nfs4}"

-export PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE PROXMOX_ISO_STORAGE \
-  PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES PROXMOX_DEFAULT_MEMORY_MB \
-  PROXMOX_DEFAULT_LXC_DISK_GB PROXMOX_DEFAULT_LXC_FEATURES \
-  PROXMOX_REMOTE_REPO_DIR
+export PVE1_HOST PVE_TEST_HOST PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE \
+  PROXMOX_ISO_STORAGE PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES \
+  PROXMOX_DEFAULT_MEMORY_MB PROXMOX_DEFAULT_LXC_DISK_GB \
+  PROXMOX_DEFAULT_LXC_FEATURES PROXMOX_REMOTE_REPO_DIR

 # Matches variables.nix's nixCacheHost -- update both if it ever changes.
 : "${NIX_CACHE_HOST:=nix-cache}"
@@ -34,3 +34,21 @@ flake_target_hostname() {
  nix eval --raw "${NIX_EVAL_FLAGS[@]}" \
    "${flake_ref}#nixosConfigurations.${target}.config.networking.hostName" 2>/dev/null
 }
+
+# flake_target_lxc_privileged <flake_ref> <target>
+# Prints "true" or "false" for one lxc-* target's config.proxmoxLXC.privileged
+# (modules/platforms/lxc.nix is the single source of truth -- e.g.
+# lxc-docker sets this true so it can NFS-mount; every other lxc-* host
+# stays unprivileged). Only meaningful for lxc-* targets -- the option
+# doesn't exist for linode-*/proxmox-* (nixpkgs' proxmox-lxc.nix, which
+# declares it, is only ever imported by modules/platforms/lxc.nix). Empty
+# (not an error under set -e) if the eval fails.
+flake_target_lxc_privileged() {
+  local flake_ref="$1" target="$2"
+  # Not --raw: the option is a Nix boolean, and --raw can only coerce
+  # strings ("cannot coerce a Boolean to a string"). Plain `nix eval`
+  # prints a bare `true`/`false` for a boolean, which is exactly the
+  # string this needs.
+  nix eval "${NIX_EVAL_FLAGS[@]}" \
+    "${flake_ref}#nixosConfigurations.${target}.config.proxmoxLXC.privileged" 2>/dev/null
+}
@@ -0,0 +1,221 @@
+#!/usr/bin/env bash
+# Ad hoc clone of a single VM/CT from pve1 (production) to pve-test
+# (sandbox), via vzdump + qmrestore/pct restore -- not a general-purpose
+# backup tool, just a quick "give me a disposable copy of this thing on
+# pve-test" for testing against real-ish data without touching prod.
+#
+# Flow:
+#   1. vzdump the resource on pve1 into its "local" storage (--mode
+#      snapshot by default, so the source keeps running throughout --
+#      see --mode below for when that's not possible).
+#   2. Stream the resulting archive straight from pve1 to pve-test
+#      (ssh pve1 cat ... | ssh pve-test cat > ...) -- this machine is
+#      just the relay, no separate on-disk staging copy here.
+#   3. qmrestore / pct restore it on pve-test under --new-vmid (default:
+#      same VMID as the source -- pve-test is a separate node/cluster, so
+#      no collision unless that VMID is already in use there too).
+#      Always restored with --unique 1 (fresh MAC addresses) since the
+#      source is typically still running on the same LAN -- restoring
+#      with the *same* MAC would put two live guests on the wire with
+#      identical hardware addresses.
+#   4. Delete the vzdump archive from pve1's local storage and the
+#      relayed copy on pve-test, so neither node accumulates ad hoc
+#      backup files from this script. Only the pve1 original is
+#      preserved on any failure after step 1, so a failed
+#      transfer/restore can be retried without re-running the backup.
+#
+# This script's own defaults are pve1 -> pve-test, unlike
+# create-proxmox-resource.sh's --node (which defaults to production) --
+# see CLAUDE.md's "Two Proxmox nodes" section. pve1 is only ever touched
+# here after typing the source VMID back to confirm; pve-test is treated
+# as disposable, matching this repo's usual policy for that node.
+#
+# See --help for the full option list.
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
+# shellcheck source=../env.sh
+source "${repo_root}/scripts/env.sh"
+# shellcheck source=../lib/confirm.sh
+source "${repo_root}/scripts/lib/confirm.sh"
+
+usage() {
+  cat <<EOF
+Usage: $0 --vmid <n> [options]
+
+  --vmid <n>              Required: VMID on the source node to clone.
+                           Kind (qemu VM vs LXC CT) is auto-detected.
+  --new-vmid <n>           VMID to restore as on the target node
+                           (default: same as --vmid).
+  --mode snapshot|suspend|stop
+                           vzdump backup mode (default: snapshot -- the
+                           source resource keeps running throughout;
+                           requires snapshot-capable storage, e.g.
+                           ZFS/LVM-thin/Ceph/qcow2). Fall back to
+                           "suspend" (brief pause) or "stop" (source
+                           goes down for the duration) if the source's
+                           storage doesn't support live snapshots --
+                           vzdump's own error will say so.
+  --source-node <host>      (default: \$PVE1_HOST, ${PVE1_HOST})
+  --target-node <host>       (default: \$PVE_TEST_HOST, ${PVE_TEST_HOST})
+  --source-storage <pool>      Where vzdump writes the backup on the
+                               source node (default: local).
+  --target-storage <pool>       Where the restored disk/rootfs lands on
+                               the target node (default: \$PROXMOX_STORAGE, ${PROXMOX_STORAGE}).
+  --keep-backup                  Don't delete the vzdump archive from
+                                 either node afterward (debugging aid).
+  --yes                            Skip the typed VMID confirmation
+                                   before touching the source node.
+  --dry-run                          Print the full plan and skip every
+                                     mutating step (vzdump, transfer,
+                                     restore, delete) and the confirm
+                                     prompt. Still makes read-only SSH
+                                     calls to look up the source kind
+                                     and check the target VMID is free
+                                     -- harmless on either node.
+  -h, --help
+EOF
+}
+
+vmid=""
+new_vmid=""
+mode="snapshot"
+source_node="$PVE1_HOST"
+target_node="$PVE_TEST_HOST"
+source_storage="local"
+target_storage="$PROXMOX_STORAGE"
+keep_backup=0
+skip_confirm=0
+dry_run=0
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+  --vmid) vmid="$2"; shift 2 ;;
+  --new-vmid) new_vmid="$2"; shift 2 ;;
+  --mode) mode="$2"; shift 2 ;;
+  --source-node) source_node="$2"; shift 2 ;;
+  --target-node) target_node="$2"; shift 2 ;;
+  --source-storage) source_storage="$2"; shift 2 ;;
+  --target-storage) target_storage="$2"; shift 2 ;;
+  --keep-backup) keep_backup=1; shift ;;
+  --yes) skip_confirm=1; shift ;;
+  --dry-run) dry_run=1; shift ;;
+  -h | --help) usage; exit 0 ;;
+  *) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;;
+  esac
+done
+
+if [[ -z "$vmid" ]]; then
+  echo "ERROR: --vmid is required." >&2
+  usage >&2
+  exit 1
+fi
+if [[ "$mode" != "snapshot" && "$mode" != "suspend" && "$mode" != "stop" ]]; then
+  echo "ERROR: --mode must be snapshot, suspend, or stop." >&2
+  exit 1
+fi
+[[ -z "$new_vmid" ]] && new_vmid="$vmid"
+
+source_target="${PROXMOX_SSH_USER}@${source_node}"
+target_target="${PROXMOX_SSH_USER}@${target_node}"
+
+# No dry-run wrapper needed for the calls below: every mutating step
+# (vzdump, transfer, restore, delete) is reached only after the --dry-run
+# early-exit further down, so a plain `ssh` call is never in the dry-run
+# path.
+
+# --- identify the resource kind on the source node -----------------------
+echo "==> Looking up VMID ${vmid} on ${source_node}..."
+kind=""
+if ssh "$source_target" "qm status ${vmid}" >/dev/null 2>&1; then
+  kind="vm"
+elif ssh "$source_target" "pct status ${vmid}" >/dev/null 2>&1; then
+  kind="lxc"
+else
+  echo "ERROR: VMID ${vmid} doesn't exist on ${source_node} as either a VM or CT." >&2
+  exit 1
+fi
+echo "VMID ${vmid} on ${source_node} is a ${kind}."
+
+# --- refuse to clobber an existing resource on the target node -----------
+if ssh "$target_target" "qm status ${new_vmid}" >/dev/null 2>&1 \
+  || ssh "$target_target" "pct status ${new_vmid}" >/dev/null 2>&1; then
+  echo "ERROR: VMID ${new_vmid} already exists on ${target_node}. Pass --new-vmid" >&2
+  echo "with a free ID, or remove the existing resource there first." >&2
+  exit 1
+fi
+
+echo
+echo "Plan:"
+echo "  source: ${kind} VMID ${vmid} on ${source_node} (storage: ${source_storage}, mode: ${mode})"
+echo "  target: VMID ${new_vmid} on ${target_node} (storage: ${target_storage}, fresh MAC via --unique)"
+[[ "$keep_backup" -eq 1 ]] && echo "  backup archives are kept on both nodes afterward (--keep-backup)"
+
+if [[ "$dry_run" -eq 1 ]]; then
+  echo
+  echo "[dry-run] No backup, transfer, restore, or delete was performed."
+  exit 0
+fi
+
+if [[ "$skip_confirm" -ne 1 ]]; then
+  echo
+  if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) to confirm backing it up from ${source_node}: "; then
+    echo "Cancelled -- input didn't match ${vmid}." >&2
+    exit 1
+  fi
+fi
+
+# --- vzdump on the source node --------------------------------------------
+echo
+echo "==> Backing up VMID ${vmid} on ${source_node} (mode=${mode}, storage=${source_storage})..."
+vzdump_log="$(ssh "$source_target" \
+  "vzdump ${vmid} --mode ${mode} --storage ${source_storage} --compress zstd" 2>&1)" \
+  || {
+    echo "$vzdump_log" >&2
+    echo "ERROR: vzdump failed on ${source_node}." >&2
+    exit 1
+  }
+echo "$vzdump_log"
+
+archive="$(echo "$vzdump_log" | grep -oP "creating vzdump archive '\K[^']+" | tail -n1)"
+if [[ -z "$archive" ]]; then
+  echo "ERROR: couldn't find the archive path in vzdump's output above." >&2
+  exit 1
+fi
+archive_basename="$(basename "$archive")"
+target_tmp_archive="/var/tmp/${archive_basename}"
+echo "Archive: ${archive}"
+
+# Always clean up the relayed copy on the target node, success or failure
+# -- it's only ever a working copy, restored or not.
+cleanup_target_tmp() {
+  if [[ "$keep_backup" -ne 1 ]]; then
+    ssh "$target_target" "rm -f '${target_tmp_archive}'" >/dev/null 2>&1 || true
+  fi
+}
+trap cleanup_target_tmp EXIT
+
+# --- relay the archive from source to target ------------------------------
+echo
+echo "==> Transferring archive to ${target_node}..."
+ssh "$source_target" "cat '${archive}'" | ssh "$target_target" "cat > '${target_tmp_archive}'"
+
+# --- restore on the target node --------------------------------------------
+echo
+echo "==> Restoring as VMID ${new_vmid} on ${target_node} (storage=${target_storage})..."
+if [[ "$kind" == "vm" ]]; then
+  ssh "$target_target" "qmrestore '${target_tmp_archive}' ${new_vmid} --storage ${target_storage} --unique 1"
+else
+  ssh "$target_target" "pct restore ${new_vmid} '${target_tmp_archive}' --storage ${target_storage} --unique 1"
+fi
+
+# --- clean up the source backup now that the restore succeeded -----------
+if [[ "$keep_backup" -ne 1 ]]; then
+  echo
+  echo "==> Deleting backup archive from ${source_node}'s ${source_storage} storage..."
+  ssh "$source_target" "rm -f '${archive}' '${archive}.notes' '${archive}.log'" >/dev/null 2>&1 || true
+fi
+
+echo
+echo "Done. VMID ${new_vmid} (${kind}) is now on ${target_node}, cloned from" \
+  "VMID ${vmid} on ${source_node}."
@@ -15,6 +15,16 @@
 # this repo -- update both if nix-cache is ever rebuilt with a new host
 # key or the cache signing key is rotated (see docs/nix-cache.md).
 #
+# REMOTE_BUILDER_KEY defaults to this machine's own default root SSH
+# identity (matches modules/nix-cache/remote-builder-client.nix's
+# convention for real NixOS clients: authenticate as nixremote with the
+# host's own default key, added individually to
+# vars.remoteBuilderAuthorizedKeys, rather than a separately-named or
+# shared keypair) -- generate one with
+# `ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519` if this machine
+# doesn't have one yet, then add its .pub to vars.remoteBuilderAuthorizedKeys
+# and rebuild nix-cache.
+#
 # Usage:
 #   sudo ./configure-nix-cache-client.sh [--dry-run] [--no-remote-builder] [--no-restart]
 #
@@ -24,9 +34,9 @@
 set -euo pipefail

 : "${NIX_CACHE_HOST:=nix-cache}"
-: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache}"
+: "${NIX_CACHE_HOST_KEY:=ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache}"
 : "${REMOTE_BUILDER_USER:=nixremote}"
-: "${REMOTE_BUILDER_KEY:=/root/.ssh/nixremote}"
+: "${REMOTE_BUILDER_KEY:=/root/.ssh/id_ed25519}"

 CACHE_PUB_KEY="cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4="
 FALLBACK_URL="https://cache.nixos.org/"
@@ -12,6 +12,17 @@
 # copies over the locally-managed host-keys/ (gitignored, so a git pull
 # alone wouldn't carry it) before building.
 #
+# --node (default: $PROXMOX_HOST, see scripts/env.sh) picks which of the two
+# LAN Proxmox nodes this runs against: production, pve1.sweet.home
+# ($PVE1_HOST, PROXMOX_HOST's own default), or the sandbox node,
+# pve-test.sweet.home ($PVE_TEST_HOST) -- pass --node "$PVE_TEST_HOST" (or
+# set PROXMOX_HOST=$PVE_TEST_HOST) to target the sandbox instead. See
+# CLAUDE.md's "Two Proxmox nodes" section: an agent session should default
+# to pve-test and only touch pve1 when the operator has explicitly said so
+# for the current task -- this script itself doesn't enforce that (its own
+# default is production, matching this repo's behavior before pve-test
+# existed), it's a policy for whoever/whatever is driving it.
+#
 # Usage:
 #   scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --host <name> [options]
 #   scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --list
@@ -120,7 +131,9 @@ Shared:
  --iso-storage <pool>            (default: \$PROXMOX_ISO_STORAGE, ${PROXMOX_ISO_STORAGE})
  --bridge <bridge>                (default: \$PROXMOX_BRIDGE, ${PROXMOX_BRIDGE})
  --node <host>                     Proxmox node to SSH into (default:
-                                    \$PROXMOX_HOST, ${PROXMOX_HOST})
+                                    \$PROXMOX_HOST, ${PROXMOX_HOST} --
+                                    production; the sandbox node is
+                                    \$PVE_TEST_HOST, ${PVE_TEST_HOST}).
  --dry-run                           Print the full plan; touch nothing
                                      local or remote, no prompts.
  -h, --help
@@ -766,13 +779,22 @@ if [[ "$type" == "lxc" ]]; then
  # 512M default otherwise (confirmed live: --memory 2048 left swap at
  # 512). Default to matching whatever --memory resolved to above.
  local_swap="${swap:-$memory}"
-  # --unprivileged 1: modules/platforms/lxc.nix sets proxmoxLXC.privileged
-  # = false, so the NixOS config inside the image assumes it's running as
-  # an unprivileged container (cgroup/capability/mount expectations baked
-  # in at boot). `pct create`'s own CLI default for this flag is
-  # privileged (unlike the web UI, which defaults its checkbox the other
-  # way) -- leaving it unset creates a privileged container running a
-  # NixOS config that assumes unprivileged, a real mismatch.
+  # --unprivileged: read back from modules/platforms/lxc.nix's own
+  # proxmoxLXC.privileged (via flake_target_lxc_privileged) rather than
+  # hardcoded, since that's no longer the same for every lxc-* target --
+  # lxc-docker sets it true so the container's NFS mounts work at all (the
+  # kernel's NFS client can't mount from inside any unprivileged
+  # container's user namespace, no matter what AppArmor allows -- see that
+  # option's own comment). The NixOS config inside the image bakes in
+  # cgroup/capability/mount expectations matching whichever value it was
+  # built with, so this must stay in sync with it -- `pct create`'s own
+  # CLI default for this flag is privileged (unlike the web UI, which
+  # defaults its checkbox the other way), so leaving it unset would create
+  # a privileged container running a NixOS config that assumes
+  # unprivileged for every target except lxc-docker, a real mismatch.
+  privileged_eval="$(flake_target_lxc_privileged "$repo_root" "$flake_target")"
+  unprivileged_flag=1
+  [[ "$privileged_eval" == "true" ]] && unprivileged_flag=0
  #
  # --features nesting=1,keyctl=1: required for a modern (v247+) systemd
  # guest to actually boot unprivileged -- confirmed live: without this,
@@ -790,16 +812,22 @@ if [[ "$type" == "lxc" ]]; then
  # hands the whole string to `ssh` as a single command for the *remote*
  # shell to parse -- unquoted, that `;` would be read as a remote
  # command separator and silently truncate this into two commands.
-  create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --unprivileged 1 --features '${PROXMOX_DEFAULT_LXC_FEATURES}' --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --swap ${local_swap} --net0 name=eth0,bridge=${bridge},ip=dhcp"
+  create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --unprivileged ${unprivileged_flag} --features '${PROXMOX_DEFAULT_LXC_FEATURES}' --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --swap ${local_swap} --net0 name=eth0,bridge=${bridge},ip=dhcp"
  remote "$create_cmd"
  remote "pct start ${vmid}"
 else
  echo "==> Creating VM ${vmid} (${name})..."
  # pre-enrolled-keys=0 disables OVMF's Secure Boot key pre-enrollment --
  # required, or systemd-boot (unsigned) can't be trusted by the firmware.
+  # --agent 1: wires up the virtio-serial channel QEMU exposes to the guest.
+  # modules/common/configuration.nix sets services.qemuGuest.enable = true
+  # on every host, so the guest-side qemu-ga daemon is already running --
+  # without this flag Proxmox never creates the channel it listens on, so
+  # `qm guest exec`/`qm agent` and the UI's IP-address display silently
+  # never work for any VM this script creates.
  remote "qm create ${vmid} --name ${name} --memory ${memory} --cores ${cores} \
 --net0 virtio,bridge=${bridge} --bios ovmf --machine q35 --scsihw virtio-scsi-pci \
--efidisk0 ${storage}:1,efitype=4m,pre-enrolled-keys=0"
+--efidisk0 ${storage}:1,efitype=4m,pre-enrolled-keys=0 --agent enabled=1"

  if [[ "$dry_run" -eq 1 ]]; then
    echo "[dry-run] ssh ${ssh_target} -- qm importdisk ${vmid} ${remote_path} ${storage}"
@@ -0,0 +1,110 @@
+#!/usr/bin/env bash
+# Detects and fixes drift between the ed25519 SSH host key nix-cache is
+# actually serving right now and vars.nixCacheHostKey (variables.nix) --
+# the value modules/nix-cache/remote-builder-client.nix bakes into every
+# client's declarative programs.ssh.knownHosts, and
+# scripts/proxmox/configure-nix-cache-client.sh hardcodes as its own
+# default for non-NixOS clients.
+#
+# This value has no automatic source of truth: nix-cache's host key is
+# generated once (first boot / container recreate) and never touches this
+# repo again unless someone remembers to update it by hand afterwards. It
+# drifted silently once already -- confirmed live: variables.nix recorded
+# a key that no longer matched what nix-cache actually presented, which
+# would fail every real client's SSH host-key verification for
+# distributed builds without ever producing an obvious error pointing
+# back here (a client just sees "Host key verification failed" against
+# *some* key, with no hint that the trusted value itself was stale).
+#
+# codex-maintenance.sh runs this in --check mode on every invocation so
+# that drift surfaces as a warning instead of a future debugging session.
+#
+# Usage:
+#   scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run] [--host <name>]
+#
+#   --check     Only report drift (exit 1 if found, 2 if nix-cache is
+#               unreachable); never writes. For CI/maintenance use.
+#   --dry-run   Show what would change; never writes.
+#   --host      Override the hostname to scan (default: variables.nix's
+#               nixCacheHost / env.sh's NIX_CACHE_HOST).
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
+variables_nix="${repo_root}/variables.nix"
+client_script="${repo_root}/scripts/proxmox/configure-nix-cache-client.sh"
+
+# shellcheck source=../env.sh
+source "${repo_root}/scripts/env.sh"
+
+check_only=0
+dry_run=0
+host="${NIX_CACHE_HOST}"
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --check) check_only=1; shift ;;
+    --dry-run) dry_run=1; shift ;;
+    --host)
+      host="${2:?--host requires a hostname}"
+      shift 2
+      ;;
+    -h|--help)
+      sed -n '2,23p' "$0"
+      exit 0
+      ;;
+    *)
+      echo "ERROR: unknown argument: $1" >&2
+      exit 1
+      ;;
+  esac
+done
+
+current_value="$(grep -oE 'nixCacheHostKey = "[^"]+"' "$variables_nix" | sed -E 's/nixCacheHostKey = "(.*)"/\1/')"
+if [[ -z "$current_value" ]]; then
+  echo "ERROR: couldn't find nixCacheHostKey in $variables_nix" >&2
+  exit 1
+fi
+current_type_blob="$(awk '{print $1, $2}' <<<"$current_value")"
+current_label="$(awk '{print $3}' <<<"$current_value")"
+
+echo "Scanning ${host} for its current ed25519 SSH host key..."
+nix_extra_opts
+scanned="$(nix-shell "${NIX_OPTS[@]}" -p openssh --run "ssh-keyscan -t ed25519 -T 5 '${host}'" 2>/dev/null | grep -v '^#' | head -1 || true)"
+if [[ -z "$scanned" ]]; then
+  echo "ERROR: couldn't reach ${host} (or got no ed25519 host key back) via ssh-keyscan." >&2
+  exit 2
+fi
+scanned_type_blob="$(awk '{print $2, $3}' <<<"$scanned")"
+
+if [[ "$current_type_blob" == "$scanned_type_blob" ]]; then
+  echo "Up to date: ${host}'s host key matches variables.nix's nixCacheHostKey."
+  exit 0
+fi
+
+echo "DRIFT DETECTED:"
+echo "  variables.nix has: $current_type_blob"
+echo "  ${host} is now:    $scanned_type_blob"
+
+if [[ "$check_only" -eq 1 ]]; then
+  echo
+  echo "Run 'scripts/secrets/sync-nix-cache-host-key.sh' (no flags) to fix." >&2
+  exit 1
+fi
+
+new_value="${scanned_type_blob} ${current_label}"
+
+if [[ "$dry_run" -eq 1 ]]; then
+  echo "(--dry-run: would update variables.nix and ${client_script##*/} to:)"
+  echo "  $new_value"
+  exit 0
+fi
+
+sed -i "s|nixCacheHostKey = \"[^\"]*\"|nixCacheHostKey = \"${new_value}\"|" "$variables_nix"
+sed -i "s|NIX_CACHE_HOST_KEY:=[^}]*}|NIX_CACHE_HOST_KEY:=${new_value}}|" "$client_script"
+
+echo "Updated variables.nix and ${client_script##*/} to:"
+echo "  $new_value"
+echo
+echo "This only takes effect on already-deployed NixOS clients after their"
+echo "next rebuild (programs.ssh.knownHosts is declarative). Review with"
+echo "'git diff', then run 'bash scripts/codex-maintenance.sh' before committing."
@@ -5,85 +5,121 @@ sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaMUpxYmUzaVY4d1pRY3g3
-            empaVUJSN0JaSlNQQlJtZENwc0VMYVhJSkFvCmJQMkpSd3RoYzlKQjBSZWErQzR2
-            aG5SVndOQWpSRTBDSWJVQkg0c0hiNFUKLS0tIHhiaEdpY3gwZkpCcHl2TW5CNThn
-            aVVUQy9Qd0trb0RNdUpVTXdrdGlrTmcK1uphQAyDV+Gk5+K1YOqw1Z8ynGP5sAPF
-            q5icujja/SGexX18hPYXbkyUtOrBYjW62gCuGJinSBPROoFUJbiP7g==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQUFhWVFVlVlBnNE5FTnMz
+            VkxkTmxpRXlzZ3pSNTVZWFUrSllsYWo0alRnCkJSc25TYktSTFFJdkQydHcxOUlj
+            ajhQU1ZIb1lodEpHTnVhQjJ6WEthaDQKLS0tIDJCY1E2UVBaU3BoMzhXUXlIdnMv
+            djZTcE1rcWNTOXFPMmFDYTVoRGo4ZTQKYy8g6pqP3VpTKDIBPbnC8NzCdDvOCKnL
+            14kSrKmKlzefTrbkVyriz2Jdl2s0F374yfQQFreZ3m4AffSACCxziQ==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKOWZxcFlhRHhKelhDS29M
-            NGRlTEhJUGUzcFV6QVFCdTNkdEN3MlljWnl3CnI3LzVxSHBwSi9TYlRUcFdyN2py
-            cVZBV0Q3Z3FhRXA0T1NFQzd2R00yeEUKLS0tIHkxL2dHV3RkVFYxZTJNTVZvVFIr
-            TXpJZzdnYlpJaXBmcjdWWUtxNkc4dWMKVsImJiavzUzSFn78pciNJPHaS3KWqJer
-            VkxF6kF3tl2HmW46eVXtsQowdu+zVR9HS35i/HvQ1r1TyP6qT5ofcQ==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBVQnpVWC9wcnIreG9GeE9U
+            UWhuRytkc2Flc0hyQm5yMjZnelNwaXhlWWc4CjBkWnd4cHNRRXQ0UXFkZGp4QlR1
+            eW5NNnE1WFhnb054M1pac2ZidFg4Y3MKLS0tIFdTNmk2V1l2WC9rUk8yd0ZnOEJS
+            VkNnejVGVUZPZkorQkltVEplN2FmdTAKRY7DPP5HeFQntn2f/fXLjU6M1V6iug86
+            BD09PI+T2DbIBQPotRZisw8IzHu9gY/O3+h0TccyIsXjI9wy/XPCAQ==
            -----END AGE ENCRYPTED FILE-----
          recipient: age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyWjdSanRRaEI4VHFZVkdw
-            NkxMODRuakFJTFNoTHZIRHptZFNxaGZ3K3lnCjJGR1FJalRsYmtLcHFxZW8yNU5W
-            MzBjWmltbzI3MTByUjB6djlEdllHNXMKLS0tIEhEOVhLVjZkRi9vUEtDWUhxT0NX
-            eUZ3bGdEaHdGbXZIYkowYUZuTWJ3d2MKMwgxOqlMH7GfLlseD4J277Dcg0KCD3d5
-            jwJDb82kRWoBMicTJZXoq/5oe5blJNa7dWYoqgkYxMA2O+0igT+I6g==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkczFSTVhxWHlIWjhRcWlV
+            V2JPQXd5Wnk5R3NwWC81T3Z0MW4vYnd5S1ZZClV1NlU1Tzd6UkxPQ2M4MmhLV01G
+            d3VIb0RhR1RiNTZqNjlQcmg2YjdPeGsKLS0tIDQ1RTFTWGN4MnEvWkRUR3VnN204
+            WVdFOXdmNC9FVFhBSGNEUUgyYWpYYzAKfdpeaFL/RrIbqpD9hNj8L7UxpmiBjE2I
+            go/dR2E1LLXsDPtnSuJb2EZYoFvSsjsIQQQDt+YwRv0fplRtssKdxQ==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsZzdRMm9zdlJHckFoREVs
-            UkVXL3dCeWRuN2NqcnVsOWVNT1FLUHY3andrCmNFY0ZCby9SdnUyYlV6R05YNDB0
-            em43SjRCNVhYbmpRZUVWT0cxQlhGdmsKLS0tIDZHazNyck5VNHBuNVM5bmRZUlpR
-            QzBSNy84VDdLVkZZbnNlUFYydXlreEEKYZaR2b7tyRAhPdP+ytpP0veUTi7pY9Nw
-            pK0h4hcegLNYJL1AfOYwFQoW7vb256GdmwdcuuBl1YBGXWGraaBnZg==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB6TjhmOTJ0bUpMQWowb0hB
+            T3Era1loU1pMdmkxdnAvRkViekpqZWZjaWgwCkFRQXhFUy9PRVBma2JMUDhqY2F1
+            VVFDRFNVbWpNaEczY1JVQUMyck9XdEkKLS0tIGpxc0tGdVFKK3FteVJKM1Fxa2ky
+            a21WLy9qV05hUURCTVBvcVh3cE45Z3cKXCYfXSjhApBoLbHDu2OOd57Y1zN54yy+
+            WDQvz8PpMxhc1nU5Kw/cI+WmL1KvN0qQZfOx/7D4W+dy/ZDWX27TpA==
            -----END AGE ENCRYPTED FILE-----
          recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFSnZiOGpsL1BESWVEdUxx
-            dXc5LzBHTlNodGJ1S3hOeHd5Wmgxb3BPSHl3CnBEWDNTc3NDdzN3RHQxUlNLOEs5
-            RU1SS2tVT29XbDJCWVliWnVkekN4L1kKLS0tIGFqY2pNK2h2S2ZWMndrRVN6eUdN
-            M0NLY2EvTTVIYlZvdk9XY0NwNE14ZlEKcOwgCK3g56kId/4tEt+2iCcylisn+Fca
-            5VWamsXdbwxbxmCEEUbgN3aHrdVz3mV4+8FZqA34yXz8pc45/PUcFg==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZK0FYQnBHMHZ6dlpMYTlC
+            WFdOWDFkRVBuY1pmdTFiUndLV3JXcndZa3pNCnJsd0tHN0FveWV6UUNQSEdpdWw5
+            dWZITkxWelNIRlpKS1pnN0ZmVlQvZjAKLS0tIEUwMXdtNFdkUWdIRjlxc0owdTRr
+            c1o1TmptWWd1ZGxzcWJJNzJ0K25PTTAKoos5rnkyQBCm+ZuhCCaMJwqJBo1fpnsl
+            G74wu5vbTBG4VjVhI5KqyiuiTRU4jPcGxysECqe7AyZUBGp7ndewgw==
            -----END AGE ENCRYPTED FILE-----
-          recipient: age1qz9d4ka4xgexujyd247s7lp737sulp5fhxl5d65fj2ykvc4j4edqrsdks8
+          recipient: age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJU1cydFFuRnpCUU5mbjFu
-            TTIxREkxZ1hMalVtNjNjcHVaUXpPNWdib2tzCkRScWRTWHhxN3lhckI4UFpMSFRa
-            RGxWcldlWnd5SVJFNWxQWjB3R0pMb3MKLS0tIGdjLzlhTHgyT2hjTmRTK1RiZmxS
-            OWRWYklBWkoyUG8raFNzd3JlRUtEUEUKw+NdAp1Mz0dEOUGPbqCV8y7029I3Kye0
-            keU2T29JGCN1D6x//1NcMUYSaFZKgv0ZVSjVUCl1EnmlJ0nBdbaDXA==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTVkVXaCtGTXVkRXRZMFZ1
+            ZUdXM1hRcDhaSURqcGh2eksxTUNCckk5SVJjCnhkSDRKdDFSckxUWXd4SmVxWG5p
+            bU5OV0hzaWR6VDFwcDY2WlY4WnN3cFEKLS0tIElOVzRCcXR4U0dhajJySUhaZGps
+            MW9rQk1JVDFWRnFxVzhCUkRIS09EamsK1rVidD48PqwlEWQyjF7iQWU7aBdPqQHy
+            z5LaSi3LvJX3rNE/+q0E8/gbZyjGpbEn3AUI5mBF64GY3IZkRxZSXQ==
            -----END AGE ENCRYPTED FILE-----
          recipient: age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnNFlzYVBqaGFDVE9xbmt5
-            OVd4ZlZRdEVJd0JFdkRIYVA5ZUEwYjJjWVJjCnZzTExNbDlYbGN6c1dYU2ZlNEVs
-            ZUo4MXdpc0tzbDI0MHl5eXc3d09VRDQKLS0tIGs4SGMya1RZQ0hpa2NtQk9Wa29C
-            bmxIYWk1VThsaFhxRDlOOGhGQ2tYamsKs+PcOiaeNFujCWwZBr+nq1MzrGohl+ch
-            TASI7eNsiHmSVRzSMHv4mX+8yXDHIRgHbFuUty2gdgGFLRjYXe4gDA==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvbnJCWU9UMXJXMG92b3hm
+            WWNBTFpQamVWQThITmt5QUVwR3h2OHI2SlVRCjcxOEJTVkFjN0NhamFZQ1plK29w
+            dk5XYkYxOXQ0YkVzcVc3VnhCQWlsV0UKLS0tIHZ5cWtFZUhDKzZkOE1BK2Y5TStR
+            SFlDRjE4ZHpiVEJOQk5TUGNEN1B4amMKUCJ8CL8QpmRpFs83HD9TUn7NrPguuP8S
+            JQH/bzPorXTXJuyOKuKAZq1hK8BmiMUFksaZ03yN6YaFVIOeelEEMg==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkcmVQYXV1NzRrQ2IxTElt
-            QUJRMkhiUUJqRDdwQW03K0lCTEtkSTU5UDJrCjh6SldpQkF1NzIybTJTdnBlOHdQ
-            VlREMy9hM0ZaL21Pa0VPUzcvb0pQQjQKLS0tIFcrK0I3Z3Nhem45eHpZYmdWeit6
-            eXFtRHVxODhaNTVDdVR1a2tOb1N1U1EKZBdLHq8PRMKX2ndFr3AxFVAZRyvhFa1u
-            72R5tordo4IR3HSxG3Z06rokOITd+KKhaQ8NEWEC8qioAAMxEC9QWw==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxWHV0S2dxeHZEMDNRSkhO
+            ZFJHaiszd0tzYzlzd3gvbXNSQmlMVlJUNGcwCnVrM29MdFZCR1NBYnpkQ1k5VFZQ
+            b3Z2Q3ZGekVQZkZKWGlka3NDOHJ0R1EKLS0tIDlXTmNzUk0wVXo0UWhkd0ZvK3FI
+            UzJxU3RkdWs4aTZYVVkrS056bTN1ek0KgKJNz8GvynX5pK33aW9x3v6yr2Ox0LCT
+            GGrt+ddbKLcwpBpYjfWkFhffO330EKui73S+c/qMf8N9j6wzalOTpQ==
            -----END AGE ENCRYPTED FILE-----
          recipient: age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0WjJPbXhXNTNyQUZzMmxS
-            Rkd3Yk5Ga3pPRXFtMFpXcmpBZ3RJYnJlaHhjClRDbFBHRVZ0SDdrNFdVWmUwalpT
-            WUt5a2RUTnQ4VFBGMlowR3JycDM3aXMKLS0tIERSeFd4cGdSbzRpSmpSQnB1Yng5
-            OHBma3lYeWEwQ2RvelpzZHRkN1JPWFEKLXkJfEkH3lo5Z4mj8PZXTfAfZn6B46To
-            U0G54tUh0U1EeMnI5ZPRGwDxi5K0eD68HjeE9MBvFBysOChP/ANDPQ==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3ZWJlbDl1TEkvWEpBMC9Y
+            ejY1MnFMUExBeDhITndxNy9YUi9hU0tXM1JJCjcwNEppcDdxYzlCSVMrMExWa3A4
+            cmFPSjV3LzkyMXZCUDU2QmtHRmpHRmsKLS0tIFFTdXBOaDJJTERseXlGbmdrQzhD
+            TWtnRFdIRXpsNkY0U1BiczNsdUk1V1kKGpndKmT8kj/oIxQuxQALfzscw+CsVmnj
+            cyPC3bF+tG6LcqqoKLjPSJfcIgzhnX7cAr/wwESavemLn8L/zQMe4w==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpQVVyY3UrSHFWSmpDdmRE
+            dlh5akFqVFdDVFNMSHE1eVJnZzR6YzFHSVFRCko4UE9EdXNxZzF2MW5PTTN6dEdU
+            ZHM1MGowcVB2Y1ZlOTVHdnNtY3diM2cKLS0tIDBYSmh5dVVPaTM3d0ErcC8wMDNB
+            eUpHWnZlYnJsbHZuS3pwbG15UGtwN2MKVPQA1MpjIfYAsNacoAbpvZNuAIkvx7ER
+            CvWBKEHUVm6m8905BXzv8MdGTAk0EyCIP3aMmYqTIYfv2k9pP0T08A==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0MTQ4MzN0bDJNL1l1bmpX
+            ekxDUXRNa0JHWWltZFNGTVltTFdSSE82SVZvCmdUWUdja3JIajMzY09IMUE5elox
+            MDdEakFJTmtkRWF2R1BGNkQ4U3grNWsKLS0tIE9hZUhkVGI1ZEpzdDhRU21EZm91
+            VnJNb1kyQ05MM0RJa1lLUEtjWWxkSTAKHVAKcGcWl6LncJALRBU9RKP7ot6C6GSE
+            1iZtj1SNX6wzEWrhOEnV37aQ8bKZj6u+Y/q6/vJ4qiBs78y/drdIzA==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBNUEVWY2NpVEU4OTZPZTZR
+            UzdKcktpUGJOcnJ0aEhkQlhvUWdUYkV2SkdzCm51eGJVeHJMcVRRRld0dFRCYUxr
+            TTN2WEhOVjRqV0FtQXowZWNTbkJneEUKLS0tIFpUazZpTUNWZUZBSFE0VDZZbkJu
+            SFVlUVhySnNqUENYOG9qUm5ZMDc1ZW8Kv0lY5dhnCEheM0sttfr4p7IL+EVog16T
+            OapUdbuXL2l7t7URzHnvfG/nbOtJIjH8a0XFsWyJChtNXpF2d/vf2g==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvTzc2aExGTHFrc0ZtL3pq
+            YlZLbzd3MVZHZTB0VUQvSXZ4NGVsZk42c2hBCk1vbTk0Tnl3b01vbVZaMkJ5aldE
+            WmZBMGFjb2pjQXpYcnBxWmp0UUsrdXcKLS0tIG16SG9JbFdkbmVidCsxUnpBR3V3
+            RzNOY3hIRWk4UXh6N3NrcjNSU3ZwWTgKaExY4U2s8E6ojljJ+4TU+YJhcLXyuVA1
+            ROB70jQCjFvQOeo6thjQohSSUoPKhxSl1/nr4ZiGBO3/VskzihckKg==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
    lastmodified: "2026-07-19T02:30:40Z"
    mac: ENC[AES256_GCM,data:UiL3VMDF6rq4Nr87KspcDx434q3tfNXeb5pwH2O+4ssNQ6xzcYDdzXBnhAY3zLBsqPMKrvHBd4Ot/gEMcq3FMIVe7Q6p9yWKpep66KZ/yWEhAlwIVhD79Oj8VS+1CHKjf25zpRdhZorp04oeFQQd9VfjJB4EE/Q1aVbwTGlpIic=,iv:i/0conaFgFia+wzNTdUL6tlSTw35HTK3Ap1Sr5RGHf8=,tag:ULbz5FllShA/JjlSRdxA0g==,type:str]
    unencrypted_suffix: _unencrypted
@@ -4,31 +4,31 @@ sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmOHlPcGZvN3o5aEZCcFdz
-            WlYydFdoQ0ZHbElMdTRkZWljMmoxTHZqNjJZCnE3cmgzKzZya3FROEczbHVveDND
-            VnEzRW12cnRKdzhkZm5uTXpkTEtrNUEKLS0tIHFDbkc4Mk4wVlM2R29zZXYwR2Ri
-            ODBML1p4eUZiZldYUERQTUhTU1llV0UKxjvH6zbW6wKghzR1o34CyKPEa2FqZmo0
-            PxgqyuXkIwas9soXVAkScx7ElaV09Fjaj+mDrKwi4a+DwdoSP7czyA==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBPWE1HTUhiSUp5ZEUwWEpI
+            bGpkZlBIMUo5ZlYrQ09SN3Q1a0ZkQ0ZnOEhVCnJPNEZQenVWWGZiODlzQzNEc1Zq
+            c3l4OWZJTElJc2Y2UE15OGtEUzhyY1EKLS0tIHNJUStyWnlQWjZBbEZjQ3UwdUpz
+            ZndoUDR6bisrNGJCUHk3TGI4bTZaMFUK87fFsm9ne9s+PK2pcwtrDjqyGBss2r2E
+            8lhqoeiKZ2j96z8kP/7ChzovwTCmqdcmAQuyNQD+ZAFijseipSvfbQ==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0VDBXYUZNbnpEZHlqV05C
-            WWxQZjVYcWlZeXlGc0RiaUpERzR0T21CWWtBCkM0dW5kYlFNK1RGSzRudFZ2Z0t3
-            aVBreElGM1BIT0RzQkxTVTA3S2NhQncKLS0tIERyTU8zWWRTRm95SmRZQ1BhalVV
-            SVdBajN1V1BuM2s4K216S3c0VHczNlkKM5jvsSEfCBA5uZRjBJNbM91lLRQkj+jK
-            rM5uSfGLTvSjPgXIMIq03OXxH1CE7GoKxAPwFrJdAFMMQcutIethhw==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUYi9SRFFGV3Z6cFd2Znk5
+            b2FLbWtzTllJMDBUaGk0NTViOTNBa2hQclVZClZHKzNhbGVjQUJhWkFWdTFBMG5a
+            cUFJdUdyVG5HQXJRRnJId3hqRTN2cXMKLS0tIEFMRjh3WE1ON0U2TTNTZ3hxMTR4
+            ZGRlemlIbDZKeExmVHROc3Eyak5DdzQKaLwIVDi6BN4cxpVxJoqTYvJETPOp4thc
+            l9uVMvIGuEsEZgDsvShw1dYLljd+uGy/A+dXbcxIUCP/mmPkwmd1Pw==
            -----END AGE ENCRYPTED FILE-----
          recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvbk9tRE9jYkpxNThQeElH
-            dWpKYThDb1ZkUWtMWlRxc2tYbjhBbEFEcWtVCkNiK0NBRko2TkJQdUVtdGtHNGcv
-            andSaUlKSFA4THRyZXNTYmI0Yk5WbmsKLS0tIHlQTUtpWDdPeTVZL0M1RElRdFNk
-            QWlGdFo5NkZWSmY0YXdpNzNUQnlsK3MKtzC0bM7Ek+K73nMranOA1Mc98RUnYnq1
-            hAt0QEFKWK4QVKubaN/rG3AzE0U7qPKWHTzoxgnAiL3WyV9teLW+iA==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArOWovSW9DeFpxL0VDUDQ3
+            SHUwTzJVZUtPV01ZRkdCUXZGL2lTRCtCNFNnCjBSNExqRW5mTEN5SFVucHJHSzZt
+            cDlNc3BjY3M1c1k1Z2tkVEg4R1pacGsKLS0tIEFWbHNKZW0vbVh1Y2VhQW93OUwx
+            MWV0eW9sOXdQd0l2ZjlWOEVVc1dwcTgK2s4p9xoNkawH2OkGsl80bNIo3ad5vn4W
+            Z2w+jwppSoUmbQnD3WFbLmSSxmuobmU8HILwElv6SZu+KE3aspF6XA==
            -----END AGE ENCRYPTED FILE-----
-          recipient: age164px2a8e48ptsf9ngtan38aa6jls4jdl26mzrgzf6sn3vcvt49hqjrgr8w
+          recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
    lastmodified: "2026-07-19T23:30:21Z"
    mac: ENC[AES256_GCM,data:kLGE2xawQT7mx+sfw68hmGk5nCEGiEjZrqTEl9B1dtQmTrMwmoVr/1RISi4LfJrwxy31mDgff4lcIL4wIJuM373uk3X8j4RNyYQNTfKEkORT6r8NHeepNs267O77pKGd7OmcM4MT/BqOnB8ELS7Wlf2ect7CAlvUUVyc8icxgZE=,iv:EYLDsHYHZ1XOQXafOTqHHWpk/OBNq/R6IJnOBYV33E4=,tag:thxrCPC5oGvDjhK7Dz87YA==,type:str]
    unencrypted_suffix: _unencrypted
@@ -3,22 +3,40 @@ sops:
    age:
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBURmMzN3hrSlNrUkkvVWNl
-            L3M1dEhWeW14N0RFNVRPci9QK1YyTFdqRVVJCk5WaWswT2NicldkYzZjbVhYU2xu
-            MGFsNmUzeTN2TS9wOEdvRURpVUVYZXMKLS0tIDZ6MEdPTVhCaTQ2UXFWTUFtc0pm
-            MFlJb0c2WXJtMGRLZEZYY0pZWWpFWm8K/mlYZIe8UC0QU+1mq3NtrtTF5b2m5hCK
-            +K0QiZLTKmmDcr4bRhZ32VE7R7GRwtMNnOP/mElZvPAyWyHHhRiOHg==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaYU9HR0lETDhkYXR5NWlj
+            L0diSE5JQU9KWjRoU3hrVjdqZ2tPNUtOdFZvCjdUVGNFbDVYa3pVSC9ZWVNORER4
+            QWozUlhoSEtjTk9IRlM3VkZoYlc1RTAKLS0tIDlHS01WVWlOMWFEQU5GTVRLZVhV
+            VTcwekhrRHB3SlVYT2MzOW5GbE52dkkKKCWehPhpdGapdyzpll20NJUcZwvW/7X8
+            KQ1EqAgI2fewnbwuIDYCleN0b0SLJNUeSV/tFKDDoTMnHWCdeD4ECg==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
        - enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBURHlUL0RNMUtNallEcG5p
-            eG4xby8yVzdQUTBaNkl4ano0YjBMcDd0Wm5NCkZvaXNPZm9wemJkMmNSdGdOaTI4
-            Z1RwUnhiRUpCMWZaeWtlSVBmNW5KOXMKLS0tIDk0R0k3ZHczTFNCWUZxSWF0M0FJ
-            MGlZMmtuSFYrcG1meDNMWDNqSjFxcE0KDu2dAc0gqmmPkpbpBe4YohM7rYmUwEkI
-            V2FUQwjlvh50svtjCVdYbx2xuq4sQLnKelk/q1onLw60FwsVfzD8sQ==
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4YndybFBTQ2p4SGZ4SDJs
+            a2p0eFRQOVVWcGd2a25ESW9ESGx1RWw0Zno4CnhpVVh3cGI5UjY0YmFINFFPMTh4
+            b1B5SjJ3NTNvUE1QUmJjVFozY1dYS2MKLS0tIC9HZGNpOFZhZGNFZGt5blJuZXVV
+            SXpkRzV4d2ppV3ZQZSt1dmxYNGVFMUEKmSe9dkrmkND81Hw2/ATAmFvcmhk1tUC1
+            LxxBw54IVHUqwYKgRYUYRNu+pykDT5OnFDPiskd49Xso99LY87PyiA==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLd1FSSHVTSGdHWm8wMVQz
+            d0dlOEsxeGwxdHU0eUlFSUxka0ZmcVpnOTM0CndYNUVjVy92QVhNY2orQmpSQjYy
+            VS9KM0NUTXhuM0lCSDBZMWtISEdtWTgKLS0tIExTL2wvS3FEdVViUmRYZEFsR3R0
+            YTJFM200RjF6MjNxOFA4eHRpWVhtRkUKOqBIT445HnPXrrH/qV6FIuAhAuJmSL6V
+            +PQopM/m3PAnK5m5Mu3cfjYfDiB8+GWTABhljfT+GbcoK7CqWLehrw==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
+        - enc: |
+            -----BEGIN AGE ENCRYPTED FILE-----
+            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvVXF4a2NmaW15R2VXS3FP
+            SDVPUGpUWTlIWjl2N2N2SXR1UlRkdllWNVRJClZDbVo4dUhrZytqZkMrYWtpOEZx
+            c0dSZGViN04zQ1B1WEZEWm1QM1lsejQKLS0tIE0wa0k5Rm1xZmw3OHFESkVXc25j
+            NlFpYTJSckQ4MVlZQ01reDlkaWY3TkEKsmQlreRhRAjVZ/q5x52FNATDF2sLhbHo
+            djOZigZx2rs6shqQ6It/XRJ8CiPaXZBPOt529Gwmplu+hWlAU1+l2Q==
+            -----END AGE ENCRYPTED FILE-----
+          recipient: age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
    lastmodified: "2026-07-19T02:30:40Z"
    mac: ENC[AES256_GCM,data:rKHZjU/MH08ASTlu32HZO9uWmsBYuMCEC6M8gwVhzuWvmablnP05tS2z13XfaWaCEUXk6kmGJKuU0zu5+IKVZgamCF6DAMtxQb6bVCaLsoAm/GSqWQ5VI9eHqgnSSdN/o3ul/33Rf8iBQo4aw8FFAmDVuNz8bfAn0QefFTj0ByI=,iv:JD2gtqRinOY77etg6PUmZNovkYl1Q3F6ZvRi4x7RznQ=,tag:/5IMpWKRVt+l1luCTQE0BA==,type:str]
    unencrypted_suffix: _unencrypted
@@ -26,7 +26,7 @@
  # fresh client that has never manually ssh'd to nix-cache before. Update
  # this if nix-cache's host key is ever rotated or the host is rebuilt
  # from scratch.
-  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
+  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPeWgMsdaiz4axT/deFc1+0B5bN+GX/NOeW9bbQ0c/IT lxc-nix-cache";

  # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
  # host (modules/nix-cache/server.nix) — one per client host that's allowed
@@ -38,6 +38,7 @@
    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGtOWOCS+ImHc7NehguoyD7PbonGosKMZqc9+QR3v/h root@nixos"
    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHxXTQxFnArK5HXG7czeoybZebCGfxpUdusJkPn+BCSp root@server"
    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEj26SL/emsVjW2YhRucJVp2kTz8WgcEQgjBEBLRikk root@claude"
  ];

  # Admin SSH public key, authorized on the primary user of every host and
Author	SHA1	Message	Date
beatzaplenty	453c7b5513	Fix PXE netboot installer hanging at boot (ISO/netboot module conflict) Check NixOS configurations / eval-hosts (pull_request) Successful in 11m1s Details The netboot build composed ./modules/installer/iso.nix (which pulls in nixpkgs' installation-cd-minimal.nix) together with nixpkgs' netboot-minimal.nix. Both installation-cd-base.nix and netboot.nix set fileSystems."/" via the identical lib.mkImageMediaOverride (mkOverride 60) priority - genuinely conflicting root-filesystem strategies (ISO-by-label vs. netboot-tmpfs) at the same priority, and the ISO one was winning. Every netboot boot hung waiting for a device that can never exist outside a real CD/USB: A start job is running for /dev/disk/by-label/nixos-minimal-... Reproduced live: deployed a scratch lxc-pxe-boot on pve-test, pulled its built kernel/initrd, and booted them directly with QEMU to confirm the hang and capture full console output. netboot-minimal.nix's own chain (netboot-base.nix) already imports profiles/installation-device.nix independently, so common.nix's initialHashedPassword override still applies correctly with iso.nix removed from this composition. Rebuilt and re-booted the same way after the fix - full boot to a working shell with SSH up, no hang.	2026-07-21 22:15:44 +00:00
beatzaplenty	b3463e4b33	Merge pull request 'Add ad hoc pve1 -> pve-test clone script (vzdump + qmrestore/pct restore)' (#29 ) from worktree-clone-pve1-to-pve-test into main Check NixOS configurations / eval-hosts (push) Successful in 10m21s Details Reviewed-on: #29	2026-07-21 21:55:25 +00:00
beatzaplentyandClaude Sonnet 5	013b2c7009	Add ad hoc pve1 -> pve-test clone script (vzdump + qmrestore/pct restore) Check NixOS configurations / eval-hosts (pull_request) Successful in 10m21s Details Backs up a VM/CT on pve1 (snapshot mode by default, so the source stays online), relays the archive to pve-test, restores it there with fresh MAC addresses (--unique), and deletes both the source and relayed backup copies afterward -- no ad hoc backup files left behind on either node. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-21 20:22:56 +00:00
beatzaplenty	12f9153957	claude audit report Check NixOS configurations / eval-hosts (push) Successful in 10m36s Details	2026-07-21 20:07:30 +00:00
beatzaplenty	1004538f00	updated sops secrets	2026-07-21 20:07:14 +00:00
beatzaplentyandClaude Sonnet 5	87873300e1	Add pve-test.sweet.home as a second Proxmox target pve1.sweet.home is production; scripts/env.sh now also defines PVE_TEST_HOST for a separate sandbox node, individually targetable via --node/PROXMOX_HOST. Tooling defaults are unchanged (still pve1) -- the new restriction (Claude defaults to pve-test unless explicitly told to use pve1) is documented as policy in CLAUDE.md, not enforced in the scripts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-21 19:57:15 +00:00
beatzaplenty	e9e2312163	Merge pull request 'Make lxc-docker a privileged container: unprivileged can't NFS-mount at all' (#28 ) from worktree-fix-container-dns-search-domain into main Check NixOS configurations / eval-hosts (push) Successful in 10m24s Details Reviewed-on: #28	2026-07-21 08:50:14 +00:00
beatzaplentyandClaude Sonnet 5	6b09a808ed	Make lxc-docker a privileged container: unprivileged can't NFS-mount at all Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s Details The kernel's NFS client filesystem doesn't set FS_USERNS_MOUNT, so mounting NFS from inside any non-init user namespace -- exactly what an unprivileged LXC container's UID-mapped root runs in -- is rejected at the VFS layer with EPERM, regardless of Proxmox's mount=nfs;nfs4 container feature (which only patches the AppArmor layer). Confirmed live on the redeployed lxc-docker container: TCP to the NFS server's port 2049 succeeds, the server's export table matches the container's IP, and mount.nfs: Operation not permitted still fires immediately with no corresponding denial anywhere in the server's own logs -- a kernel-level rejection that no amount of DNS/ automount/export tweaking (this branch's earlier commits) could ever fix. modules/platforms/lxc.nix now keys proxmoxLXC.privileged off hostName ("docker" -> true) rather than a blanket false, since build-types/docker.nix is also composed for linode-docker/proxmox-docker, which don't import proxmox-lxc.nix at all -- setting this option there would break their eval. create-proxmox-resource.sh reads the value back via a new flake_target_lxc_privileged helper instead of hardcoding --unprivileged 1, so the two stay in sync automatically for every lxc-* target. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE	2026-07-21 05:57:06 +00:00
beatzaplenty	9496efdd22	Merge pull request 'Add pve.sweet.home guard rails to CLAUDE.md' (#26 ) from worktree-claude-md-pve-guardrails into main Check NixOS configurations / eval-hosts (push) Successful in 10m13s Details Reviewed-on: #26	2026-07-21 04:28:14 +00:00
beatzaplenty	33c9506c7d	Merge pull request 'Fix NFS mount device strings on lxc-docker: use FQDN, not search domain' (#25 ) from worktree-fix-container-dns-search-domain into main Check NixOS configurations / eval-hosts (push) Failing after 7m3s Details Reviewed-on: #25	2026-07-21 04:26:32 +00:00
beatzaplentyandClaude Sonnet 5	abe3763cb3	Add pve.sweet.home guard rails to CLAUDE.md Check NixOS configurations / eval-hosts (pull_request) Successful in 10m24s Details Codifies read-only access to existing Proxmox config/VMs/containers, allows scratch test VMs/containers as long as they're torn down again, and forbids any change to production on the node. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-21 02:01:15 +00:00
beatzaplenty	744904b19f	Merge pull request 'Enable QEMU guest agent on Proxmox VMs; register lxc-gui sops key' (#24 ) from worktree-flake-e2e-audit into main Check NixOS configurations / eval-hosts (push) Successful in 16m3s Details Reviewed-on: #24	2026-07-21 01:46:06 +00:00
beatzaplentyandClaude Sonnet 5	9cbaf1a070	Enable QEMU guest agent on Proxmox VMs; register lxc-gui sops key Check NixOS configurations / eval-hosts (pull_request) Successful in 16m19s Details create-proxmox-resource.sh's `qm create` never passed --agent, so despite services.qemuGuest.enable = true being set on every host, Proxmox never created the virtio-serial channel the guest agent needs -- qm guest exec and the UI's IP-address display silently never worked for any VM this script created. Found while live-testing every lxc-/proxmox- build type against pve.sweet.home for an end-to-end flake audit. Also registers a fresh sops age key for lxc-gui (no prior registration existed), generated while testing that target live -- needed before lxc-gui can be deployed with working secrets. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CZHwwAM7cacQKkqKfQ5e8a	2026-07-21 01:25:16 +00:00
beatzaplentyandClaude Sonnet 5	42da626397	Fix NFS mount device strings on lxc-docker: use FQDN, not search domain Check NixOS configurations / eval-hosts (pull_request) Successful in 17m29s Details The previous commit's networking.search fix was wrong. Confirmed live on lxc-docker (vmid 105) after redeploying with it: `resolvectl query server.sweet.home` started failing again, even though `resolvectl query --interface=eth0 server.sweet.home` still resolved correctly to the right IP via the LAN's real DNS server. The debug log showed why -- adding a global search domain via networking.search gave systemd-resolved a domain-matched but server-less "global" scope, which it now prioritizes over eth0's correctly-configured scope for every "*.sweet.home" query, silently sending them to public fallback DNS (1.1.1.1 et al) instead, which of course returns NXDOMAIN for an internal-only name. Bare single-label names (e.g. "server") were never going to work either way -- systemd-resolved only ever tries LLMNR for those, never DNS search-suffixing, regardless of configuration. Reverts the networking.search addition and instead has modules/docker/mount-data.nix build each NFS device string from "${vars.nfsServerHost}.${vars.homeDomain}" (a plain FQDN, no dependency on search-domain behavior at all) -- the same pattern modules/raspi/mount-data.nix already uses for the Raspberry Pi's share and for the identical reason. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE	2026-07-21 01:19:42 +00:00
beatzaplenty	d7aba8554d	Merge pull request 'Add global DNS search domain to fix NFS mounts on lxc-docker' (#23 ) from worktree-fix-container-dns-search-domain into main Check NixOS configurations / eval-hosts (push) Successful in 18m29s Details Reviewed-on: #23	2026-07-21 00:11:16 +00:00
beatzaplentyandClaude Sonnet 5	e176ff723d	Add global DNS search domain to fix NFS mounts on lxc-docker Check NixOS configurations / eval-hosts (pull_request) Failing after 40m2s Details Cross-host references throughout this repo (vars.nfsServerHost, vars.nixCacheHost, vars.dockerHost) are bare short names, not FQDNs. Resolving them has always depended on whatever network stack happens to be in play picking up the DHCP-advertised domain as a search suffix -- NetworkManager does this by default, which is why it went unnoticed everywhere else, but LXC containers force-disable NetworkManager and get their systemd-networkd config written directly by Proxmox instead, which never sets one. Confirmed live on lxc-docker (vmid 105) after today's earlier fix for the automount/mount=nfs bugs: systemd-resolved had no search domain for eth0, so "server" failed to resolve ("Name or service not known") while "server.sweet.home" resolved fine via the same DNS server -- every NFS mount in modules/docker/mount-data.nix was still failing. networking.search sets systemd-resolved's own Domains= globally rather than depending on the per-link DHCP path, so it isn't at the mercy of whichever component owns a given host's interface file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE	2026-07-20 18:22:12 +00:00
beatzaplenty	61bbe5e6da	clean up in ailse 3 Check NixOS configurations / eval-hosts (push) Failing after 8m41s Details	2026-07-20 17:49:35 +00:00
beatzaplenty	ab719cc8eb	Merge branch 'worktree-maintenance-script-changed-files-only' into worktree-resolve-maintenance-merge Check NixOS configurations / eval-hosts (push) Successful in 21m49s Details # Conflicts: # scripts/codex-maintenance.sh	2026-07-20 17:46:07 +00:00
beatzaplenty	91c977e5e7	Merge pull request 'Fix nix-cache remote-builder trust: stale host key + wrong sshKey path' (#21 ) from worktree-fix-nix-cache-host-key into main Check NixOS configurations / eval-hosts (push) Failing after 12m42s Details Reviewed-on: #21	2026-07-20 17:32:25 +00:00
beatzaplenty	7e9c0c2a6f	Rewrite codex-maintenance.sh to scope CI checks to changed files Check NixOS configurations / eval-hosts (pull_request) Canceled after 0s Details CI was running a full eval of every host + package on every push/PR, which was slow enough to routinely time out the Gitea runner. Default mode now diffs against a base ref and scopes nixpkgs-fmt/statix/eval to the files that changed and the hosts/packages they can affect; a change to flake.nix/flake.lock/variables.nix/modules/common/* (or any other modules/*.nix outside platforms//build-types, whose blast radius isn't inferable from the path) falls back to evaluating everything. The old full sweep moves behind --full-check, which CI never passes; --dry-run adds build-planning on top of whichever scope is active. Also trims codex-setup.sh's redundant full host eval loop -- that's what codex-maintenance.sh is for; setup should just install tooling.	2026-07-20 17:25:22 +00:00
beatzaplentyandClaude Sonnet 5	fd773b65da	Fix nix-cache remote-builder trust: stale host key + wrong sshKey path Check NixOS configurations / eval-hosts (pull_request) Failing after 11m12s Details variables.nix's nixCacheHostKey no longer matched nix-cache's actual SSH host key (confirmed via ssh-keyscan against the live container), so every declaratively-configured client's programs.ssh.knownHosts trusted the wrong key -- distributed builds would fail host-key verification. Also, modules/nix-cache/remote-builder-client.nix hardcoded sshKey to /root/.ssh/nixremote, but the `server` host only has its own default /root/.ssh/id_ed25519 installed (confirmed live via qm guest-agent) -- that file was never even present, so the build machine config pointed at nothing. Standardize on each client's own default identity, matching the per-host-key pattern vars.remoteBuilderAuthorizedKeys already uses instead of a shared/differently-named keypair, and add scripts/secrets/sync-nix-cache-host-key.sh (wired into codex-maintenance.sh's --check) so the host-key drift doesn't silently recur next time nix-cache is rebuilt or recreated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V7yVH71vGrDVzovh9UaMu8	2026-07-20 17:21:12 +00:00
beatzaplenty	d340aca403	Merge pull request 'Authorize this box's SSH key as a nix-cache remote-builder client' (#20 ) from add-claude-builder-key into main Check NixOS configurations / eval-hosts (push) Failing after 18m59s Details Reviewed-on: #20	2026-07-20 16:31:25 +00:00
beatzaplenty	bf8ee3ce48	Merge pull request 'Worktree refactor scripts shared lib' (#19 ) from worktree-refactor-scripts-shared-lib into main Check NixOS configurations / eval-hosts (push) Failing after 11m3s Details Reviewed-on: #19	2026-07-20 16:29:42 +00:00
beatzaplentyandClaude Sonnet 5	98d4545e8f	Authorize this box's SSH key as a nix-cache remote-builder client Check NixOS configurations / eval-hosts (pull_request) Failing after 11m4s Details Following the existing per-client-key pattern in vars.remoteBuilderAuthorizedKeys (one entry per host's own default root key, not a shared dedicated keypair -- docs/nix-cache.md's "./nixremote" filename is just a generic placeholder in the instructions). This box had no entry yet, which is why its earlier `ssh-copy-id nixremote@nix-cache` never stuck: nix-cache's nixremote authorized_keys is NixOS-declarative and gets overwritten on every rebuild. Public key only; requires a nix-cache rebuild to take effect. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-20 16:29:42 +00:00
beatzaplenty	1d44523181	Merge pull request 'Re-encrypt secrets/nix-cache.yaml for rotated lxc-nix-cache key' (#18 ) from worktree-nix-cache-502-fix into main Check NixOS configurations / eval-hosts (push) Failing after 11m15s Details Reviewed-on: #18	2026-07-20 15:33:31 +00:00
beatzaplentyandClaude Sonnet 5	222a3ced69	Re-encrypt secrets/nix-cache.yaml for current lxc-nix-cache key Check NixOS configurations / eval-hosts (pull_request) Failing after 11m53s Details .sops.yaml's &lxc-nix-cache anchor was rotated (commit `b3c8145`) after today's LXC rebuild, but secrets/nix-cache.yaml was never re-run through sops updatekeys -- it stayed encrypted for the pre-rotation key. Per docs/nix-cache.md and docs/auto-installer.md, sops-nix decrypt failures are silent: cache-priv-key never materialized, nix-serve had nothing to bind its secretKeyFile to and failed to start, and nginx proxy_pass had no upstream -- the 502 Bad Gateway seen from clients. Confirmed the new key matches the currently-deployed host by deriving the age identity from host-keys/lxc-nix-cache_ssh_host_ed25519_key.pub via ssh-to-age and comparing to .sops.yaml. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-07-20 15:29:34 +00:00