Archived
beatzaplentyandClaude Sonnet 5
fd773b65da
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m12s
variables.nix's nixCacheHostKey no longer matched nix-cache's actual SSH host key (confirmed via ssh-keyscan against the live container), so every declaratively-configured client's programs.ssh.knownHosts trusted the wrong key -- distributed builds would fail host-key verification. Also, modules/nix-cache/remote-builder-client.nix hardcoded sshKey to /root/.ssh/nixremote, but the `server` host only has its own default /root/.ssh/id_ed25519 installed (confirmed live via qm guest-agent) -- that file was never even present, so the build machine config pointed at nothing. Standardize on each client's own default identity, matching the per-host-key pattern vars.remoteBuilderAuthorizedKeys already uses instead of a shared/differently-named keypair, and add scripts/secrets/sync-nix-cache-host-key.sh (wired into codex-maintenance.sh's --check) so the host-key drift doesn't silently recur next time nix-cache is rebuilt or recreated. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01V7yVH71vGrDVzovh9UaMu8
92 lines
3.9 KiB
Markdown
92 lines
3.9 KiB
Markdown
# nix-cache architecture
|
|
|
|
This repository configures `nix-cache` as a **binary cache server** and a **remote builder** for other hosts.
|
|
|
|
## Important design notes
|
|
|
|
- This is **not** a shared `/nix/store` setup.
|
|
- Every machine still keeps and uses its own local `/nix/store`.
|
|
- Clients prefer `http://nix-cache` for substitutes and keep `https://cache.nixos.org/` as fallback.
|
|
- Clients can offload builds to `nix-cache` through SSH (`nix.distributedBuilds`).
|
|
- Client hosts import `modules/nix-cache/client.nix` and, when remote building is enabled, `modules/nix-cache/remote-builder-client.nix`.
|
|
- The `nix-cache` host imports `modules/nix-cache/server.nix`.
|
|
|
|
## Binary cache signing key
|
|
|
|
`modules/nix-cache/client.nix` hardcodes every client's trust in one
|
|
specific public key (`cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4=`).
|
|
That means whichever host is currently playing the `nix-cache` role has to
|
|
use that *exact* keypair — not a freshly generated one — or no client will
|
|
accept substitutes from it (they'd just silently fall back to building
|
|
from source). So unlike most per-host secrets, this one can't be
|
|
self-generated on first boot; it's managed via sops-nix like every other
|
|
secret in this repo, sourced from `secrets/nix-cache.yaml`'s
|
|
`cache-priv-key` entry (`modules/nix-cache/server.nix`).
|
|
|
|
**Adding or rotating the value:**
|
|
|
|
```bash
|
|
nix-shell -p sops --run 'sops secrets/nix-cache.yaml'
|
|
```
|
|
|
|
Add (or replace) a `cache-priv-key` entry with the private key file's exact
|
|
contents. If you don't have it yet, generate a keypair once:
|
|
|
|
```bash
|
|
nix-store --generate-binary-cache-key nix-cache-1 cache-priv.pem cache-pub.pem
|
|
```
|
|
|
|
— paste `cache-priv.pem`'s contents into the `cache-priv-key` entry above,
|
|
delete both local files afterward, and update
|
|
`trusted-public-keys` in `modules/nix-cache/client.nix` (and every already-built
|
|
client) to match `cache-pub.pem` if this is a genuine rotation rather than
|
|
a first-time bootstrap. Any `nixos-configurations.*-nix-cache` host picks
|
|
the new key up automatically on next activation — no more manual
|
|
`/etc/nix/cache-priv.pem` install step.
|
|
|
|
## Remote builder SSH keys
|
|
|
|
Each client authenticates as `nixremote` using its **own default root SSH
|
|
identity** (`/root/.ssh/id_ed25519`) — not a separately-named or shared
|
|
keypair. If a client doesn't have one yet:
|
|
|
|
```bash
|
|
sudo ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519
|
|
```
|
|
|
|
Then add its `.pub` contents as a new entry in `vars.remoteBuilderAuthorizedKeys`
|
|
(`variables.nix`) and rebuild `nix-cache` to pick it up (that list is
|
|
declarative — an imperative `ssh-copy-id nixremote@nix-cache` won't stick;
|
|
it gets overwritten on every rebuild). Verify with:
|
|
|
|
```bash
|
|
sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
|
|
```
|
|
|
|
The committed `remoteBuilderAuthorizedKeys` entries are public SSH keys
|
|
only. Keep the matching private keys on client hosts and out of the
|
|
repository.
|
|
|
|
nix-cache's own SSH *host* key is trusted declaratively via
|
|
`programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
|
|
sourced from `vars.nixCacheHostKey` (`variables.nix`) — every client rebuild
|
|
picks it up automatically, so distributed builds don't fail with "Host key
|
|
verification failed" on a client that has never manually SSH'd to nix-cache
|
|
before. If nix-cache's host key is ever rotated or the host rebuilt from
|
|
scratch, update `vars.nixCacheHostKey` to match its new
|
|
`/etc/ssh/ssh_host_ed25519_key.pub`.
|
|
|
|
## Manual verification
|
|
|
|
After deployment:
|
|
|
|
```bash
|
|
curl http://nix-cache/nix-cache-info
|
|
nix store ping --store http://nix-cache
|
|
nix show-config | grep -E 'substituters|trusted-public-keys|builders-use-substitutes'
|
|
sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
|
|
nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/id_ed25519 4 2 big-parallel,kvm,nixos-test,benchmark' -L
|
|
nix path-info -r nixpkgs#hello
|
|
curl -I "http://nix-cache/$(basename "$(nix path-info nixpkgs#hello)").narinfo"
|
|
```
|