beatzaplentyandClaude Sonnet 5
53b9a64826
nix-serve's secretKeyFile was a manual, undocumented-outside-a-comment `nix-store --generate-binary-cache-key` step per host -- easy to miss on a fresh nix-cache instance (as lxc-nix-cache testing just found: systemd fails the unit with EXIT_CREDENTIALS when LoadCredential can't find the source file, which nginx then reports as a 502 from clients). It also can't be regenerated per-host safely: modules/nix-cache/client.nix hardcodes every client's trust in one specific public key, so every nix-cache instance has to share the exact same keypair. Sourced from secrets/nix-cache.yaml's new cache-priv-key entry instead, via the same sops-nix pattern every other secret in this repo already uses. Verified the added value derives to the exact public key modules/nix-cache/client.nix already trusts before committing. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
3.1 KiB
nix-cache architecture
This repository configures nix-cache as a binary cache server and a remote builder for other hosts.
Important design notes
- This is not a shared
/nix/storesetup. - Every machine still keeps and uses its own local
/nix/store. - Clients prefer
http://nix-cachefor substitutes and keephttps://cache.nixos.org/as fallback. - Clients can offload builds to
nix-cachethrough SSH (nix.distributedBuilds). - Client hosts import
modules/nix-cache/client.nixand, when remote building is enabled,modules/nix-cache/remote-builder-client.nix. - The
nix-cachehost importsmodules/nix-cache/server.nix.
Binary cache signing key
modules/nix-cache/client.nix hardcodes every client's trust in one
specific public key (cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4=).
That means whichever host is currently playing the nix-cache role has to
use that exact keypair — not a freshly generated one — or no client will
accept substitutes from it (they'd just silently fall back to building
from source). So unlike most per-host secrets, this one can't be
self-generated on first boot; it's managed via sops-nix like every other
secret in this repo, sourced from secrets/nix-cache.yaml's
cache-priv-key entry (modules/nix-cache/server.nix).
Adding or rotating the value:
nix-shell -p sops --run 'sops secrets/nix-cache.yaml'
Add (or replace) a cache-priv-key entry with the private key file's exact
contents. If you don't have it yet, generate a keypair once:
nix-store --generate-binary-cache-key nix-cache-1 cache-priv.pem cache-pub.pem
— paste cache-priv.pem's contents into the cache-priv-key entry above,
delete both local files afterward, and update
trusted-public-keys in modules/nix-cache/client.nix (and every already-built
client) to match cache-pub.pem if this is a genuine rotation rather than
a first-time bootstrap. Any nixos-configurations.*-nix-cache host picks
the new key up automatically on next activation — no more manual
/etc/nix/cache-priv.pem install step.
Remote builder SSH keys
On each client, install the private key used to authenticate as nixremote:
sudo install -d -m 0700 /root/.ssh
sudo install -m 0600 ./nixremote /root/.ssh/nixremote
sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
On nix-cache, install the matching public key used by nixremote authorized keys.
The committed nixremote authorized keys are public SSH keys only. Keep the
matching private keys on client hosts and out of the repository.
Manual verification
After deployment:
curl http://nix-cache/nix-cache-info
nix store ping --store http://nix-cache
nix show-config | grep -E 'substituters|trusted-public-keys|builders-use-substitutes'
sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/nixremote 4 2 big-parallel,kvm,nixos-test,benchmark' -L
nix path-info -r nixpkgs#hello
curl -I "http://nix-cache/$(basename "$(nix path-info nixpkgs#hello)").narinfo"