nixos/docs/nix-cache.md

nix-cache architecture

This repository configures nix-cache as a binary cache server and a remote builder for other hosts.

Important design notes

• This is not a shared /nix/store setup.
• Every machine still keeps and uses its own local /nix/store.
• Clients prefer http://nix-cache for substitutes and keep https://cache.nixos.org/ as fallback.
• Clients can offload builds to nix-cache through SSH (nix.distributedBuilds).
• Client hosts import modules/nix-cache/client.nix and, when remote building is enabled, modules/nix-cache/remote-builder-client.nix.
• The nix-cache host imports modules/nix-cache/server.nix.

Binary cache signing key

modules/nix-cache/client.nix hardcodes every client's trust in one specific public key (cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4=). That means whichever host is currently playing the nix-cache role has to use that exact keypair — not a freshly generated one — or no client will accept substitutes from it (they'd just silently fall back to building from source). So unlike most per-host secrets, this one can't be self-generated on first boot; it's managed via sops-nix like every other secret in this repo, sourced from secrets/nix-cache.yaml's cache-priv-key entry (modules/nix-cache/server.nix).

Adding or rotating the value:

nix-shell -p sops --run 'sops secrets/nix-cache.yaml'

Add (or replace) a cache-priv-key entry with the private key file's exact contents. If you don't have it yet, generate a keypair once:

nix-store --generate-binary-cache-key nix-cache-1 cache-priv.pem cache-pub.pem

— paste cache-priv.pem's contents into the cache-priv-key entry above, delete both local files afterward, and update trusted-public-keys in modules/nix-cache/client.nix (and every already-built client) to match cache-pub.pem if this is a genuine rotation rather than a first-time bootstrap. Any nixos-configurations.*-nix-cache host picks the new key up automatically on next activation — no more manual /etc/nix/cache-priv.pem install step.

Remote builder SSH keys

Each client authenticates as nixremote using its own default root SSH identity (/root/.ssh/id_ed25519) — not a separately-named or shared keypair. If a client doesn't have one yet:

sudo ssh-keygen -t ed25519 -N '' -f /root/.ssh/id_ed25519

Then add its .pub contents as a new entry in vars.remoteBuilderAuthorizedKeys (variables.nix) and rebuild nix-cache to pick it up (that list is declarative — an imperative ssh-copy-id nixremote@nix-cache won't stick; it gets overwritten on every rebuild). Verify with:

sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version

The committed remoteBuilderAuthorizedKeys entries are public SSH keys only. Keep the matching private keys on client hosts and out of the repository.

nix-cache's own SSH host key is trusted declaratively via programs.ssh.knownHosts in modules/nix-cache/remote-builder-client.nix, sourced from vars.nixCacheHostKey (variables.nix) — every client rebuild picks it up automatically, so distributed builds don't fail with "Host key verification failed" on a client that has never manually SSH'd to nix-cache before. If nix-cache's host key is ever rotated or the host rebuilt from scratch, update vars.nixCacheHostKey to match its new /etc/ssh/ssh_host_ed25519_key.pub.

Manual verification

After deployment:

curl http://nix-cache/nix-cache-info
nix store ping --store http://nix-cache
nix show-config | grep -E 'substituters|trusted-public-keys|builders-use-substitutes'
sudo ssh -i /root/.ssh/id_ed25519 nixremote@nix-cache nix-store --version
nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/id_ed25519 4 2 big-parallel,kvm,nixos-test,benchmark' -L
nix path-info -r nixpkgs#hello
curl -I "http://nix-cache/$(basename "$(nix path-info nixpkgs#hello)").narinfo"