nixos/docs/nix-cache.md

nix-cache architecture

This repository configures nix-cache as a binary cache server and a remote builder for other hosts.

Important design notes

• This is not a shared /nix/store setup.
• Every machine still keeps and uses its own local /nix/store.
• Clients prefer http://nix-cache for substitutes and keep https://cache.nixos.org/ as fallback.
• Clients can offload builds to nix-cache through SSH (nix.distributedBuilds).
• Client hosts import modules/nix-cache/client.nix and, when remote building is enabled, modules/nix-cache/remote-builder-client.nix.
• The nix-cache host imports modules/nix-cache/server.nix.

Binary cache signing key

modules/nix-cache/client.nix hardcodes every client's trust in one specific public key (cache.local-1:usoWYanY3Kpq2+kDIS2nhWoLZiRxanmdysdzqCFBHW4=). That means whichever host is currently playing the nix-cache role has to use that exact keypair — not a freshly generated one — or no client will accept substitutes from it (they'd just silently fall back to building from source). So unlike most per-host secrets, this one can't be self-generated on first boot; it's managed via sops-nix like every other secret in this repo, sourced from secrets/nix-cache.yaml's cache-priv-key entry (modules/nix-cache/server.nix).

Adding or rotating the value:

nix-shell -p sops --run 'sops secrets/nix-cache.yaml'

Add (or replace) a cache-priv-key entry with the private key file's exact contents. If you don't have it yet, generate a keypair once:

nix-store --generate-binary-cache-key nix-cache-1 cache-priv.pem cache-pub.pem

— paste cache-priv.pem's contents into the cache-priv-key entry above, delete both local files afterward, and update trusted-public-keys in modules/nix-cache/client.nix (and every already-built client) to match cache-pub.pem if this is a genuine rotation rather than a first-time bootstrap. Any nixos-configurations.*-nix-cache host picks the new key up automatically on next activation — no more manual /etc/nix/cache-priv.pem install step.

Remote builder SSH keys

On each client, install the private key used to authenticate as nixremote:

sudo install -d -m 0700 /root/.ssh
sudo install -m 0600 ./nixremote /root/.ssh/nixremote
sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version

On nix-cache, install the matching public key used by nixremote authorized keys.

The committed nixremote authorized keys are public SSH keys only. Keep the matching private keys on client hosts and out of the repository.

nix-cache's own SSH host key is trusted declaratively via programs.ssh.knownHosts in modules/nix-cache/remote-builder-client.nix, sourced from vars.nixCacheHostKey (variables.nix) — every client rebuild picks it up automatically, so distributed builds don't fail with "Host key verification failed" on a client that has never manually SSH'd to nix-cache before. If nix-cache's host key is ever rotated or the host rebuilt from scratch, update vars.nixCacheHostKey to match its new /etc/ssh/ssh_host_ed25519_key.pub.

Manual verification

After deployment:

curl http://nix-cache/nix-cache-info
nix store ping --store http://nix-cache
nix show-config | grep -E 'substituters|trusted-public-keys|builders-use-substitutes'
sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
nix build nixpkgs#hello --builders 'ssh://nixremote@nix-cache x86_64-linux /root/.ssh/nixremote 4 2 big-parallel,kvm,nixos-test,benchmark' -L
nix path-info -r nixpkgs#hello
curl -I "http://nix-cache/$(basename "$(nix path-info nixpkgs#hello)").narinfo"