Archived
Fix lxc-* hosts having no host-key pre-seeding mechanism at all
The real root cause behind the original nix-cache 502, traced all the way through: modules/installer/host-keys.nix (which NIXOS_HOST_KEYS_DIR=... --impure actually wires up) is only ever imported by the installer's own modules/installer/common.nix -- modules/platforms/lxc.nix, which every real lxc-* host build actually uses, never imported anything like it. docs/auto-installer.md previously claimed NIXOS_HOST_KEYS_DIR bakes a key into lxc-* tarballs "the same way it does for the ISO/PXE installer images" -- that was never actually true; I wrote it without verifying the mechanism existed for lxc.nix specifically. In practice this meant every lxc-* container booted with a freshly self-generated SSH host key that could never match whatever .sops.yaml actually trusts for that target, so *every* secret -- not just cache-priv-key -- silently failed to decrypt. No error surfaces in the boot log for this: the activation step that installs secrets only runs on a genuinely fresh first activation and silently no-ops once /run/current-system already exists, so by the time anyone looks the window has closed. Found by manually invoking sops-install-secrets directly: "Error getting data key: 0 successful groups required, got 0". Fixed by giving modules/platforms/lxc.nix the same key-baking mechanism the installer has, but keyed to its own exact flake target and placing the key directly at /etc/ssh/ssh_host_ed25519_key (no copy step to stage for, unlike the installer's /etc/host-keys/ staging area -- an lxc-* tarball has no install step). The target name comes in via specialArgs.flakeTarget (new, set by flake.nix's mkTarget) rather than being read back from config.environment.etc."flake-target" -- reading that back from within a module that also contributes to environment.etc is circular (confirmed: "infinite recursion encountered"). Verified live end-to-end against the real test container (lxc-nix-cache, VMID 100 on pve.sweet.home): destroyed it, rebuilt the tarball fresh with the fix, recreated it, and confirmed /run/secrets/ now has all three secrets this host needs (beszel-token, cache-priv-key, nix-github-token), nix-serve is active (running), and curl http://localhost/nix-cache-info succeeds both directly and through nginx. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
This commit is contained in:
+27
-5
@@ -77,11 +77,33 @@ system profile) — there's no separate activation step to run yourself.
|
||||
of this (build, host-key handling, upload, `pct create` with the flags
|
||||
above) — see its `--help`.
|
||||
|
||||
Host keys still need pre-seeding the same way as any other host (see "Host
|
||||
keys" below) — the sops-nix activation-vs-first-boot race is identical
|
||||
regardless of how the image reaches the machine. `NIXOS_HOST_KEYS_DIR=...
|
||||
nix build ... --impure` bakes the matching key into the tarball the same way
|
||||
it does for the ISO/PXE installer images.
|
||||
Host keys still need pre-seeding the same way as any other host — the
|
||||
sops-nix activation-vs-first-boot race is identical regardless of how the
|
||||
image reaches the machine. Unlike the ISO/PXE installer (where
|
||||
`modules/installer/host-keys.nix` bakes *every* `host-keys/` entry into
|
||||
`/etc/host-keys/` for `auto-install.sh` to pick from and copy at install
|
||||
time — see "Host keys" below), an `lxc-*` tarball has no install step to
|
||||
copy anything during, so `modules/platforms/lxc.nix` bakes this *one*
|
||||
target's key straight into `/etc/ssh/ssh_host_ed25519_key(.pub)` directly,
|
||||
keyed by its own exact flake target name (`config.environment.etc` can't
|
||||
be read back from within a module still contributing to it, so this comes
|
||||
in via `specialArgs.flakeTarget`, set by `flake.nix`'s `mkTarget`):
|
||||
|
||||
```sh
|
||||
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
|
||||
nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure
|
||||
```
|
||||
|
||||
Confirmed the hard way: without this, the tarball's own built-in system
|
||||
just generates a fresh host key at first boot like any host would, which
|
||||
can never match whatever `.sops.yaml` actually trusts for that target —
|
||||
`sops-install-secrets` fails with `Error getting data key: 0 successful
|
||||
groups required, got 0`, and *every* secret (including this host's own
|
||||
login) permanently fails to decrypt, silently — no error in the boot log
|
||||
at all, since the activation step that would install secrets only runs on
|
||||
a from-scratch first activation and skips silently once `/run/current-system`
|
||||
already exists. `scripts/create-proxmox-resource.sh` always builds with
|
||||
`NIXOS_HOST_KEYS_DIR` set for this reason.
|
||||
|
||||
## Layout
|
||||
|
||||
|
||||
Reference in New Issue
Block a user