Migrate host-key tooling from nix-auto-installer, bake keys into images

Finishes the nix-auto-installer migration: scripts/prepare-host-key.sh
and the local host-keys/ directory (gitignored, private key material,
never committed — moved as plain files, not through git history)
weren't carried over in the initial migration.

Also implements automatic key staging, replacing the manual
scp-after-boot step:

- modules/installer/host-keys.nix reads host-keys/ via
  builtins.getEnv, which Nix silently returns as "" under normal
  (non---impure) evaluation — the module is a no-op by default, safe
  for CI, until explicitly opted into:

    NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build .#iso --impure

  When built this way every key present gets baked into the image at
  /etc/host-keys/, and auto-install.sh installs whichever one matches
  the flake target selected at install time — no manual per-host scp.

- This deliberately includes the PXE netboot variant, even though
  pxe-boot serves it unauthenticated over LAN HTTP: accepted
  explicitly as a reasonable trade-off for a network that sits behind
  LAN-only infrastructure, not the open internet. auto-install.sh
  still falls back to /root/host-keys (manual scp) if a key isn't
  baked in, so images built without --impure keep working exactly as
  before.

- docs/auto-installer.md replaces nix-auto-installer's README,
  updated for in-repo paths and the new build flow.

Verified: normal `nix eval` (no --impure) evaluates identically across
all 19 nixosConfigurations + 4 packages with zero host-keys/* entries
(CI-unaffected); with --impure + the env var set, all three installer
variants (installer/ISO, proxmox-lxc, pxe) correctly embed every key
in host-keys/.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot

.gitignore

@@ -6,9 +6,9 @@ result-*
 # Ignore automatically generated direnv output
 .direnv
 
-auto-installer/flake.lock
+# Locally-generated SSH host keys staged for transfer to a new machine
-auto-installer/result
+# during install (see scripts/prepare-host-key.sh) — never commit these.
-auto-installer/nixos-auto.iso
+host-keys/
 
 # Temporary Milestone 1 audit checklist (remove-sensetive-info-refactor.md)
 # - working notes only, never committed, deleted once every row is rotated.

README.md

@@ -53,8 +53,10 @@ nix eval --json .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]'
 | `modules/build-types/` | Build-type-specific config: what makes a system minimal/server/docker/gui/pxe-boot/nix-cache |
 | `modules/common/` | Shared NixOS config, Home Manager, aliases imported by every host |
 | `modules/nix-cache/` | Binary cache and remote builder client/server modules |
-| `docs/` | Operational notes for cache, builders, lock updates, and boot services |
+| `modules/installer/` | Auto-installer environment (ISO/netboot/Proxmox LXC) — see `docs/auto-installer.md` |
-| `scripts/` | Codex setup and validation helpers |
+| `host-keys/` | Gitignored, locally-generated SSH host keys for the auto-installer — see `docs/auto-installer.md` |
+| `docs/` | Operational notes for cache, builders, lock updates, boot services, and the auto-installer |
+| `scripts/` | Codex setup, validation, and host-key-prep helpers |
 
 ## Validation
 

docs/auto-installer.md

@@ -0,0 +1,167 @@
+# Auto-installer
+
+This flake builds a self-contained NixOS installer environment that can
+install any host exposed by its own `nixosConfigurations`. It was migrated
+from a formerly-separate `nix-auto-installer` repo — everything it did now
+lives here.
+
+The installer provides a small NixOS install environment (ISO/netboot/Proxmox
+LXC) with SSH access, Git support, and an interactive installation script.
+Logging in as any user (root or `nixos`) runs `/etc/auto-install.sh`,
+discovers available hosts from this same flake, lets the operator choose a
+target, applies that host's Disko storage configuration, installs NixOS, and
+reboots.
+
+## Layout
+
+- `modules/installer/common.nix` — shared by every installer target: SSH
+  access, users, the generated `/etc/auto-install.sh` script, and the
+  `programs.bash.loginShellInit` hook that runs it on login.
+- `modules/installer/iso.nix` — ISO/netboot-specific: imports the stock
+  `installation-cd-minimal.nix` module plus `common.nix`. Also used, paired
+  with `netboot-minimal.nix`, to build the PXE netboot variant (see
+  `docs/pxe-boot.md`).
+- `modules/installer/proxmox-lxc.nix` — Proxmox LXC-specific: disables
+  bootloader options (containers don't need one) plus `common.nix`.
+- `modules/installer/host-keys.nix` — optionally bakes pre-generated SSH
+  host keys into the image; see "Host keys" below.
+- `scripts/prepare-host-key.sh` — admin-workstation pre-flight tool, see
+  "Creating a New Machine" below.
+
+Flake outputs:
+
+```nix
+nixosConfigurations.installer     # ISO/netboot installer image
+nixosConfigurations.proxmox-lxc   # Proxmox LXC-based installer image
+
+packages.x86_64-linux.iso   # installer ISO/netboot image
+packages.x86_64-linux.lxc   # Proxmox LXC installer tarball
+packages.x86_64-linux.pxe   # netboot-ipxe + netboot-initrd + netboot-kernel, bundled
+packages.x86_64-linux.all   # all three of the above, bundled
+```
+
+```sh
+nix build .#iso
+nix build .#lxc
+nix build .#pxe
+nix build .#all
+```
+
+The `pxe` variant is also built automatically as part of the `pxe-boot` host
+itself (`modules/pxe-boot/stage-installer-artifacts.nix`) and served over
+iPXE — see `docs/pxe-boot.md`.
+
+## Host keys
+
+`sops-nix` derives each host's decryption key from its own
+`/etc/ssh/ssh_host_ed25519_key`, generated at **activation** time — before
+systemd would otherwise generate one on first boot. Without pre-seeding this
+key, secrets (including the root/nixos login password) fail to decrypt on a
+genuinely fresh install.
+
+Generated host keys live in `host-keys/` at the repo root (`ssh_host_ed25519_key`
++ `.pub` pairs per hostname). This directory is **gitignored on purpose** —
+private key material must never be committed — which also means flakes can't
+see it through a normal relative path. `modules/installer/host-keys.nix`
+reads it through `builtins.getEnv`, which Nix silently returns as an empty
+string under normal (non-`--impure`) evaluation, so the module is a no-op —
+safe by default, including in CI — unless explicitly opted into:
+
+```sh
+NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
+```
+
+When built this way, every key currently in `host-keys/` is baked into the
+image at `/etc/host-keys/<hostname>_ssh_host_ed25519_key(.pub)`, and
+`auto-install.sh` automatically installs whichever one matches the flake
+target selected at install time — no manual per-host scp step needed.
+
+**Trade-off, accepted deliberately for this LAN-only setup:** baking keys in
+means every key present in `host-keys/` at build time becomes readable by
+anyone who can reach the built image — including, for the PXE variant, anyone
+who can reach the `pxe-boot` host's unauthenticated HTTP server. This is
+considered acceptable here because `pxe-boot` sits behind LAN-only network
+infrastructure, not the open internet. If that ever changes, reconsider this
+default.
+
+`auto-install.sh` still supports the older manual path as a fallback: if a
+host's key isn't baked in (`/etc/host-keys`), it checks `/root/host-keys`
+next, where you can `scp` a key in after boot, same as before this migration.
+
+## Storage
+
+Disk partitioning is handled by Disko — the installer has no hardcoded
+`parted`/`mkfs`/`mkswap`/`mount` commands. Each host defines its own layout
+via `disko.devices`; a host without one is installed in place via bind-mount
+(this is how every LXC container target works, since they have no raw block
+device to partition).
+
+## Installer process
+
+`/etc/auto-install.sh`:
+
+1. Queries `nixosConfigurations` from this flake over the network (`git+https://<lanDomain>/beatzaplenty/nixos.git`) — this happens at *install* time, not build time, so a generic installer image always sees whatever hosts are currently committed, without needing a rebuild.
+2. Presents them as a menu; confirms the choice.
+3. Skips the `nix-cache` substituter when installing a `nix-cache` host itself (consistent with that host's own runtime config).
+4. Runs `disko --mode destroy,format,mount` if the target has a Disko config, otherwise bind-mounts `/` onto `/mnt`.
+5. Installs the target's SSH host key from `/etc/host-keys` or `/root/host-keys` (see "Host keys" above).
+6. Runs `nixos-install --flake <url>#<choice> --no-root-password`.
+7. Cleans up and reboots.
+
+## Creating a new machine
+
+Do this instead of jumping straight to a plain install whenever the target
+host consumes any sops-nix secret — as of this writing, that's every host
+(`modules/common/configuration.nix` puts the root/nixos password hash and the
+GitHub token behind sops-nix for all of them).
+
+1. **On your admin workstation**, decide the new machine's flake target name
+   (`<platform>-<buildtype>`, e.g. `proxmox-server`) and generate + register
+   its host key:
+
+   ```sh
+   ./scripts/prepare-host-key.sh <hostname>
+   ```
+
+   This generates `host-keys/<hostname>_ssh_host_ed25519_key(.pub)` and
+   prints the `.sops.yaml` snippet to add, which secrets files to
+   re-encrypt it into, and the next steps below.
+
+2. **Edit `.sops.yaml`** with the printed snippet, then re-encrypt every
+   secrets file you added the new host to:
+
+   ```sh
+   nix-shell -p sops --run 'sops updatekeys secrets/common.yaml'
+   ```
+
+3. **Commit and push.** The flake build the installer uses has to see the
+   new recipient before you install, or decryption fails on first boot
+   regardless of the next step.
+
+4. **Build the installer image with keys baked in** (or reuse an already-serving `pxe-boot` host, which does this automatically once redeployed):
+
+   ```sh
+   NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
+   ```
+
+5. **Boot it on the target machine**, log in, select the new host's flake
+   target from the menu, confirm. `auto-install.sh` finds the baked-in key,
+   runs Disko + `nixos-install`, and reboots.
+
+6. **Verify after reboot:**
+
+   ```sh
+   ssh <new-host> ls /run/secrets/
+   ```
+
+   If that's empty or login fails, the host's age key most likely wasn't in
+   `.sops.yaml` (or wasn't re-encrypted into the secrets file it needs) when
+   `nixos-install` ran — fix `.sops.yaml`/`secrets/*.yaml`, push, then re-run
+   `nixos-install --flake .#<hostname> --no-root-password` from a rescue
+   environment against the existing `/mnt`, or just redo the install.
+
+## Safety
+
+This installer is destructive: `disko --mode destroy,format,mount` erases
+any disk defined by the selected host's Disko configuration. Always verify
+the selected host profile and target machine before confirming.

modules/installer/common.nix

@@ -1,6 +1,10 @@
 { pkgs, lib, vars, ... }:
 
 {
+  imports = [
+    ./host-keys.nix
+  ];
+
   networking.useDHCP = lib.mkDefault true;
 
   time.timeZone = vars.timeZone;
@@ -112,19 +116,29 @@
         # at *activation* time, which runs before systemd would otherwise
         # generate one on first boot. Without pre-seeding it here, secrets
         # (including the login password) fail to decrypt on first boot.
-        # Generate the key + register it with `nixos`'s sops-nix setup ahead of
+        # Generate the key with scripts/prepare-host-key.sh first.
-        # time (see nix-auto-installer/scripts/prepare-host-key.sh), then scp it
+        #
-        # to /root/host-keys/ on this machine before continuing.
+        # Two places a key can come from, checked in order:
+        #   /etc/host-keys  — baked into this image at build time (see
+        #                      modules/installer/host-keys.nix; only present
+        #                      if built with NIXOS_HOST_KEYS_DIR set)
+        #   /root/host-keys — scp'd in manually after boot (older fallback,
+        #                      still supported for images built without keys)
         mkdir -p /root/host-keys
-        if [[ -f "/root/host-keys/''${choice}_ssh_host_ed25519_key" ]]; then
+        if [[ -f "/etc/host-keys/''${choice}_ssh_host_ed25519_key" ]]; then
+            echo "Found baked-in SSH host key for ''${choice}, installing to target..."
+            install -D -m 0600 "/etc/host-keys/''${choice}_ssh_host_ed25519_key" /mnt/etc/ssh/ssh_host_ed25519_key
+            install -D -m 0644 "/etc/host-keys/''${choice}_ssh_host_ed25519_key.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub
+        elif [[ -f "/root/host-keys/''${choice}_ssh_host_ed25519_key" ]]; then
             echo "Found pre-seeded SSH host key for ''${choice}, installing to target..."
             install -D -m 0600 "/root/host-keys/''${choice}_ssh_host_ed25519_key" /mnt/etc/ssh/ssh_host_ed25519_key
             install -D -m 0644 "/root/host-keys/''${choice}_ssh_host_ed25519_key.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub
         else
-            echo "WARNING: no pre-seeded host key found at /root/host-keys/''${choice}_ssh_host_ed25519_key"
+            echo "WARNING: no SSH host key found for ''${choice} (checked /etc/host-keys and /root/host-keys)"
             echo "sops-nix secrets (including the login password) will NOT decrypt on first boot."
             echo "Run scripts/prepare-host-key.sh for host ''${choice} on your admin workstation first,"
-            echo "then scp the result here, if this host needs sops-nix secrets."
+            echo "then either rebuild this image with NIXOS_HOST_KEYS_DIR set, or scp the result to"
+            echo "/root/host-keys/ on this machine."
             read -rp "Continue without a pre-seeded key anyway? (y/N): " skip_key
             if [[ ! "$skip_key" =~ ^[Yy]$ ]]; then
                 echo "Aborted."

modules/installer/host-keys.nix

@@ -0,0 +1,37 @@
+{ lib, ... }:
+
+let
+  # host-keys/ is gitignored (private key material must never be committed),
+  # which means flakes' git-filtered source tree can never see it via a
+  # normal relative path — referencing it at all requires stepping outside
+  # pure evaluation. builtins.getEnv is neutered to "" under normal
+  # `nix build`/`nix eval` (no error, just empty), so this whole module is a
+  # silent no-op unless the operator explicitly opts in with --impure and
+  # the env var set — safe by default, including in CI.
+  #
+  #   NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build .#iso --impure
+  #
+  # See docs/auto-installer.md.
+  hostKeysDirStr = builtins.getEnv "NIXOS_HOST_KEYS_DIR";
+  hasHostKeysDir = hostKeysDirStr != "" && builtins.pathExists hostKeysDirStr;
+  hostKeysDir = /. + hostKeysDirStr;
+
+  keyFileNames =
+    if hasHostKeysDir
+    then
+      lib.filter
+        (name: lib.hasSuffix "_ssh_host_ed25519_key" name || lib.hasSuffix "_ssh_host_ed25519_key.pub" name)
+        (lib.attrNames (builtins.readDir hostKeysDir))
+    else [ ];
+in
+{
+  environment.etc = lib.listToAttrs (map
+    (name: {
+      name = "host-keys/${name}";
+      value = {
+        source = hostKeysDir + "/${name}";
+        mode = "0400";
+      };
+    })
+    keyFileNames);
+}

scripts/prepare-host-key.sh

@@ -0,0 +1,75 @@
+#!/usr/bin/env bash
+# Generates a new machine's SSH host key ahead of installing it, so
+# sops-nix has something to derive an age key from before the target
+# ever boots.
+#
+# Why this is needed: sops-nix derives each host's decryption key from
+# its own /etc/ssh/ssh_host_ed25519_key at *activation* time, but that
+# activation runs before systemd would otherwise generate this key on
+# first boot (sshd-keygen is a normal systemd service gated behind
+# multi-user.target; activation scripts run earlier than that). Without
+# pre-seeding, secrets — including the root/nixos login password — fail
+# to decrypt on the machine's very first boot.
+#
+# This script only touches your admin workstation and this repo's
+# .sops.yaml (it never contacts the target machine). Run it, follow the
+# printed next steps, then use the resulting key with the auto-install.sh
+# prompt (see modules/installer/common.nix) when you actually install the
+# new machine.
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "$0")/.." && pwd)"
+
+hostname="${1:?usage: scripts/prepare-host-key.sh <hostname> [path-to-nixos-repo]}"
+nixos_repo="${2:-$repo_root}"
+sops_yaml="${nixos_repo}/.sops.yaml"
+
+if [[ ! -f "$sops_yaml" ]]; then
+  echo "ERROR: $sops_yaml not found. Pass the nixos repo path as the 2nd argument." >&2
+  exit 1
+fi
+
+keydir="${repo_root}/host-keys"
+mkdir -p "$keydir"
+keyfile="${keydir}/${hostname}_ssh_host_ed25519_key"
+
+if [[ -f "$keyfile" ]]; then
+  echo "ERROR: $keyfile already exists. Remove it first if you want to regenerate." >&2
+  exit 1
+fi
+
+nix-shell -p openssh --run "ssh-keygen -t ed25519 -N '' -C '${hostname}' -f '${keyfile}'" >/dev/null
+
+age_pub="$(nix-shell -p ssh-to-age --run "ssh-to-age -i '${keyfile}.pub'")"
+
+cat <<EOF
+
+Generated: ${keyfile}(.pub)
+
+=== 1. Add this line under keys: in ${sops_yaml} ===
+  - &${hostname} ${age_pub}
+
+=== 2. Add *${hostname} to whichever creation_rules key_groups this host needs ===
+(e.g. secrets/common.yaml always; add a per-host secrets/${hostname}.yaml
+ block too if this host will get its own secrets, same pattern as
+ nix-cache/server.)
+
+=== 3. Re-encrypt every secrets file you just added it to ===
+  nix-shell -p sops --run 'sops updatekeys ${nixos_repo}/secrets/common.yaml'
+
+=== 4. Commit + push this repo so the flake build picks up the new recipient ===
+
+=== 5. Get the key onto the installer, one of two ways ===
+  a) Rebuild the installer image with all host-keys/ baked in (see
+     docs/auto-installer.md):
+       NIXOS_HOST_KEYS_DIR="${keydir}" nix build .#iso --impure
+     (or .#lxc / .#pxe / .#all — --impure is required since host-keys/
+     is gitignored and flakes can't see it otherwise)
+
+  b) Or, for an image already built without keys, scp it in after boot:
+       scp ${keyfile}{,.pub} root@<target-ip>:/root/host-keys/
+
+Then continue with /etc/auto-install.sh as normal — it checks
+/etc/host-keys (baked in) before /root/host-keys (scp'd) and installs
+whichever it finds before running nixos-install.
+EOF