diff --git a/.gitignore b/.gitignore index 7a5862a..85d9d70 100644 --- a/.gitignore +++ b/.gitignore @@ -6,9 +6,9 @@ result-* # Ignore automatically generated direnv output .direnv -auto-installer/flake.lock -auto-installer/result -auto-installer/nixos-auto.iso +# Locally-generated SSH host keys staged for transfer to a new machine +# during install (see scripts/prepare-host-key.sh) — never commit these. +host-keys/ # Temporary Milestone 1 audit checklist (remove-sensetive-info-refactor.md) # - working notes only, never committed, deleted once every row is rotated. diff --git a/README.md b/README.md index 5964e05..a6bb5c5 100644 --- a/README.md +++ b/README.md @@ -53,8 +53,10 @@ nix eval --json .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]' | `modules/build-types/` | Build-type-specific config: what makes a system minimal/server/docker/gui/pxe-boot/nix-cache | | `modules/common/` | Shared NixOS config, Home Manager, aliases imported by every host | | `modules/nix-cache/` | Binary cache and remote builder client/server modules | -| `docs/` | Operational notes for cache, builders, lock updates, and boot services | -| `scripts/` | Codex setup and validation helpers | +| `modules/installer/` | Auto-installer environment (ISO/netboot/Proxmox LXC) — see `docs/auto-installer.md` | +| `host-keys/` | Gitignored, locally-generated SSH host keys for the auto-installer — see `docs/auto-installer.md` | +| `docs/` | Operational notes for cache, builders, lock updates, boot services, and the auto-installer | +| `scripts/` | Codex setup, validation, and host-key-prep helpers | ## Validation diff --git a/docs/auto-installer.md b/docs/auto-installer.md new file mode 100644 index 0000000..3f0080e --- /dev/null +++ b/docs/auto-installer.md @@ -0,0 +1,167 @@ +# Auto-installer + +This flake builds a self-contained NixOS installer environment that can +install any host exposed by its own `nixosConfigurations`. It was migrated +from a formerly-separate `nix-auto-installer` repo — everything it did now +lives here. + +The installer provides a small NixOS install environment (ISO/netboot/Proxmox +LXC) with SSH access, Git support, and an interactive installation script. +Logging in as any user (root or `nixos`) runs `/etc/auto-install.sh`, +discovers available hosts from this same flake, lets the operator choose a +target, applies that host's Disko storage configuration, installs NixOS, and +reboots. + +## Layout + +- `modules/installer/common.nix` — shared by every installer target: SSH + access, users, the generated `/etc/auto-install.sh` script, and the + `programs.bash.loginShellInit` hook that runs it on login. +- `modules/installer/iso.nix` — ISO/netboot-specific: imports the stock + `installation-cd-minimal.nix` module plus `common.nix`. Also used, paired + with `netboot-minimal.nix`, to build the PXE netboot variant (see + `docs/pxe-boot.md`). +- `modules/installer/proxmox-lxc.nix` — Proxmox LXC-specific: disables + bootloader options (containers don't need one) plus `common.nix`. +- `modules/installer/host-keys.nix` — optionally bakes pre-generated SSH + host keys into the image; see "Host keys" below. +- `scripts/prepare-host-key.sh` — admin-workstation pre-flight tool, see + "Creating a New Machine" below. + +Flake outputs: + +```nix +nixosConfigurations.installer # ISO/netboot installer image +nixosConfigurations.proxmox-lxc # Proxmox LXC-based installer image + +packages.x86_64-linux.iso # installer ISO/netboot image +packages.x86_64-linux.lxc # Proxmox LXC installer tarball +packages.x86_64-linux.pxe # netboot-ipxe + netboot-initrd + netboot-kernel, bundled +packages.x86_64-linux.all # all three of the above, bundled +``` + +```sh +nix build .#iso +nix build .#lxc +nix build .#pxe +nix build .#all +``` + +The `pxe` variant is also built automatically as part of the `pxe-boot` host +itself (`modules/pxe-boot/stage-installer-artifacts.nix`) and served over +iPXE — see `docs/pxe-boot.md`. + +## Host keys + +`sops-nix` derives each host's decryption key from its own +`/etc/ssh/ssh_host_ed25519_key`, generated at **activation** time — before +systemd would otherwise generate one on first boot. Without pre-seeding this +key, secrets (including the root/nixos login password) fail to decrypt on a +genuinely fresh install. + +Generated host keys live in `host-keys/` at the repo root (`ssh_host_ed25519_key` ++ `.pub` pairs per hostname). This directory is **gitignored on purpose** — +private key material must never be committed — which also means flakes can't +see it through a normal relative path. `modules/installer/host-keys.nix` +reads it through `builtins.getEnv`, which Nix silently returns as an empty +string under normal (non-`--impure`) evaluation, so the module is a no-op — +safe by default, including in CI — unless explicitly opted into: + +```sh +NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure +``` + +When built this way, every key currently in `host-keys/` is baked into the +image at `/etc/host-keys/_ssh_host_ed25519_key(.pub)`, and +`auto-install.sh` automatically installs whichever one matches the flake +target selected at install time — no manual per-host scp step needed. + +**Trade-off, accepted deliberately for this LAN-only setup:** baking keys in +means every key present in `host-keys/` at build time becomes readable by +anyone who can reach the built image — including, for the PXE variant, anyone +who can reach the `pxe-boot` host's unauthenticated HTTP server. This is +considered acceptable here because `pxe-boot` sits behind LAN-only network +infrastructure, not the open internet. If that ever changes, reconsider this +default. + +`auto-install.sh` still supports the older manual path as a fallback: if a +host's key isn't baked in (`/etc/host-keys`), it checks `/root/host-keys` +next, where you can `scp` a key in after boot, same as before this migration. + +## Storage + +Disk partitioning is handled by Disko — the installer has no hardcoded +`parted`/`mkfs`/`mkswap`/`mount` commands. Each host defines its own layout +via `disko.devices`; a host without one is installed in place via bind-mount +(this is how every LXC container target works, since they have no raw block +device to partition). + +## Installer process + +`/etc/auto-install.sh`: + +1. Queries `nixosConfigurations` from this flake over the network (`git+https:///beatzaplenty/nixos.git`) — this happens at *install* time, not build time, so a generic installer image always sees whatever hosts are currently committed, without needing a rebuild. +2. Presents them as a menu; confirms the choice. +3. Skips the `nix-cache` substituter when installing a `nix-cache` host itself (consistent with that host's own runtime config). +4. Runs `disko --mode destroy,format,mount` if the target has a Disko config, otherwise bind-mounts `/` onto `/mnt`. +5. Installs the target's SSH host key from `/etc/host-keys` or `/root/host-keys` (see "Host keys" above). +6. Runs `nixos-install --flake # --no-root-password`. +7. Cleans up and reboots. + +## Creating a new machine + +Do this instead of jumping straight to a plain install whenever the target +host consumes any sops-nix secret — as of this writing, that's every host +(`modules/common/configuration.nix` puts the root/nixos password hash and the +GitHub token behind sops-nix for all of them). + +1. **On your admin workstation**, decide the new machine's flake target name + (`-`, e.g. `proxmox-server`) and generate + register + its host key: + + ```sh + ./scripts/prepare-host-key.sh + ``` + + This generates `host-keys/_ssh_host_ed25519_key(.pub)` and + prints the `.sops.yaml` snippet to add, which secrets files to + re-encrypt it into, and the next steps below. + +2. **Edit `.sops.yaml`** with the printed snippet, then re-encrypt every + secrets file you added the new host to: + + ```sh + nix-shell -p sops --run 'sops updatekeys secrets/common.yaml' + ``` + +3. **Commit and push.** The flake build the installer uses has to see the + new recipient before you install, or decryption fails on first boot + regardless of the next step. + +4. **Build the installer image with keys baked in** (or reuse an already-serving `pxe-boot` host, which does this automatically once redeployed): + + ```sh + NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure + ``` + +5. **Boot it on the target machine**, log in, select the new host's flake + target from the menu, confirm. `auto-install.sh` finds the baked-in key, + runs Disko + `nixos-install`, and reboots. + +6. **Verify after reboot:** + + ```sh + ssh ls /run/secrets/ + ``` + + If that's empty or login fails, the host's age key most likely wasn't in + `.sops.yaml` (or wasn't re-encrypted into the secrets file it needs) when + `nixos-install` ran — fix `.sops.yaml`/`secrets/*.yaml`, push, then re-run + `nixos-install --flake .# --no-root-password` from a rescue + environment against the existing `/mnt`, or just redo the install. + +## Safety + +This installer is destructive: `disko --mode destroy,format,mount` erases +any disk defined by the selected host's Disko configuration. Always verify +the selected host profile and target machine before confirming. diff --git a/modules/installer/common.nix b/modules/installer/common.nix index b7118b1..8c58dc1 100644 --- a/modules/installer/common.nix +++ b/modules/installer/common.nix @@ -1,6 +1,10 @@ { pkgs, lib, vars, ... }: { + imports = [ + ./host-keys.nix + ]; + networking.useDHCP = lib.mkDefault true; time.timeZone = vars.timeZone; @@ -112,19 +116,29 @@ # at *activation* time, which runs before systemd would otherwise # generate one on first boot. Without pre-seeding it here, secrets # (including the login password) fail to decrypt on first boot. - # Generate the key + register it with `nixos`'s sops-nix setup ahead of - # time (see nix-auto-installer/scripts/prepare-host-key.sh), then scp it - # to /root/host-keys/ on this machine before continuing. + # Generate the key with scripts/prepare-host-key.sh first. + # + # Two places a key can come from, checked in order: + # /etc/host-keys — baked into this image at build time (see + # modules/installer/host-keys.nix; only present + # if built with NIXOS_HOST_KEYS_DIR set) + # /root/host-keys — scp'd in manually after boot (older fallback, + # still supported for images built without keys) mkdir -p /root/host-keys - if [[ -f "/root/host-keys/''${choice}_ssh_host_ed25519_key" ]]; then + if [[ -f "/etc/host-keys/''${choice}_ssh_host_ed25519_key" ]]; then + echo "Found baked-in SSH host key for ''${choice}, installing to target..." + install -D -m 0600 "/etc/host-keys/''${choice}_ssh_host_ed25519_key" /mnt/etc/ssh/ssh_host_ed25519_key + install -D -m 0644 "/etc/host-keys/''${choice}_ssh_host_ed25519_key.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub + elif [[ -f "/root/host-keys/''${choice}_ssh_host_ed25519_key" ]]; then echo "Found pre-seeded SSH host key for ''${choice}, installing to target..." install -D -m 0600 "/root/host-keys/''${choice}_ssh_host_ed25519_key" /mnt/etc/ssh/ssh_host_ed25519_key install -D -m 0644 "/root/host-keys/''${choice}_ssh_host_ed25519_key.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub else - echo "WARNING: no pre-seeded host key found at /root/host-keys/''${choice}_ssh_host_ed25519_key" + echo "WARNING: no SSH host key found for ''${choice} (checked /etc/host-keys and /root/host-keys)" echo "sops-nix secrets (including the login password) will NOT decrypt on first boot." echo "Run scripts/prepare-host-key.sh for host ''${choice} on your admin workstation first," - echo "then scp the result here, if this host needs sops-nix secrets." + echo "then either rebuild this image with NIXOS_HOST_KEYS_DIR set, or scp the result to" + echo "/root/host-keys/ on this machine." read -rp "Continue without a pre-seeded key anyway? (y/N): " skip_key if [[ ! "$skip_key" =~ ^[Yy]$ ]]; then echo "Aborted." diff --git a/modules/installer/host-keys.nix b/modules/installer/host-keys.nix new file mode 100644 index 0000000..4e05422 --- /dev/null +++ b/modules/installer/host-keys.nix @@ -0,0 +1,37 @@ +{ lib, ... }: + +let + # host-keys/ is gitignored (private key material must never be committed), + # which means flakes' git-filtered source tree can never see it via a + # normal relative path — referencing it at all requires stepping outside + # pure evaluation. builtins.getEnv is neutered to "" under normal + # `nix build`/`nix eval` (no error, just empty), so this whole module is a + # silent no-op unless the operator explicitly opts in with --impure and + # the env var set — safe by default, including in CI. + # + # NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build .#iso --impure + # + # See docs/auto-installer.md. + hostKeysDirStr = builtins.getEnv "NIXOS_HOST_KEYS_DIR"; + hasHostKeysDir = hostKeysDirStr != "" && builtins.pathExists hostKeysDirStr; + hostKeysDir = /. + hostKeysDirStr; + + keyFileNames = + if hasHostKeysDir + then + lib.filter + (name: lib.hasSuffix "_ssh_host_ed25519_key" name || lib.hasSuffix "_ssh_host_ed25519_key.pub" name) + (lib.attrNames (builtins.readDir hostKeysDir)) + else [ ]; +in +{ + environment.etc = lib.listToAttrs (map + (name: { + name = "host-keys/${name}"; + value = { + source = hostKeysDir + "/${name}"; + mode = "0400"; + }; + }) + keyFileNames); +} diff --git a/scripts/prepare-host-key.sh b/scripts/prepare-host-key.sh new file mode 100755 index 0000000..5ee83f0 --- /dev/null +++ b/scripts/prepare-host-key.sh @@ -0,0 +1,75 @@ +#!/usr/bin/env bash +# Generates a new machine's SSH host key ahead of installing it, so +# sops-nix has something to derive an age key from before the target +# ever boots. +# +# Why this is needed: sops-nix derives each host's decryption key from +# its own /etc/ssh/ssh_host_ed25519_key at *activation* time, but that +# activation runs before systemd would otherwise generate this key on +# first boot (sshd-keygen is a normal systemd service gated behind +# multi-user.target; activation scripts run earlier than that). Without +# pre-seeding, secrets — including the root/nixos login password — fail +# to decrypt on the machine's very first boot. +# +# This script only touches your admin workstation and this repo's +# .sops.yaml (it never contacts the target machine). Run it, follow the +# printed next steps, then use the resulting key with the auto-install.sh +# prompt (see modules/installer/common.nix) when you actually install the +# new machine. +set -euo pipefail + +repo_root="$(cd "$(dirname "$0")/.." && pwd)" + +hostname="${1:?usage: scripts/prepare-host-key.sh [path-to-nixos-repo]}" +nixos_repo="${2:-$repo_root}" +sops_yaml="${nixos_repo}/.sops.yaml" + +if [[ ! -f "$sops_yaml" ]]; then + echo "ERROR: $sops_yaml not found. Pass the nixos repo path as the 2nd argument." >&2 + exit 1 +fi + +keydir="${repo_root}/host-keys" +mkdir -p "$keydir" +keyfile="${keydir}/${hostname}_ssh_host_ed25519_key" + +if [[ -f "$keyfile" ]]; then + echo "ERROR: $keyfile already exists. Remove it first if you want to regenerate." >&2 + exit 1 +fi + +nix-shell -p openssh --run "ssh-keygen -t ed25519 -N '' -C '${hostname}' -f '${keyfile}'" >/dev/null + +age_pub="$(nix-shell -p ssh-to-age --run "ssh-to-age -i '${keyfile}.pub'")" + +cat <:/root/host-keys/ + +Then continue with /etc/auto-install.sh as normal — it checks +/etc/host-keys (baked in) before /root/host-keys (scp'd) and installs +whichever it finds before running nixos-install. +EOF