Archived
beatzaplentyandClaude Sonnet 5
2858891c20
Check NixOS configurations / eval-hosts (push) Failing after 10m48s
sync-host-keys.sh: generates/registers SSH host keys and their .sops.yaml/secrets/*.yaml recipients for flake targets, idempotently. --all, <target>, --remove, --regenerate-all-keys, all with --dry-run (verified zero-side-effect via a sandboxed git-status check across every mode). Only ever touches anchors with a corresponding host-keys/ file -- &admin and any hand-registered real-host anchor are never listed, removed, or regenerated. Supersedes running prepare-host-key.sh one host at a time for any target that already has a flake entry. create-proxmox-resource.sh: builds a lxc-*/proxmox-* target's tarball/disk image and creates it on a real Proxmox node, or reconfigures an existing resource's cores/memory/disk (--modify, always requires typing the VMID back to confirm). Refuses to create a new resource for a VMID that already exists, and refuses to duplicate a host identity that already has a real deployment elsewhere (variables.nix's new deployedTargets, checked by hostName so it also catches cross-platform duplicates) unless --allow-duplicate-host is passed. --dry-run throughout. scripts/env.sh centralizes the Proxmox connection config both scripts (and future ones) share. Also fixes an unrelated gap found along the way: proxmox-* Disko image builds write their .raw file straight into the repo root, and .gitignore never covered it. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
227 lines
11 KiB
Markdown
227 lines
11 KiB
Markdown
# Auto-installer
|
|
|
|
This flake builds a self-contained NixOS installer environment that can
|
|
install any host exposed by its own `nixosConfigurations`. It was migrated
|
|
from a formerly-separate `nix-auto-installer` repo — everything it did now
|
|
lives here.
|
|
|
|
The installer provides a small NixOS install environment (ISO, or the same
|
|
image netbooted via PXE) with SSH access, Git support, and an interactive
|
|
installation script.
|
|
Logging in as any user (root or `nixos`) runs `/etc/auto-install.sh`,
|
|
discovers available hosts from this same flake, lets the operator choose a
|
|
target, applies that host's Disko storage configuration, installs NixOS, and
|
|
reboots.
|
|
|
|
**This applies to every `nixosConfigurations` target except `lxc-*` hosts —
|
|
see "LXC hosts" immediately below for why those are different.**
|
|
|
|
## LXC hosts
|
|
|
|
`lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`,
|
|
`lxc-gui`, `lxc-pxe-boot`) are **not** installed via `auto-install.sh` — the
|
|
interactive menu deliberately excludes them. Don't try to select one there;
|
|
`nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw
|
|
disk to partition) and then refuse to touch the filesystem it's currently
|
|
running on — it's designed to protect exactly this case, so it just fails.
|
|
|
|
`modules/platforms/lxc.nix` imports nixpkgs' own
|
|
`virtualisation/proxmox-lxc.nix` module, which gives every `lxc-*` host a
|
|
`config.system.build.tarball` output — a complete, directly Proxmox-importable
|
|
container image, no install step at all:
|
|
|
|
```sh
|
|
nix build .#nixosConfigurations.lxc-minimal.config.system.build.tarball
|
|
```
|
|
|
|
Then, on the Proxmox host, `pct restore` (or the GUI's "Create CT" → upload
|
|
template flow) that tarball directly as a new container. First boot runs
|
|
`boot.postBootCommands` (registers the Nix store DB and system profile) —
|
|
there's no separate activation step to run yourself.
|
|
|
|
Host keys still need pre-seeding the same way as any other host (see "Host
|
|
keys" below) — the sops-nix activation-vs-first-boot race is identical
|
|
regardless of how the image reaches the machine. `NIXOS_HOST_KEYS_DIR=...
|
|
nix build ... --impure` bakes the matching key into the tarball the same way
|
|
it does for the ISO/PXE installer images.
|
|
|
|
## Layout
|
|
|
|
- `modules/installer/common.nix` — shared by every installer target: SSH
|
|
access, users, the generated `/etc/auto-install.sh` script, and the
|
|
`programs.bash.loginShellInit` hook that runs it on login.
|
|
- `modules/installer/iso.nix` — ISO/netboot-specific: imports the stock
|
|
`installation-cd-minimal.nix` module plus `common.nix`. Also used, paired
|
|
with `netboot-minimal.nix`, to build the PXE netboot variant (see
|
|
`docs/pxe-boot.md`).
|
|
- `modules/installer/host-keys.nix` — optionally bakes pre-generated SSH
|
|
host keys into the image; see "Host keys" below.
|
|
- `scripts/sync-host-keys.sh` — admin-workstation tool that generates,
|
|
registers, and (via `--remove`/`--regenerate-all-keys`) retires host
|
|
keys; see "Creating a New Machine" below.
|
|
- `scripts/prepare-host-key.sh` — narrower predecessor: generates a single
|
|
key by an arbitrary name without touching `.sops.yaml`. Still useful for
|
|
pre-generating a key *before* its flake target exists (`sync-host-keys.sh`
|
|
can only act on targets `nixosConfigurations` already has); otherwise
|
|
`sync-host-keys.sh` does the same thing and more.
|
|
|
|
Flake outputs:
|
|
|
|
```nix
|
|
nixosConfigurations.installer # ISO/netboot installer image
|
|
|
|
packages.x86_64-linux.iso # installer ISO/netboot image
|
|
packages.x86_64-linux.pxe # netboot-ipxe + netboot-initrd + netboot-kernel, bundled
|
|
```
|
|
|
|
```sh
|
|
nix build .#iso
|
|
nix build .#pxe
|
|
```
|
|
|
|
There's no `nixosConfigurations.proxmox-lxc` (installer-boots-as-an-LXC-
|
|
container) or `packages.x86_64-linux.lxc`/`.all` anymore. Both existed only
|
|
to let the installer itself run as an LXC container so you could
|
|
`nixos-install` some *other* host from within it — but LXC targets are
|
|
excluded from the install menu (same bind-mount problem as any LXC
|
|
`nixos-install`), and now have their own direct tarball path anyway (see
|
|
"LXC hosts" above), which left the installer's own LXC form with no real
|
|
use case.
|
|
|
|
The `pxe` variant is also built automatically as part of the `pxe-boot` host
|
|
itself (`modules/pxe-boot/stage-installer-artifacts.nix`) and served over
|
|
iPXE — see `docs/pxe-boot.md`.
|
|
|
|
## Host keys
|
|
|
|
`sops-nix` derives each host's decryption key from its own
|
|
`/etc/ssh/ssh_host_ed25519_key`, generated at **activation** time — before
|
|
systemd would otherwise generate one on first boot. Without pre-seeding this
|
|
key, secrets (including the root/nixos login password) fail to decrypt on a
|
|
genuinely fresh install.
|
|
|
|
Generated host keys live in `host-keys/` at the repo root (`ssh_host_ed25519_key`
|
|
+ `.pub` pairs per hostname). This directory is **gitignored on purpose** —
|
|
private key material must never be committed — which also means flakes can't
|
|
see it through a normal relative path. `modules/installer/host-keys.nix`
|
|
reads it through `builtins.getEnv`, which Nix silently returns as an empty
|
|
string under normal (non-`--impure`) evaluation, so the module is a no-op —
|
|
safe by default, including in CI — unless explicitly opted into:
|
|
|
|
```sh
|
|
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
|
|
```
|
|
|
|
When built this way, every key currently in `host-keys/` is baked into the
|
|
image at `/etc/host-keys/<hostname>_ssh_host_ed25519_key(.pub)`, and
|
|
`auto-install.sh` automatically installs whichever one matches the flake
|
|
target selected at install time — no manual per-host scp step needed.
|
|
|
|
**Trade-off, accepted deliberately for this LAN-only setup:** baking keys in
|
|
means every key present in `host-keys/` at build time becomes readable by
|
|
anyone who can reach the built image — including, for the PXE variant, anyone
|
|
who can reach the `pxe-boot` host's unauthenticated HTTP server. This is
|
|
considered acceptable here because `pxe-boot` sits behind LAN-only network
|
|
infrastructure, not the open internet. If that ever changes, reconsider this
|
|
default.
|
|
|
|
`auto-install.sh` still supports the older manual path as a fallback: if a
|
|
host's key isn't baked in (`/etc/host-keys`), it checks `/root/host-keys`
|
|
next, where you can `scp` a key in after boot, same as before this migration.
|
|
|
|
## Storage
|
|
|
|
Disk partitioning is handled by Disko — the installer has no hardcoded
|
|
`parted`/`mkfs`/`mkswap`/`mount` commands, and `auto-install.sh` runs
|
|
`disko --mode destroy,format,mount` unconditionally, no branching on whether
|
|
the target has a Disko config. Every host reachable through this menu has
|
|
one:
|
|
|
|
- `proxmox-*` (`modules/disko/proxmox.nix`): a real GPT partition table
|
|
(ESP + swap + root) on `/dev/sda`.
|
|
- `linode-*` (`modules/disko/linode.nix`): Linode provisions and sizes
|
|
`/dev/sda`/`/dev/sdb` itself as whole, unpartitioned block devices before
|
|
the OS ever boots, so this declares them with `destroy = false` (skips
|
|
disko's wipe stage for these disks entirely — see the option's own docs)
|
|
and a bare `filesystem`/`swap` content type with no partition table, and
|
|
the format step it does run only calls `mkfs`/`mkswap` if `blkid` shows
|
|
the device isn't already formatted — a re-run against an
|
|
already-provisioned Linode disk is a no-op, not a wipe.
|
|
|
|
`lxc-*` is the only category without one — it's excluded from this menu
|
|
entirely (see "LXC hosts" above), so it never reaches this code path.
|
|
|
|
## Installer process
|
|
|
|
`/etc/auto-install.sh`:
|
|
|
|
1. Queries `nixosConfigurations` from this flake over the network (`git+https://<lanDomain>/beatzaplenty/nixos.git`) — this happens at *install* time, not build time, so a generic installer image always sees whatever hosts are currently committed, without needing a rebuild.
|
|
2. Presents them as a menu; confirms the choice.
|
|
3. Skips the `nix-cache` substituter when installing a `nix-cache` host itself (consistent with that host's own runtime config).
|
|
4. Runs `disko --mode destroy,format,mount` (see "Storage" above — every host reachable through this menu has a Disko config, so this is unconditional).
|
|
5. Installs the target's SSH host key from `/etc/host-keys` or `/root/host-keys` (see "Host keys" above).
|
|
6. Runs `nixos-install --flake <url>#<choice> --no-root-password`.
|
|
7. Cleans up and reboots.
|
|
|
|
## Creating a new machine
|
|
|
|
Do this instead of jumping straight to a plain install whenever the target
|
|
host consumes any sops-nix secret — as of this writing, that's every host
|
|
(`modules/common/configuration.nix` puts the root/nixos password hash and the
|
|
GitHub token behind sops-nix for all of them).
|
|
|
|
1. **Add the flake target** — `hosts/<name>/host.nix` plus the matching
|
|
`mkTarget { ... }` entry in `flake.nix`'s `generatedTargets` (see
|
|
"Composition pattern" in `CLAUDE.md`). No secrets involved yet, so this
|
|
is safe to commit on its own if you want a clean history.
|
|
|
|
2. **On your admin workstation, generate and register its host key:**
|
|
|
|
```sh
|
|
./scripts/sync-host-keys.sh <flake-target>
|
|
```
|
|
|
|
This generates `host-keys/<flake-target>_ssh_host_ed25519_key(.pub)`,
|
|
adds it as a new `.sops.yaml` anchor, works out which `secrets/*.yaml`
|
|
files this specific host actually references (from its own
|
|
`config.sops.secrets`, not guessed), adds it to each one's
|
|
`key_groups`, and re-encrypts them with `sops updatekeys` — no manual
|
|
YAML editing. Safe to re-run; it only fills in what's missing.
|
|
|
|
Doing this for every host that needs one at once — after adding several
|
|
new targets, or just to catch up any that were missed — is
|
|
`./scripts/sync-host-keys.sh --all`. See `scripts/sync-host-keys.sh --help`
|
|
for its other modes (`--remove`, `--regenerate-all-keys`).
|
|
|
|
3. **Commit and push.** The flake build the installer uses has to see the
|
|
new recipient before you install, or decryption fails on first boot
|
|
regardless of the next step.
|
|
|
|
4. **Build the installer image with keys baked in** (or reuse an already-serving `pxe-boot` host, which does this automatically once redeployed):
|
|
|
|
```sh
|
|
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
|
|
```
|
|
|
|
5. **Boot it on the target machine**, log in, select the new host's flake
|
|
target from the menu, confirm. `auto-install.sh` finds the baked-in key,
|
|
runs Disko + `nixos-install`, and reboots.
|
|
|
|
6. **Verify after reboot:**
|
|
|
|
```sh
|
|
ssh <new-host> ls /run/secrets/
|
|
```
|
|
|
|
If that's empty or login fails, the host's age key most likely wasn't in
|
|
`.sops.yaml` (or wasn't re-encrypted into the secrets file it needs) when
|
|
`nixos-install` ran — fix `.sops.yaml`/`secrets/*.yaml`, push, then re-run
|
|
`nixos-install --flake .#<hostname> --no-root-password` from a rescue
|
|
environment against the existing `/mnt`, or just redo the install.
|
|
|
|
## Safety
|
|
|
|
This installer is destructive: `disko --mode destroy,format,mount` erases
|
|
any disk defined by the selected host's Disko configuration. Always verify
|
|
the selected host profile and target machine before confirming.
|