Archived
beatzaplenty
f565e9c2a1
Check NixOS configurations / eval-hosts (pull_request) Successful in 11m6s
Two real bugs, both hit live: 1. Shebang: #!/run/current-system/sw/bin/bash only resolves on an already-activated NixOS system -- running the checked-out script directly (e.g. from a stock ISO, cloned repo) failed with "cannot execute: required file not found" on a non-NixOS box. Switched to #!/usr/bin/env bash, which resolves identically on NixOS (environment.usrbinenv's own default) and any normal Linux distro. Also fixed the file's missing executable bit. 2. FLAKE_BASE_URL: previously depended on pkgs.replaceVars substituting a Nix-templated @lanDomain@ placeholder at build time -- meaning it only ever worked when baked into the built installer image, not when run straight from a checkout (the literal, unexpanded "@lanDomain@" string reached git as a bogus hostname). Replaced with LAN_DOMAIN in scripts/env.sh (manually kept in sync with variables.nix's lanDomain, same pattern as NIX_CACHE_HOST/nixCacheHost already), sourced by the script itself like every other script in scripts/. Dropped pkgs.replaceVars from modules/installer/common.nix entirely -- scripts/env.sh is now baked into the image alongside auto-install.sh at a matching relative path (/etc/nixos-installer/env.sh next to /etc/nixos-installer/installer/auto-install.sh) so the script's own relative `source` line resolves the same way in both contexts. loginShellInit's invocation path and docs/auto-installer.md updated to match. Verified: shellcheck clean on both scripts, the baked files are byte-identical to their checked-in sources (no templating left to verify), and codex-maintenance.sh (secret grep, fmt, statix, full eval of every host/package including the installer/pxe artifacts) passes clean.
308 lines
15 KiB
Markdown
308 lines
15 KiB
Markdown
# Auto-installer
|
|
|
|
This flake builds a self-contained NixOS installer environment that can
|
|
install any host exposed by its own `nixosConfigurations`. It was migrated
|
|
from a formerly-separate `nix-auto-installer` repo — everything it did now
|
|
lives here.
|
|
|
|
The installer provides a small NixOS install environment (ISO, or the same
|
|
image netbooted via PXE) with SSH access, Git support, and an interactive
|
|
installation script.
|
|
Logging in as any user (root or `nixos`) runs
|
|
`/etc/nixos-installer/installer/auto-install.sh` (the same file as
|
|
`scripts/installer/auto-install.sh` in this repo — see "Installer process"
|
|
below for why it's baked in at that path rather than a flat
|
|
`/etc/auto-install.sh`), discovers available hosts from this same flake,
|
|
lets the operator choose a target, applies that host's Disko storage
|
|
configuration, installs NixOS, and reboots.
|
|
|
|
**This applies to every `nixosConfigurations` target except `lxc-*` hosts —
|
|
see "LXC hosts" immediately below for why those are different.**
|
|
|
|
## LXC hosts
|
|
|
|
`lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`,
|
|
`lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-exit-node`, `lxc-tor-relay`) are **not** installed via `auto-install.sh` — the
|
|
interactive menu deliberately excludes them. Don't try to select one there;
|
|
`nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw
|
|
disk to partition) and then refuse to touch the filesystem it's currently
|
|
running on — it's designed to protect exactly this case, so it just fails.
|
|
|
|
`modules/platforms/lxc.nix` imports nixpkgs' own
|
|
`virtualisation/proxmox-lxc.nix` module, which gives every `lxc-*` host a
|
|
`config.system.build.tarball` output — a complete, directly Proxmox-importable
|
|
container image, no install step at all:
|
|
|
|
```sh
|
|
nix build .#nixosConfigurations.lxc-minimal.config.system.build.tarball
|
|
```
|
|
|
|
This is a plain rootfs tarball, not a `vzdump` backup archive — restoring it
|
|
with `pct restore` fails ("archive contains no configuration file"), since
|
|
that command expects backup-archive metadata this tarball doesn't have. Use
|
|
it as a CT *template* instead: drop it under Proxmox's template storage
|
|
(conventionally `/var/lib/vz/template/cache/` for the `local` storage, or
|
|
the GUI's "Create CT" → upload-as-template flow) and create a container
|
|
from it, supplying all config on the command line since a template has none
|
|
of its own:
|
|
|
|
```sh
|
|
pct create <vmid> local:vztmpl/<file>.tar.xz \
|
|
--unprivileged 1 --features nesting=1,keyctl=1 \
|
|
--rootfs local-lvm:8 --hostname <name> --cores 2 --memory 2048 --swap 2048 \
|
|
--net0 name=eth0,bridge=vmbr0,ip=dhcp
|
|
pct start <vmid>
|
|
```
|
|
|
|
Every one of those extra flags is load-bearing, confirmed by actually
|
|
booting one:
|
|
|
|
- `--unprivileged 1` — `modules/platforms/lxc.nix` sets
|
|
`proxmoxLXC.privileged = false`, so the image assumes it's running
|
|
unprivileged. `pct create`'s own CLI default for this flag is
|
|
privileged (unlike the web UI, whose checkbox defaults the other way)
|
|
— omit it and you get a privileged container running a NixOS config
|
|
that assumes unprivileged, a real mismatch.
|
|
- `--features nesting=1,keyctl=1` — required for a modern (v247+)
|
|
systemd guest to boot unprivileged at all. Without it, AppArmor denies
|
|
the nested user namespaces and credential mounts systemd routinely
|
|
uses (even plain getty units) — every getty crash-loops on a denied
|
|
`/run/credentials/*` mount every ~3s (this is what garbage on the
|
|
console turns out to be) while core services like `nsncd` fail the
|
|
same way, and the system never finishes activating.
|
|
- `--swap 2048` — `--memory` doesn't touch swap; it silently stays at
|
|
Proxmox's own 512M default otherwise. Match it to `--memory` unless
|
|
you deliberately want otherwise.
|
|
|
|
First boot runs `boot.postBootCommands` (registers the Nix store DB and
|
|
system profile) — there's no separate activation step to run yourself.
|
|
`scripts/proxmox/create-proxmox-resource.sh --type lxc --host <name>` automates all
|
|
of this (host-key handling, building the tarball directly on the Proxmox
|
|
node itself, `pct create` with the flags above) — see its `--help`.
|
|
|
|
Host keys still need pre-seeding the same way as any other host — the
|
|
sops-nix activation-vs-first-boot race is identical regardless of how the
|
|
image reaches the machine. Unlike the ISO/PXE installer (where
|
|
`modules/installer/host-keys.nix` bakes *every* `host-keys/` entry into
|
|
`/etc/host-keys/` for `auto-install.sh` to pick from and copy at install
|
|
time — see "Host keys" below), an `lxc-*` tarball has no install step to
|
|
copy anything during, so `modules/platforms/lxc.nix` bakes this *one*
|
|
target's key straight into `/etc/ssh/ssh_host_ed25519_key(.pub)` directly,
|
|
keyed by its own exact flake target name (`config.environment.etc` can't
|
|
be read back from within a module still contributing to it, so this comes
|
|
in via `specialArgs.flakeTarget`, set by `flake.nix`'s `mkTarget`):
|
|
|
|
```sh
|
|
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
|
|
nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure
|
|
```
|
|
|
|
Confirmed the hard way: without this, the tarball's own built-in system
|
|
just generates a fresh host key at first boot like any host would, which
|
|
can never match whatever `.sops.yaml` actually trusts for that target —
|
|
`sops-install-secrets` fails with `Error getting data key: 0 successful
|
|
groups required, got 0`, and *every* secret (including this host's own
|
|
login) permanently fails to decrypt, silently — no error in the boot log
|
|
at all, since the activation step that would install secrets only runs on
|
|
a from-scratch first activation and skips silently once `/run/current-system`
|
|
already exists. `scripts/proxmox/create-proxmox-resource.sh` always builds with
|
|
`NIXOS_HOST_KEYS_DIR` set for this reason.
|
|
|
|
## Layout
|
|
|
|
- `modules/installer/common.nix` — shared by every installer target: SSH
|
|
access, users, the generated `/etc/auto-install.sh` script, and the
|
|
`programs.bash.loginShellInit` hook that runs it on login.
|
|
- `modules/installer/iso.nix` — ISO/netboot-specific: imports the stock
|
|
`installation-cd-minimal.nix` module plus `common.nix`. Also used, paired
|
|
with `netboot-minimal.nix`, to build the PXE netboot variant (see
|
|
`docs/pxe-boot.md`).
|
|
- `modules/installer/host-keys.nix` — optionally bakes pre-generated SSH
|
|
host keys into the image; see "Host keys" below.
|
|
- `scripts/secrets/sync-host-keys.sh` — admin-workstation tool that generates,
|
|
registers, and (via `--remove`/`--regenerate-all-keys`) retires host
|
|
keys; see "Creating a New Machine" below.
|
|
- `scripts/secrets/prepare-host-key.sh` — narrower predecessor: generates a single
|
|
key by an arbitrary name without touching `.sops.yaml`. Still useful for
|
|
pre-generating a key *before* its flake target exists (`sync-host-keys.sh`
|
|
can only act on targets `nixosConfigurations` already has); otherwise
|
|
`sync-host-keys.sh` does the same thing and more.
|
|
|
|
Flake outputs:
|
|
|
|
```nix
|
|
nixosConfigurations.installer # ISO/netboot installer image
|
|
|
|
packages.x86_64-linux.iso # installer ISO/netboot image
|
|
packages.x86_64-linux.pxe # netboot-ipxe + netboot-initrd + netboot-kernel, bundled
|
|
```
|
|
|
|
```sh
|
|
nix build .#iso
|
|
nix build .#pxe
|
|
```
|
|
|
|
There's no `nixosConfigurations.proxmox-lxc` (installer-boots-as-an-LXC-
|
|
container) or `packages.x86_64-linux.lxc`/`.all` anymore. Both existed only
|
|
to let the installer itself run as an LXC container so you could
|
|
`nixos-install` some *other* host from within it — but LXC targets are
|
|
excluded from the install menu (same bind-mount problem as any LXC
|
|
`nixos-install`), and now have their own direct tarball path anyway (see
|
|
"LXC hosts" above), which left the installer's own LXC form with no real
|
|
use case.
|
|
|
|
The `pxe` variant is also built automatically as part of the `pxe-boot` host
|
|
itself (`modules/pxe-boot/stage-installer-artifacts.nix`) and served over
|
|
iPXE as the menu's "NixOS Auto-Installer" entry — see `docs/pxe-boot.md`.
|
|
That same host also builds and serves `packages.x86_64-linux.pxe-minimal`,
|
|
a vanilla NixOS minimal netboot image with none of this auto-installer's
|
|
wiring, as a separate "NixOS Minimal" menu entry — also documented in
|
|
`docs/pxe-boot.md`, not covered further here since it's not this installer.
|
|
|
|
## Host keys
|
|
|
|
`sops-nix` derives each host's decryption key from its own
|
|
`/etc/ssh/ssh_host_ed25519_key`, generated at **activation** time — before
|
|
systemd would otherwise generate one on first boot. Without pre-seeding this
|
|
key, secrets (including the root/nixos login password) fail to decrypt on a
|
|
genuinely fresh install.
|
|
|
|
Generated host keys live in `host-keys/` at the repo root (`ssh_host_ed25519_key`
|
|
+ `.pub` pairs per hostname). This directory is **gitignored on purpose** —
|
|
private key material must never be committed — which also means flakes can't
|
|
see it through a normal relative path. `modules/installer/host-keys.nix`
|
|
reads it through `builtins.getEnv`, which Nix silently returns as an empty
|
|
string under normal (non-`--impure`) evaluation, so the module is a no-op —
|
|
safe by default, including in CI — unless explicitly opted into:
|
|
|
|
```sh
|
|
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
|
|
```
|
|
|
|
When built this way, every key currently in `host-keys/` is baked into the
|
|
image at `/etc/host-keys/<hostname>_ssh_host_ed25519_key(.pub)`, and
|
|
`auto-install.sh` automatically installs whichever one matches the flake
|
|
target selected at install time — no manual per-host scp step needed.
|
|
|
|
**Trade-off, accepted deliberately for this LAN-only setup:** baking keys in
|
|
means every key present in `host-keys/` at build time becomes readable by
|
|
anyone who can reach the built image — including, for the PXE variant, anyone
|
|
who can reach the `pxe-boot` host's unauthenticated HTTP server. This is
|
|
considered acceptable here because `pxe-boot` sits behind LAN-only network
|
|
infrastructure, not the open internet. If that ever changes, reconsider this
|
|
default.
|
|
|
|
`auto-install.sh` still supports the older manual path as a fallback: if a
|
|
host's key isn't baked in (`/etc/host-keys`), it checks `/root/host-keys`
|
|
next, where you can `scp` a key in after boot, same as before this migration.
|
|
|
|
## Storage
|
|
|
|
Disk partitioning is handled by Disko — the installer has no hardcoded
|
|
`parted`/`mkfs`/`mkswap`/`mount` commands, and `auto-install.sh` runs
|
|
`disko --mode destroy,format,mount` unconditionally, no branching on whether
|
|
the target has a Disko config. Every host reachable through this menu has
|
|
one:
|
|
|
|
- `proxmox-*` (`modules/disko/proxmox.nix`): a real GPT partition table
|
|
(ESP + swap + root) on `/dev/sda`.
|
|
- `linode-*` (`modules/disko/linode.nix`): Linode provisions and sizes
|
|
`/dev/sda`/`/dev/sdb` itself as whole, unpartitioned block devices before
|
|
the OS ever boots, so this declares them with `destroy = false` (skips
|
|
disko's wipe stage for these disks entirely — see the option's own docs)
|
|
and a bare `filesystem`/`swap` content type with no partition table, and
|
|
the format step it does run only calls `mkfs`/`mkswap` if `blkid` shows
|
|
the device isn't already formatted — a re-run against an
|
|
already-provisioned Linode disk is a no-op, not a wipe.
|
|
|
|
`lxc-*` is the only category without one — it's excluded from this menu
|
|
entirely (see "LXC hosts" above), so it never reaches this code path.
|
|
|
|
## Installer process
|
|
|
|
`scripts/installer/auto-install.sh` is a real, version-controlled shell
|
|
script — not an inline Nix string. It sources `scripts/env.sh` for
|
|
`LAN_DOMAIN` itself (same as every other script in `scripts/`), so it
|
|
behaves identically whether it's run straight from a git checkout (e.g.
|
|
manually, from a stock NixOS ISO that isn't this repo's own installer
|
|
image) or from inside the built installer image. That's also why it's
|
|
baked in at `/etc/nixos-installer/installer/auto-install.sh` rather than a
|
|
flat `/etc/auto-install.sh` — `modules/installer/common.nix` bakes
|
|
`scripts/env.sh` in alongside it at `/etc/nixos-installer/env.sh`,
|
|
preserving the same relative layout (`installer/auto-install.sh` ->
|
|
`../env.sh`) the checked-out repo has, so the script's own
|
|
`source ".../env.sh"` line resolves correctly in both places without any
|
|
Nix-level templating.
|
|
|
|
Once running, it:
|
|
|
|
1. Queries `nixosConfigurations` from this flake over the network (`git+https://<lanDomain>/beatzaplenty/nixos.git`) — this happens at *install* time, not build time, so a generic installer image always sees whatever hosts are currently committed, without needing a rebuild.
|
|
2. Presents them as a menu; confirms the choice.
|
|
3. Skips the `nix-cache` substituter when installing a `nix-cache` host itself (consistent with that host's own runtime config).
|
|
4. Runs `disko --mode destroy,format,mount` (see "Storage" above — every host reachable through this menu has a Disko config, so this is unconditional).
|
|
5. Installs the target's SSH host key from `/etc/host-keys` or `/root/host-keys` (see "Host keys" above).
|
|
6. Runs `nixos-install --flake <url>#<choice> --no-root-password`.
|
|
7. Cleans up and reboots.
|
|
|
|
## Creating a new machine
|
|
|
|
Do this instead of jumping straight to a plain install whenever the target
|
|
host consumes any sops-nix secret — as of this writing, that's every host
|
|
(`modules/common/configuration.nix` puts the root/nixos password hash and the
|
|
GitHub token behind sops-nix for all of them).
|
|
|
|
1. **Add the flake target** — `hosts/<name>/host.nix` plus the matching
|
|
`mkTarget { ... }` entry in `flake.nix`'s `generatedTargets` (see
|
|
"Composition pattern" in `CLAUDE.md`). No secrets involved yet, so this
|
|
is safe to commit on its own if you want a clean history.
|
|
|
|
2. **On your admin workstation, generate and register its host key:**
|
|
|
|
```sh
|
|
./scripts/secrets/sync-host-keys.sh <flake-target>
|
|
```
|
|
|
|
This generates `host-keys/<flake-target>_ssh_host_ed25519_key(.pub)`,
|
|
adds it as a new `.sops.yaml` anchor, works out which `secrets/*.yaml`
|
|
files this specific host actually references (from its own
|
|
`config.sops.secrets`, not guessed), adds it to each one's
|
|
`key_groups`, and re-encrypts them with `sops updatekeys` — no manual
|
|
YAML editing. Safe to re-run; it only fills in what's missing.
|
|
|
|
Doing this for every host that needs one at once — after adding several
|
|
new targets, or just to catch up any that were missed — is
|
|
`./scripts/secrets/sync-host-keys.sh --all`. See `scripts/secrets/sync-host-keys.sh --help`
|
|
for its other modes (`--remove`, `--regenerate-all-keys`).
|
|
|
|
3. **Commit and push.** The flake build the installer uses has to see the
|
|
new recipient before you install, or decryption fails on first boot
|
|
regardless of the next step.
|
|
|
|
4. **Build the installer image with keys baked in** (or reuse an already-serving `pxe-boot` host, which does this automatically once redeployed):
|
|
|
|
```sh
|
|
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
|
|
```
|
|
|
|
5. **Boot it on the target machine**, log in, select the new host's flake
|
|
target from the menu, confirm. `auto-install.sh` finds the baked-in key,
|
|
runs Disko + `nixos-install`, and reboots.
|
|
|
|
6. **Verify after reboot:**
|
|
|
|
```sh
|
|
ssh <new-host> ls /run/secrets/
|
|
```
|
|
|
|
If that's empty or login fails, the host's age key most likely wasn't in
|
|
`.sops.yaml` (or wasn't re-encrypted into the secrets file it needs) when
|
|
`nixos-install` ran — fix `.sops.yaml`/`secrets/*.yaml`, push, then re-run
|
|
`nixos-install --flake .#<hostname> --no-root-password` from a rescue
|
|
environment against the existing `/mnt`, or just redo the install.
|
|
|
|
## Safety
|
|
|
|
This installer is destructive: `disko --mode destroy,format,mount` erases
|
|
any disk defined by the selected host's Disko configuration. Always verify
|
|
the selected host profile and target machine before confirming.
|