nixos/docs/auto-installer.md

# Auto-installer

This flake builds a self-contained NixOS installer environment that can
install any host exposed by its own `nixosConfigurations`. It was migrated
from a formerly-separate `nix-auto-installer` repo — everything it did now
lives here.

The installer provides a small NixOS install environment (ISO, or the same
image netbooted via PXE) with SSH access, Git support, and an interactive
installation script.
Logging in as any user (root or `nixos`) runs
`/etc/nixos-installer/installer/auto-install.sh` (the same file as
`scripts/installer/auto-install.sh` in this repo — see "Installer process"
below for why it's baked in at that path rather than a flat
`/etc/auto-install.sh`), discovers available hosts from this same flake,
lets the operator choose a target, applies that host's Disko storage
configuration, installs NixOS, and reboots.

**This applies to every `nixosConfigurations` target except `lxc-*` hosts —
see "LXC hosts" immediately below for why those are different.**

## LXC hosts

`lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`,
`lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-router`, `lxc-tor-relay`) are **not** installed via `auto-install.sh` — the
interactive menu deliberately excludes them. Don't try to select one there;
`nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw
disk to partition) and then refuse to touch the filesystem it's currently
running on — it's designed to protect exactly this case, so it just fails.

`modules/platforms/lxc.nix` imports nixpkgs' own
`virtualisation/proxmox-lxc.nix` module, which gives every `lxc-*` host a
`config.system.build.tarball` output — a complete, directly Proxmox-importable
container image, no install step at all:

```sh
nix build .#nixosConfigurations.lxc-minimal.config.system.build.tarball
```

This is a plain rootfs tarball, not a `vzdump` backup archive — restoring it
with `pct restore` fails ("archive contains no configuration file"), since
that command expects backup-archive metadata this tarball doesn't have. Use
it as a CT *template* instead: drop it under Proxmox's template storage
(conventionally `/var/lib/vz/template/cache/` for the `local` storage, or
the GUI's "Create CT" → upload-as-template flow) and create a container
from it, supplying all config on the command line since a template has none
of its own:

```sh
pct create <vmid> local:vztmpl/<file>.tar.xz \
  --unprivileged 1 --features nesting=1,keyctl=1 \
  --rootfs local-lvm:8 --hostname <name> --cores 2 --memory 2048 --swap 2048 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp
pct start <vmid>
```

Every one of those extra flags is load-bearing, confirmed by actually
booting one:

- `--unprivileged 1` — `modules/platforms/lxc.nix` sets
  `proxmoxLXC.privileged = false`, so the image assumes it's running
  unprivileged. `pct create`'s own CLI default for this flag is
  privileged (unlike the web UI, whose checkbox defaults the other way)
  — omit it and you get a privileged container running a NixOS config
  that assumes unprivileged, a real mismatch.
- `--features nesting=1,keyctl=1` — required for a modern (v247+)
  systemd guest to boot unprivileged at all. Without it, AppArmor denies
  the nested user namespaces and credential mounts systemd routinely
  uses (even plain getty units) — every getty crash-loops on a denied
  `/run/credentials/*` mount every ~3s (this is what garbage on the
  console turns out to be) while core services like `nsncd` fail the
  same way, and the system never finishes activating.
- `--swap 2048` — `--memory` doesn't touch swap; it silently stays at
  Proxmox's own 512M default otherwise. Match it to `--memory` unless
  you deliberately want otherwise.

First boot runs `boot.postBootCommands` (registers the Nix store DB and
system profile) — there's no separate activation step to run yourself.
`scripts/proxmox/create-proxmox-resource.sh --type lxc --host <name>` automates all
of this (host-key handling, building the tarball directly on the Proxmox
node itself, `pct create` with the flags above) — see its `--help`.

Host keys still need pre-seeding the same way as any other host — the
sops-nix activation-vs-first-boot race is identical regardless of how the
image reaches the machine. Unlike the ISO/PXE installer (where
`modules/installer/host-keys.nix` bakes *every* `host-keys/` entry into
`/etc/host-keys/` for `auto-install.sh` to pick from and copy at install
time — see "Host keys" below), an `lxc-*` tarball has no install step to
copy anything during, so `modules/platforms/lxc.nix` bakes this *one*
target's key straight into `/etc/ssh/ssh_host_ed25519_key(.pub)` directly,
keyed by its own exact flake target name (`config.environment.etc` can't
be read back from within a module still contributing to it, so this comes
in via `specialArgs.flakeTarget`, set by `flake.nix`'s `mkTarget`):

```sh
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
  nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure
```

Confirmed the hard way: without this, the tarball's own built-in system
just generates a fresh host key at first boot like any host would, which
can never match whatever `.sops.yaml` actually trusts for that target —
`sops-install-secrets` fails with `Error getting data key: 0 successful
groups required, got 0`, and *every* secret (including this host's own
login) permanently fails to decrypt, silently — no error in the boot log
at all, since the activation step that would install secrets only runs on
a from-scratch first activation and skips silently once `/run/current-system`
already exists. `scripts/proxmox/create-proxmox-resource.sh` always builds with
`NIXOS_HOST_KEYS_DIR` set for this reason.

## Layout

- `modules/installer/common.nix` — shared by every installer target: SSH
  access, users, the generated `/etc/auto-install.sh` script, and the
  `programs.bash.loginShellInit` hook that runs it on login.
- `modules/installer/iso.nix` — ISO/netboot-specific: imports the stock
  `installation-cd-minimal.nix` module plus `common.nix`. Also used, paired
  with `netboot-minimal.nix`, to build the PXE netboot variant (see
  `docs/pxe-boot.md`).
- `modules/installer/host-keys.nix` — optionally bakes pre-generated SSH
  host keys into the image; see "Host keys" below.
- `scripts/secrets/sync-host-keys.sh` — admin-workstation tool that generates,
  registers, and (via `--remove`/`--regenerate-all-keys`) retires host
  keys; see "Creating a New Machine" below.
- `scripts/secrets/prepare-host-key.sh` — narrower predecessor: generates a single
  key by an arbitrary name without touching `.sops.yaml`. Still useful for
  pre-generating a key *before* its flake target exists (`sync-host-keys.sh`
  can only act on targets `nixosConfigurations` already has); otherwise
  `sync-host-keys.sh` does the same thing and more.

Flake outputs:

```nix
nixosConfigurations.installer   # ISO/netboot installer image

packages.x86_64-linux.iso       # installer ISO/netboot image
packages.x86_64-linux.pxe       # auto-installer netboot bundle (kernel + initrd + ipxe script)
packages.x86_64-linux.pxe-minimal  # vanilla NixOS minimal netboot bundle (no installer wiring)
```

```sh
nix build .#iso
nix build .#pxe
nix build .#pxe-minimal
```

There's no `nixosConfigurations.proxmox-lxc` (installer-boots-as-an-LXC-
container) or `packages.x86_64-linux.lxc`/`.all` anymore. Both existed only
to let the installer itself run as an LXC container so you could
`nixos-install` some *other* host from within it — but LXC targets are
excluded from the install menu (same bind-mount problem as any LXC
`nixos-install`), and now have their own direct tarball path anyway (see
"LXC hosts" above), which left the installer's own LXC form with no real
use case.

The `pxe` variant is also built automatically as part of the `pxe-boot` host
itself (`modules/pxe-boot/stage-installer-artifacts.nix`) and served over
iPXE as the menu's "NixOS Auto-Installer" entry — see `docs/pxe-boot.md`.
That same host also builds and serves `packages.x86_64-linux.pxe-minimal`,
a vanilla NixOS minimal netboot image with none of this auto-installer's
wiring, as a separate "NixOS Minimal" menu entry — also documented in
`docs/pxe-boot.md`, not covered further here since it's not this installer.

## Host keys

`sops-nix` derives each host's decryption key from its own
`/etc/ssh/ssh_host_ed25519_key`, generated at **activation** time — before
systemd would otherwise generate one on first boot. Without pre-seeding this
key, secrets (including the root/nixos login password) fail to decrypt on a
genuinely fresh install.

Generated host keys live in `host-keys/` at the repo root (`ssh_host_ed25519_key`
+ `.pub` pairs per hostname). This directory is **gitignored on purpose** —
private key material must never be committed — which also means flakes can't
see it through a normal relative path. `modules/installer/host-keys.nix`
reads it through `builtins.getEnv`, which Nix silently returns as an empty
string under normal (non-`--impure`) evaluation, so the module is a no-op —
safe by default, including in CI — unless explicitly opted into:

```sh
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
```

When built this way, every key currently in `host-keys/` is baked into the
image at `/etc/host-keys/<hostname>_ssh_host_ed25519_key(.pub)`, and
`auto-install.sh` automatically installs whichever one matches the flake
target selected at install time — no manual per-host scp step needed.

**Trade-off, accepted deliberately for this LAN-only setup:** baking keys in
means every key present in `host-keys/` at build time becomes readable by
anyone who can reach the built image — including, for the PXE variant, anyone
who can reach the `pxe-boot` host's unauthenticated HTTP server. This is
considered acceptable here because `pxe-boot` sits behind LAN-only network
infrastructure, not the open internet. If that ever changes, reconsider this
default.

`auto-install.sh` still supports the older manual path as a fallback: if a
host's key isn't baked in (`/etc/host-keys`), it checks `/root/host-keys`
next, where you can `scp` a key in after boot, same as before this migration.
If neither has it and the script is running interactively (an actual
operator at the other end of stdin, not an unattended run), it prompts for
an arbitrary directory to check (a mounted USB stick, another filesystem,
etc.) and copies the key pair into `/root/host-keys` from there if found.

## Storage

Disk partitioning is handled by Disko — the installer has no hardcoded
`parted`/`mkfs`/`mkswap`/`mount` commands, and `auto-install.sh` runs
`disko --mode destroy,format,mount` unconditionally, no branching on whether
the target has a Disko config. Every host reachable through this menu has
one:

- `proxmox-*` (`modules/disko/proxmox.nix`): a real GPT partition table
  (ESP + swap + root) on `/dev/sda`.
- `linode-*` (`modules/disko/linode.nix`): Linode provisions and sizes
  `/dev/sda`/`/dev/sdb` itself as whole, unpartitioned block devices before
  the OS ever boots, so this declares them with `destroy = false` (skips
  disko's wipe stage for these disks entirely — see the option's own docs)
  and a bare `filesystem`/`swap` content type with no partition table, and
  the format step it does run only calls `mkfs`/`mkswap` if `blkid` shows
  the device isn't already formatted — a re-run against an
  already-provisioned Linode disk is a no-op, not a wipe.

`lxc-*` is the only category without one — it's excluded from this menu
entirely (see "LXC hosts" above), so it never reaches this code path.

## Installer process

`scripts/installer/auto-install.sh` is a real, version-controlled shell
script — not an inline Nix string. It sources `scripts/env.sh` for
`LAN_DOMAIN` itself (same as every other script in `scripts/`), so it
behaves identically whether it's run straight from a git checkout (e.g.
manually, from a stock NixOS ISO that isn't this repo's own installer
image) or from inside the built installer image. That's also why it's
baked in at `/etc/nixos-installer/installer/auto-install.sh` rather than a
flat `/etc/auto-install.sh` — `modules/installer/common.nix` bakes
`scripts/env.sh` in alongside it at `/etc/nixos-installer/env.sh`,
preserving the same relative layout (`installer/auto-install.sh` ->
`../env.sh`) the checked-out repo has, so the script's own
`source ".../env.sh"` line resolves correctly in both places without any
Nix-level templating.

Once running, it:

1. Queries `nixosConfigurations` from this flake over the network (`git+https://<lanDomain>/beatzaplenty/nixos.git`) — this happens at *install* time, not build time, so a generic installer image always sees whatever hosts are currently committed, without needing a rebuild.
2. Presents them as a menu; confirms the choice.
3. Skips the `nix-cache` substituter when installing a `nix-cache` host itself (consistent with that host's own runtime config).
4. Runs `disko --mode destroy,format,mount` (see "Storage" above — every host reachable through this menu has a Disko config, so this is unconditional).
5. Installs the target's SSH host key from `/etc/host-keys` or `/root/host-keys` (see "Host keys" above).
6. Runs `nixos-install --flake <url>#<choice> --no-root-password`.
7. Cleans up and reboots.

## Creating a new machine

Do this instead of jumping straight to a plain install whenever the target
host consumes any sops-nix secret — as of this writing, that's every host
(`modules/common/configuration.nix` puts the root/nixos password hash and the
GitHub token behind sops-nix for all of them).

1. **Add the flake target** — `hosts/<name>/host.nix` plus the matching
   `mkTarget { ... }` entry in `flake.nix`'s `generatedTargets` (see
   "Composition pattern" in `CLAUDE.md`). No secrets involved yet, so this
   is safe to commit on its own if you want a clean history.

2. **On your admin workstation, generate and register its host key:**

   ```sh
   ./scripts/secrets/sync-host-keys.sh <flake-target>
   ```

   This generates `host-keys/<flake-target>_ssh_host_ed25519_key(.pub)`,
   adds it as a new `.sops.yaml` anchor, works out which `secrets/*.yaml`
   files this specific host actually references (from its own
   `config.sops.secrets`, not guessed), adds it to each one's
   `key_groups`, and re-encrypts them with `sops updatekeys` — no manual
   YAML editing. Safe to re-run; it only fills in what's missing.

   Doing this for every host that needs one at once — after adding several
   new targets, or just to catch up any that were missed — is
   `./scripts/secrets/sync-host-keys.sh --all`. See `scripts/secrets/sync-host-keys.sh --help`
   for its other modes (`--remove`, `--regenerate-all-keys`).

3. **Commit and push.** The flake build the installer uses has to see the
   new recipient before you install, or decryption fails on first boot
   regardless of the next step.

4. **Build the installer image with keys baked in** (or reuse an already-serving `pxe-boot` host, which does this automatically once redeployed):

   ```sh
   NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
   ```

5. **Boot it on the target machine**, log in, select the new host's flake
   target from the menu, confirm. `auto-install.sh` finds the baked-in key,
   runs Disko + `nixos-install`, and reboots.

6. **Verify after reboot:**

   ```sh
   ssh <new-host> ls /run/secrets/
   ```

   If that's empty or login fails, the host's age key most likely wasn't in
   `.sops.yaml` (or wasn't re-encrypted into the secrets file it needs) when
   `nixos-install` ran — fix `.sops.yaml`/`secrets/*.yaml`, push, then re-run
   `nixos-install --flake .#<hostname> --no-root-password` from a rescue
   environment against the existing `/mnt`, or just redo the install.

## Safety

This installer is destructive: `disko --mode destroy,format,mount` erases
any disk defined by the selected host's Disko configuration. Always verify
the selected host profile and target machine before confirming.