nixos/docs/auto-installer.md

# Auto-installer

This flake builds a self-contained NixOS installer environment that can
install any host exposed by its own `nixosConfigurations`. It was migrated
from a formerly-separate `nix-auto-installer` repo — everything it did now
lives here.

The installer provides a small NixOS install environment (ISO, or the same
image netbooted via PXE) with SSH access, Git support, and an interactive
installation script.
Logging in as any user (root or `nixos`) runs `/etc/auto-install.sh`,
discovers available hosts from this same flake, lets the operator choose a
target, applies that host's Disko storage configuration, installs NixOS, and
reboots.

**This applies to every `nixosConfigurations` target except `lxc-*` hosts —
see "LXC hosts" immediately below for why those are different.**

## LXC hosts

`lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`,
`lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-exit-node`) are **not** installed via `auto-install.sh` — the
interactive menu deliberately excludes them. Don't try to select one there;
`nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw
disk to partition) and then refuse to touch the filesystem it's currently
running on — it's designed to protect exactly this case, so it just fails.

`modules/platforms/lxc.nix` imports nixpkgs' own
`virtualisation/proxmox-lxc.nix` module, which gives every `lxc-*` host a
`config.system.build.tarball` output — a complete, directly Proxmox-importable
container image, no install step at all:

```sh
nix build .#nixosConfigurations.lxc-minimal.config.system.build.tarball
```

This is a plain rootfs tarball, not a `vzdump` backup archive — restoring it
with `pct restore` fails ("archive contains no configuration file"), since
that command expects backup-archive metadata this tarball doesn't have. Use
it as a CT *template* instead: drop it under Proxmox's template storage
(conventionally `/var/lib/vz/template/cache/` for the `local` storage, or
the GUI's "Create CT" → upload-as-template flow) and create a container
from it, supplying all config on the command line since a template has none
of its own:

```sh
pct create <vmid> local:vztmpl/<file>.tar.xz \
  --unprivileged 1 --features nesting=1,keyctl=1 \
  --rootfs local-lvm:8 --hostname <name> --cores 2 --memory 2048 --swap 2048 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp
pct start <vmid>
```

Every one of those extra flags is load-bearing, confirmed by actually
booting one:

- `--unprivileged 1` — `modules/platforms/lxc.nix` sets
  `proxmoxLXC.privileged = false`, so the image assumes it's running
  unprivileged. `pct create`'s own CLI default for this flag is
  privileged (unlike the web UI, whose checkbox defaults the other way)
  — omit it and you get a privileged container running a NixOS config
  that assumes unprivileged, a real mismatch.
- `--features nesting=1,keyctl=1` — required for a modern (v247+)
  systemd guest to boot unprivileged at all. Without it, AppArmor denies
  the nested user namespaces and credential mounts systemd routinely
  uses (even plain getty units) — every getty crash-loops on a denied
  `/run/credentials/*` mount every ~3s (this is what garbage on the
  console turns out to be) while core services like `nsncd` fail the
  same way, and the system never finishes activating.
- `--swap 2048` — `--memory` doesn't touch swap; it silently stays at
  Proxmox's own 512M default otherwise. Match it to `--memory` unless
  you deliberately want otherwise.

First boot runs `boot.postBootCommands` (registers the Nix store DB and
system profile) — there's no separate activation step to run yourself.
`scripts/create-proxmox-resource.sh --type lxc --host <name>` automates all
of this (host-key handling, building the tarball directly on the Proxmox
node itself, `pct create` with the flags above) — see its `--help`.

Host keys still need pre-seeding the same way as any other host — the
sops-nix activation-vs-first-boot race is identical regardless of how the
image reaches the machine. Unlike the ISO/PXE installer (where
`modules/installer/host-keys.nix` bakes *every* `host-keys/` entry into
`/etc/host-keys/` for `auto-install.sh` to pick from and copy at install
time — see "Host keys" below), an `lxc-*` tarball has no install step to
copy anything during, so `modules/platforms/lxc.nix` bakes this *one*
target's key straight into `/etc/ssh/ssh_host_ed25519_key(.pub)` directly,
keyed by its own exact flake target name (`config.environment.etc` can't
be read back from within a module still contributing to it, so this comes
in via `specialArgs.flakeTarget`, set by `flake.nix`'s `mkTarget`):

```sh
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
  nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure
```

Confirmed the hard way: without this, the tarball's own built-in system
just generates a fresh host key at first boot like any host would, which
can never match whatever `.sops.yaml` actually trusts for that target —
`sops-install-secrets` fails with `Error getting data key: 0 successful
groups required, got 0`, and *every* secret (including this host's own
login) permanently fails to decrypt, silently — no error in the boot log
at all, since the activation step that would install secrets only runs on
a from-scratch first activation and skips silently once `/run/current-system`
already exists. `scripts/create-proxmox-resource.sh` always builds with
`NIXOS_HOST_KEYS_DIR` set for this reason.

## Layout

- `modules/installer/common.nix` — shared by every installer target: SSH
  access, users, the generated `/etc/auto-install.sh` script, and the
  `programs.bash.loginShellInit` hook that runs it on login.
- `modules/installer/iso.nix` — ISO/netboot-specific: imports the stock
  `installation-cd-minimal.nix` module plus `common.nix`. Also used, paired
  with `netboot-minimal.nix`, to build the PXE netboot variant (see
  `docs/pxe-boot.md`).
- `modules/installer/host-keys.nix` — optionally bakes pre-generated SSH
  host keys into the image; see "Host keys" below.
- `scripts/sync-host-keys.sh` — admin-workstation tool that generates,
  registers, and (via `--remove`/`--regenerate-all-keys`) retires host
  keys; see "Creating a New Machine" below.
- `scripts/prepare-host-key.sh` — narrower predecessor: generates a single
  key by an arbitrary name without touching `.sops.yaml`. Still useful for
  pre-generating a key *before* its flake target exists (`sync-host-keys.sh`
  can only act on targets `nixosConfigurations` already has); otherwise
  `sync-host-keys.sh` does the same thing and more.

Flake outputs:

```nix
nixosConfigurations.installer   # ISO/netboot installer image

packages.x86_64-linux.iso   # installer ISO/netboot image
packages.x86_64-linux.pxe   # netboot-ipxe + netboot-initrd + netboot-kernel, bundled
```

```sh
nix build .#iso
nix build .#pxe
```

There's no `nixosConfigurations.proxmox-lxc` (installer-boots-as-an-LXC-
container) or `packages.x86_64-linux.lxc`/`.all` anymore. Both existed only
to let the installer itself run as an LXC container so you could
`nixos-install` some *other* host from within it — but LXC targets are
excluded from the install menu (same bind-mount problem as any LXC
`nixos-install`), and now have their own direct tarball path anyway (see
"LXC hosts" above), which left the installer's own LXC form with no real
use case.

The `pxe` variant is also built automatically as part of the `pxe-boot` host
itself (`modules/pxe-boot/stage-installer-artifacts.nix`) and served over
iPXE — see `docs/pxe-boot.md`.

## Host keys

`sops-nix` derives each host's decryption key from its own
`/etc/ssh/ssh_host_ed25519_key`, generated at **activation** time — before
systemd would otherwise generate one on first boot. Without pre-seeding this
key, secrets (including the root/nixos login password) fail to decrypt on a
genuinely fresh install.

Generated host keys live in `host-keys/` at the repo root (`ssh_host_ed25519_key`
+ `.pub` pairs per hostname). This directory is **gitignored on purpose** —
private key material must never be committed — which also means flakes can't
see it through a normal relative path. `modules/installer/host-keys.nix`
reads it through `builtins.getEnv`, which Nix silently returns as an empty
string under normal (non-`--impure`) evaluation, so the module is a no-op —
safe by default, including in CI — unless explicitly opted into:

```sh
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
```

When built this way, every key currently in `host-keys/` is baked into the
image at `/etc/host-keys/<hostname>_ssh_host_ed25519_key(.pub)`, and
`auto-install.sh` automatically installs whichever one matches the flake
target selected at install time — no manual per-host scp step needed.

**Trade-off, accepted deliberately for this LAN-only setup:** baking keys in
means every key present in `host-keys/` at build time becomes readable by
anyone who can reach the built image — including, for the PXE variant, anyone
who can reach the `pxe-boot` host's unauthenticated HTTP server. This is
considered acceptable here because `pxe-boot` sits behind LAN-only network
infrastructure, not the open internet. If that ever changes, reconsider this
default.

`auto-install.sh` still supports the older manual path as a fallback: if a
host's key isn't baked in (`/etc/host-keys`), it checks `/root/host-keys`
next, where you can `scp` a key in after boot, same as before this migration.

## Storage

Disk partitioning is handled by Disko — the installer has no hardcoded
`parted`/`mkfs`/`mkswap`/`mount` commands, and `auto-install.sh` runs
`disko --mode destroy,format,mount` unconditionally, no branching on whether
the target has a Disko config. Every host reachable through this menu has
one:

- `proxmox-*` (`modules/disko/proxmox.nix`): a real GPT partition table
  (ESP + swap + root) on `/dev/sda`.
- `linode-*` (`modules/disko/linode.nix`): Linode provisions and sizes
  `/dev/sda`/`/dev/sdb` itself as whole, unpartitioned block devices before
  the OS ever boots, so this declares them with `destroy = false` (skips
  disko's wipe stage for these disks entirely — see the option's own docs)
  and a bare `filesystem`/`swap` content type with no partition table, and
  the format step it does run only calls `mkfs`/`mkswap` if `blkid` shows
  the device isn't already formatted — a re-run against an
  already-provisioned Linode disk is a no-op, not a wipe.

`lxc-*` is the only category without one — it's excluded from this menu
entirely (see "LXC hosts" above), so it never reaches this code path.

## Installer process

`/etc/auto-install.sh`:

1. Queries `nixosConfigurations` from this flake over the network (`git+https://<lanDomain>/beatzaplenty/nixos.git`) — this happens at *install* time, not build time, so a generic installer image always sees whatever hosts are currently committed, without needing a rebuild.
2. Presents them as a menu; confirms the choice.
3. Skips the `nix-cache` substituter when installing a `nix-cache` host itself (consistent with that host's own runtime config).
4. Runs `disko --mode destroy,format,mount` (see "Storage" above — every host reachable through this menu has a Disko config, so this is unconditional).
5. Installs the target's SSH host key from `/etc/host-keys` or `/root/host-keys` (see "Host keys" above).
6. Runs `nixos-install --flake <url>#<choice> --no-root-password`.
7. Cleans up and reboots.

## Creating a new machine

Do this instead of jumping straight to a plain install whenever the target
host consumes any sops-nix secret — as of this writing, that's every host
(`modules/common/configuration.nix` puts the root/nixos password hash and the
GitHub token behind sops-nix for all of them).

1. **Add the flake target** — `hosts/<name>/host.nix` plus the matching
   `mkTarget { ... }` entry in `flake.nix`'s `generatedTargets` (see
   "Composition pattern" in `CLAUDE.md`). No secrets involved yet, so this
   is safe to commit on its own if you want a clean history.

2. **On your admin workstation, generate and register its host key:**

   ```sh
   ./scripts/sync-host-keys.sh <flake-target>
   ```

   This generates `host-keys/<flake-target>_ssh_host_ed25519_key(.pub)`,
   adds it as a new `.sops.yaml` anchor, works out which `secrets/*.yaml`
   files this specific host actually references (from its own
   `config.sops.secrets`, not guessed), adds it to each one's
   `key_groups`, and re-encrypts them with `sops updatekeys` — no manual
   YAML editing. Safe to re-run; it only fills in what's missing.

   Doing this for every host that needs one at once — after adding several
   new targets, or just to catch up any that were missed — is
   `./scripts/sync-host-keys.sh --all`. See `scripts/sync-host-keys.sh --help`
   for its other modes (`--remove`, `--regenerate-all-keys`).

3. **Commit and push.** The flake build the installer uses has to see the
   new recipient before you install, or decryption fails on first boot
   regardless of the next step.

4. **Build the installer image with keys baked in** (or reuse an already-serving `pxe-boot` host, which does this automatically once redeployed):

   ```sh
   NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
   ```

5. **Boot it on the target machine**, log in, select the new host's flake
   target from the menu, confirm. `auto-install.sh` finds the baked-in key,
   runs Disko + `nixos-install`, and reboots.

6. **Verify after reboot:**

   ```sh
   ssh <new-host> ls /run/secrets/
   ```

   If that's empty or login fails, the host's age key most likely wasn't in
   `.sops.yaml` (or wasn't re-encrypted into the secrets file it needs) when
   `nixos-install` ran — fix `.sops.yaml`/`secrets/*.yaml`, push, then re-run
   `nixos-install --flake .#<hostname> --no-root-password` from a rescue
   environment against the existing `/mnt`, or just redo the install.

## Safety

This installer is destructive: `disko --mode destroy,format,mount` erases
any disk defined by the selected host's Disko configuration. Always verify
the selected host profile and target machine before confirming.