nixos/docs/auto-installer.md

Auto-installer

This flake builds a self-contained NixOS installer environment that can install any host exposed by its own nixosConfigurations. It was migrated from a formerly-separate nix-auto-installer repo — everything it did now lives here.

The installer provides a small NixOS install environment (ISO, or the same image netbooted via PXE) with SSH access, Git support, and an interactive installation script. Logging in as any user (root or nixos) runs /etc/auto-install.sh, discovers available hosts from this same flake, lets the operator choose a target, applies that host's Disko storage configuration, installs NixOS, and reboots.

This applies to every nixosConfigurations target except lxc-* hosts — see "LXC hosts" immediately below for why those are different.

LXC hosts

lxc-* targets (lxc-minimal, lxc-nix-cache, lxc-server, lxc-docker, lxc-gui, lxc-pxe-boot, lxc-tailscale-exit-node, lxc-tor-relay) are not installed via auto-install.sh — the interactive menu deliberately excludes them. Don't try to select one there; nixos-install would bind-mount / onto /mnt (LXC containers have no raw disk to partition) and then refuse to touch the filesystem it's currently running on — it's designed to protect exactly this case, so it just fails.

modules/platforms/lxc.nix imports nixpkgs' own virtualisation/proxmox-lxc.nix module, which gives every lxc-* host a config.system.build.tarball output — a complete, directly Proxmox-importable container image, no install step at all:

nix build .#nixosConfigurations.lxc-minimal.config.system.build.tarball

This is a plain rootfs tarball, not a vzdump backup archive — restoring it with pct restore fails ("archive contains no configuration file"), since that command expects backup-archive metadata this tarball doesn't have. Use it as a CT template instead: drop it under Proxmox's template storage (conventionally /var/lib/vz/template/cache/ for the local storage, or the GUI's "Create CT" → upload-as-template flow) and create a container from it, supplying all config on the command line since a template has none of its own:

pct create <vmid> local:vztmpl/<file>.tar.xz \
  --unprivileged 1 --features nesting=1,keyctl=1 \
  --rootfs local-lvm:8 --hostname <name> --cores 2 --memory 2048 --swap 2048 \
  --net0 name=eth0,bridge=vmbr0,ip=dhcp
pct start <vmid>

Every one of those extra flags is load-bearing, confirmed by actually booting one:

• --unprivileged 1 — modules/platforms/lxc.nix sets proxmoxLXC.privileged = false, so the image assumes it's running unprivileged. pct create's own CLI default for this flag is privileged (unlike the web UI, whose checkbox defaults the other way) — omit it and you get a privileged container running a NixOS config that assumes unprivileged, a real mismatch.
• --features nesting=1,keyctl=1 — required for a modern (v247+) systemd guest to boot unprivileged at all. Without it, AppArmor denies the nested user namespaces and credential mounts systemd routinely uses (even plain getty units) — every getty crash-loops on a denied /run/credentials/* mount every ~3s (this is what garbage on the console turns out to be) while core services like nsncd fail the same way, and the system never finishes activating.
• --swap 2048 — --memory doesn't touch swap; it silently stays at Proxmox's own 512M default otherwise. Match it to --memory unless you deliberately want otherwise.

First boot runs boot.postBootCommands (registers the Nix store DB and system profile) — there's no separate activation step to run yourself. scripts/proxmox/create-proxmox-resource.sh --type lxc --host <name> automates all of this (host-key handling, building the tarball directly on the Proxmox node itself, pct create with the flags above) — see its --help.

Host keys still need pre-seeding the same way as any other host — the sops-nix activation-vs-first-boot race is identical regardless of how the image reaches the machine. Unlike the ISO/PXE installer (where modules/installer/host-keys.nix bakes every host-keys/ entry into /etc/host-keys/ for auto-install.sh to pick from and copy at install time — see "Host keys" below), an lxc-* tarball has no install step to copy anything during, so modules/platforms/lxc.nix bakes this one target's key straight into /etc/ssh/ssh_host_ed25519_key(.pub) directly, keyed by its own exact flake target name (config.environment.etc can't be read back from within a module still contributing to it, so this comes in via specialArgs.flakeTarget, set by flake.nix's mkTarget):

NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
  nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure

Confirmed the hard way: without this, the tarball's own built-in system just generates a fresh host key at first boot like any host would, which can never match whatever .sops.yaml actually trusts for that target — sops-install-secrets fails with Error getting data key: 0 successful groups required, got 0, and every secret (including this host's own login) permanently fails to decrypt, silently — no error in the boot log at all, since the activation step that would install secrets only runs on a from-scratch first activation and skips silently once /run/current-system already exists. scripts/proxmox/create-proxmox-resource.sh always builds with NIXOS_HOST_KEYS_DIR set for this reason.

Layout

• modules/installer/common.nix — shared by every installer target: SSH access, users, the generated /etc/auto-install.sh script, and the programs.bash.loginShellInit hook that runs it on login.
• modules/installer/iso.nix — ISO/netboot-specific: imports the stock installation-cd-minimal.nix module plus common.nix. Also used, paired with netboot-minimal.nix, to build the PXE netboot variant (see docs/pxe-boot.md).
• modules/installer/host-keys.nix — optionally bakes pre-generated SSH host keys into the image; see "Host keys" below.
• scripts/secrets/sync-host-keys.sh — admin-workstation tool that generates, registers, and (via --remove/--regenerate-all-keys) retires host keys; see "Creating a New Machine" below.
• scripts/secrets/prepare-host-key.sh — narrower predecessor: generates a single key by an arbitrary name without touching .sops.yaml. Still useful for pre-generating a key before its flake target exists (sync-host-keys.sh can only act on targets nixosConfigurations already has); otherwise sync-host-keys.sh does the same thing and more.

Flake outputs:

nixosConfigurations.installer   # ISO/netboot installer image

packages.x86_64-linux.iso   # installer ISO/netboot image
packages.x86_64-linux.pxe   # netboot-ipxe + netboot-initrd + netboot-kernel, bundled

nix build .#iso
nix build .#pxe

There's no nixosConfigurations.proxmox-lxc (installer-boots-as-an-LXC- container) or packages.x86_64-linux.lxc/.all anymore. Both existed only to let the installer itself run as an LXC container so you could nixos-install some other host from within it — but LXC targets are excluded from the install menu (same bind-mount problem as any LXC nixos-install), and now have their own direct tarball path anyway (see "LXC hosts" above), which left the installer's own LXC form with no real use case.

The pxe variant is also built automatically as part of the pxe-boot host itself (modules/pxe-boot/stage-installer-artifacts.nix) and served over iPXE — see docs/pxe-boot.md.

Host keys

sops-nix derives each host's decryption key from its own /etc/ssh/ssh_host_ed25519_key, generated at activation time — before systemd would otherwise generate one on first boot. Without pre-seeding this key, secrets (including the root/nixos login password) fail to decrypt on a genuinely fresh install.

Generated host keys live in host-keys/ at the repo root (ssh_host_ed25519_key

• .pub pairs per hostname). This directory is gitignored on purpose — private key material must never be committed — which also means flakes can't see it through a normal relative path. modules/installer/host-keys.nix reads it through builtins.getEnv, which Nix silently returns as an empty string under normal (non---impure) evaluation, so the module is a no-op — safe by default, including in CI — unless explicitly opted into:

NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure

When built this way, every key currently in host-keys/ is baked into the image at /etc/host-keys/<hostname>_ssh_host_ed25519_key(.pub), and auto-install.sh automatically installs whichever one matches the flake target selected at install time — no manual per-host scp step needed.

Trade-off, accepted deliberately for this LAN-only setup: baking keys in means every key present in host-keys/ at build time becomes readable by anyone who can reach the built image — including, for the PXE variant, anyone who can reach the pxe-boot host's unauthenticated HTTP server. This is considered acceptable here because pxe-boot sits behind LAN-only network infrastructure, not the open internet. If that ever changes, reconsider this default.

auto-install.sh still supports the older manual path as a fallback: if a host's key isn't baked in (/etc/host-keys), it checks /root/host-keys next, where you can scp a key in after boot, same as before this migration.

Storage

Disk partitioning is handled by Disko — the installer has no hardcoded parted/mkfs/mkswap/mount commands, and auto-install.sh runs disko --mode destroy,format,mount unconditionally, no branching on whether the target has a Disko config. Every host reachable through this menu has one:

• proxmox-* (modules/disko/proxmox.nix): a real GPT partition table (ESP + swap + root) on /dev/sda.
• linode-* (modules/disko/linode.nix): Linode provisions and sizes /dev/sda//dev/sdb itself as whole, unpartitioned block devices before the OS ever boots, so this declares them with destroy = false (skips disko's wipe stage for these disks entirely — see the option's own docs) and a bare filesystem/swap content type with no partition table, and the format step it does run only calls mkfs/mkswap if blkid shows the device isn't already formatted — a re-run against an already-provisioned Linode disk is a no-op, not a wipe.

lxc-* is the only category without one — it's excluded from this menu entirely (see "LXC hosts" above), so it never reaches this code path.

Installer process

/etc/auto-install.sh:

1. Queries nixosConfigurations from this flake over the network (git+https://<lanDomain>/beatzaplenty/nixos.git) — this happens at install time, not build time, so a generic installer image always sees whatever hosts are currently committed, without needing a rebuild.
2. Presents them as a menu; confirms the choice.
3. Skips the nix-cache substituter when installing a nix-cache host itself (consistent with that host's own runtime config).
4. Runs disko --mode destroy,format,mount (see "Storage" above — every host reachable through this menu has a Disko config, so this is unconditional).
5. Installs the target's SSH host key from /etc/host-keys or /root/host-keys (see "Host keys" above).
6. Runs nixos-install --flake <url>#<choice> --no-root-password.
7. Cleans up and reboots.

Creating a new machine

Do this instead of jumping straight to a plain install whenever the target host consumes any sops-nix secret — as of this writing, that's every host (modules/common/configuration.nix puts the root/nixos password hash and the GitHub token behind sops-nix for all of them).

1. Add the flake target — hosts/<name>/host.nix plus the matching mkTarget { ... } entry in flake.nix's generatedTargets (see "Composition pattern" in CLAUDE.md). No secrets involved yet, so this is safe to commit on its own if you want a clean history.

2. On your admin workstation, generate and register its host key:

   ./scripts/secrets/sync-host-keys.sh <flake-target>
   

   This generates host-keys/<flake-target>_ssh_host_ed25519_key(.pub), adds it as a new .sops.yaml anchor, works out which secrets/*.yaml files this specific host actually references (from its own config.sops.secrets, not guessed), adds it to each one's key_groups, and re-encrypts them with sops updatekeys — no manual YAML editing. Safe to re-run; it only fills in what's missing.

   Doing this for every host that needs one at once — after adding several new targets, or just to catch up any that were missed — is ./scripts/secrets/sync-host-keys.sh --all. See scripts/secrets/sync-host-keys.sh --help for its other modes (--remove, --regenerate-all-keys).

3. Commit and push. The flake build the installer uses has to see the new recipient before you install, or decryption fails on first boot regardless of the next step.

4. Build the installer image with keys baked in (or reuse an already-serving pxe-boot host, which does this automatically once redeployed):

   NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
   

5. Boot it on the target machine, log in, select the new host's flake target from the menu, confirm. auto-install.sh finds the baked-in key, runs Disko + nixos-install, and reboots.

6. Verify after reboot:

   ssh <new-host> ls /run/secrets/
   

   If that's empty or login fails, the host's age key most likely wasn't in .sops.yaml (or wasn't re-encrypted into the secrets file it needs) when nixos-install ran — fix .sops.yaml/secrets/*.yaml, push, then re-run nixos-install --flake .#<hostname> --no-root-password from a rescue environment against the existing /mnt, or just redo the install.

Safety

This installer is destructive: disko --mode destroy,format,mount erases any disk defined by the selected host's Disko configuration. Always verify the selected host profile and target machine before confirming.