Archived
beatzaplentyandClaude Sonnet 5
120240f14a
LXC hosts (device busy fix): modules/platforms/lxc.nix now imports nixpkgs' own virtualisation/proxmox-lxc.nix, giving every lxc-* host a real config.system.build.tarball output — a directly `pct restore`-able Proxmox container image. This is the actual bug fix behind the "cannot remove real root directory: device busy or in use" error: lxc-* targets were only reachable through nixos-install, which bind-mounts / onto /mnt for containers (no raw disk to partition) and then correctly refuses to modify the filesystem it's currently running on. auto-install.sh's menu now excludes lxc-* targets entirely (they deploy via nix build + pct restore instead, see docs/auto-installer.md) — and, on the same reasoning, also excludes `installer`/`proxmox-lxc`, which are the installer image's own flake targets, not deployable hosts. manageHostName = true keeps host.nix's declared hostnames (upstream's default would let Proxmox's ambient container config win instead); privileged = false matches how these containers are actually created. Eval warnings, now zero across all 19 nixosConfigurations + 4 packages: - Multiple password options (root/nixos in the installer): nixpkgs' own installer profile sets initialHashedPassword = "" for passwordless login, conflicting with our explicit hashedPassword. Force-nulled the upstream option rather than adopting passwordless login, since this image now also boots over LAN PXE with PasswordAuthentication enabled. - boot.zfs.forceImportRoot default value: set explicitly to false (matching the two places that already did) in modules/common/configuration.nix and modules/installer/common.nix, covering every host and the installer alike. - Deprecated pkgs.system in modules/build-types/gui.nix: switched to pkgs.stdenv.hostPlatform.system. All confirmed non-behavioral where it matters: unrelated hosts' drvPaths are byte-identical to their pre-existing baselines throughout. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
200 lines
8.8 KiB
Markdown
200 lines
8.8 KiB
Markdown
# Auto-installer
|
|
|
|
This flake builds a self-contained NixOS installer environment that can
|
|
install any host exposed by its own `nixosConfigurations`. It was migrated
|
|
from a formerly-separate `nix-auto-installer` repo — everything it did now
|
|
lives here.
|
|
|
|
The installer provides a small NixOS install environment (ISO/netboot/Proxmox
|
|
LXC) with SSH access, Git support, and an interactive installation script.
|
|
Logging in as any user (root or `nixos`) runs `/etc/auto-install.sh`,
|
|
discovers available hosts from this same flake, lets the operator choose a
|
|
target, applies that host's Disko storage configuration, installs NixOS, and
|
|
reboots.
|
|
|
|
**This applies to every `nixosConfigurations` target except `lxc-*` hosts —
|
|
see "LXC hosts" immediately below for why those are different.**
|
|
|
|
## LXC hosts
|
|
|
|
`lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`,
|
|
`lxc-gui`, `lxc-pxe-boot`) are **not** installed via `auto-install.sh` — the
|
|
interactive menu deliberately excludes them. Don't try to select one there;
|
|
`nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw
|
|
disk to partition) and then refuse to touch the filesystem it's currently
|
|
running on — it's designed to protect exactly this case, so it just fails.
|
|
|
|
`modules/platforms/lxc.nix` imports nixpkgs' own
|
|
`virtualisation/proxmox-lxc.nix` module, which gives every `lxc-*` host a
|
|
`config.system.build.tarball` output — a complete, directly Proxmox-importable
|
|
container image, no install step at all:
|
|
|
|
```sh
|
|
nix build .#nixosConfigurations.lxc-minimal.config.system.build.tarball
|
|
```
|
|
|
|
Then, on the Proxmox host, `pct restore` (or the GUI's "Create CT" → upload
|
|
template flow) that tarball directly as a new container. First boot runs
|
|
`boot.postBootCommands` (registers the Nix store DB and system profile) —
|
|
there's no separate activation step to run yourself.
|
|
|
|
Host keys still need pre-seeding the same way as any other host (see "Host
|
|
keys" below) — the sops-nix activation-vs-first-boot race is identical
|
|
regardless of how the image reaches the machine. `NIXOS_HOST_KEYS_DIR=...
|
|
nix build ... --impure` bakes the matching key into the tarball the same way
|
|
it does for the ISO/PXE installer images.
|
|
|
|
## Layout
|
|
|
|
- `modules/installer/common.nix` — shared by every installer target: SSH
|
|
access, users, the generated `/etc/auto-install.sh` script, and the
|
|
`programs.bash.loginShellInit` hook that runs it on login.
|
|
- `modules/installer/iso.nix` — ISO/netboot-specific: imports the stock
|
|
`installation-cd-minimal.nix` module plus `common.nix`. Also used, paired
|
|
with `netboot-minimal.nix`, to build the PXE netboot variant (see
|
|
`docs/pxe-boot.md`).
|
|
- `modules/installer/proxmox-lxc.nix` — Proxmox LXC-specific: disables
|
|
bootloader options (containers don't need one) plus `common.nix`.
|
|
- `modules/installer/host-keys.nix` — optionally bakes pre-generated SSH
|
|
host keys into the image; see "Host keys" below.
|
|
- `scripts/prepare-host-key.sh` — admin-workstation pre-flight tool, see
|
|
"Creating a New Machine" below.
|
|
|
|
Flake outputs:
|
|
|
|
```nix
|
|
nixosConfigurations.installer # ISO/netboot installer image
|
|
nixosConfigurations.proxmox-lxc # Proxmox LXC-based installer image
|
|
|
|
packages.x86_64-linux.iso # installer ISO/netboot image
|
|
packages.x86_64-linux.lxc # Proxmox LXC installer tarball
|
|
packages.x86_64-linux.pxe # netboot-ipxe + netboot-initrd + netboot-kernel, bundled
|
|
packages.x86_64-linux.all # all three of the above, bundled
|
|
```
|
|
|
|
```sh
|
|
nix build .#iso
|
|
nix build .#lxc
|
|
nix build .#pxe
|
|
nix build .#all
|
|
```
|
|
|
|
The `pxe` variant is also built automatically as part of the `pxe-boot` host
|
|
itself (`modules/pxe-boot/stage-installer-artifacts.nix`) and served over
|
|
iPXE — see `docs/pxe-boot.md`.
|
|
|
|
## Host keys
|
|
|
|
`sops-nix` derives each host's decryption key from its own
|
|
`/etc/ssh/ssh_host_ed25519_key`, generated at **activation** time — before
|
|
systemd would otherwise generate one on first boot. Without pre-seeding this
|
|
key, secrets (including the root/nixos login password) fail to decrypt on a
|
|
genuinely fresh install.
|
|
|
|
Generated host keys live in `host-keys/` at the repo root (`ssh_host_ed25519_key`
|
|
+ `.pub` pairs per hostname). This directory is **gitignored on purpose** —
|
|
private key material must never be committed — which also means flakes can't
|
|
see it through a normal relative path. `modules/installer/host-keys.nix`
|
|
reads it through `builtins.getEnv`, which Nix silently returns as an empty
|
|
string under normal (non-`--impure`) evaluation, so the module is a no-op —
|
|
safe by default, including in CI — unless explicitly opted into:
|
|
|
|
```sh
|
|
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
|
|
```
|
|
|
|
When built this way, every key currently in `host-keys/` is baked into the
|
|
image at `/etc/host-keys/<hostname>_ssh_host_ed25519_key(.pub)`, and
|
|
`auto-install.sh` automatically installs whichever one matches the flake
|
|
target selected at install time — no manual per-host scp step needed.
|
|
|
|
**Trade-off, accepted deliberately for this LAN-only setup:** baking keys in
|
|
means every key present in `host-keys/` at build time becomes readable by
|
|
anyone who can reach the built image — including, for the PXE variant, anyone
|
|
who can reach the `pxe-boot` host's unauthenticated HTTP server. This is
|
|
considered acceptable here because `pxe-boot` sits behind LAN-only network
|
|
infrastructure, not the open internet. If that ever changes, reconsider this
|
|
default.
|
|
|
|
`auto-install.sh` still supports the older manual path as a fallback: if a
|
|
host's key isn't baked in (`/etc/host-keys`), it checks `/root/host-keys`
|
|
next, where you can `scp` a key in after boot, same as before this migration.
|
|
|
|
## Storage
|
|
|
|
Disk partitioning is handled by Disko — the installer has no hardcoded
|
|
`parted`/`mkfs`/`mkswap`/`mount` commands. Each host defines its own layout
|
|
via `disko.devices`; a host without one is installed in place via bind-mount
|
|
(this is how every LXC container target works, since they have no raw block
|
|
device to partition).
|
|
|
|
## Installer process
|
|
|
|
`/etc/auto-install.sh`:
|
|
|
|
1. Queries `nixosConfigurations` from this flake over the network (`git+https://<lanDomain>/beatzaplenty/nixos.git`) — this happens at *install* time, not build time, so a generic installer image always sees whatever hosts are currently committed, without needing a rebuild.
|
|
2. Presents them as a menu; confirms the choice.
|
|
3. Skips the `nix-cache` substituter when installing a `nix-cache` host itself (consistent with that host's own runtime config).
|
|
4. Runs `disko --mode destroy,format,mount` if the target has a Disko config, otherwise bind-mounts `/` onto `/mnt`.
|
|
5. Installs the target's SSH host key from `/etc/host-keys` or `/root/host-keys` (see "Host keys" above).
|
|
6. Runs `nixos-install --flake <url>#<choice> --no-root-password`.
|
|
7. Cleans up and reboots.
|
|
|
|
## Creating a new machine
|
|
|
|
Do this instead of jumping straight to a plain install whenever the target
|
|
host consumes any sops-nix secret — as of this writing, that's every host
|
|
(`modules/common/configuration.nix` puts the root/nixos password hash and the
|
|
GitHub token behind sops-nix for all of them).
|
|
|
|
1. **On your admin workstation**, decide the new machine's flake target name
|
|
(`<platform>-<buildtype>`, e.g. `proxmox-server`) and generate + register
|
|
its host key:
|
|
|
|
```sh
|
|
./scripts/prepare-host-key.sh <hostname>
|
|
```
|
|
|
|
This generates `host-keys/<hostname>_ssh_host_ed25519_key(.pub)` and
|
|
prints the `.sops.yaml` snippet to add, which secrets files to
|
|
re-encrypt it into, and the next steps below.
|
|
|
|
2. **Edit `.sops.yaml`** with the printed snippet, then re-encrypt every
|
|
secrets file you added the new host to:
|
|
|
|
```sh
|
|
nix-shell -p sops --run 'sops updatekeys secrets/common.yaml'
|
|
```
|
|
|
|
3. **Commit and push.** The flake build the installer uses has to see the
|
|
new recipient before you install, or decryption fails on first boot
|
|
regardless of the next step.
|
|
|
|
4. **Build the installer image with keys baked in** (or reuse an already-serving `pxe-boot` host, which does this automatically once redeployed):
|
|
|
|
```sh
|
|
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build .#iso --impure
|
|
```
|
|
|
|
5. **Boot it on the target machine**, log in, select the new host's flake
|
|
target from the menu, confirm. `auto-install.sh` finds the baked-in key,
|
|
runs Disko + `nixos-install`, and reboots.
|
|
|
|
6. **Verify after reboot:**
|
|
|
|
```sh
|
|
ssh <new-host> ls /run/secrets/
|
|
```
|
|
|
|
If that's empty or login fails, the host's age key most likely wasn't in
|
|
`.sops.yaml` (or wasn't re-encrypted into the secrets file it needs) when
|
|
`nixos-install` ran — fix `.sops.yaml`/`secrets/*.yaml`, push, then re-run
|
|
`nixos-install --flake .#<hostname> --no-root-password` from a rescue
|
|
environment against the existing `/mnt`, or just redo the install.
|
|
|
|
## Safety
|
|
|
|
This installer is destructive: `disko --mode destroy,format,mount` erases
|
|
any disk defined by the selected host's Disko configuration. Always verify
|
|
the selected host profile and target machine before confirming.
|