Archived
Document the sops-nix host-key pre-seeding process end-to-end
Adds a "Creating a New Machine" walkthrough tying together the steps that were previously scattered or missing entirely: running scripts/prepare-host-key.sh, editing .sops.yaml + sops updatekeys, pushing nixos, scp'ing the key to the live installer, and verifying /run/secrets after first boot. Also fixes several places that still described the pre-refactor layout (installer.nix as the sole config file, only the nixos user triggering the installer, "Pre-Seeding..." section name that no longer existed) to match the current common.nix/installer.nix/ proxmox-lxc.nix split. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
beatzaplentyandClaude Sonnet 5 committed
1 parent
9db659d54e
commit
5a7bdea454
1 file changed
+128
-44
@@ -4,10 +4,10 @@ This repository builds a custom NixOS installer environment that can automatical
|
|||||||
install any host exposed by a separate NixOS flake.
|
install any host exposed by a separate NixOS flake.
|
||||||
|
|
||||||
The installer provides a small NixOS netboot/install environment with SSH access,
|
The installer provides a small NixOS netboot/install environment with SSH access,
|
||||||
Git support, and an interactive installation script. When the `nixos` user logs
|
Git support, and an interactive installation script. Logging in as any user
|
||||||
in, it runs `/etc/auto-install.sh`, discovers available hosts from the target
|
(root or `nixos`) runs `/etc/auto-install.sh`, discovers available hosts from
|
||||||
flake, allows the operator to choose a system profile, applies the matching
|
the target flake, allows the operator to choose a system profile, applies the
|
||||||
Disko storage configuration, installs NixOS, and reboots.
|
matching Disko storage configuration, installs NixOS, and reboots.
|
||||||
|
|
||||||
The installer is designed for a mixed environment where different machines may
|
The installer is designed for a mixed environment where different machines may
|
||||||
have different storage layouts, including Proxmox VMs, Linode instances, and
|
have different storage layouts, including Proxmox VMs, Linode instances, and
|
||||||
@@ -18,17 +18,35 @@ defined by each host's NixOS configuration using Disko.
|
|||||||
|
|
||||||
* `flake.nix`
|
* `flake.nix`
|
||||||
|
|
||||||
* Defines the installer build outputs.
|
* Defines the installer build outputs directly via `nixosSystem` (no
|
||||||
* Builds an `install-iso` image using `nixos-generators`.
|
external `nixos-generators` dependency).
|
||||||
|
* `nixosConfigurations.installer` — ISO/netboot installer image.
|
||||||
|
* `nixosConfigurations.proxmox-lxc` — Proxmox LXC-based installer image.
|
||||||
* Builds netboot kernel, initrd, and iPXE scripts.
|
* Builds netboot kernel, initrd, and iPXE scripts.
|
||||||
|
|
||||||
|
* `common.nix`
|
||||||
|
|
||||||
|
* Shared by every installer target (imported by both `installer.nix`
|
||||||
|
and `proxmox-lxc.nix`).
|
||||||
|
* Enables SSH access, configures Git credentials for the target flake
|
||||||
|
repository, creates the `nixos` and `root` users.
|
||||||
|
* Provides the generated `/etc/auto-install.sh` installation script and
|
||||||
|
the `programs.bash.loginShellInit` hook that runs it on login.
|
||||||
|
|
||||||
* `installer.nix`
|
* `installer.nix`
|
||||||
|
|
||||||
* Defines the installer environment.
|
* ISO/netboot-specific: imports the stock `installation-cd-minimal.nix`
|
||||||
* Enables SSH access.
|
module plus `common.nix`.
|
||||||
* Configures Git access to the target flake repository.
|
|
||||||
* Creates the `nixos` and `root` users.
|
* `proxmox-lxc.nix`
|
||||||
* Provides the generated `/etc/auto-install.sh` installation script.
|
|
||||||
|
* Proxmox LXC-specific: disables bootloader options (containers don't
|
||||||
|
need one) plus `common.nix`.
|
||||||
|
|
||||||
|
* `scripts/prepare-host-key.sh`
|
||||||
|
|
||||||
|
* Admin-workstation pre-flight tool — see "Creating a New Machine"
|
||||||
|
below.
|
||||||
|
|
||||||
The installed system configuration lives in a separate NixOS flake. Each target
|
The installed system configuration lives in a separate NixOS flake. Each target
|
||||||
host is exposed through:
|
host is exposed through:
|
||||||
@@ -87,7 +105,7 @@ These can be used to PXE boot the installer environment.
|
|||||||
|
|
||||||
1. Boot the generated ISO or netboot environment on the target machine.
|
1. Boot the generated ISO or netboot environment on the target machine.
|
||||||
|
|
||||||
2. Log in as the `nixos` user.
|
2. Log in as `root` or `nixos` (over SSH or local console).
|
||||||
|
|
||||||
3. The login shell launches:
|
3. The login shell launches:
|
||||||
|
|
||||||
@@ -200,9 +218,9 @@ The generated `/etc/auto-install.sh` performs the following steps:
|
|||||||
6. Prepare `/mnt` using the Disko-generated mount configuration.
|
6. Prepare `/mnt` using the Disko-generated mount configuration.
|
||||||
|
|
||||||
7. Check for a pre-seeded SSH host key at `/root/host-keys/<host>_ssh_host_ed25519_key`
|
7. Check for a pre-seeded SSH host key at `/root/host-keys/<host>_ssh_host_ed25519_key`
|
||||||
and install it to `/mnt/etc/ssh/` if present (see "Pre-Seeding SSH Host
|
and install it to `/mnt/etc/ssh/` if present (see "Creating a New
|
||||||
Keys for sops-nix" below) — prompts for confirmation before continuing
|
Machine" below) — prompts for confirmation before continuing without
|
||||||
without one.
|
one.
|
||||||
|
|
||||||
8. Install NixOS:
|
8. Install NixOS:
|
||||||
|
|
||||||
@@ -216,42 +234,98 @@ The generated `/etc/auto-install.sh` performs the following steps:
|
|||||||
|
|
||||||
10. Reboot.
|
10. Reboot.
|
||||||
|
|
||||||
## Pre-Seeding SSH Host Keys for sops-nix
|
## Creating a New Machine
|
||||||
|
|
||||||
|
Full walkthrough for installing a new host, including the sops-nix
|
||||||
|
pre-seeding step. Do this instead of jumping straight to "Use The
|
||||||
|
Installer" whenever the target host consumes any secret managed by the
|
||||||
|
`nixos` flake's sops-nix setup (as of this writing, that's every host —
|
||||||
|
`modules/common/configuration.nix` puts the root/nixos password hash and
|
||||||
|
the GitHub token behind sops-nix for all of them).
|
||||||
|
|
||||||
|
### Why the extra step exists
|
||||||
|
|
||||||
The `nixos` flake manages secrets with sops-nix, using an age key derived
|
The `nixos` flake manages secrets with sops-nix, using an age key derived
|
||||||
from each host's own `/etc/ssh/ssh_host_ed25519_key`. That key is normally
|
from each host's own `/etc/ssh/ssh_host_ed25519_key`. That key is normally
|
||||||
generated by a systemd service (`sshd-keygen`) the first time a host boots
|
generated by a systemd service (`sshd-keygen`) the first time a host boots
|
||||||
— but sops-nix decrypts secrets (including the login password) earlier
|
— but sops-nix decrypts secrets during system *activation*, which runs
|
||||||
than that, during system *activation*, which runs before systemd starts
|
**before** systemd starts pursuing the target `sshd-keygen` is gated
|
||||||
pursuing the target that `sshd-keygen` is gated behind. On a genuinely
|
behind. On a genuinely fresh install, that means secrets — including the
|
||||||
fresh install, this means secrets fail to decrypt on the very first boot
|
login password — fail to decrypt on the very first boot unless the host
|
||||||
unless the host key already exists beforehand.
|
key already exists beforehand. Pre-seeding it is what fixes that.
|
||||||
|
|
||||||
To avoid this, generate the key ahead of time and register it with the
|
Machines that don't consume any sops-nix secret don't strictly need this,
|
||||||
`nixos` flake's sops-nix setup **before** installing:
|
but since the shared password hash currently applies to every host, treat
|
||||||
|
it as required unless you've specifically confirmed otherwise.
|
||||||
|
|
||||||
```sh
|
### Steps
|
||||||
./scripts/prepare-host-key.sh <hostname> [path-to-nixos-repo]
|
|
||||||
```
|
|
||||||
|
|
||||||
This generates `host-keys/<hostname>_ssh_host_ed25519_key(.pub)` locally
|
**1. On your admin workstation** (the machine with `~/.config/sops/age/keys.txt`
|
||||||
and prints the exact steps to add its derived age key to `nixos/.sops.yaml`,
|
from the `nixos` repo's sops-nix setup), decide the new machine's flake
|
||||||
re-encrypt the relevant `secrets/*.yaml` files with `sops updatekeys`, and
|
target name (`<platform>-<buildtype>`, e.g. `proxmox-server`) and
|
||||||
commit + push the `nixos` repo.
|
generate + register its host key:
|
||||||
|
|
||||||
Once that's done and the target machine is booted into the installer,
|
```sh
|
||||||
copy the generated key onto it before running (or continuing)
|
./scripts/prepare-host-key.sh <hostname> [path-to-nixos-repo]
|
||||||
`/etc/auto-install.sh`:
|
```
|
||||||
|
|
||||||
```sh
|
This generates `host-keys/<hostname>_ssh_host_ed25519_key(.pub)`
|
||||||
scp host-keys/<hostname>_ssh_host_ed25519_key{,.pub} root@<target-ip>:/root/host-keys/
|
locally and prints:
|
||||||
```
|
|
||||||
|
|
||||||
`auto-install.sh` checks for this file automatically and installs it to
|
- the line to add under `keys:` in `nixos/.sops.yaml`
|
||||||
the target's `/mnt/etc/ssh/` before running `nixos-install`. If it's
|
- which `creation_rules` key_groups to add it to (`secrets/common.yaml`
|
||||||
missing, the script warns and asks for confirmation before continuing —
|
always; a per-host `secrets/<hostname>.yaml` too if this build type
|
||||||
useful for hosts that don't consume any sops-nix secrets, but skipping it
|
has its own secrets, same pattern as `nix-cache`/`server`)
|
||||||
for a host that does will lock secrets out of decrypting on first boot.
|
|
||||||
|
**2. Edit `nixos/.sops.yaml`** by hand with the printed snippet, then
|
||||||
|
re-encrypt every secrets file you added the new host to:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
nix-shell -p sops --run 'sops updatekeys nixos/secrets/common.yaml'
|
||||||
|
```
|
||||||
|
|
||||||
|
**3. Commit and push the `nixos` repo.** The flake build the installer
|
||||||
|
uses has to see the new recipient before you install, or decryption
|
||||||
|
will fail on first boot regardless of step 4 below.
|
||||||
|
|
||||||
|
**4. Build and boot the installer image** on the target machine (ISO,
|
||||||
|
netboot, or Proxmox LXC — see "Build The Installer ISO" / "Build
|
||||||
|
Netboot Components" above).
|
||||||
|
|
||||||
|
**5. Copy the generated private key onto the live installer environment**
|
||||||
|
once it's reachable over SSH:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
scp host-keys/<hostname>_ssh_host_ed25519_key{,.pub} root@<target-ip>:/root/host-keys/
|
||||||
|
```
|
||||||
|
|
||||||
|
**6. Log in** (as `root` or `nixos`) to trigger `/etc/auto-install.sh`
|
||||||
|
automatically, or run it manually if it doesn't (see Troubleshooting).
|
||||||
|
Select `<hostname>`'s flake target from the menu and confirm. The
|
||||||
|
script will:
|
||||||
|
|
||||||
|
- run Disko to format and mount `/mnt`
|
||||||
|
- find the key you copied in at `/root/host-keys/<hostname>_ssh_host_ed25519_key`
|
||||||
|
and install it to `/mnt/etc/ssh/` — if it's missing, the script
|
||||||
|
warns and asks for confirmation before continuing without it (fine
|
||||||
|
for a host with no sops-nix secrets; for any other host, continuing
|
||||||
|
anyway means the machine will boot with no working login password)
|
||||||
|
- run `nixos-install` and reboot
|
||||||
|
|
||||||
|
**7. Verify after reboot.** SSH into the new machine and confirm secrets
|
||||||
|
actually decrypted:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
ls /run/secrets/
|
||||||
|
```
|
||||||
|
|
||||||
|
If that's empty or login fails, the most likely cause is the host's
|
||||||
|
age key not being in `.sops.yaml` (or not re-encrypted into the
|
||||||
|
secrets file it needs) at the time `nixos-install` ran — re-check
|
||||||
|
steps 1–3, fix `nixos/.sops.yaml`/`secrets/*.yaml`, push, then
|
||||||
|
re-run `nixos-install --flake <flake>#<hostname> --no-root-password`
|
||||||
|
from a rescue/live environment against the existing `/mnt` (or just
|
||||||
|
redo the install).
|
||||||
|
|
||||||
`host-keys/` is gitignored — never commit private key material.
|
`host-keys/` is gitignored — never commit private key material.
|
||||||
|
|
||||||
@@ -285,11 +359,11 @@ The installer currently assumes:
|
|||||||
|
|
||||||
* The target host is responsible for defining its own storage layout.
|
* The target host is responsible for defining its own storage layout.
|
||||||
|
|
||||||
Change these values in `installer.nix` if your environment differs.
|
Change these values in `common.nix` if your environment differs.
|
||||||
|
|
||||||
## Security Notes
|
## Security Notes
|
||||||
|
|
||||||
`installer.nix` currently contains:
|
`common.nix` currently contains:
|
||||||
|
|
||||||
* Git credentials
|
* Git credentials
|
||||||
* Password hashes
|
* Password hashes
|
||||||
@@ -369,8 +443,18 @@ Verify:
|
|||||||
|
|
||||||
### Installer does not start automatically
|
### Installer does not start automatically
|
||||||
|
|
||||||
Run manually:
|
It's triggered by `programs.bash.loginShellInit` (rendered into
|
||||||
|
`/etc/profile`), which only fires for genuine login shells. If you're not
|
||||||
|
seeing it — e.g. you attached to an existing shell rather than logging in
|
||||||
|
fresh — run it manually:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
sudo /etc/auto-install.sh
|
sudo /etc/auto-install.sh
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Login succeeds but the machine has no working password after install
|
||||||
|
|
||||||
|
The new host's sops-nix secrets didn't decrypt on first boot — almost
|
||||||
|
always because its age key wasn't registered in `nixos/.sops.yaml` and
|
||||||
|
re-encrypted into the secrets file(s) it needs before `nixos-install` ran.
|
||||||
|
See "Creating a New Machine" above, specifically step 7.
|
||||||
Reference in new issue
Block a user