Consolidate CI on codex-maintenance.sh; clean up dead scripts and docs

Both check-nixos.yml workflows (GitHub + Gitea) now call
scripts/codex-maintenance.sh instead of a hand-rolled eval-only loop,
closing a real gap: CI previously enforced none of the secret grep,
nixpkgs-fmt, or statix checks that codex-maintenance.sh already runs
locally — nothing was stopping that from regressing. One script now
backs both, instead of two copies that can drift from each other.

codex-maintenance.sh itself is extended to cover buildable surface
that wasn't validated anywhere before: packages.x86_64-linux.*, plus
config.system.build.tarball (lxc-* hosts) and
config.system.build.diskoImagesScript (proxmox-*, excluding the
installer's own proxmox-lxc target, which has no disko config).

Also:
- scripts/prepare-host-key.sh: dropped the redundant
  [path-to-nixos-repo] parameter — it always defaults to the repo the
  script itself lives in now, so a second argument never made sense
  after the nix-auto-installer migration.
- Removed prepare.sh (dead pre-disko manual parted/mkfs/mkswap
  partitioning, fully superseded) and
  scripts/create-linode-installer-disk.sh (incomplete draft for an
  abandoned dd-via-rescue-mode approach; Linode hosts already deploy
  fine through the normal auto-installer flow).
- docs/pxe-boot.md: fixed a stale `nixosConfigurations.pxe-boot` eval
  command (pre-refactor flat name, not a real flake attribute
  anymore) and added a cross-reference to docs/auto-installer.md.
- CLAUDE.md/README.md: full documentation pass reconciling this
  session's changes — modules/installer/, modules/pxe-boot/, the
  LXC/Proxmox image-building deployment paths, corrected the
  password-hash/SSH-key locations in the safety-rules section (both
  had drifted to reference files/paths that no longer exist), and
  added session-workflow guidance to prefer targeted host evals over
  full-repo sweeps for incremental changes (explicitly scoped to
  interactive sessions, not CI).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
This commit is contained in:
2026-07-20 07:02:23 +10:00
co-authored by Claude Sonnet 5
parent 559c538a3d
commit 288835db29
9 changed files with 128 additions and 74 deletions
+2 -10
View File
@@ -17,13 +17,5 @@ jobs:
- name: Install Nix - name: Install Nix
uses: DeterminateSystems/nix-installer-action@v19 uses: DeterminateSystems/nix-installer-action@v19
- name: Evaluate all NixOS hosts - name: Run maintenance checks (secrets, fmt, lint, eval)
run: | run: bash scripts/codex-maintenance.sh
set -euo pipefail
hosts="$(nix --extra-experimental-features 'nix-command flakes' eval --json \
.#nixosConfigurations --apply builtins.attrNames | jq -r '.[]')"
for host in $hosts; do
echo "Evaluating ${host}"
nix --extra-experimental-features 'nix-command flakes' eval \
".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath" --raw
done
+2 -10
View File
@@ -17,13 +17,5 @@ jobs:
- name: Install Nix - name: Install Nix
uses: DeterminateSystems/nix-installer-action@v19 uses: DeterminateSystems/nix-installer-action@v19
- name: Evaluate all NixOS hosts - name: Run maintenance checks (secrets, fmt, lint, eval)
run: | run: bash scripts/codex-maintenance.sh
set -euo pipefail
hosts="$(nix --extra-experimental-features 'nix-command flakes' eval --json \
.#nixosConfigurations --apply builtins.attrNames | jq -r '.[]')"
for host in $hosts; do
echo "Evaluating ${host}"
nix --extra-experimental-features 'nix-command flakes' eval \
".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath" --raw
done
+47 -7
View File
@@ -17,11 +17,19 @@ machines when deployed.
- Validation is limited to evaluation, linting, formatting checks, and - Validation is limited to evaluation, linting, formatting checks, and
`nix build --dry-run --no-link`. `nix build --dry-run --no-link`.
- Do not add secrets, tokens, private keys, or new password hashes to the repo. - Do not add secrets, tokens, private keys, or new password hashes to the repo.
- This repo currently contains **committed password hashes** (e.g. - This repo currently contains **committed password hashes** in
`prepare.sh`, `hosts/nixos/configuration.nix`) and SSH public keys (e.g. `modules/installer/common.nix` (the auto-installer's own root/nixos login —
`modules/nix-cache/server.nix`). The hashes are known tech debt — do not use a deliberate, documented choice, see `docs/auto-installer.md`, not
them as a template for new hosts, and flag any *new* secret-like string you accidental tech debt) and **SSH public keys** in `variables.nix`
encounter instead of committing it. (`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`) plus a couple of
per-host `KEY` values for beszel-agent auth (`hosts/server/host.nix`,
`hosts/nix-cache/host.nix`). Don't use the installer's hardcoded hash as a
template for a *real* host — every other host uses sops-nix
(`hashedPasswordFile`, see "Security Notes" in `README.md`). Flag any *new*
secret-like string you encounter instead of committing it.
- `host-keys/` is gitignored — locally-generated *private* SSH host keys for
the auto-installer (see `docs/auto-installer.md`). Never commit its
contents; if `git status` ever shows it as trackable, something is wrong.
## Commands ## Commands
@@ -50,6 +58,18 @@ maintenance script pulls them via `nix run github:NixOS/nixpkgs/nixos-25.11#<too
There is no test suite — "correctness" here means the flake evaluates and There is no test suite — "correctness" here means the flake evaluates and
`nixpkgs-fmt`/`statix` are clean. `nixpkgs-fmt`/`statix` are clean.
**In an interactive agent session**, prefer targeted checks over full-repo
sweeps: after editing one or two hosts/modules, evaluate just the
`nixosConfigurations.<host>` you touched (plus any `config.system.build.tarball`
/`diskoImagesScript`/package output affected) rather than looping over every
host — `codex-maintenance.sh` evaluates 19 hosts plus every package/tarball/
image variant now and is slow to run after each small change. Reserve a full
`codex-maintenance.sh` run for changes that plausibly affect every host
(`modules/common/*`, `flake.nix`, `variables.nix`) or as a final check before
committing. This is a session-workflow preference only — it does not apply to
CI, which should keep running the full script on every push/PR regardless of
diff size; that's the point of it.
## Architecture ## Architecture
`flake.nix` is the single entry point. It generates one `flake.nix` is the single entry point. It generates one
@@ -93,7 +113,10 @@ removing a host.
hardware config, imported directly by the platform module itself hardware config, imported directly by the platform module itself
(`../hardware-configuration/vm/{proxmox,linode}.nix`) — **not** wired in (`../hardware-configuration/vm/{proxmox,linode}.nix`) — **not** wired in
from `flake.nix`. `lxc.nix` has no hardware-configuration counterpart since from `flake.nix`. `lxc.nix` has no hardware-configuration counterpart since
containers share the host kernel. containers share the host kernel; instead it imports nixpkgs' own
`virtualisation/proxmox-lxc.nix`, which gives every `lxc-*` host a
`config.system.build.tarball` output (`pct restore`-ready, no install step
— see `docs/auto-installer.md`).
- `modules/build-types/*.nix` — what a system is for: - `modules/build-types/*.nix` — what a system is for:
minimal/server/docker/gui/pxe-boot/nix-cache. minimal/server/docker/gui/pxe-boot/nix-cache.
- `modules/common/configuration.nix` — base NixOS config imported by every - `modules/common/configuration.nix` — base NixOS config imported by every
@@ -102,8 +125,19 @@ removing a host.
the `nixos` user; the `nixos` workstation (`gui` build type) has its own, the `nixos` user; the `nixos` workstation (`gui` build type) has its own,
other hosts share `modules/common/home.nix`. other hosts share `modules/common/home.nix`.
- `modules/disko/proxmox.nix` — declarative disk layout (GPT: ESP + swap + - `modules/disko/proxmox.nix` — declarative disk layout (GPT: ESP + swap +
ext4 root) via disko, used by all Proxmox-VM hosts. ext4 root) via disko, used by all Proxmox-VM hosts (`proxmox-*`, not
`lxc-*`). Also carries `imageSize`/`imageName`, letting every `proxmox-*`
host be built as a standalone, `qm importdisk`-ready `.raw` image with no
install step — see `docs/proxmox-images.md`.
- `modules/boot/efi.nix` — systemd-boot + EFI vars, paired with the disko module. - `modules/boot/efi.nix` — systemd-boot + EFI vars, paired with the disko module.
- `modules/installer/` — the auto-installer environment (ISO/netboot/Proxmox
LXC): `common.nix` (shared config + the generated `auto-install.sh`),
`iso.nix`/`proxmox-lxc.nix` (per-variant), `host-keys.nix` (optionally
bakes `host-keys/` into the image under `--impure`). See
`docs/auto-installer.md`.
- `modules/pxe-boot/stage-installer-artifacts.nix` — builds the installer's
netboot image and stages it on the `pxe-boot` host so its iPXE menu can
chain straight to it. See `docs/pxe-boot.md`.
- `modules/nix-cache/{client,server,remote-builder-client}.nix` — binary cache - `modules/nix-cache/{client,server,remote-builder-client}.nix` — binary cache
substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the
full design (per-host local stores, no shared `/nix/store`, and how the full design (per-host local stores, no shared `/nix/store`, and how the
@@ -129,6 +163,12 @@ duplicating config.
handling. handling.
- `docs/pxe-boot.md` — the `pxe-boot` host's iPXE/TFTP/HTTP boot chain and - `docs/pxe-boot.md` — the `pxe-boot` host's iPXE/TFTP/HTTP boot chain and
directory layout under `/srv/pxe`. directory layout under `/srv/pxe`.
- `docs/auto-installer.md` — the installer environment (ISO/netboot/Proxmox
LXC), `host-keys/` and the sops-nix pre-seeding problem it solves, and why
`lxc-*` hosts are deliberately excluded from its menu.
- `docs/proxmox-images.md` — building `proxmox-*` hosts as standalone `.raw`
disk images (disko's image builder) instead of installing, and deploying
the result to Proxmox.
- `docs/flake-lock-automation.md` — how `flake.lock` updates flow through CI - `docs/flake-lock-automation.md` — how `flake.lock` updates flow through CI
(scheduled `nix flake update` PR + host-eval-on-PR workflow) and why hosts (scheduled `nix flake update` PR + host-eval-on-PR workflow) and why hosts
should track the committed lock file rather than `nixos-rebuild --upgrade-all`. should track the committed lock file rather than `nixos-rebuild --upgrade-all`.
+22 -1
View File
@@ -55,7 +55,7 @@ nix eval --json .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]'
| `modules/nix-cache/` | Binary cache and remote builder client/server modules | | `modules/nix-cache/` | Binary cache and remote builder client/server modules |
| `modules/installer/` | Auto-installer environment (ISO/netboot/Proxmox LXC) — see `docs/auto-installer.md` | | `modules/installer/` | Auto-installer environment (ISO/netboot/Proxmox LXC) — see `docs/auto-installer.md` |
| `host-keys/` | Gitignored, locally-generated SSH host keys for the auto-installer — see `docs/auto-installer.md` | | `host-keys/` | Gitignored, locally-generated SSH host keys for the auto-installer — see `docs/auto-installer.md` |
| `docs/` | Operational notes for cache, builders, lock updates, boot services, and the auto-installer | | `docs/` | Operational notes for cache, builders, lock updates, boot services, the auto-installer, and Proxmox image builds |
| `scripts/` | Codex setup, validation, and host-key-prep helpers | | `scripts/` | Codex setup, validation, and host-key-prep helpers |
## Validation ## Validation
@@ -87,6 +87,20 @@ review sessions.
client hosts. client hosts.
- `pxe-boot` serves iPXE boot files over HTTP from `/srv/pxe`. - `pxe-boot` serves iPXE boot files over HTTP from `/srv/pxe`.
### Deploying a new host
Three different paths depending on target, none of them involving a manual
`nixos-rebuild switch` from this repo:
- Most hosts: boot the auto-installer, pick the target from its menu — see
`docs/auto-installer.md`.
- `lxc-*` targets: not installed at all — build a ready-to-run container
tarball and `pct restore` it directly. `docs/auto-installer.md` covers why
(and the installer's menu excludes them for the same reason).
- `proxmox-*` targets: can alternatively be built as a standalone `.raw`
disk image and attached to a new VM with no install step — see
`docs/proxmox-images.md`.
## Security Notes ## Security Notes
Do not commit tokens, private keys, live credentials, or new password hashes Do not commit tokens, private keys, live credentials, or new password hashes
@@ -107,6 +121,13 @@ enabled via `git config core.hooksPath .githooks`, done automatically by
`scripts/codex-setup.sh`) runs `gitleaks protect --staged` to catch mistakes `scripts/codex-setup.sh`) runs `gitleaks protect --staged` to catch mistakes
before they're committed. before they're committed.
The auto-installer environment is the one deliberate exception to
sops-nix-everywhere: it has a hardcoded login password instead (no stable
per-boot host key for sops-nix to derive from on ephemeral media) — see
"Host keys" in `docs/auto-installer.md` for why, and how the private keys it
*does* pre-seed for target hosts stay out of git via the gitignored
`host-keys/` directory.
This repository's git *history* still contains secrets committed before this This repository's git *history* still contains secrets committed before this
migration (see `remove-sensetive-info-refactor.md`) — those are being migration (see `remove-sensetive-info-refactor.md`) — those are being
scrubbed and rotated separately; don't treat the repo as safe to make public scrubbed and rotated separately; don't treat the repo as safe to make public
+5 -2
View File
@@ -1,6 +1,9 @@
# pxe-boot # pxe-boot
The `pxe-boot` host serves HTTP boot assets for iPXE clients. The `pxe-boot` host serves HTTP boot assets for iPXE clients — including a
self-staged copy of this flake's own auto-installer netboot image, see
`docs/auto-installer.md` for what that image actually is and does once
booted.
## Host Role ## Host Role
@@ -83,7 +86,7 @@ uses `archiso_http_srv` to fetch the squashfs payload over HTTP.
Safe evaluation check: Safe evaluation check:
```bash ```bash
nix eval .#nixosConfigurations.pxe-boot.config.system.build.toplevel.drvPath --raw nix eval .#nixosConfigurations.proxmox-pxe-boot.config.system.build.toplevel.drvPath --raw
``` ```
After deployment by an operator, basic service checks are: After deployment by an operator, basic service checks are:
-30
View File
@@ -1,30 +0,0 @@
#create MBR table
parted /dev/sda -- mklabel msdos
#create nixos partition
parted /dev/sda -- mkpart primary 1MB -8GB
#set nixos partition to bootable
parted /dev/sda -- set 1 boot on
# create swap partition
parted /dev/sda -- mkpart primary linux-swap -8GB 100%
#format OS partition
mkfs.ext4 -L nixos /dev/sda1
#format swap
mkswap -L swap /dev/sda2
#activate swap
swapon /dev/sda2
#mount nixos partition
mount /dev/disk/by-label/nixos /mnt
export TMPDIR=/mnt/install-tmp
mkdir -p /mnt/install-tmp
#Generate config
#nixos-generate-config --root /mnt/
#copy customised configuration over
#cp configuration.nix /mnt/etc/nixos/configuration.nix
#nixos-install --no-root-passwd
#reboot
+46
View File
@@ -55,6 +55,33 @@ echo "Evaluating host toplevel derivations..."
for host in $hosts; do for host in $hosts; do
echo "==> $host" echo "==> $host"
nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath" nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
# lxc-* hosts deploy via a directly pct-restore-able tarball instead of
# nixos-install (see docs/auto-installer.md); proxmox-* hosts (excluding
# proxmox-lxc, the installer's own LXC target, which has no disko config)
# can alternatively be built as a standalone disk image (see
# docs/proxmox-images.md). Both are otherwise-unvalidated buildable
# surface, easy to silently break without this.
case "$host" in
lxc-*)
echo "==> $host (tarball)"
nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.tarball.drvPath"
;;
proxmox-lxc) ;;
proxmox-*)
echo "==> $host (diskoImagesScript)"
nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript.drvPath"
;;
esac
done
echo
echo "Evaluating buildable packages..."
packages_json="$(nix eval --json --no-use-registries --no-accept-flake-config .#packages.x86_64-linux --apply builtins.attrNames)"
packages="$(echo "$packages_json" | jq -r '.[]')"
for pkg in $packages; do
echo "==> packages.x86_64-linux.${pkg}"
nix eval --raw --no-use-registries --no-accept-flake-config ".#packages.x86_64-linux.${pkg}"
done done
if [[ "$MODE" == "dry-run" ]]; then if [[ "$MODE" == "dry-run" ]]; then
@@ -63,6 +90,25 @@ if [[ "$MODE" == "dry-run" ]]; then
for host in $hosts; do for host in $hosts; do
echo "==> Dry-run build: $host" echo "==> Dry-run build: $host"
nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.toplevel" nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.toplevel"
case "$host" in
lxc-*)
echo "==> Dry-run build: $host (tarball)"
nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.tarball"
;;
proxmox-lxc) ;;
proxmox-*)
echo "==> Dry-run build: $host (diskoImagesScript)"
nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript"
;;
esac
done
echo
echo "Running dry-run builds for all packages."
for pkg in $packages; do
echo "==> Dry-run build: packages.x86_64-linux.${pkg}"
nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#packages.x86_64-linux.${pkg}"
done done
fi fi
-9
View File
@@ -1,9 +0,0 @@
#!/usr/bin/env bash
set -euo pipefail
#boot to rescue mode
# set root password
scp $RESULT_ISO root@$LINODE_IP:/tmp/nixos-auto.iso
#in LISH or ssh to rescue mode
dd if=/tmp/nixos.iso of=/dev/sda bs=4M status=progress conv=fsync
+4 -5
View File
@@ -20,12 +20,11 @@ set -euo pipefail
repo_root="$(cd "$(dirname "$0")/.." && pwd)" repo_root="$(cd "$(dirname "$0")/.." && pwd)"
hostname="${1:?usage: scripts/prepare-host-key.sh <hostname> [path-to-nixos-repo]}" hostname="${1:?usage: scripts/prepare-host-key.sh <hostname>}"
nixos_repo="${2:-$repo_root}" sops_yaml="${repo_root}/.sops.yaml"
sops_yaml="${nixos_repo}/.sops.yaml"
if [[ ! -f "$sops_yaml" ]]; then if [[ ! -f "$sops_yaml" ]]; then
echo "ERROR: $sops_yaml not found. Pass the nixos repo path as the 2nd argument." >&2 echo "ERROR: $sops_yaml not found — is this script still under nixos/scripts/?" >&2
exit 1 exit 1
fi fi
@@ -55,7 +54,7 @@ Generated: ${keyfile}(.pub)
nix-cache/server.) nix-cache/server.)
=== 3. Re-encrypt every secrets file you just added it to === === 3. Re-encrypt every secrets file you just added it to ===
nix-shell -p sops --run 'sops updatekeys ${nixos_repo}/secrets/common.yaml' nix-shell -p sops --run 'sops updatekeys ${repo_root}/secrets/common.yaml'
=== 4. Commit + push this repo so the flake build picks up the new recipient === === 4. Commit + push this repo so the flake build picks up the new recipient ===