Run per-host/per-package nix eval and dry-run build concurrently

codex-maintenance.sh evaluated each affected host/package one at a time,
even though those calls are independent. Added scripts/lib/nix-parallel.sh
(run_nix_parallel) and wired it into the host-eval, package-eval, and
dry-run-build loops.

Concurrency defaults to core count capped by available memory (~1GB/job)
rather than plain nproc: empirically, nproc-many concurrent full-flake
evals OOM-killed each other on a 4GB/6-core box, while 3-4 ran clean and
were still ~2x faster than serial. Override via NIX_PARALLEL_JOBS.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CLAUDE.md

@@ -164,39 +164,13 @@ before committing.
 
 Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` is
 organized by purpose: `scripts/secrets/` (sops/age + SSH host-key
-management), `scripts/proxmox/` (Proxmox deployment), `scripts/installer/`
-(the auto-installer's own shell script, templated into the image — see
-below), `scripts/lib/` (shared helpers, sourced by the scripts below — not
-run directly), and a handful of repo-wide scripts left at the top level
-(`env.sh`, `bump-nixpkgs-release.sh`, plus `codex-setup.sh`/
-`codex-maintenance.sh` above). When adding a new script, put it in the
-matching subfolder rather than the top level, and if it duplicates logic
-another script already has, lift the shared part into `scripts/lib/`
-instead of copying it.
-
-### `scripts/installer/`
-
-- `scripts/installer/auto-install.sh` — the interactive install script
-  baked into the auto-installer image (see `docs/auto-installer.md`), kept
-  as a real, version-controlled shell file rather than inline in
-  `modules/installer/common.nix`'s Nix. It sources `scripts/env.sh` itself
-  for `LAN_DOMAIN` (`export LAN_DOMAIN`/`: "${LAN_DOMAIN:=...}"`, matching
-  `variables.nix`'s `lanDomain` — manually kept in sync, same pattern as
-  `NIX_CACHE_HOST` mirroring `nixCacheHost`), rather than Nix-level string
-  substitution — that's what makes it work identically whether run
-  straight from a git checkout or from inside the built installer image.
-  `common.nix` bakes `scripts/env.sh` in alongside it at a matching
-  relative path (`/etc/nixos-installer/env.sh` next to
-  `/etc/nixos-installer/installer/auto-install.sh`) so the script's own
-  `source "$(dirname ...)/../env.sh"` line resolves the same way in both
-  contexts — this is also why it's invoked from
-  `/etc/nixos-installer/installer/auto-install.sh` rather than a flat
-  `/etc/auto-install.sh`. `#!/usr/bin/env bash`, not
-  `#!/run/current-system/sw/bin/bash`: the latter only resolves on an
-  already-activated NixOS system, breaking the checked-out-file case
-  entirely (confirmed live: "cannot execute: required file not found" on
-  a non-NixOS box); `/usr/bin/env` is reliably present on both NixOS
-  (`environment.usrbinenv`'s own default) and any normal Linux distro.
+management), `scripts/proxmox/` (Proxmox deployment), `scripts/lib/`
+(shared helpers, sourced by the scripts below — not run directly), and a
+handful of repo-wide scripts left at the top level (`env.sh`,
+`bump-nixpkgs-release.sh`, plus `codex-setup.sh`/`codex-maintenance.sh`
+above). When adding a new script, put it in the matching subfolder rather
+than the top level, and if it duplicates logic another script already has,
+lift the shared part into `scripts/lib/` instead of copying it.
 
 ### `scripts/secrets/`
 
@@ -308,9 +282,8 @@ Sourced by the scripts above, never run directly:
 ### Top level
 
 - `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
-  default cores/memory, `NIX_CACHE_HOST`, `LAN_DOMAIN`) sourced by
-  `create-proxmox-resource.sh` and `scripts/installer/auto-install.sh`. Add
-  new cross-script config here instead of duplicating it per-script.
+  default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
+  cross-script config here instead of duplicating it per-script.
 - `scripts/bump-nixpkgs-release.sh` — bumps `flake.nix`'s `nixpkgs.url`/
   `home-manager.url` in place. Exists because flake input URLs can't
   reference `variables.nix` (confirmed empirically — `nix flake metadata`

docs/auto-installer.md

@@ -8,13 +8,10 @@ lives here.
 The installer provides a small NixOS install environment (ISO, or the same
 image netbooted via PXE) with SSH access, Git support, and an interactive
 installation script.
-Logging in as any user (root or `nixos`) runs
-`/etc/nixos-installer/installer/auto-install.sh` (the same file as
-`scripts/installer/auto-install.sh` in this repo — see "Installer process"
-below for why it's baked in at that path rather than a flat
-`/etc/auto-install.sh`), discovers available hosts from this same flake,
-lets the operator choose a target, applies that host's Disko storage
-configuration, installs NixOS, and reboots.
+Logging in as any user (root or `nixos`) runs `/etc/auto-install.sh`,
+discovers available hosts from this same flake, lets the operator choose a
+target, applies that host's Disko storage configuration, installs NixOS, and
+reboots.
 
 **This applies to every `nixosConfigurations` target except `lxc-*` hosts —
 see "LXC hosts" immediately below for why those are different.**
@@ -195,10 +192,6 @@ default.
 `auto-install.sh` still supports the older manual path as a fallback: if a
 host's key isn't baked in (`/etc/host-keys`), it checks `/root/host-keys`
 next, where you can `scp` a key in after boot, same as before this migration.
-If neither has it and the script is running interactively (an actual
-operator at the other end of stdin, not an unattended run), it prompts for
-an arbitrary directory to check (a mounted USB stick, another filesystem,
-etc.) and copies the key pair into `/root/host-keys` from there if found.
 
 ## Storage
 
@@ -224,21 +217,7 @@ entirely (see "LXC hosts" above), so it never reaches this code path.
 
 ## Installer process
 
-`scripts/installer/auto-install.sh` is a real, version-controlled shell
-script — not an inline Nix string. It sources `scripts/env.sh` for
-`LAN_DOMAIN` itself (same as every other script in `scripts/`), so it
-behaves identically whether it's run straight from a git checkout (e.g.
-manually, from a stock NixOS ISO that isn't this repo's own installer
-image) or from inside the built installer image. That's also why it's
-baked in at `/etc/nixos-installer/installer/auto-install.sh` rather than a
-flat `/etc/auto-install.sh` — `modules/installer/common.nix` bakes
-`scripts/env.sh` in alongside it at `/etc/nixos-installer/env.sh`,
-preserving the same relative layout (`installer/auto-install.sh` ->
-`../env.sh`) the checked-out repo has, so the script's own
-`source ".../env.sh"` line resolves correctly in both places without any
-Nix-level templating.
-
-Once running, it:
+`/etc/auto-install.sh`:
 
 1. Queries `nixosConfigurations` from this flake over the network (`git+https://<lanDomain>/beatzaplenty/nixos.git`) — this happens at *install* time, not build time, so a generic installer image always sees whatever hosts are currently committed, without needing a rebuild.
 2. Presents them as a menu; confirms the choice.

modules/installer/common.nix

@@ -45,21 +45,140 @@
       disko
     ];
 
-    # Auto-install script, kept as a real, version-controlled shell file at
-    # scripts/installer/auto-install.sh rather than an inline Nix string.
-    # It sources scripts/env.sh itself (for LAN_DOMAIN, same as every other
-    # script in this repo) rather than relying on Nix-level templating, so
-    # it behaves identically whether it's run straight from a git checkout
-    # or from here -- baking scripts/env.sh in alongside it at a matching
-    # relative path (installer/auto-install.sh -> ../env.sh) is what makes
-    # that resolve correctly in both places.
-    etc = {
-      "nixos-installer/env.sh".source = ../../scripts/env.sh;
+    # Write auto-install script to /root
+    etc."auto-install.sh" = {
+      text = ''
+            #!/run/current-system/sw/bin/bash
+            set -eux
 
-      "nixos-installer/installer/auto-install.sh" = {
-        source = ../../scripts/installer/auto-install.sh;
-        mode = "0755";
-      };
+            set -euo pipefail
+
+        export FLAKE_BASE_URL="git+https://${vars.lanDomain}/beatzaplenty/nixos.git"
+
+        echo "Fetching available NixOS hosts from flake..."
+        # Two categories deliberately excluded from the menu:
+        #   lxc-*     — these build a config.system.build.tarball meant for
+        #               `pct restore` on Proxmox directly, not an install.
+        #               Running nixos-install against one here would
+        #               bind-mount / onto /mnt and then refuse to touch the
+        #               filesystem it's currently running on — see
+        #               docs/auto-installer.md.
+        #   installer — this *is* the installer image's own flake target,
+        #               not a deployable host; "installing" it means
+        #               nixos-install-ing a copy of the installer into
+        #               itself.
+        mapfile -t options < <(
+          nix eval --json --no-use-registries --no-accept-flake-config --extra-experimental-features "flakes nix-command" \
+            "''${FLAKE_BASE_URL}#nixosConfigurations" \
+            --apply builtins.attrNames \
+          | jq -r '.[]
+              | select(startswith("lxc-") | not)
+              | select(. != "installer")'
+        )
+
+        if [[ ''${#options[@]} -eq 0 ]]; then
+          echo "ERROR: No NixOS hosts found in ''${FLAKE_BASE_URL}#nixosConfigurations" >&2
+          exit 1
+        fi
+
+        echo "Note: lxc-* targets aren't installed this way — build them with"
+        echo "  nix build .#nixosConfigurations.<name>.config.system.build.tarball"
+        echo "and 'pct restore' the result on Proxmox directly. See docs/auto-installer.md."
+
+        echo "Choose the flake profile to install:"
+        select choice in "''${options[@]}"; do
+          if [[ -n "$choice" ]]; then
+            echo "You selected: $choice"
+            break
+          else
+            echo "Invalid selection. Try again."
+          fi
+        done
+
+        echo "Starting install with flake: ''${FLAKE_BASE_URL}#''${choice}"
+
+        # Optional: confirm before proceeding
+        read -rp "Proceed with installation? (y/N): " confirm
+        if [[ ! "$confirm" =~ ^[Yy]$ ]]; then
+          echo "Aborted."
+          exit 1
+        fi
+
+        # A nix-cache host is *the* substituter/remote-builder for every other
+        # host once installed (its own config explicitly excludes itself from
+        # using either — see buildType != "nix-cache" in the nixos flake.nix).
+        # Installing one shouldn't depend on a nix-cache substituter either,
+        # for the same reason — plus in practice "nix-cache" only resolves over
+        # Tailscale, which a fresh installer environment was never connected to
+        # anyway, so it's dead weight even for non-nix-cache installs until
+        # that's sorted out. Override it away here specifically for nix-cache
+        # targets to keep install-time behaviour consistent with run-time.
+        nix_extra_opts=()
+        if [[ "''${choice}" == *-nix-cache ]]; then
+            echo "Installing a nix-cache host — skipping the nix-cache substituter."
+            nix_extra_opts+=(--option substituters "https://cache.nixos.org/")
+        fi
+
+        # Every host reachable through this menu has a Disko config (lxc-*
+        # is filtered out above, and is the only category that doesn't —
+        # see docs/auto-installer.md), so this can run unconditionally: no
+        # need to probe the flake first and branch on whether Disko applies.
+        disko --mode destroy,format,mount \
+            --flake "''${FLAKE_BASE_URL}#''${choice}" "''${nix_extra_opts[@]}" --yes-wipe-all-disks
+
+        # sops-nix derives this host's decryption key from its own SSH host key
+        # at *activation* time, which runs before systemd would otherwise
+        # generate one on first boot. Without pre-seeding it here, secrets
+        # (including the login password) fail to decrypt on first boot.
+        # Generate the key with scripts/secrets/prepare-host-key.sh first.
+        #
+        # Two places a key can come from, checked in order:
+        #   /etc/host-keys  — baked into this image at build time (see
+        #                      modules/installer/host-keys.nix; only present
+        #                      if built with NIXOS_HOST_KEYS_DIR set)
+        #   /root/host-keys — scp'd in manually after boot (older fallback,
+        #                      still supported for images built without keys)
+        mkdir -p /root/host-keys
+        if [[ -f "/etc/host-keys/''${choice}_ssh_host_ed25519_key" ]]; then
+            echo "Found baked-in SSH host key for ''${choice}, installing to target..."
+            install -D -m 0600 "/etc/host-keys/''${choice}_ssh_host_ed25519_key" /mnt/etc/ssh/ssh_host_ed25519_key
+            install -D -m 0644 "/etc/host-keys/''${choice}_ssh_host_ed25519_key.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub
+        elif [[ -f "/root/host-keys/''${choice}_ssh_host_ed25519_key" ]]; then
+            echo "Found pre-seeded SSH host key for ''${choice}, installing to target..."
+            install -D -m 0600 "/root/host-keys/''${choice}_ssh_host_ed25519_key" /mnt/etc/ssh/ssh_host_ed25519_key
+            install -D -m 0644 "/root/host-keys/''${choice}_ssh_host_ed25519_key.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub
+        else
+            echo "WARNING: no SSH host key found for ''${choice} (checked /etc/host-keys and /root/host-keys)"
+            echo "sops-nix secrets (including the login password) will NOT decrypt on first boot."
+            echo "Run scripts/secrets/prepare-host-key.sh for host ''${choice} on your admin workstation first,"
+            echo "then either rebuild this image with NIXOS_HOST_KEYS_DIR set, or scp the result to"
+            echo "/root/host-keys/ on this machine."
+            read -rp "Continue without a pre-seeded key anyway? (y/N): " skip_key
+            if [[ ! "$skip_key" =~ ^[Yy]$ ]]; then
+                echo "Aborted."
+                exit 1
+            fi
+        fi
+
+        mkdir -p /mnt/install-tmp
+        export TMPDIR=/mnt/install-tmp
+
+        nixos-install \
+            --flake "''${FLAKE_BASE_URL}#''${choice}" \
+            "''${nix_extra_opts[@]}" \
+            --no-root-password
+
+
+            rm -rf /mnt/install-tmp
+            # Redundant copy of the host's private key — the real one is now at
+            # /etc/ssh/ssh_host_ed25519_key. Nothing NixOS-managed ever cleans this
+            # up on its own since it was written imperatively, not declaratively.
+            rm -rf /root/host-keys
+            sleep 10
+            reboot
+      '';
+
+      mode = "0755";
     };
   };
 
@@ -73,7 +192,7 @@
   # file-copying/chown.
   programs.bash.loginShellInit = ''
     if [ -n "$PS1" ] && [ ! -e "$HOME/.auto_install_ran" ]; then
-      sudo /etc/nixos-installer/installer/auto-install.sh
+      sudo /etc/auto-install.sh
       touch "$HOME/.auto_install_ran"
     fi
   '';

scripts/codex-maintenance.sh

@@ -16,6 +16,14 @@
 #
 # --dry-run: adds `nix build --dry-run --no-link` for whatever scope is
 # active (changed-files scope by default, full scope under --full-check).
+#
+# Per-host/per-package eval and dry-run build calls run concurrently (see
+# scripts/lib/nix-parallel.sh) since they're independent of each other.
+# Concurrency defaults to core count capped by available memory (~1GB/job)
+# rather than plain core count, since each concurrent `nix eval` evaluates a
+# whole NixOS system closure and can OOM a small/memory-constrained CI
+# runner otherwise; override via NIX_PARALLEL_JOBS if a runner has more (or
+# less) room than that estimate assumes.
 set -euo pipefail
 
 script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
@@ -23,6 +31,8 @@ script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
 source "${script_dir}/lib/nix-bootstrap.sh"
 # shellcheck source=lib/nix-eval.sh
 source "${script_dir}/lib/nix-eval.sh"
+# shellcheck source=lib/nix-parallel.sh
+source "${script_dir}/lib/nix-parallel.sh"
 
 repo_root="$(cd "${script_dir}/.." && pwd)"
 cd "$repo_root"
@@ -246,66 +256,64 @@ echo
 if [[ ${#hosts[@]} -eq 0 ]]; then
   echo "No hosts affected by changed files; skipping host eval."
 else
-  echo "Evaluating host toplevel derivations (${scope_desc})..."
+  echo "Evaluating host toplevel derivations (${scope_desc}, up to ${NIX_PARALLEL_JOBS} at a time)..."
+  # lxc-* hosts deploy via a directly pct-restore-able tarball instead of
+  # nixos-install (see docs/auto-installer.md); proxmox-* hosts can
+  # alternatively be built as a standalone disk image (see
+  # docs/proxmox-images.md). Both are otherwise-unvalidated buildable
+  # surface, easy to silently break without this.
+  declare -a host_eval_jobs=()
   for host in "${hosts[@]}"; do
-    echo "==> $host"
-    nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
-
-    # lxc-* hosts deploy via a directly pct-restore-able tarball instead of
-    # nixos-install (see docs/auto-installer.md); proxmox-* hosts can
-    # alternatively be built as a standalone disk image (see
-    # docs/proxmox-images.md). Both are otherwise-unvalidated buildable
-    # surface, easy to silently break without this.
+    host_eval_jobs+=("${host}${NIX_PARALLEL_SEP}.#nixosConfigurations.${host}.config.system.build.toplevel.drvPath")
     case "$host" in
       lxc-*)
-        echo "==> $host (tarball)"
-        nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.tarball.drvPath"
+        host_eval_jobs+=("${host} (tarball)${NIX_PARALLEL_SEP}.#nixosConfigurations.${host}.config.system.build.tarball.drvPath")
         ;;
       proxmox-*)
-        echo "==> $host (diskoImagesScript)"
-        nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript.drvPath"
+        host_eval_jobs+=("${host} (diskoImagesScript)${NIX_PARALLEL_SEP}.#nixosConfigurations.${host}.config.system.build.diskoImagesScript.drvPath")
         ;;
     esac
   done
+  run_nix_parallel host_eval_jobs eval --raw "${NIX_EVAL_FLAGS[@]}"
 fi
 
 echo
 if ! $eval_packages; then
   echo "No packages affected by changed files; skipping package eval."
 else
-  echo "Evaluating buildable packages..."
+  echo "Evaluating buildable packages (up to ${NIX_PARALLEL_JOBS} at a time)..."
+  declare -a package_eval_jobs=()
   for pkg in "${all_packages[@]}"; do
-    echo "==> packages.x86_64-linux.${pkg}"
-    nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
+    package_eval_jobs+=("packages.x86_64-linux.${pkg}${NIX_PARALLEL_SEP}.#packages.x86_64-linux.${pkg}")
   done
+  run_nix_parallel package_eval_jobs eval --raw "${NIX_EVAL_FLAGS[@]}"
 fi
 
 if $dry_run; then
   echo
-  echo "Running dry-run builds for the active scope. This will not create result symlinks."
+  echo "Running dry-run builds for the active scope (up to ${NIX_PARALLEL_JOBS} at a time). This will not create result symlinks."
+  declare -a host_build_jobs=()
   for host in "${hosts[@]:-}"; do
-    echo "==> Dry-run build: $host"
-    nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel"
-
+    host_build_jobs+=("Dry-run build: ${host}${NIX_PARALLEL_SEP}.#nixosConfigurations.${host}.config.system.build.toplevel")
     case "$host" in
       lxc-*)
-        echo "==> Dry-run build: $host (tarball)"
-        nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.tarball"
+        host_build_jobs+=("Dry-run build: ${host} (tarball)${NIX_PARALLEL_SEP}.#nixosConfigurations.${host}.config.system.build.tarball")
         ;;
       proxmox-*)
-        echo "==> Dry-run build: $host (diskoImagesScript)"
-        nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript"
+        host_build_jobs+=("Dry-run build: ${host} (diskoImagesScript)${NIX_PARALLEL_SEP}.#nixosConfigurations.${host}.config.system.build.diskoImagesScript")
         ;;
     esac
   done
+  run_nix_parallel host_build_jobs build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}"
 
   if $eval_packages; then
     echo
     echo "Running dry-run builds for packages."
+    declare -a package_build_jobs=()
     for pkg in "${all_packages[@]}"; do
-      echo "==> Dry-run build: packages.x86_64-linux.${pkg}"
-      nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
+      package_build_jobs+=("Dry-run build: packages.x86_64-linux.${pkg}${NIX_PARALLEL_SEP}.#packages.x86_64-linux.${pkg}")
     done
+    run_nix_parallel package_build_jobs build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}"
   fi
 fi
 

scripts/env.sh

@@ -82,12 +82,6 @@ export PVE1_HOST PVE_TEST_HOST PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE \
 : "${NIX_CACHE_HOST:=nix-cache}"
 export NIX_CACHE_HOST
 
-# Matches variables.nix's lanDomain (the Gitea host this flake's own repo
-# is served from -- see scripts/installer/auto-install.sh's FLAKE_BASE_URL)
-# -- update both if it ever changes.
-: "${LAN_DOMAIN:=gitea.lan.ddnsgeek.com}"
-export LAN_DOMAIN
-
 # nix_extra_opts: call as a plain statement (NOT inside $(...)/<(...) --
 # that forks a subshell, and the whole point is exporting a decision back
 # into *this* shell) to populate the global NIX_OPTS array with whatever

scripts/installer/auto-install.sh

@@ -1,166 +0,0 @@
-#!/usr/bin/env nix-shell
-#!nix-shell -i bash -p jq disko nixos-install-tools
-# shellcheck shell=bash
-# The only genuinely external tools this script calls directly: `jq`
-# (parsing the `nix eval` host list) and `disko`/`nixos-install` (the
-# install itself). Everything disko shells out to internally
-# (parted/sgdisk/mkfs.*/zfs/...) is self-contained -- disko's own
-# generated scripts hardcode absolute Nix store paths for those, they
-# don't rely on this script's PATH at all (confirmed by inspecting a
-# generated system.build.formatScript). The built installer image
-# (modules/installer/common.nix) already has all three in
-# environment.systemPackages, so this nix-shell wrapper is a fast no-op
-# there; it's what makes the script also work standalone (e.g. run
-# directly from a checkout on a stock ISO), where they aren't.
-set -eux
-
-set -euo pipefail
-
-# shellcheck source=../env.sh
-source "$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/env.sh"
-
-export FLAKE_BASE_URL="git+https://${LAN_DOMAIN}/beatzaplenty/nixos.git"
-
-echo "Fetching available NixOS hosts from flake..."
-# Two categories deliberately excluded from the menu:
-#   lxc-*     — these build a config.system.build.tarball meant for
-#               `pct restore` on Proxmox directly, not an install.
-#               Running nixos-install against one here would
-#               bind-mount / onto /mnt and then refuse to touch the
-#               filesystem it's currently running on — see
-#               docs/auto-installer.md.
-#   installer — this *is* the installer image's own flake target,
-#               not a deployable host; "installing" it means
-#               nixos-install-ing a copy of the installer into
-#               itself.
-mapfile -t options < <(
-  nix eval --json --no-use-registries --no-accept-flake-config --extra-experimental-features "flakes nix-command" \
-    "${FLAKE_BASE_URL}#nixosConfigurations" \
-    --apply builtins.attrNames \
-  | jq -r '.[]
-      | select(startswith("lxc-") | not)
-      | select(. != "installer")'
-)
-
-if [[ ${#options[@]} -eq 0 ]]; then
-  echo "ERROR: No NixOS hosts found in ${FLAKE_BASE_URL}#nixosConfigurations" >&2
-  exit 1
-fi
-
-echo "Note: lxc-* targets aren't installed this way — build them with"
-echo "  nix build .#nixosConfigurations.<name>.config.system.build.tarball"
-echo "and 'pct restore' the result on Proxmox directly. See docs/auto-installer.md."
-
-echo "Choose the flake profile to install:"
-select choice in "${options[@]}"; do
-  if [[ -n "$choice" ]]; then
-    echo "You selected: $choice"
-    break
-  else
-    echo "Invalid selection. Try again."
-  fi
-done
-
-echo "Starting install with flake: ${FLAKE_BASE_URL}#${choice}"
-
-# Optional: confirm before proceeding
-read -rp "Proceed with installation? (y/N): " confirm
-if [[ ! "$confirm" =~ ^[Yy]$ ]]; then
-  echo "Aborted."
-  exit 1
-fi
-
-# A nix-cache host is *the* substituter/remote-builder for every other
-# host once installed (its own config explicitly excludes itself from
-# using either — see buildType != "nix-cache" in the nixos flake.nix).
-# Installing one shouldn't depend on a nix-cache substituter either,
-# for the same reason — plus in practice "nix-cache" only resolves over
-# Tailscale, which a fresh installer environment was never connected to
-# anyway, so it's dead weight even for non-nix-cache installs until
-# that's sorted out. Override it away here specifically for nix-cache
-# targets to keep install-time behaviour consistent with run-time.
-nix_extra_opts=()
-if [[ "${choice}" == *-nix-cache ]]; then
-    echo "Installing a nix-cache host — skipping the nix-cache substituter."
-    nix_extra_opts+=(--option substituters "https://cache.nixos.org/")
-fi
-
-# Every host reachable through this menu has a Disko config (lxc-*
-# is filtered out above, and is the only category that doesn't —
-# see docs/auto-installer.md), so this can run unconditionally: no
-# need to probe the flake first and branch on whether Disko applies.
-disko --mode destroy,format,mount \
-    --flake "${FLAKE_BASE_URL}#${choice}" "${nix_extra_opts[@]}" --yes-wipe-all-disks
-
-# sops-nix derives this host's decryption key from its own SSH host key
-# at *activation* time, which runs before systemd would otherwise
-# generate one on first boot. Without pre-seeding it here, secrets
-# (including the login password) fail to decrypt on first boot.
-# Generate the key with scripts/secrets/prepare-host-key.sh first.
-#
-# Two places a key can come from, checked in order:
-#   /etc/host-keys  — baked into this image at build time (see
-#                      modules/installer/host-keys.nix; only present
-#                      if built with NIXOS_HOST_KEYS_DIR set)
-#   /root/host-keys — scp'd in manually after boot (older fallback,
-#                      still supported for images built without keys)
-mkdir -p /root/host-keys
-if [[ -f "/etc/host-keys/${choice}_ssh_host_ed25519_key" ]]; then
-    echo "Found baked-in SSH host key for ${choice}, installing to target..."
-    install -D -m 0600 "/etc/host-keys/${choice}_ssh_host_ed25519_key" /mnt/etc/ssh/ssh_host_ed25519_key
-    install -D -m 0644 "/etc/host-keys/${choice}_ssh_host_ed25519_key.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub
-elif [[ -f "/root/host-keys/${choice}_ssh_host_ed25519_key" ]]; then
-    echo "Found pre-seeded SSH host key for ${choice}, installing to target..."
-    install -D -m 0600 "/root/host-keys/${choice}_ssh_host_ed25519_key" /mnt/etc/ssh/ssh_host_ed25519_key
-    install -D -m 0644 "/root/host-keys/${choice}_ssh_host_ed25519_key.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub
-else
-    # Third place a key can come from: an arbitrary path the operator
-    # points at interactively (e.g. a USB stick, a mount from another
-    # machine) -- only offered when there's an actual human at the other
-    # end of stdin to ask, never in a non-interactive run.
-    key_copied=0
-    if [[ -t 0 ]]; then
-        echo "No SSH host key found for ${choice} (checked /etc/host-keys and /root/host-keys)."
-        read -rp "Path to a directory containing ${choice}_ssh_host_ed25519_key(.pub) (blank to skip): " key_src_dir
-        if [[ -n "$key_src_dir" && -f "${key_src_dir}/${choice}_ssh_host_ed25519_key" && -f "${key_src_dir}/${choice}_ssh_host_ed25519_key.pub" ]]; then
-            cp "${key_src_dir}/${choice}_ssh_host_ed25519_key" "${key_src_dir}/${choice}_ssh_host_ed25519_key.pub" /root/host-keys/
-            key_copied=1
-        elif [[ -n "$key_src_dir" ]]; then
-            echo "WARNING: ${choice}_ssh_host_ed25519_key(.pub) not found in ${key_src_dir}."
-        fi
-    fi
-
-    if [[ "$key_copied" -eq 1 ]]; then
-        echo "Copied SSH host key for ${choice} from ${key_src_dir}, installing to target..."
-        install -D -m 0600 "/root/host-keys/${choice}_ssh_host_ed25519_key" /mnt/etc/ssh/ssh_host_ed25519_key
-        install -D -m 0644 "/root/host-keys/${choice}_ssh_host_ed25519_key.pub" /mnt/etc/ssh/ssh_host_ed25519_key.pub
-    else
-        echo "WARNING: no SSH host key found for ${choice} (checked /etc/host-keys and /root/host-keys)"
-        echo "sops-nix secrets (including the login password) will NOT decrypt on first boot."
-        echo "Run scripts/secrets/prepare-host-key.sh for host ${choice} on your admin workstation first,"
-        echo "then either rebuild this image with NIXOS_HOST_KEYS_DIR set, scp the result to"
-        echo "/root/host-keys/ on this machine, or point at it when prompted above."
-        read -rp "Continue without a pre-seeded key anyway? (y/N): " skip_key
-        if [[ ! "$skip_key" =~ ^[Yy]$ ]]; then
-            echo "Aborted."
-            exit 1
-        fi
-    fi
-fi
-
-mkdir -p /mnt/install-tmp
-export TMPDIR=/mnt/install-tmp
-
-nixos-install \
-    --flake "${FLAKE_BASE_URL}#${choice}" \
-    "${nix_extra_opts[@]}" \
-    --no-root-password
-
-
-    rm -rf /mnt/install-tmp
-    # Redundant copy of the host's private key — the real one is now at
-    # /etc/ssh/ssh_host_ed25519_key. Nothing NixOS-managed ever cleans this
-    # up on its own since it was written imperatively, not declaratively.
-    rm -rf /root/host-keys
-    sleep 10
-    reboot

scripts/lib/nix-parallel.sh

@@ -0,0 +1,95 @@
+#!/usr/bin/env bash
+# Shared parallel-nix-invocation helper for scripts/codex-maintenance.sh.
+# Source alongside nix-eval.sh:
+#   source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/nix-parallel.sh"
+#
+# The per-host/per-package `nix eval`/`nix build --dry-run` calls in
+# codex-maintenance.sh are independent of each other, so running them one at
+# a time leaves most cores idle for most of the sweep -- run_nix_parallel
+# fans a batch of them out across up to NIX_PARALLEL_JOBS processes instead.
+
+# NIX_PARALLEL_JOBS: how many `nix` invocations run_nix_parallel runs at
+# once. Defaults to core count capped by available memory (~1GB/job,
+# floor 1) rather than plain `nproc` -- each concurrent `nix eval` here
+# evaluates a whole NixOS system closure from scratch, and on a small/
+# memory-constrained CI runner, `nproc` concurrent evals can OOM-kill each
+# other (confirmed empirically: on a 4GB/6-core box, 5-6 concurrent evals
+# started getting killed while 3-4 ran clean and were still ~2x faster than
+# serial). Override via env if a given machine/CI runner has room to spare
+# or needs a tighter cap.
+default_nix_parallel_jobs() {
+  local cores mem_avail_kb mem_cap
+  cores="$(nproc 2>/dev/null || echo 4)"
+  mem_avail_kb="$(awk '/^MemAvailable:/ {print $2}' /proc/meminfo 2>/dev/null)"
+  if [[ -z "$mem_avail_kb" ]]; then
+    echo "$cores"
+    return
+  fi
+  mem_cap=$((mem_avail_kb / 1024 / 1024))
+  ((mem_cap < 1)) && mem_cap=1
+  ((mem_cap < cores)) && echo "$mem_cap" || echo "$cores"
+}
+NIX_PARALLEL_JOBS="${NIX_PARALLEL_JOBS:-$(default_nix_parallel_jobs)}"
+
+# Separator between a job's label and its flake attr in the arrays
+# run_nix_parallel takes -- a control character so it can't collide with
+# anything a label or attr path would plausibly contain.
+NIX_PARALLEL_SEP=$'\x1f'
+
+# run_nix_parallel <jobs_array_name> <nix subcommand + flags...>
+#
+# jobs_array_name: name of an already-populated bash array whose entries are
+# "<label>${NIX_PARALLEL_SEP}<attr>" pairs, e.g.
+#   jobs=("proxmox-docker${NIX_PARALLEL_SEP}.#nixosConfigurations.proxmox-docker...drvPath")
+# Remaining args are passed to `nix` before the attr, e.g.:
+#   run_nix_parallel jobs eval --raw "${NIX_EVAL_FLAGS[@]}"
+#   run_nix_parallel jobs build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}"
+#
+# Prints "==> <label>" followed by that job's stdout+stderr for every job,
+# in submission order (not completion order) so a run stays readable and
+# diffable across invocations even though the work itself doesn't finish in
+# that order. Returns non-zero if any job failed, only after every job has
+# finished and been printed -- same "surface everything, then fail" contract
+# a `set -e` caller gets, just parallelized instead of stopping at the first
+# failure.
+run_nix_parallel() {
+  local -n jobs_ref="$1"
+  shift
+  local -a nix_args=("$@")
+
+  local n=${#jobs_ref[@]}
+  [[ $n -eq 0 ]] && return 0
+
+  local tmp_dir
+  tmp_dir="$(mktemp -d)"
+
+  local i=0 running=0
+  for job in "${jobs_ref[@]}"; do
+    local attr="${job#*"${NIX_PARALLEL_SEP}"}"
+    printf '%s\n' "${job%%"${NIX_PARALLEL_SEP}"*}" >"${tmp_dir}/${i}.label"
+    (
+      if nix "${nix_args[@]}" "$attr" >"${tmp_dir}/${i}.out" 2>&1; then
+        echo 0 >"${tmp_dir}/${i}.status"
+      else
+        echo 1 >"${tmp_dir}/${i}.status"
+      fi
+    ) &
+    i=$((i + 1))
+    running=$((running + 1))
+    if ((running >= NIX_PARALLEL_JOBS)); then
+      wait -n
+      running=$((running - 1))
+    fi
+  done
+  wait
+
+  local failed=0 j
+  for ((j = 0; j < n; j++)); do
+    echo "==> $(cat "${tmp_dir}/${j}.label")"
+    cat "${tmp_dir}/${j}.out"
+    [[ "$(cat "${tmp_dir}/${j}.status")" -ne 0 ]] && failed=1
+  done
+
+  rm -rf "$tmp_dir"
+  return $failed
+}

variables.nix

@@ -45,20 +45,12 @@
   # the installer image's nixos/root users.
   adminSshKey = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCq/Q5LvIXlZwO2kdeAN5nLGZ59nZB7JHYMEszHxmNtGMzv1lM31jiPNsr0z2EKVZhE7OOfa2IF9rhWYD7JUA9G0yzdZ4WTXFNGVVOJoOVH6vAF3XCxoVilOEwTc7h2Wiy+rzd0B28/3spffzQQWJhY6GRQVa8j+6xAGF60Fcvl1vLosYT9Bn2ZbK4TCWOwAn2jqXIieGpZdn/UNZbGOeKRiCvhktDfMAzuQzN/9jMu/oF4pkPn2X1UrsQdNlvp0Ci8md612MozIpncQJyAF1ADhunr3sMx0isUXiqD29R5DS4TftpekqLNLak+zcxFa8N7DcRNp3DcKfJvyTkwQrR4r+b7lFLYOLHLagSso9CzeW/paAS2q9I5SBm/2DtE1diLLg2jZikYcstsu/G5RgvbzbKqjiaMwTdXC3AMvDxQrs7U5pDRZFzoofG3cpODbTm+uy3m0kP70z0M1K45UbDG0p+itnTu9x40JbQEgefbx38AItNvAIx1A8HO4I1VX28= wayne@stream";
 
-  # Prestaged wifi SSID for the gui host's NetworkManager profile
-  # (modules/networking/wifi.nix). The password is not here -- it's
-  # sops-encrypted in secrets/gui.yaml (wifi-password) instead, since this
-  # file isn't a secret store.
+  # Prestaged wifi credentials for the gui host's NetworkManager profile
+  # (modules/networking/wifi.nix). Leave blank until the bare-metal
+  # hardware profile is wired up — an empty ssid disables the profile
+  # rather than creating a broken empty one.
   wifiSsid = "nbn-fttp-net-5G";
 
-  # Bare-metal gui host's two disks for a ZFS RAID0 (striped) root pool
-  # (modules/disko/baremetal.nix). Only used transiently at disko-format
-  # time (partitioning); the resulting fileSystems/zpool import reference
-  # by-partlabel/by-id paths afterward regardless, same as
-  # modules/disko/proxmox.nix's own plain "/dev/sda".
-  guiRootDisk1 = "/dev/sda";
-  guiRootDisk2 = "/dev/sdb";
-
   # System
   timeZone = "Australia/Brisbane";