Archived
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
- variables.nix: switch to rec {}, extract giteaDomain/giteaRepoPath,
extraAdminSshKeys, haLanNfsFqdn, tailscaleResolverIp, ports.dhcp,
ports.dns; ipaServer now derives from homeDomain ref; section headers
- modules: use new vars throughout (pxe-boot, ts-dns-forwarder,
cluster-config, configuration.nix, mount-pxe-images) — eval unchanged
- docs: delete ephemeral planning docs (AUDIT_REPORT, ha-network-audit,
network-cutover); add docs/ha.md; drop migration reference table from
ip-addressing.md; remove stale server example from beszel.md
- CLAUDE.md/README.md/AGENTS.md: fix build types (tailscale-router,
ha-server, drop server); document scripts/ha/, scripts/ipa/, and
all previously undocumented top-level and lib scripts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
557 lines
32 KiB
Markdown
557 lines
32 KiB
Markdown
# CLAUDE.md
|
||
|
||
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||
|
||
## Repo purpose
|
||
|
||
Flake-based NixOS configuration for Wayne's LAN servers and workstation. There is
|
||
no application code here — changes are Nix module edits that affect real
|
||
machines when deployed.
|
||
|
||
## Safety rules (read before touching anything)
|
||
|
||
- **Never** run `nixos-rebuild switch|boot|test`, `nixos-install`, `parted`,
|
||
`mkfs`, `mkswap`, `swapon`, `mount`, or any other destructive disk/deploy
|
||
command from an agent session, even if asked indirectly. Deployment is done
|
||
manually by the operator on the target host.
|
||
- Validation is limited to evaluation, linting, formatting checks, and
|
||
`nix build --dry-run --no-link`.
|
||
- Do not add secrets, tokens, private keys, or new password hashes to the repo.
|
||
- This repo currently contains **committed password hashes** in
|
||
`modules/installer/common.nix` (the auto-installer's own root/nixos login —
|
||
a deliberate, documented choice, see `docs/auto-installer.md`, not
|
||
accidental tech debt) and **SSH public keys** in `variables.nix`
|
||
(`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`, `vars.beszelHubKey`). Don't use the installer's hardcoded hash as a
|
||
template for a *real* host — every other host uses sops-nix
|
||
(`hashedPasswordFile`, see "Security Notes" in `README.md`). Flag any *new*
|
||
secret-like string you encounter instead of committing it.
|
||
- `host-keys/` is gitignored — used only by the auto-installer's own
|
||
environment for pre-seeding non-LXC host keys before first boot (see
|
||
`docs/auto-installer.md`). Never commit its contents; if `git status`
|
||
ever shows it as trackable, something is wrong. All deployed hosts use
|
||
clan vars (`vars/per-machine/<target>/openssh/`, committed and
|
||
sops-encrypted) for their SSH host keys — those ARE tracked by git and
|
||
belong in the repo.
|
||
|
||
### Two Proxmox nodes: `pve1.sweet.home` (production) and `pve-test.sweet.home` (sandbox)
|
||
|
||
There are two SSH-reachable Proxmox nodes on the LAN, both defined in
|
||
`scripts/env.sh` (`PVE1_HOST` / `PVE_TEST_HOST`), individually targetable
|
||
via `scripts/proxmox/create-proxmox-resource.sh --node <host>` or by
|
||
overriding `PROXMOX_HOST`. `PROXMOX_HOST` itself still defaults to
|
||
`PVE1_HOST` (production) — that default, and every other script behavior,
|
||
is unchanged from before `pve-test` existed; the only thing new is that
|
||
`pve-test` can now be reached at all. They are **not interchangeable** —
|
||
one is real production infrastructure, the other exists specifically so
|
||
there's somewhere safe to test. The restriction below is a policy for
|
||
Claude specifically, not a change to the tooling's own default or
|
||
anything the operator needs to opt into.
|
||
|
||
#### `pve1.sweet.home` (production — off-limits to Claude)
|
||
|
||
A real, live Proxmox node hosting production VMs/containers — not a
|
||
sandbox, and not Claude's to touch by default.
|
||
|
||
- **Off-limits at all times unless the operator has given explicit,
|
||
same-session instructions to act on this specific host.** That
|
||
authorization is scoped to the task it was given for — don't carry it
|
||
forward to unrelated later work in the same conversation, and never
|
||
assume it from a previous session.
|
||
- **Read-only for existing state is always fine, authorization or not.**
|
||
You may SSH in (or use `pvesm`, `qm list`, `pct list`, `qm config`, `pct
|
||
config`, the Proxmox API, etc.) to inspect the node's config, storage,
|
||
and any existing VM/container — including ones this repo didn't create.
|
||
- **Never** modify, stop, restart, delete, reconfigure, or create anything
|
||
on this node (`qm set`, `pct set`, `qm destroy`, `pct destroy`, `qm
|
||
stop`, `pct stop`, `qm create`, `pct create`, snapshot operations,
|
||
storage changes, etc.) — including scratch/test resources — without
|
||
that explicit go-ahead. Use `pve-test.sweet.home` for anything
|
||
exploratory instead; it exists precisely so `pve1` never has to be the
|
||
answer to "where do I test this."
|
||
- **This is a Claude-specific policy, not something the scripts enforce.**
|
||
`scripts/env.sh`/`create-proxmox-resource.sh` default to `pve1` exactly
|
||
as they did before `pve-test` existed, with no extra flag or prompt
|
||
required — that's deliberate, so the operator's own existing workflows
|
||
don't change. Claude, however, must never rely on that default: every
|
||
Proxmox action Claude takes on its own initiative — not explicitly
|
||
pointed at `pve1` by the operator this session — targets `pve-test`
|
||
instead (e.g. `--node "$PVE_TEST_HOST"`, or `PROXMOX_HOST=$PVE_TEST_HOST`).
|
||
Claude's own default is `pve-test`, full stop, regardless of what the
|
||
tooling's own unqualified default happens to be.
|
||
|
||
#### `pve-test.sweet.home` (sandbox — Claude's default target)
|
||
|
||
A separate Proxmox node set aside for testing. The *tooling's* default is
|
||
still production (`PROXMOX_HOST` → `PVE1_HOST`, see above) — but
|
||
**Claude's own default is this node**: absent an explicit, same-session
|
||
instruction to use `pve1`, every Proxmox action Claude initiates targets
|
||
`pve-test`. Once targeted, it's safe to create, interrogate, and destroy
|
||
resources on without asking first.
|
||
|
||
- **Test VMs/containers are allowed, but must be torn down.** Create a
|
||
scratch VM or container here (e.g. via
|
||
`scripts/proxmox/create-proxmox-resource.sh` or raw `qm`/`pct create`)
|
||
to validate something. Anything created this way must be destroyed
|
||
again in the same session, before ending the task — never leave a test
|
||
resource running. Use a VMID/name that's obviously scratch (and doesn't
|
||
collide with a real flake target) so it's unambiguous what's safe to
|
||
remove.
|
||
- **Node-level config is still not yours to change.** Creating/destroying
|
||
your own scratch guests is fine; Proxmox host config, storage pools, and
|
||
networking on `pve-test` itself are still the operator's call to make
|
||
manually, same as on `pve1`.
|
||
|
||
## Commands
|
||
|
||
```bash
|
||
# One-time environment bootstrap (installs Nix if missing, prints hosts)
|
||
bash scripts/codex-setup.sh
|
||
|
||
# Changed-files-only validation: secret grep (whole repo), nixpkgs-fmt --check
|
||
# and statix on changed *.nix files, eval of the hosts/packages those changes
|
||
# can affect. This is what CI runs on every push/PR.
|
||
bash scripts/codex-maintenance.sh
|
||
|
||
# Full sweep: nixpkgs-fmt --check/statix over the whole tree, eval every host
|
||
# and package. Slow (minutes) -- CI never runs this; use it locally before a
|
||
# release or after touching modules/common/*, flake.nix, or variables.nix for
|
||
# extra confidence beyond the automatic full-fallback those paths already
|
||
# trigger in the default mode (see below).
|
||
bash scripts/codex-maintenance.sh --full-check
|
||
|
||
# Either mode, plus a dry-run build (no result symlink) of every host/package
|
||
# in whichever scope is active
|
||
bash scripts/codex-maintenance.sh --dry-run
|
||
bash scripts/codex-maintenance.sh --full-check --dry-run
|
||
|
||
# List the hosts the flake currently exposes
|
||
nix eval --json .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]'
|
||
|
||
# Evaluate a single host without building (fast sanity check)
|
||
nix eval .#nixosConfigurations.<host>.config.system.build.toplevel.drvPath --raw
|
||
|
||
# Dry-run build a single host
|
||
nix build --dry-run --no-link .#nixosConfigurations.<host>.config.system.build.toplevel
|
||
```
|
||
|
||
Formatting/lint tools (`nixpkgs-fmt`, `statix`) are not installed locally; the
|
||
maintenance script pulls them via `nix run github:NixOS/nixpkgs/nixos-25.11#<tool>`.
|
||
There is no test suite — "correctness" here means the flake evaluates and
|
||
`nixpkgs-fmt`/`statix` are clean.
|
||
|
||
With no flags, `codex-maintenance.sh` diffs against a base ref (env
|
||
`MAINT_BASE_SHA`, else the PR base SHA in CI, else `HEAD^` locally) and scopes
|
||
fmt-check/statix to the changed `*.nix` files and eval to the hosts/packages
|
||
those changes can affect — a `hosts/<name>/host.nix` edit only evals that
|
||
host's targets, a `modules/platforms/<platform>.nix` edit only evals that
|
||
platform's hosts, and so on. A change to `flake.nix`, `flake.lock`,
|
||
`variables.nix`, `modules/common/*`, or any other `modules/*.nix` file outside
|
||
`platforms/`/`build-types/` (whose blast radius isn't safely inferable from
|
||
the path alone) falls back to evaluating every host and package, same as
|
||
`--full-check` would, just without the whole-tree fmt/statix sweep. This
|
||
exists because the whole-tree sweep is what was timing out CI; **CI always
|
||
runs the plain, no-flag form and never passes `--full-check`.**
|
||
|
||
The default mode's diff is against the working tree (uncommitted and staged
|
||
edits included, not just committed ones), so it's already the right tool for
|
||
an interactive session too: after editing one or two hosts/modules, plain
|
||
`bash scripts/codex-maintenance.sh` naturally scopes to just what you
|
||
touched. Reserve `--full-check` for changes that plausibly affect every host
|
||
(`modules/common/*`, `flake.nix`, `variables.nix` — though the default mode
|
||
already falls back to evaluating everything for those paths, `--full-check`
|
||
additionally re-checks fmt/statix over the whole tree) or as a final check
|
||
before committing.
|
||
|
||
## Scripts
|
||
|
||
Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` is
|
||
organized by purpose: `scripts/secrets/` (sops/age + SSH host-key
|
||
management), `scripts/proxmox/` (Proxmox deployment), `scripts/installer/`
|
||
(the auto-installer's own shell script, templated into the image — see
|
||
below), `scripts/lib/` (shared helpers, sourced by the scripts below — not
|
||
run directly), and a handful of repo-wide scripts left at the top level
|
||
(`env.sh`, `bump-nixpkgs-release.sh`, plus `codex-setup.sh`/
|
||
`codex-maintenance.sh` above). When adding a new script, put it in the
|
||
matching subfolder rather than the top level, and if it duplicates logic
|
||
another script already has, lift the shared part into `scripts/lib/`
|
||
instead of copying it.
|
||
|
||
### `scripts/installer/`
|
||
|
||
- `scripts/installer/auto-install.sh` — the interactive install script
|
||
baked into the auto-installer image (see `docs/auto-installer.md`), kept
|
||
as a real, version-controlled shell file rather than inline in
|
||
`modules/installer/common.nix`'s Nix. It sources `scripts/env.sh` itself
|
||
for `LAN_DOMAIN` (`export LAN_DOMAIN`/`: "${LAN_DOMAIN:=...}"`, matching
|
||
`variables.nix`'s `lanDomain` — manually kept in sync, same pattern as
|
||
`NIX_CACHE_HOST` mirroring `nixCacheHost`), rather than Nix-level string
|
||
substitution — that's what makes it work identically whether run
|
||
straight from a git checkout or from inside the built installer image.
|
||
`common.nix` bakes `scripts/env.sh` in alongside it at a matching
|
||
relative path (`/etc/nixos-installer/env.sh` next to
|
||
`/etc/nixos-installer/installer/auto-install.sh`) so the script's own
|
||
`source "$(dirname ...)/../env.sh"` line resolves the same way in both
|
||
contexts — this is also why it's invoked from
|
||
`/etc/nixos-installer/installer/auto-install.sh` rather than a flat
|
||
`/etc/auto-install.sh`. `#!/usr/bin/env bash`, not
|
||
`#!/run/current-system/sw/bin/bash`: the latter only resolves on an
|
||
already-activated NixOS system, breaking the checked-out-file case
|
||
entirely (confirmed live: "cannot execute: required file not found" on
|
||
a non-NixOS box); `/usr/bin/env` is reliably present on both NixOS
|
||
(`environment.usrbinenv`'s own default) and any normal Linux distro.
|
||
|
||
### `scripts/secrets/`
|
||
|
||
- `scripts/secrets/sync-host-keys.sh` — generates/registers SSH host keys
|
||
and their `.sops.yaml`/`secrets/*.yaml` recipients for flake targets,
|
||
idempotently (`--all`, `<target>`, `--remove`, `--regenerate-all-keys`,
|
||
all with `--dry-run`). Stores keys as clan vars
|
||
(`vars/per-machine/<target>/openssh/`, committed and sops-encrypted) for
|
||
all flake targets. The primary tool for provisioning a new host's
|
||
secrets access — see "Creating a new machine" in
|
||
`docs/auto-installer.md`.
|
||
- `scripts/secrets/prepare-host-key.sh` — narrower predecessor: generates a
|
||
key by an arbitrary name without touching `.sops.yaml`. Still useful to
|
||
pre-generate a key before its flake target exists yet, since
|
||
`sync-host-keys.sh` can only act on targets `nixosConfigurations` already
|
||
has.
|
||
- `scripts/secrets/rotate-admin-key.sh <backup-admin-key> [--new-key-file
|
||
<path>] [--dry-run]` — rotates `.sops.yaml`'s `&admin` age key: decrypts
|
||
with a backed-up copy of the key currently trusted as `&admin` (verified
|
||
by deriving its public key and comparing, not taken on faith), replaces
|
||
the `&admin` line with a new key already present in the environment
|
||
(defaults to wherever sops/age itself would look), and runs
|
||
`sops updatekeys` on every `secrets/*.yaml`. One-way: the old key can no
|
||
longer decrypt anything re-encrypted this way. This is the automation
|
||
for the manual steps `sync-host-keys.sh`/`create-proxmox-resource.sh`
|
||
print when they bootstrap a brand-new, not-yet-trusted key on a machine
|
||
with no prior admin access.
|
||
- `scripts/secrets/backup-admin-key.sh <dest-path> [--key-file <path>]
|
||
[--force] [--dry-run]` — copies the local sops age key (source
|
||
resolution matches sops/age itself: `$SOPS_AGE_KEY` inline, then
|
||
`--key-file`, then `$SOPS_AGE_KEY_FILE`, then the XDG default) to an
|
||
arbitrary destination path with `0600` permissions, validating it's a
|
||
real age identity and round-tripping the public key before and after the
|
||
write. Refuses to overwrite an existing `<dest-path>` without `--force`.
|
||
Purely a local filesystem copy — never touches `.sops.yaml`/
|
||
`secrets/*.yaml` or the repo at all. The resulting file is exactly what
|
||
`rotate-admin-key.sh` expects as its backup-key argument.
|
||
- `scripts/secrets/sync-nix-cache-host-key.sh [--check] [--dry-run]
|
||
[--host <name>]` — detects drift between the ed25519 SSH host key
|
||
nix-cache is actually serving right now (via `ssh-keyscan`) and
|
||
`vars.nixCacheHostKey` (`variables.nix`), the value
|
||
`modules/nix-cache/remote-builder-client.nix` bakes into every real
|
||
client's declarative `programs.ssh.knownHosts` and
|
||
`configure-nix-cache-client.sh` hardcodes as its own default for
|
||
non-NixOS clients. That value has no automatic source of truth — it's
|
||
set once from whatever nix-cache's host key happened to be at the time,
|
||
and silently goes stale if the host is ever rebuilt/recreated with a new
|
||
key, breaking every client's distributed-build SSH trust with no error
|
||
that points back here. `--check` (used by `codex-maintenance.sh`, which
|
||
treats an unreachable nix-cache — e.g. from a non-LAN CI runner — as a
|
||
silent skip rather than a failure) only reports drift; the no-flags form
|
||
updates both files in place. Declarative clients still need a rebuild to
|
||
pick up the fix.
|
||
- `scripts/secrets/push-host-keys.sh [--all | <target>] [--dry-run]
|
||
[--skip-git-check]` — pushes newly-generated SSH host keys from
|
||
`host-keys/` to already-running NixOS hosts, so they can decrypt sops
|
||
secrets after a rebuild following `sync-host-keys.sh
|
||
--regenerate-all-keys`. Verifies that `.sops.yaml` and `secrets/*.yaml`
|
||
are committed and pushed to the remote first (hosts rebuild from the
|
||
remote Gitea flake, so recipient changes must land there before any key
|
||
push).
|
||
|
||
### `scripts/proxmox/`
|
||
|
||
- `scripts/proxmox/create-proxmox-resource.sh` — builds a `lxc-*`/
|
||
`proxmox-*` target's tarball/disk image and creates it on a real Proxmox
|
||
node (`pct create` against the tarball as a CT template / `qm create`+
|
||
`importdisk`), or reconfigures an existing resource's cores/memory/disk
|
||
size (`--modify`, always requires typing the VMID back to confirm).
|
||
Checks for an already-uploaded image on the node before building
|
||
(`--force-rebuild` to skip that and always rebuild), and probes
|
||
nix-cache's substituter/remote-builder reachability once up front rather
|
||
than letting every `nix build` call retry against it individually.
|
||
Refuses to create a target whose host identity already exists live on
|
||
the node (checked directly via `qm`/`pct`, not any file in this repo)
|
||
unless `--allow-duplicate-host` is passed. `--dry-run` throughout both
|
||
modes. The first time it has to bootstrap build tooling on a node (i.e.
|
||
`nix` wasn't already on its `PATH`), it also runs
|
||
`scripts/proxmox/configure-nix-cache-client.sh` there (non-fatally — a
|
||
failure just falls back to building from source / `cache.nixos.org`) so
|
||
the node substitutes from and can offload builds to nix-cache on every
|
||
subsequent run, not just this one.
|
||
- `scripts/proxmox/clone-pve1-to-pve-test.sh <vmid> [--new-vmid <id>]
|
||
[--mode snapshot|suspend|stop] [--dry-run]` — ad-hoc clone of a single
|
||
VM or CT from pve1 (production) to pve-test (sandbox) via vzdump +
|
||
qmrestore/pct restore. Streams the archive directly between nodes (no
|
||
local staging copy). Always restores with `--unique 1` (fresh MAC
|
||
addresses) since the original is still running on the LAN. Cleans up
|
||
the vzdump archive from both nodes after a successful restore. The
|
||
script's own default is pve1 → pve-test, matching CLAUDE.md's policy
|
||
(unlike `create-proxmox-resource.sh`, which defaults to production for
|
||
the operator's own unqualified use).
|
||
- `scripts/proxmox/configure-nix-cache-client.sh [--dry-run]
|
||
[--no-remote-builder] [--no-restart]` — the non-NixOS equivalent of
|
||
`modules/nix-cache/client.nix`/`remote-builder-client.nix`, for a plain
|
||
Debian machine with the Nix package manager (not NixOS) already
|
||
installed: run as root *on that machine* to add nix-cache as a
|
||
substituter in `/etc/nix/nix.conf` (`https://cache.nixos.org/` kept as
|
||
fallback) via `extra-substituters`/`extra-trusted-public-keys` so it
|
||
layers on top of whatever's already there instead of clobbering it, and,
|
||
if `/root/.ssh/nixremote` is already present (see docs/nix-cache.md
|
||
"Remote builder SSH keys"), configures it as a distributed-build
|
||
machine too and trusts nix-cache's SSH host key in
|
||
`/etc/ssh/ssh_known_hosts`. Idempotent (re-running replaces its own
|
||
marked block rather than duplicating it); restarts `nix-daemon` by
|
||
default so the change takes effect immediately.
|
||
|
||
### `scripts/ha/`
|
||
|
||
HA cluster lifecycle and operational scripts. All mutate real cluster state
|
||
when run for real — always run against pve-test first unless the operator
|
||
explicitly targets pve1.
|
||
|
||
- `scripts/ha/deploy.sh [--skip-*] [--destroy] [--dry-run]` — full
|
||
lifecycle manager: phases through bridge creation, key sync, VM creation
|
||
(via `create-proxmox-resource.sh`), NIC/disk attachment, and cluster
|
||
initialisation. `--destroy` tears it back down. Safe to rerun
|
||
idempotently; each phase can be individually skipped.
|
||
- `scripts/ha/cluster-init.sh` — one-time cluster bootstrap run **as root
|
||
on ha-server-1** after both VMs are booted. Generates/distributes the
|
||
Corosync authkey, initialises DRBD metadata, creates XFS on `/dev/drbd0`,
|
||
configures LIO iSCSI, and registers all Pacemaker resources (DRBD → XFS
|
||
→ iSCSI → NFS → VIPs).
|
||
- `scripts/ha/health.sh` — read-only cluster health snapshot: SSH
|
||
reachability, quorum, DRBD state, Pacemaker resources, and VIP port
|
||
reachability. Safe to run from the workstation at any time.
|
||
- `scripts/ha/failover.sh [--to node1|node2] [--force] [--timeout <s>]
|
||
[--dry-run]` — graceful failover by putting the active node into
|
||
Pacemaker standby and waiting for resources to appear on the target.
|
||
- `scripts/ha/acceptance-tests.sh` — T1–T7 acceptance tests (failover,
|
||
NFS/iSCSI connectivity, DRBD sync, etc.) that must all pass before the
|
||
cluster is considered production-ready.
|
||
- `scripts/ha/resize-data-disk.sh --size +NNg [--force] [--dry-run]` —
|
||
online data-disk resize: `qm resize` on both VMs, guest block-device
|
||
rescan, `drbdadm resize`, `xfs_growfs`. No downtime required.
|
||
- `scripts/ha/cluster-enable-stonith.sh` — enables the `fence_pve_ssh`
|
||
STONITH resource after the fence SSH key is deployed to both nodes and
|
||
authorised on the Proxmox host. Run once after `cluster-init.sh`.
|
||
- `scripts/ha/fence-pve-ssh.py` — Python STONITH fence agent for Pacemaker.
|
||
Deploy to `/etc/pacemaker/fence_pve_ssh` on both HA nodes (`chmod +x`).
|
||
SSHes to the Proxmox host and runs `qm stop/start <vmid>`.
|
||
|
||
### `scripts/ipa/`
|
||
|
||
- `scripts/ipa/create-nixos-ipa-host-account.sh [options] <hostname>` —
|
||
adds a NixOS host to the FreeIPA domain and produces a sops-encrypted
|
||
keytab at `secrets/<hostname>.keytab`, ready for `modules/ipa/client.nix`.
|
||
Replaces three error-prone manual steps: `ipa host-add`, `ipa-getkeytab`
|
||
(run on the DC, SCP'd back), and `sops encrypt` in the correct location
|
||
(must be at `secrets/<hostname>.keytab` for the creation rule to match).
|
||
|
||
### `scripts/lib/`
|
||
|
||
Sourced by the scripts above, never run directly:
|
||
|
||
- `nix-bootstrap.sh` — `NIX_CONFIG`/`ensure_nix_profile`, shared by
|
||
`codex-setup.sh`/`codex-maintenance.sh` and the remote build commands
|
||
`create-proxmox-resource.sh` runs over SSH.
|
||
- `nix-eval.sh` — `NIX_EVAL_FLAGS` plus `list_flake_targets`/
|
||
`flake_target_hostname` flake-introspection helpers.
|
||
- `nix-parallel.sh` — `run_nix_parallel`: fans out independent `nix eval`/
|
||
`nix build --dry-run` calls across up to `NIX_PARALLEL_JOBS` processes,
|
||
capped by available memory (~1 GB/job) rather than raw `nproc` to avoid
|
||
OOM on constrained CI runners. Used by `codex-maintenance.sh`.
|
||
- `clan-vars.sh` — helpers for reading/writing SSH host keys stored as clan
|
||
vars (`vars/per-machine/<target>/openssh/`, sops-encrypted) instead of
|
||
the gitignored `host-keys/` directory. Sourced by
|
||
`create-proxmox-resource.sh` and `sync-host-keys.sh`; depends on
|
||
`sops-age.sh` and `ssh-host-keys.sh` being sourced first.
|
||
- `ssh-host-keys.sh` — `generate_host_ed25519_key`/`ssh_pubkey_to_age`,
|
||
shared by `sync-host-keys.sh` and `prepare-host-key.sh`.
|
||
- `sops-age.sh` — `age_pubkey_from_identity_file`/`sops_yaml_admin_pubkey`/
|
||
`sops_updatekeys` plus the shared sops/age default key-file resolution,
|
||
shared by `backup-admin-key.sh`, `rotate-admin-key.sh`, and
|
||
`sync-host-keys.sh`.
|
||
- `confirm.sh` — `confirm_typed`, the "type X back to confirm" destructive-
|
||
action prompt shared by `create-proxmox-resource.sh` and
|
||
`sync-host-keys.sh`.
|
||
- `sync-host-keys-edit-sops.py` — the `.sops.yaml` anchor/key_groups editor
|
||
`sync-host-keys.sh` shells out to (see that script for why: precise,
|
||
idempotent YAML edits are impractical in bash).
|
||
|
||
### Top level
|
||
|
||
- `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
|
||
default cores/memory, `NIX_CACHE_HOST`, `LAN_DOMAIN`) sourced by
|
||
`create-proxmox-resource.sh` and `scripts/installer/auto-install.sh`. Add
|
||
new cross-script config here instead of duplicating it per-script.
|
||
- `scripts/recover-hosts.sh [<hostname> ...]` — fixes sops/SSH-key/GitHub-token
|
||
issues on deployed NixOS hosts and triggers a `Switch-nix` rebuild on each.
|
||
With no args discovers every known hostname; with args checks only those.
|
||
Fixes applied automatically (prompts before rebuilding): SSH host key drift
|
||
(restores the registered key) and stale GitHub access tokens (empties the
|
||
rendered `nix-github-token.conf` so Nix falls back to unauthenticated requests
|
||
until sops-nix re-renders the correct token after the next successful rebuild).
|
||
- `scripts/gc-hosts.sh [--dry-run]` — runs `nix-collect-garbage -d` on all live
|
||
NixOS hosts (workstation first, then pve1, then all Proxmox guests). Excludes
|
||
`nix-cache` (gc-ing the shared binary cache evicts store paths other hosts
|
||
depend on). Uses passwordless sudo where available; falls back to user-level gc.
|
||
- `scripts/bump-nixpkgs-release.sh` — bumps `flake.nix`'s `nixpkgs.url`/
|
||
`home-manager.url` in place. Exists because flake input URLs can't
|
||
reference `variables.nix` (confirmed empirically — `nix flake metadata`
|
||
errors on it), so this is the closest equivalent to a single source of
|
||
truth for the tracked release.
|
||
|
||
`sync-host-keys.sh`, `create-proxmox-resource.sh`, and
|
||
`rotate-admin-key.sh` genuinely mutate real state when run for real (not
|
||
`--dry-run`): real `secrets/*.yaml` recipients, real Proxmox VMs/
|
||
containers, real revocation of decrypt access. They require the
|
||
operator's own SSH/sops access, which an agent session doesn't have — but
|
||
don't suggest running any of them non-dry-run without the operator's
|
||
explicit go-ahead even if it becomes technically reachable.
|
||
`backup-admin-key.sh` only writes a key copy to a path the operator gives
|
||
it — lower-stakes than the others, but it still handles a real private
|
||
key, so treat its destination path choice as the operator's call too.
|
||
|
||
## Architecture
|
||
|
||
`flake.nix` is the single entry point. It generates one
|
||
`nixosConfigurations.<platform>-<buildtype>` attribute per target via the
|
||
`mkTarget` function, composed from:
|
||
|
||
```
|
||
nixosSystem {
|
||
modules = [
|
||
disko.nixosModules.disko
|
||
sops-nix.nixosModules.sops
|
||
./modules/common/configuration.nix
|
||
./modules/platforms/${platform}.nix # what it runs on
|
||
./modules/build-types/${buildType}.nix # what it's for
|
||
hostPath # hosts/<name>/host.nix — per-machine identity
|
||
home-manager.nixosModules.home-manager { ... }
|
||
] ++ (client-only modules, for every buildType except "nix-cache" itself)
|
||
}
|
||
```
|
||
|
||
Platforms: `linode`, `proxmox`, `lxc`, `baremetal`. Build types: `minimal`,
|
||
`nix-cache`, `docker`, `gui`, `pxe-boot`, `tailscale-router`, `tor-relay`,
|
||
`ha-server`. Not every combination is built — e.g. `pxe-boot` has no `linode`
|
||
variant (PXE/DHCP/TFTP need LAN L2 adjacency a Linode VPS doesn't have),
|
||
`tor-relay` only exists as `lxc-tor-relay`, `ha-server` only exists as
|
||
`proxmox-ha-server-{1,2}`, and `baremetal` only exists as `baremetal-gui`
|
||
(the real gui-host hardware — see `hosts/nixos/host.nix` and
|
||
`modules/platforms/baremetal.nix`). Treat
|
||
`flake.nix`'s
|
||
`generatedTargets` as the source
|
||
of truth for which hosts exist — `README.md`, `AGENTS.md`,
|
||
`docs/flake-lock-automation.md`, and the CI eval workflows
|
||
(`.github/workflows/check-nixos.yml`, `.gitea/workflows/check-nixos.yml`) list
|
||
hosts by hand (or, for the CI workflows, evaluate the flake dynamically) and
|
||
can drift from it, so re-check them against `flake.nix` when adding or
|
||
removing a host.
|
||
|
||
### Composition pattern
|
||
|
||
- `hosts/<name>/host.nix` — per-machine identity **only**: hostname, hostId,
|
||
per-machine secrets, `system.stateVersion`. These files carry no `imports`
|
||
of their own — all shared behavior comes from the platform/build-type modules
|
||
composed in `flake.nix`, not from the host file.
|
||
- `modules/platforms/{linode,proxmox,lxc,baremetal}.nix` — platform-specific
|
||
config: boot method, guest tooling, and the hardware config, imported
|
||
directly by the platform module itself — **not** wired in from
|
||
`flake.nix`. VM platforms use `../hardware-configuration/vm/{proxmox,linode}.nix`;
|
||
`baremetal.nix` uses `../hardware-configuration/baremetal.nix` (adapted
|
||
from a real `nixos-generate-config` run on the actual hardware, not a
|
||
vm/ file, since it isn't a VM) plus `hardware.enableRedistributableFirmware
|
||
= true` for real wifi/GPU/microcode firmware that VMs never needed.
|
||
`lxc.nix` has no hardware-configuration counterpart since containers
|
||
share the host kernel; instead it imports nixpkgs' own
|
||
`virtualisation/proxmox-lxc.nix`, which gives every `lxc-*` host a
|
||
`config.system.build.tarball` output — a plain rootfs tarball, used as a
|
||
`pct create ... vztmpl` CT template (**not** `pct restore`, which expects
|
||
`vzdump` backup-archive metadata this doesn't have), no install step —
|
||
see `docs/auto-installer.md`.
|
||
- `modules/build-types/*.nix` — what a system is for:
|
||
minimal/docker/gui/pxe-boot/nix-cache/tailscale-router/tor-relay/ha-server.
|
||
- `modules/common/configuration.nix` — base NixOS config imported by every
|
||
host: locale, users, nix settings, git.
|
||
- `modules/common/home.nix` / `hosts/nixos/home.nix` — Home Manager config for
|
||
the `nixos` user; the `nixos` workstation (`gui` build type) has its own,
|
||
other hosts share `modules/common/home.nix`.
|
||
- `modules/disko/proxmox.nix` — declarative disk layout (GPT: ESP + swap +
|
||
ext4 root) via disko, used by all Proxmox-VM hosts (`proxmox-*`, not
|
||
`lxc-*`). Also carries `imageSize`/`imageName`, letting every `proxmox-*`
|
||
host be built as a standalone, `qm importdisk`-ready `.raw` image with no
|
||
install step — see `docs/proxmox-images.md`.
|
||
- `modules/disko/linode.nix` — `linode-*`'s disko config, deliberately
|
||
different in kind from the Proxmox one: Linode provisions and sizes
|
||
`/dev/sda`/`/dev/sdb` itself as whole, unpartitioned devices before the OS
|
||
boots, so this declares them with `destroy = false` (disko never wipes
|
||
them) and a bare `filesystem`/`swap` content type instead of a partition
|
||
table — idempotent against an already-provisioned disk, never destructive.
|
||
- `modules/disko/baremetal.nix` — `baremetal-gui`'s disko config: a ZFS
|
||
RAID0 (striped, no redundancy — disko's zpool `mode` defaults to `""`,
|
||
which is a plain stripe rather than `"mirror"`/`"raidz"`) root pool
|
||
across two disks, ESP + systemd-boot on the first. Device paths
|
||
(`vars.guiRootDisk1`/`guiRootDisk2`) are placeholders — fill in stable
|
||
`/dev/disk/by-id/...` paths before running disko for real.
|
||
`modules/platforms/baremetal.nix` also imports
|
||
`modules/services/zfs/enable-service.nix` for this (the `zfs_unstable`
|
||
package, autoScrub/autoSnapshot/trim) — the only other importer today is
|
||
`ha-server`'s NFS data pool, an unrelated non-root ZFS use.
|
||
- `modules/boot/efi.nix` — systemd-boot + EFI vars, paired with the disko module.
|
||
- `modules/installer/` — the auto-installer environment (ISO, also served as
|
||
PXE netboot): `common.nix` (shared config + the generated
|
||
`auto-install.sh`), `iso.nix`, `host-keys.nix` (optionally bakes
|
||
`host-keys/` into the image under `--impure`). See
|
||
`docs/auto-installer.md`.
|
||
- `modules/pxe-boot/stage-installer-artifacts.nix` — builds the installer's
|
||
netboot image and stages it on the `pxe-boot` host so its iPXE menu can
|
||
chain straight to it. See `docs/pxe-boot.md`.
|
||
- `modules/nix-cache/{client,server,remote-builder-client}.nix` — binary cache
|
||
substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the
|
||
full design (per-host local stores, no shared `/nix/store`, and how the
|
||
`nixremote` signing/SSH keys fit together).
|
||
- `modules/ha/` — HA cluster NixOS modules: `cluster-config.nix` (DRBD,
|
||
Corosync, Pacemaker, firewall rules, cluster-wide NFS/iSCSI port
|
||
authorisation — shared by both ha-server nodes), `pacemaker-stack.nix`
|
||
(Pacemaker + Corosync service enablement), and supporting modules. See
|
||
`docs/ha.md` for the cluster operational guide.
|
||
- `modules/ipa/client.nix` — FreeIPA client enrollment: sssd, Kerberos keytab,
|
||
and IPA host registration; imported by every real host via
|
||
`modules/common/configuration.nix`.
|
||
- `modules/beszel/enable-agent.nix` — enables beszel-agent, sets `HUB_URL`,
|
||
fixes the upstream `StateDirectory` bug, and wires the universal
|
||
`beszel-token` sops secret (from `secrets/common.yaml`) into the agent's
|
||
`environmentFile`; see `docs/beszel.md` for the full setup guide.
|
||
- `modules/tailscale/`, `modules/docker/`, `modules/networking/`,
|
||
`modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose,
|
||
single-host feature modules (e.g. `docker/enable-service.nix`,
|
||
`services/zfs/enable-service.nix`). Grep `modules/build-types/*.nix` for
|
||
each build type's `imports` list to see which modules apply where.
|
||
|
||
New host = new `hosts/<name>/host.nix` + a matching
|
||
`mkTarget { platform; buildType; hostPath; }` entry added to `flake.nix`'s
|
||
`generatedTargets`, composed from existing `modules/*` pieces rather than
|
||
duplicating config.
|
||
|
||
### Other docs worth reading before touching these areas
|
||
|
||
- `docs/nix-cache.md` — nix-cache binary cache/remote-builder design and key
|
||
handling.
|
||
- `docs/pxe-boot.md` — the `pxe-boot` host's iPXE/TFTP/HTTP boot chain and
|
||
directory layout under `/srv/pxe`.
|
||
- `docs/auto-installer.md` — the installer environment (ISO/netboot/Proxmox
|
||
LXC), `host-keys/` and the sops-nix pre-seeding problem it solves, and why
|
||
`lxc-*` hosts are deliberately excluded from its menu.
|
||
- `docs/proxmox-images.md` — building `proxmox-*` hosts as standalone `.raw`
|
||
disk images (disko's image builder) instead of installing, and deploying
|
||
the result to Proxmox.
|
||
- `docs/flake-lock-automation.md` — how `flake.lock` updates flow through CI
|
||
(scheduled `nix flake update` PR + host-eval-on-PR workflow) and why hosts
|
||
should track the committed lock file rather than `nixos-rebuild --upgrade-all`.
|
||
- `docs/ha.md` — HA file-server cluster: DRBD + XFS + LIO iSCSI + NFS managed
|
||
by Corosync + Pacemaker; network topology; lifecycle scripts in `scripts/ha/`.
|