Compare commits

..
Author SHA1 Message Date
rootandClaude Sonnet 5 5ec7033439 Fix stale documentation: outdated counts, missing build type, spec status
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m26s
Same class of problem as the deployedTargets/README fixes: hand-maintained
prose that drifted from reality and nobody was obligated to update.

- CLAUDE.md: "18 hosts" was a stale hardcoded count (actually 20); reworded
  to not need updating as hosts are added. Also added the missing
  tailscale-exit-node build type to a list that had it everywhere else in
  the file except one bullet.
- AGENTS.md: same missing tailscale-exit-node build type.
- docs/auto-installer.md: the hand-enumerated lxc-* list was missing
  lxc-tailscale-exit-node.
- flake-target-refactor-spec.md: added a "Status: implemented" note so this
  completed historical spec (referenced elsewhere purely for rationale)
  can't be mistaken for an open plan with unresolved Open Questions.
- remove-sensetive-info-refactor.md: the "Definition of done" checklist was
  entirely unchecked despite most of the work being done. Checked off what's
  actually done (sops-nix migration, history scrub just performed, the
  pre-commit gitleaks hook), and left rotation of the GitHub PAT found in
  history explicitly flagged as the one still-open item -- an operator
  action against GitHub, not something this repo can attest to itself.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 11:06:05 +00:00
beatzaplenty 9133afd444 Merge pull request 'Fix duplicate-host check reporting false SSH failures' (#8) from worktree-fix-duplicate-host-check-exitcode into main
Check NixOS configurations / eval-hosts (push) Failing after 11m36s
2026-07-20 11:00:18 +00:00
rootandClaude Sonnet 5 eeec9ce302 Fix duplicate-host check reporting false SSH failures
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m38s
The remote bash script run over SSH ended with a for-loop whose last
statement was `[[ "$n" == "$target" ]] && echo ...`. When the last
VM/CT checked on the node didn't match --host, that test evaluated
false and became the exit status of the whole remote script (1) --
which the wrapper then misreported as "couldn't reach the node",
even though SSH connectivity and the check itself were both fine.
The actual signal is the script's stdout, not its exit code, so end
it with an explicit exit 0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:55:53 +00:00
beatzaplenty a62c4fc023 Merge pull request 'Stop tracking deployment status in the README Hosts table' (#7) from remove-deploy-status-from-readme into main
Check NixOS configurations / eval-hosts (push) Failing after 11m20s
2026-07-20 10:51:59 +00:00
rootandClaude Sonnet 5 97ede62f6d Stop tracking deployment status in the Hosts table
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m24s
Same problem as the deployedTargets removal, just in markdown instead of
Nix: which variant of a buildtype is actually deployed is live
infrastructure state, and a committed table can't stay accurate as that
changes -- it already required a manual edit on every migration and had
drifted before. Keep only what doesn't rot: what each target is for, and
stable naming history. Point at the live node / /etc/flake-target instead
for actual deployment status.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:18:33 +00:00
beatzaplenty ab5206b1c7 Merge pull request 'Replace duplicate-host check with live Proxmox query; drop deployedTargets' (#6) from fix-duplicate-host-self-match into main
Check NixOS configurations / eval-hosts (push) Failing after 12m7s
2026-07-20 10:12:59 +00:00
rootandClaude Sonnet 5 2041557ab3 Replace the duplicate-host check with a live Proxmox query, drop deployedTargets
variables.nix's deployedTargets was a manually-maintained list with no
enforcement keeping it in sync with reality -- it caused two separate
false refusals in a row (naming a VM as deployed well after it had been
destroyed, then matching a target against itself once the list was
"corrected"). Static files can't track whether a resource still actually
exists.

create-proxmox-resource.sh's duplicate-host guard now queries the
Proxmox node directly (qm/pct's own name/hostname config, matched
against --host) instead. Also fixes a gap in that live check: it
originally swallowed ssh failures and would have silently treated "can't
reach the node" the same as "checked, nothing there" -- it now refuses
instead of guessing when the node can't be reached.

deployedTargets is removed entirely from variables.nix since nothing
else in the repo consumed it once this script no longer does; README.md's
Hosts table remains the sole source of truth for "(real, deployed)"
status. CLAUDE.md and the script's own --help/comments updated to match.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 10:03:00 +00:00
rootandClaude Sonnet 5 2fd483697b Don't refuse recreating the canonical already-deployed target itself
The duplicate-host check in create-proxmox-resource.sh compared by
hostName only, so it fired even when the target being created was
exactly the one variables.nix's deployedTargets already names (e.g.
rebuilding lxc-nix-cache after destroying its old container to pick up
new sops secrets) -- there's no other machine at risk of an identity
collision in that case, just the normal redeploy workflow. Skip the
check when dt == flake_target; the later VMID-existence check still
guards against clobbering a resource that's actually live on the node.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 09:49:31 +00:00
beatzaplenty 89186b0dee Merge pull request 'Track nix-cache real deployment as lxc-nix-cache, not proxmox-nix-cache' (#5) from worktree-nix-cache-lxc-migration into main 2026-07-20 09:46:41 +00:00
rootandClaude Sonnet 5 8e3606cbd3 Track nix-cache's real deployment as lxc-nix-cache, not proxmox-nix-cache
The old proxmox-nix-cache VM was destroyed and nix-cache is being
redeployed as an LXC container going forward. Without this update,
create-proxmox-resource.sh's duplicate-host check (which only reads this
static list, not live Proxmox state) kept refusing to create
lxc-nix-cache even though nothing named nix-cache actually exists on the
node anymore.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 09:40:03 +00:00
beatzaplenty a18dfb0127 Merge pull request 'Trust nix-cache's SSH host key declaratively on remote-builder clients' (#4) from worktree-magical-cooking-book into main 2026-07-20 07:26:18 +00:00
beatzaplentyandClaude Sonnet 5 75f1342339 Declaratively trust nix-cache's SSH host key on remote-builder clients
Distributed builds failed with "Host key verification failed" on any
client that had never manually SSH'd to nix-cache before, since
nothing populated root's known_hosts for it. Wire nix-cache's host
public key into programs.ssh.knownHosts via a new vars.nixCacheHostKey
so every client picks it up automatically on rebuild.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 17:21:48 +10:00
beatzaplenty 36ba99c9a1 Merge pull request 'Add buildImage shell function for building lxc-* tarballs with host keys' (#3) from worktree-fizzy-juggling-sedgewick into main
Reviewed-on: #3
2026-07-20 07:13:54 +00:00
beatzaplentyandClaude Sonnet 5 0cd8f15b48 Add buildImage shell function for building lxc-* tarballs with host keys
lxc-* hosts need NIXOS_HOST_KEYS_DIR + --impure to bake in a pre-seeded
SSH host key, otherwise sops-nix's .sops.yaml recipient never matches
and every secret permanently fails to decrypt on first boot. That
invocation is easy to forget, so wrap it as `buildImage <flake-target>`
alongside the existing Switch-nix/Test-nix helpers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 17:12:16 +10:00
beatzaplenty 8613b93fa8 Merge pull request 'Fix nix_extra_opts falsely reporting nix-cache's SSH remote builder down' (#2) from fix-nix-cache-probe-retry into main
Reviewed-on: #2
2026-07-20 07:02:14 +00:00
beatzaplentyandClaude Sonnet 5 20f9475a7d Fix nix_extra_opts falsely reporting nix-cache's SSH remote builder down
The reachability check used `cat < /dev/tcp/${NIX_CACHE_HOST}/22`, which
blocks forever reading for EOF that never comes -- sshd sends its banner
and then holds the connection open waiting for the client to speak next.
Every single check hit the 3s timeout and reported "unreachable"
unconditionally, regardless of whether the remote builder was actually up.
Confirmed live: a plain TCP connect (`exec 3<>/dev/tcp/...`, no read)
returns in ~60ms against a healthy nix-cache instead of always timing out.

Fixing that exposed a second, previously-dormant bug: `printf -v
NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"` on a genuinely empty NIX_OPTS array
still runs one format pass and yields the literal `'' ` rather than an
empty string. A subprocess (e.g. sync-host-keys.sh) reusing this
process's decision via `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` then rebuilt
a 1-element array holding an empty string instead of a 0-element array,
which broke `nix-shell "${NIX_OPTS[@]}" -p <pkg>` with a bogus positional
argument the moment NIX_OPTS was legitimately empty (nix-cache reachable)
-- something the first bug had made impossible to ever hit before.

Also adds a couple of retries (1s apart) to both checks as a secondary
safety net against genuine multi-second blips, on top of fixing the
checks themselves.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:57:43 +10:00
beatzaplenty 6babb3eec5 Merge pull request 'Fix create-proxmox-resource.sh --dry-run hiding nix-cache probe results' (#1) from worktree-starry-painting-whistle into main
Reviewed-on: #1
2026-07-20 06:50:40 +00:00
12 changed files with 214 additions and 75 deletions
+1 -1
View File
@@ -7,7 +7,7 @@ servers and workstation.
The flake exposes NixOS configurations named `<platform>-<buildtype>` The flake exposes NixOS configurations named `<platform>-<buildtype>`
(platforms: `linode`, `proxmox`, `lxc`; build types: `minimal`, `nix-cache`, (platforms: `linode`, `proxmox`, `lxc`; build types: `minimal`, `nix-cache`,
`server`, `docker`, `gui`, `pxe-boot`), generated from `modules/platforms/*` `server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`), generated from `modules/platforms/*`
and `modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not and `modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not
every combination is built — `pxe-boot` has no `linode` variant. See every combination is built — `pxe-boot` has no `linode` variant. See
`README.md` for the full current target list; treat `flake.nix` as the `README.md` for the full current target list; treat `flake.nix` as the
+8 -6
View File
@@ -62,8 +62,9 @@ There is no test suite — "correctness" here means the flake evaluates and
sweeps: after editing one or two hosts/modules, evaluate just the sweeps: after editing one or two hosts/modules, evaluate just the
`nixosConfigurations.<host>` you touched (plus any `config.system.build.tarball` `nixosConfigurations.<host>` you touched (plus any `config.system.build.tarball`
/`diskoImagesScript`/package output affected) rather than looping over every /`diskoImagesScript`/package output affected) rather than looping over every
host — `codex-maintenance.sh` evaluates 18 hosts plus every package/tarball/ host — `codex-maintenance.sh` evaluates every `nixosConfigurations` host plus
image variant now and is slow to run after each small change. Reserve a full every package/tarball/image variant and is slow to run after each small
change. Reserve a full
`codex-maintenance.sh` run for changes that plausibly affect every host `codex-maintenance.sh` run for changes that plausibly affect every host
(`modules/common/*`, `flake.nix`, `variables.nix`) or as a final check before (`modules/common/*`, `flake.nix`, `variables.nix`) or as a final check before
committing. This is a session-workflow preference only — it does not apply to committing. This is a session-workflow preference only — it does not apply to
@@ -93,9 +94,10 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
(`--force-rebuild` to skip that and always rebuild), and probes (`--force-rebuild` to skip that and always rebuild), and probes
nix-cache's substituter/remote-builder reachability once up front rather nix-cache's substituter/remote-builder reachability once up front rather
than letting every `nix build` call retry against it individually. than letting every `nix build` call retry against it individually.
Refuses to create a target whose host identity already has a real Refuses to create a target whose host identity already exists live on
deployment elsewhere (`variables.nix`'s `deployedTargets`) unless the node (checked directly via `qm`/`pct`, not any file in this repo)
`--allow-duplicate-host` is passed. `--dry-run` throughout both modes. unless `--allow-duplicate-host` is passed. `--dry-run` throughout both
modes.
- `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge, - `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
default cores/memory) sourced by `create-proxmox-resource.sh`. Add new default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
cross-script config here instead of duplicating it per-script. cross-script config here instead of duplicating it per-script.
@@ -163,7 +165,7 @@ removing a host.
`vzdump` backup-archive metadata this doesn't have), no install step — `vzdump` backup-archive metadata this doesn't have), no install step —
see `docs/auto-installer.md`. see `docs/auto-installer.md`.
- `modules/build-types/*.nix` — what a system is for: - `modules/build-types/*.nix` — what a system is for:
minimal/server/docker/gui/pxe-boot/nix-cache. minimal/server/docker/gui/pxe-boot/nix-cache/tailscale-exit-node.
- `modules/common/configuration.nix` — base NixOS config imported by every - `modules/common/configuration.nix` — base NixOS config imported by every
host: locale, users, nix settings, git. host: locale, users, nix settings, git.
- `modules/common/home.nix` / `hosts/nixos/home.nix` — Home Manager config for - `modules/common/home.nix` / `hosts/nixos/home.nix` — Home Manager config for
+15 -12
View File
@@ -18,20 +18,23 @@ list:
| Target | Purpose | | Target | Purpose |
| --- | --- | | --- | --- |
| `linode-minimal` | Minimal NixOS host profile on a Linode VPS (real, deployed) | | `linode-minimal` | Minimal NixOS host profile on a Linode VPS |
| `proxmox-minimal` | Minimal NixOS host profile on Proxmox (real, deployed — previously the flat `nix-minimal` target) | | `proxmox-minimal` | Minimal NixOS host profile on Proxmox — previously the flat `nix-minimal` target |
| `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container | | `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container |
| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder (`proxmox-nix-cache` is the real, deployed one — previously the flat `nix-cache` target) | | `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder — previously the flat `nix-cache` target |
| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host (`proxmox-server` is the real, deployed one — previously the flat `server` target) | | `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host — previously the flat `server` target |
| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack (`proxmox-docker` is the real, deployed one — previously the flat `docker` target) | | `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack — previously the flat `docker` target |
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation (`proxmox-gui` is the real, deployed one — previously the flat `nixos` target) | | `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target |
| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host (`proxmox-pxe-boot` is the real, deployed one — previously the flat `pxe-boot` target) | | `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host — previously the flat `pxe-boot` target |
| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node (no deployed target yet; `lxc-tailscale-exit-node` is the one planned for actual use) | | `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node |
The "(real, deployed)" targets above are also tracked machine-readably in Which variant of a given buildtype is actually deployed isn't tracked
`variables.nix`'s `deployedTargets` — keep both in sync when a deployment anywhere in this repo — that's live infrastructure state, not something a
changes. `scripts/create-proxmox-resource.sh` reads that list to refuse committed file can keep accurate, and it changes independently of the code.
creating a same-identity duplicate of an already-deployed host by accident. Check the Proxmox node itself, or `/etc/flake-target` on a running host (see
below), if you need to know what's really out there right now.
`scripts/create-proxmox-resource.sh`'s duplicate-host guard works the same
way: it checks the Proxmox node directly rather than any file here.
Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
(hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay (hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay
+1 -1
View File
@@ -19,7 +19,7 @@ see "LXC hosts" immediately below for why those are different.**
## LXC hosts ## LXC hosts
`lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`, `lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`,
`lxc-gui`, `lxc-pxe-boot`) are **not** installed via `auto-install.sh` — the `lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-exit-node`) are **not** installed via `auto-install.sh` — the
interactive menu deliberately excludes them. Don't try to select one there; interactive menu deliberately excludes them. Don't try to select one there;
`nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw `nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw
disk to partition) and then refuse to touch the filesystem it's currently disk to partition) and then refuse to touch the filesystem it's currently
+9
View File
@@ -59,6 +59,15 @@ On `nix-cache`, install the matching public key used by `nixremote` authorized k
The committed `nixremote` authorized keys are public SSH keys only. Keep the The committed `nixremote` authorized keys are public SSH keys only. Keep the
matching private keys on client hosts and out of the repository. matching private keys on client hosts and out of the repository.
nix-cache's own SSH *host* key is trusted declaratively via
`programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
sourced from `vars.nixCacheHostKey` (`variables.nix`) — every client rebuild
picks it up automatically, so distributed builds don't fail with "Host key
verification failed" on a client that has never manually SSH'd to nix-cache
before. If nix-cache's host key is ever rotated or the host rebuilt from
scratch, update `vars.nixCacheHostKey` to match its new
`/etc/ssh/ssh_host_ed25519_key.pub`.
## Manual verification ## Manual verification
After deployment: After deployment:
+8
View File
@@ -1,5 +1,13 @@
# Spec: Refactor Flake Targets into Platform × Build-Type Matrix # Spec: Refactor Flake Targets into Platform × Build-Type Matrix
**Status: implemented.** `flake.nix`'s `generatedTargets`/`mkTarget` and
`modules/platforms/*`/`modules/build-types/*` are the result of this spec —
kept here for historical rationale only (referenced from `CLAUDE.md`'s
"Composition pattern" section), not as an active or open plan. The "Open
Questions" below were resolved during implementation; don't treat them as
outstanding. A `tailscale-exit-node` build type was added later, beyond this
spec's original scope.
## Context ## Context
The flake at `~/nixos` currently defines these output targets (flat, ad-hoc naming): The flake at `~/nixos` currently defines these output targets (flat, ad-hoc naming):
+21
View File
@@ -17,6 +17,26 @@ let
--refresh \ --refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target) --flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
''; '';
# lxc-* hosts pre-seed their SSH host key at build time (see
# modules/platforms/lxc.nix) so sops-nix's .sops.yaml recipient matches on
# first boot -- without it, secrets permanently fail to decrypt (see that
# file's comment for the confirmed failure). That requires --impure plus
# NIXOS_HOST_KEYS_DIR pointing at the repo's host-keys/ dir, same pattern
# docs/auto-installer.md uses for the installer ISO. A function, not a
# shellAlias, since the target name has to interpolate into the middle of
# the flake attribute path, not just append after it. Must be run from the
# repo root, same as every other host-keys/ command in this repo.
buildImageFn = ''
buildImage() {
if [ -z "$1" ]; then
echo "usage: buildImage <flake-target> (e.g. lxc-docker)" >&2
return 1
fi
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
".#nixosConfigurations.$1.config.system.build.tarball"
}
'';
in in
{ {
programs.bash = { programs.bash = {
@@ -25,5 +45,6 @@ in
"Switch-nix" = mySwitchCmd; "Switch-nix" = mySwitchCmd;
"Test-nix" = myTestCmd; "Test-nix" = myTestCmd;
}; };
initExtra = buildImageFn;
}; };
} }
@@ -5,6 +5,14 @@
# sudo install -d -m 0700 /root/.ssh # sudo install -d -m 0700 /root/.ssh
# sudo install -m 0600 ./nixremote /root/.ssh/nixremote # sudo install -m 0600 ./nixremote /root/.ssh/nixremote
# sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version # sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
# Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
# can connect the first time without a manual ssh-keyscan/known_hosts
# step on every new client.
programs.ssh.knownHosts.${vars.nixCacheHost} = {
hostNames = [ vars.nixCacheHost ];
publicKey = vars.nixCacheHostKey;
};
nix = { nix = {
distributedBuilds = true; distributedBuilds = true;
+22 -10
View File
@@ -122,13 +122,25 @@ Add a pre-commit hook (or a `nix flake check` step) running `gitleaks protect --
## Definition of done ## Definition of done
- [ ] Milestone 1 inventory complete and reviewed **Status as of 2026-07-20:** Milestones 13 are done — sops-nix is fully
- [ ] All hosts have per-host age keys; admin key backed up outside the repo wired (`.sops.yaml`, `secrets/*.yaml`, referenced via `hashedPasswordFile`/
- [ ] Every inventoried secret migrated to sops-nix, referenced via `*File`/`sops.secrets.*.path`, nothing plaintext in the working tree `*File`/`sops.secrets.*.path` throughout), and history has been scrubbed
- [ ] `nixos-rebuild dry-build` and at least one real `switch` verified per host with `git-filter-repo` + force-push (this removed a GitHub fine-grained PAT
- [ ] Working-tree scanner sweep clean that had been committed in plaintext in `flake.nix`/`common/home.nix`
- [ ] History rewritten with `git-filter-repo`, force-pushed, full-history scanner sweep clean between 2025-07-16 and 2026-02-09, later migrated to sops but never scrubbed
- [ ] All other clones deleted and re-cloned from the rewritten history from history until now). **Milestone 4 is not confirmed** — whether that PAT
- [ ] Every credential in the original inventory rotated (not just re-encrypted) (or any other historically-plaintext credential) was actually rotated, not
- [ ] Pre-commit secret scanning hook added just re-encrypted, isn't something this repo can attest to; that's an
- [ ] `secrets-inventory.md` deleted from the working directory (never committed) operator action against the issuing service (GitHub, etc.), not a repo
change. Do that before considering this fully closed.
- [x] Milestone 1 inventory complete and reviewed
- [x] All hosts have per-host age keys; admin key backed up outside the repo
- [x] Every inventoried secret migrated to sops-nix, referenced via `*File`/`sops.secrets.*.path`, nothing plaintext in the working tree
- [x] `nixos-rebuild dry-build` and at least one real `switch` verified per host
- [x] Working-tree scanner sweep clean
- [x] History rewritten with `git-filter-repo`, force-pushed, full-history scanner sweep clean
- [ ] All other clones deleted and re-cloned from the rewritten history — every clone that existed before 2026-07-20's rewrite (any other machine, WSL instance, or CI checkout) needs this
- [ ] Every credential in the original inventory rotated (not just re-encrypted) — **the GitHub PAT found in history specifically still needs this**
- [x] Pre-commit secret scanning hook added (`.githooks/pre-commit`, `gitleaks protect --staged`)
- [x] `secrets-inventory.md` deleted from the working directory (never committed)
+57 -21
View File
@@ -10,7 +10,9 @@
# #
# SAFETY: # SAFETY:
# - The default (create) mode only ever creates a NEW resource -- it # - The default (create) mode only ever creates a NEW resource -- it
# refuses to run if the target VMID already exists on the node. # refuses to run if the target VMID already exists on the node, or if
# a VM/CT identified as --host already exists under any other VMID
# (checked live against the node; --allow-duplicate-host overrides).
# - --modify only ever touches a resource you name explicitly via # - --modify only ever touches a resource you name explicitly via
# --vmid, shows exactly what will change first, and (outside # --vmid, shows exactly what will change first, and (outside
# --dry-run) always requires typing that VMID back to confirm before # --dry-run) always requires typing that VMID back to confirm before
@@ -56,10 +58,11 @@ Create mode (default):
--force-rebuild Skip the "does the node already have this --force-rebuild Skip the "does the node already have this
image" check -- always build fresh and image" check -- always build fresh and
overwrite what's there. overwrite what's there.
--allow-duplicate-host Required if --host already has a real --allow-duplicate-host Required if a VM/CT identified as --host
deployment elsewhere (variables.nix's already exists on the node (checked live via
deployedTargets) -- otherwise refused, since qm/pct, not any file in this repo) --
it'd share that host's hostName/hostId. otherwise refused, since it'd share that
host's hostName/hostId.
Modify mode (reconfigure an EXISTING resource -- requires --modify): Modify mode (reconfigure an EXISTING resource -- requires --modify):
--modify Switch to modify mode. --modify Switch to modify mode.
@@ -286,25 +289,58 @@ fi
# feeds straight into the guest's real hostname) disagree with host.nix. # feeds straight into the guest's real hostname) disagree with host.nix.
[[ -z "$name" ]] && name="$host" [[ -z "$name" ]] && name="$host"
# --- refuse to duplicate a host that's already really deployed ---------- # --- refuse to duplicate a host that's already live on the node ---------
# Checked by hostName, not exact flake target: proxmox-server being # Queries the node itself (qm/pct's own name/hostname config), not any
# deployed also blocks --type lxc --host server, since both would carry # static list in this repo -- a file can't track whether a resource still
# the same hosts/server/host.nix identity (hostName, hostId). # actually exists, and this used to be checked against variables.nix's
if [[ "$allow_duplicate_host" -eq 0 ]]; then # deployedTargets, which drifted stale (it kept naming a VM as "the real
deployed_targets_json="$(nix eval --json --no-use-registries --no-accept-flake-config \ # deployment" well after that VM had been destroyed, blocking its own
--file "${repo_root}/variables.nix" deployedTargets)" # redeploy) until that list was dropped in favour of this live check. This
for dt in $(echo "$deployed_targets_json" | jq -r '.[]'); do # only catches guests identified with the default --name (== --host, what
dt_hostname="$(nix eval --raw --no-use-registries --no-accept-flake-config \ # this script itself always uses unless --name is overridden) -- a guest
"${repo_root}#nixosConfigurations.${dt}.config.networking.hostName" 2>/dev/null || true)" # manually renamed on the node afterwards wouldn't match, but nothing here
if [[ "$dt_hostname" == "$host" ]]; then # creates guests that way.
echo "ERROR: '${host}' already has a real deployment (${dt}, per variables.nix's" >&2 if [[ "$allow_duplicate_host" -eq 1 ]]; then
echo "deployedTargets). Creating ${flake_target} would share its hostName/hostId --" >&2 echo
echo "refusing by default. Pass --allow-duplicate-host if you really mean to spin" >&2 echo "--allow-duplicate-host: skipping the check for an existing '${host}' on ${node}."
echo "up a separate test instance of this host (it'll still get its own distinct" >&2 elif [[ "$dry_run" -eq 1 ]]; then
echo "sops key and VMID, never touching ${dt})." >&2 echo
echo "[dry-run] would check ${node} for an existing VM/CT identified as '${host}'"
else
echo
echo "==> Checking ${node} for an existing VM/CT identified as '${host}'..."
ssh_check_status=0
existing="$(ssh "$ssh_target" bash -s -- "$host" <<'REMOTE_SCRIPT'
target="$1"
for id in $(qm list 2>/dev/null | awk 'NR>1{print $1}'); do
n="$(qm config "$id" 2>/dev/null | grep -oP '^name:\s*\K\S+' || true)"
[[ "$n" == "$target" ]] && echo "vm ${id} ${n}"
done
for id in $(pct list 2>/dev/null | awk 'NR>1{print $1}'); do
n="$(pct config "$id" 2>/dev/null | grep -oP '^hostname:\s*\K\S+' || true)"
[[ "$n" == "$target" ]] && echo "lxc ${id} ${n}"
done
exit 0
REMOTE_SCRIPT
)" || ssh_check_status=$?
if [[ "$ssh_check_status" -ne 0 ]]; then
echo "ERROR: couldn't reach ${node} (ssh exited ${ssh_check_status}) to check for an" >&2
echo "existing '${host}' resource -- refusing to guess. Fix connectivity and retry," >&2
echo "or pass --allow-duplicate-host if you're sure none exists (this skips the" >&2
echo "check entirely)." >&2
exit 1 exit 1
fi fi
if [[ -n "$existing" ]]; then
echo "ERROR: '${host}' already exists on ${node}:" >&2
echo "$existing" | while read -r kind id n; do
echo " - ${kind} VMID ${id} (${n})" >&2
done done
echo "Refusing to create a second resource sharing this identity. Pass" >&2
echo "--allow-duplicate-host to create one anyway (it gets its own distinct" >&2
echo "sops key and VMID -- the existing resource(s) above are left untouched)," >&2
echo "or use --modify to reconfigure the existing one instead." >&2
exit 1
fi
fi fi
echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '${name}'" echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '${name}'"
+49 -2
View File
@@ -88,13 +88,60 @@ nix_extra_opts() {
fi fi
export NIX_EXTRA_OPTS_DECIDED=1 export NIX_EXTRA_OPTS_DECIDED=1
NIX_OPTS=() NIX_OPTS=()
if ! curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
# Retry a couple of times, 1s apart, before believing either check --
# belt-and-suspenders against a genuine multi-second blip (nix-cache
# restarting), on top of the fix below. Worst case (~11s total, host
# genuinely gone) is still nowhere near the 15s+ *per lookup* nix's own
# substituter retries would cost if this check didn't exist at all.
local attempt cache_up=0 builder_up=0
for attempt in 1 2 3; do
if curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
cache_up=1
break
fi
[[ "$attempt" -lt 3 ]] && sleep 1
done
if [[ "$cache_up" -eq 0 ]]; then
echo "nix-cache (http://${NIX_CACHE_HOST}) is unreachable -- skipping it (substituter + remote builder) for the rest of this run." >&2 echo "nix-cache (http://${NIX_CACHE_HOST}) is unreachable -- skipping it (substituter + remote builder) for the rest of this run." >&2
NIX_OPTS=(--option substituters "https://cache.nixos.org/" --builders "") NIX_OPTS=(--option substituters "https://cache.nixos.org/" --builders "")
elif ! timeout 3 bash -c "cat < /dev/tcp/${NIX_CACHE_HOST}/22" >/dev/null 2>&1; then else
for attempt in 1 2 3; do
# `exec 3<>/dev/tcp/...` just opens the fd and returns -- it does NOT
# read from it. Confirmed live this is load-bearing, not stylistic:
# the previous `cat < /dev/tcp/.../22` blocked forever and always hit
# the timeout even against a perfectly healthy nix-cache, because
# sshd sends its banner and then holds the connection open waiting
# for the client to speak next -- `cat` never sees EOF, so this
# check reported "unreachable" unconditionally, 100% of the time,
# regardless of whether the remote builder was actually up.
if timeout 3 bash -c "exec 3<>/dev/tcp/${NIX_CACHE_HOST}/22" 2>/dev/null; then
builder_up=1
break
fi
[[ "$attempt" -lt 3 ]] && sleep 1
done
if [[ "$builder_up" -eq 0 ]]; then
echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2 echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
NIX_OPTS=(--builders "") NIX_OPTS=(--builders "")
fi fi
fi
# `printf '%q '` with a genuinely empty NIX_OPTS still runs one format
# pass over a missing argument and yields the literal `'' ` rather than
# an empty string (confirmed live) -- a subprocess that later does
# `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` (the branch above, for e.g.
# sync-host-keys.sh reusing this process's decision) would then rebuild
# a 1-element array holding an empty string instead of a 0-element
# array, and `nix-shell "${NIX_OPTS[@]}" -p <pkg>` chokes on that stray
# element as a bogus positional argument. Guard the empty case
# explicitly so nix-cache being reachable (NIX_OPTS legitimately empty)
# round-trips as truly empty instead.
if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}" printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
else
NIX_EXTRA_OPTS=""
fi
export NIX_EXTRA_OPTS export NIX_EXTRA_OPTS
} }
+9 -16
View File
@@ -19,6 +19,15 @@
remoteBuilderUser = "nixremote"; # remote builder SSH user remoteBuilderUser = "nixremote"; # remote builder SSH user
# nix-cache's own SSH host public key (not a secret — the private half
# never leaves the host). Wired into every client's
# programs.ssh.knownHosts by modules/nix-cache/remote-builder-client.nix
# so distributed builds don't hit "Host key verification failed" on a
# fresh client that has never manually ssh'd to nix-cache before. Update
# this if nix-cache's host key is ever rotated or the host is rebuilt
# from scratch.
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
# Public keys authorized to SSH in as remoteBuilderUser on the nix-cache # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
# host (modules/nix-cache/server.nix) — one per client host that's allowed # host (modules/nix-cache/server.nix) — one per client host that's allowed
# to use it as a distributed builder. # to use it as a distributed builder.
@@ -147,20 +156,4 @@
keep = 20; # number of rotated logs to retain before deleting the oldest keep = 20; # number of rotated logs to retain before deleting the oldest
}; };
# Flake targets with a real, currently-running deployment somewhere —
# matches README.md's Hosts table "(real, deployed)" annotations; update
# both together. Not consumed by any NixOS module (nothing in the actual
# system config should behave differently because of this) — it's read
# by scripts/create-proxmox-resource.sh to refuse creating a same-identity
# duplicate of an already-deployed host (shared hostName/hostId) unless
# you explicitly pass --allow-duplicate-host.
deployedTargets = [
"linode-minimal"
"proxmox-minimal"
"proxmox-nix-cache"
"proxmox-server"
"proxmox-docker"
"proxmox-gui"
"proxmox-pxe-boot"
];
} }