From 87873300e1c3e04025946ce5be875f8a3337aaad Mon Sep 17 00:00:00 2001 From: beatzaplenty Date: Tue, 21 Jul 2026 19:57:15 +0000 Subject: [PATCH 1/3] Add pve-test.sweet.home as a second Proxmox target pve1.sweet.home is production; scripts/env.sh now also defines PVE_TEST_HOST for a separate sandbox node, individually targetable via --node/PROXMOX_HOST. Tooling defaults are unchanged (still pve1) -- the new restriction (Claude defaults to pve-test unless explicitly told to use pve1) is documented as policy in CLAUDE.md, not enforced in the scripts. Co-Authored-By: Claude Sonnet 5 --- CLAUDE.md | 89 ++++++++++++++++------ README.md | 5 ++ scripts/env.sh | 30 +++++--- scripts/proxmox/create-proxmox-resource.sh | 15 +++- 4 files changed, 104 insertions(+), 35 deletions(-) diff --git a/CLAUDE.md b/CLAUDE.md index 68931a9..cdc1c79 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -31,32 +31,73 @@ machines when deployed. the auto-installer (see `docs/auto-installer.md`). Never commit its contents; if `git status` ever shows it as trackable, something is wrong. -### `pve.sweet.home` (the Proxmox node) +### Two Proxmox nodes: `pve1.sweet.home` (production) and `pve-test.sweet.home` (sandbox) -`pve.sweet.home` (`PROXMOX_HOST` in `scripts/env.sh`) is a real, live Proxmox -node hosting production VMs/containers — not a sandbox. +There are two SSH-reachable Proxmox nodes on the LAN, both defined in +`scripts/env.sh` (`PVE1_HOST` / `PVE_TEST_HOST`), individually targetable +via `scripts/proxmox/create-proxmox-resource.sh --node ` or by +overriding `PROXMOX_HOST`. `PROXMOX_HOST` itself still defaults to +`PVE1_HOST` (production) — that default, and every other script behavior, +is unchanged from before `pve-test` existed; the only thing new is that +`pve-test` can now be reached at all. They are **not interchangeable** — +one is real production infrastructure, the other exists specifically so +there's somewhere safe to test. The restriction below is a policy for +Claude specifically, not a change to the tooling's own default or +anything the operator needs to opt into. -- **Read-only for existing state.** You may SSH in (or use `pvesm`, `qm - list`, `pct list`, `qm config`, `pct config`, the Proxmox API, etc.) to - inspect the node's config, storage, and any existing VM/container — - including ones this repo didn't create. Never modify, stop, restart, - delete, or reconfigure anything that already exists there (`qm set`, - `pct set`, `qm destroy`, `pct destroy`, `qm stop`, `pct stop`, snapshot - operations, storage changes, etc.) without the operator's explicit - go-ahead. -- **Test VMs/containers are allowed, but must be torn down.** You may create - a scratch VM or container on the node (e.g. via - `scripts/proxmox/create-proxmox-resource.sh` or raw `qm`/`pct create`) to - validate something. Anything you create this way must be destroyed again - in the same session, before ending the task — never leave a test - resource running on the node. Use a VMID/name that's obviously scratch - (and doesn't collide with a real flake target) so it's unambiguous what's - safe to remove. -- **No changes to production, ever, from an agent session** — this covers - both the node itself (Proxmox host config, storage pools, networking) and - any pre-existing guest. Anything beyond inspecting and tearing down your - own scratch resources is the operator's call to make manually, same as - the deploy commands above. +#### `pve1.sweet.home` (production — off-limits to Claude) + +A real, live Proxmox node hosting production VMs/containers — not a +sandbox, and not Claude's to touch by default. + +- **Off-limits at all times unless the operator has given explicit, + same-session instructions to act on this specific host.** That + authorization is scoped to the task it was given for — don't carry it + forward to unrelated later work in the same conversation, and never + assume it from a previous session. +- **Read-only for existing state is always fine, authorization or not.** + You may SSH in (or use `pvesm`, `qm list`, `pct list`, `qm config`, `pct + config`, the Proxmox API, etc.) to inspect the node's config, storage, + and any existing VM/container — including ones this repo didn't create. +- **Never** modify, stop, restart, delete, reconfigure, or create anything + on this node (`qm set`, `pct set`, `qm destroy`, `pct destroy`, `qm + stop`, `pct stop`, `qm create`, `pct create`, snapshot operations, + storage changes, etc.) — including scratch/test resources — without + that explicit go-ahead. Use `pve-test.sweet.home` for anything + exploratory instead; it exists precisely so `pve1` never has to be the + answer to "where do I test this." +- **This is a Claude-specific policy, not something the scripts enforce.** + `scripts/env.sh`/`create-proxmox-resource.sh` default to `pve1` exactly + as they did before `pve-test` existed, with no extra flag or prompt + required — that's deliberate, so the operator's own existing workflows + don't change. Claude, however, must never rely on that default: every + Proxmox action Claude takes on its own initiative — not explicitly + pointed at `pve1` by the operator this session — targets `pve-test` + instead (e.g. `--node "$PVE_TEST_HOST"`, or `PROXMOX_HOST=$PVE_TEST_HOST`). + Claude's own default is `pve-test`, full stop, regardless of what the + tooling's own unqualified default happens to be. + +#### `pve-test.sweet.home` (sandbox — Claude's default target) + +A separate Proxmox node set aside for testing. The *tooling's* default is +still production (`PROXMOX_HOST` → `PVE1_HOST`, see above) — but +**Claude's own default is this node**: absent an explicit, same-session +instruction to use `pve1`, every Proxmox action Claude initiates targets +`pve-test`. Once targeted, it's safe to create, interrogate, and destroy +resources on without asking first. + +- **Test VMs/containers are allowed, but must be torn down.** Create a + scratch VM or container here (e.g. via + `scripts/proxmox/create-proxmox-resource.sh` or raw `qm`/`pct create`) + to validate something. Anything created this way must be destroyed + again in the same session, before ending the task — never leave a test + resource running. Use a VMID/name that's obviously scratch (and doesn't + collide with a real flake target) so it's unambiguous what's safe to + remove. +- **Node-level config is still not yours to change.** Creating/destroying + your own scratch guests is fine; Proxmox host config, storage pools, and + networking on `pve-test` itself are still the operator's call to make + manually, same as on `pve1`. ## Commands diff --git a/README.md b/README.md index 04d3d2b..8747b0e 100644 --- a/README.md +++ b/README.md @@ -36,6 +36,11 @@ Check the Proxmox node itself, or `/etc/flake-target` on a running host (see below), if you need to know what's really out there right now. `scripts/proxmox/create-proxmox-resource.sh`'s duplicate-host guard works the same way: it checks the Proxmox node directly rather than any file here. +Real, production deployments live on `pve1.sweet.home`; there's a second +node, `pve-test.sweet.home`, set aside purely for scratch/test resources — +see `scripts/env.sh` (`PVE1_HOST` / `PVE_TEST_HOST`, and the +`--node`/`PROXMOX_HOST` targeting they feed into) and CLAUDE.md's Proxmox +section for which is which. Each buildtype's `hosts//host.nix` carries the per-machine identity (hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay diff --git a/scripts/env.sh b/scripts/env.sh index dc864d6..76a7ec2 100755 --- a/scripts/env.sh +++ b/scripts/env.sh @@ -6,12 +6,22 @@ # environment (e.g. PROXMOX_STORAGE=tank-nvme ./scripts/proxmox/create-proxmox-resource.sh ...) # since each one only sets a default if unset. -# SSH-reachable Proxmox node that scripts/proxmox/create-proxmox-resource.sh runs -# pct/qm on. Matches the Proxmox web UI hostname already used in -# hosts/nixos/home.nix's desktop shortcuts (pve. from -# variables.nix) -- change this if that's not actually reachable over SSH, -# or if you're targeting a different node in a multi-node cluster. -: "${PROXMOX_HOST:=pve.sweet.home}" +# Two SSH-reachable Proxmox nodes exist on the LAN: +# - pve1.sweet.home -- production. Real, live VMs/containers. +# - pve-test.sweet.home -- sandbox/test node, for scratch VMs/containers +# that don't belong on production. +# +# PROXMOX_HOST is what scripts/proxmox/create-proxmox-resource.sh actually +# targets by default -- overridable per-invocation with --node , +# or per-variable as usual (e.g. PROXMOX_HOST=$PVE_TEST_HOST). It defaults +# to production, matching this repo's behavior before pve-test existed -- +# see CLAUDE.md's "Two Proxmox nodes" section for the policy on which +# situations should target which node (in particular: Claude defaults to +# pve-test, not this variable's own default, unless explicitly told +# otherwise). +: "${PVE1_HOST:=pve1.sweet.home}" +: "${PVE_TEST_HOST:=pve-test.sweet.home}" +: "${PROXMOX_HOST:=$PVE1_HOST}" : "${PROXMOX_SSH_USER:=root}" # Where this flake repo lives on the Proxmox node itself. @@ -63,10 +73,10 @@ # nothing here forces a mount to happen. : "${PROXMOX_DEFAULT_LXC_FEATURES:=nesting=1,keyctl=1,mount=nfs;nfs4}" -export PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE PROXMOX_ISO_STORAGE \ - PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES PROXMOX_DEFAULT_MEMORY_MB \ - PROXMOX_DEFAULT_LXC_DISK_GB PROXMOX_DEFAULT_LXC_FEATURES \ - PROXMOX_REMOTE_REPO_DIR +export PVE1_HOST PVE_TEST_HOST PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE \ + PROXMOX_ISO_STORAGE PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES \ + PROXMOX_DEFAULT_MEMORY_MB PROXMOX_DEFAULT_LXC_DISK_GB \ + PROXMOX_DEFAULT_LXC_FEATURES PROXMOX_REMOTE_REPO_DIR # Matches variables.nix's nixCacheHost -- update both if it ever changes. : "${NIX_CACHE_HOST:=nix-cache}" diff --git a/scripts/proxmox/create-proxmox-resource.sh b/scripts/proxmox/create-proxmox-resource.sh index 6de5c40..79a1455 100755 --- a/scripts/proxmox/create-proxmox-resource.sh +++ b/scripts/proxmox/create-proxmox-resource.sh @@ -12,6 +12,17 @@ # copies over the locally-managed host-keys/ (gitignored, so a git pull # alone wouldn't carry it) before building. # +# --node (default: $PROXMOX_HOST, see scripts/env.sh) picks which of the two +# LAN Proxmox nodes this runs against: production, pve1.sweet.home +# ($PVE1_HOST, PROXMOX_HOST's own default), or the sandbox node, +# pve-test.sweet.home ($PVE_TEST_HOST) -- pass --node "$PVE_TEST_HOST" (or +# set PROXMOX_HOST=$PVE_TEST_HOST) to target the sandbox instead. See +# CLAUDE.md's "Two Proxmox nodes" section: an agent session should default +# to pve-test and only touch pve1 when the operator has explicitly said so +# for the current task -- this script itself doesn't enforce that (its own +# default is production, matching this repo's behavior before pve-test +# existed), it's a policy for whoever/whatever is driving it. +# # Usage: # scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --host [options] # scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --list @@ -120,7 +131,9 @@ Shared: --iso-storage (default: \$PROXMOX_ISO_STORAGE, ${PROXMOX_ISO_STORAGE}) --bridge (default: \$PROXMOX_BRIDGE, ${PROXMOX_BRIDGE}) --node Proxmox node to SSH into (default: - \$PROXMOX_HOST, ${PROXMOX_HOST}) + \$PROXMOX_HOST, ${PROXMOX_HOST} -- + production; the sandbox node is + \$PVE_TEST_HOST, ${PVE_TEST_HOST}). --dry-run Print the full plan; touch nothing local or remote, no prompts. -h, --help From 1004538f0012b1dcd629ee1b18a2acee30e193ef Mon Sep 17 00:00:00 2001 From: beatzaplenty Date: Tue, 21 Jul 2026 20:07:14 +0000 Subject: [PATCH 2/3] updated sops secrets --- .sops.yaml | 6 +++ scripts/env.sh | 2 +- secrets/common.yaml | 128 +++++++++++++++++++++++++------------------- secrets/server.yaml | 38 +++++++++---- 4 files changed, 108 insertions(+), 66 deletions(-) diff --git a/.sops.yaml b/.sops.yaml index 84be5c7..b7c1d80 100644 --- a/.sops.yaml +++ b/.sops.yaml @@ -10,6 +10,8 @@ keys: - &lxc-minimal age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep - &lxc-pxe-boot age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp - &lxc-gui age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05 + - &proxmox-server age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk + - &vm-server age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh creation_rules: # Shared across every currently-deployed host: root/nixos password hash, @@ -30,6 +32,8 @@ creation_rules: - *lxc-docker - *lxc-pxe-boot - *lxc-gui + - *proxmox-server + - *vm-server - path_regex: secrets/nix-cache\.yaml$ key_groups: @@ -43,6 +47,8 @@ creation_rules: - age: - *admin - *server + - *proxmox-server + - *vm-server - path_regex: secrets/docker\.yaml$ key_groups: diff --git a/scripts/env.sh b/scripts/env.sh index 76a7ec2..19edb8b 100755 --- a/scripts/env.sh +++ b/scripts/env.sh @@ -35,7 +35,7 @@ # Storage pool names -- Proxmox's own stock-install defaults, but this # varies a lot by setup (ZFS pool name, custom LVM-thin volume, etc.). # Verify with `pvesm status` on the node and correct these if wrong. -: "${PROXMOX_STORAGE:=local-lvm}" # VM disks / CT rootfs +: "${PROXMOX_STORAGE:=local-zfs}" # VM disks / CT rootfs : "${PROXMOX_ISO_STORAGE:=local}" # uploaded images/ISOs/CT templates : "${PROXMOX_BRIDGE:=vmbr0}" diff --git a/secrets/common.yaml b/secrets/common.yaml index 54ba946..7b70cc3 100644 --- a/secrets/common.yaml +++ b/secrets/common.yaml @@ -5,103 +5,121 @@ sops: age: - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAzdUlybVpDamVxYk94MDEv - VjNiVjJkWjZvYjN1SUhMSG5ZOS91a3FrNDJRCi9IQWh4RVdwdkhwQjd6TTBhcW9J - alhLMlArM1VvVkE5VUxOQ1ZWNmw2SmcKLS0tIFRvWklUN0xxSFdKSk1vakExQ3Rv - OWpmSGRTUHlvVysvR2N4UHRYWHgvMHMKqapmFB4ct1FTPa1hMWyylvLycUvOEFop - enZI5SV1F86HOTEhK0QbnEW3jMl4ZXOtAlwcys2oE9a783MSgVPxeg== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQUFhWVFVlVlBnNE5FTnMz + VkxkTmxpRXlzZ3pSNTVZWFUrSllsYWo0alRnCkJSc25TYktSTFFJdkQydHcxOUlj + ajhQU1ZIb1lodEpHTnVhQjJ6WEthaDQKLS0tIDJCY1E2UVBaU3BoMzhXUXlIdnMv + djZTcE1rcWNTOXFPMmFDYTVoRGo4ZTQKYy8g6pqP3VpTKDIBPbnC8NzCdDvOCKnL + 14kSrKmKlzefTrbkVyriz2Jdl2s0F374yfQQFreZ3m4AffSACCxziQ== -----END AGE ENCRYPTED FILE----- recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEc0lHazMrVUR2UUlaQ21O - T2Jha0EvUmJzVC96Wksxay9GVlEwU2lZWVRRClBQdWVqTjNTYnBrQldMeWR0dWNz - QTN6T0VtQzhpM3UwKzVjUjl0Rkp4VzQKLS0tIEg5VzhUT1VTamQzZjNYT2orMnY0 - Q1BpNzI2cVk3by9Ic3lrQ25sM05YMlkKXVofmCu7iI/my1o47p2eUzhXuP/V0NS8 - mmIerz2v2uUwoS252qSU4a2vmxPOgWM0Os1vWsYamsapofQc6TaU2Q== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBVQnpVWC9wcnIreG9GeE9U + UWhuRytkc2Flc0hyQm5yMjZnelNwaXhlWWc4CjBkWnd4cHNRRXQ0UXFkZGp4QlR1 + eW5NNnE1WFhnb054M1pac2ZidFg4Y3MKLS0tIFdTNmk2V1l2WC9rUk8yd0ZnOEJS + VkNnejVGVUZPZkorQkltVEplN2FmdTAKRY7DPP5HeFQntn2f/fXLjU6M1V6iug86 + BD09PI+T2DbIBQPotRZisw8IzHu9gY/O3+h0TccyIsXjI9wy/XPCAQ== -----END AGE ENCRYPTED FILE----- recipient: age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYZHVnTEk2SjUwdTdKdWNV - dVFPNDFvdzY2VUlzMU5nK0t0UlUxU3Ewa0g0CjJCd0ZrMjhRZldaUDNIS1pGZTZn - Z2h2MEJ6TUpna1NNdXl4R3d1S1djVDQKLS0tIFJqMlZ3b0Zzalp2N0hIc0NHMzU4 - RGRoVjdvMEkxam9DRzdYaW16enAyQ3cKUZTDqvWnmEMKHhI430coKHw3raIiD/o1 - 1BtDWOmKFFcuaF5mRx/qTUjEwU5OZWOujqLSPoVbmsbhyxsG3XQSHg== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkczFSTVhxWHlIWjhRcWlV + V2JPQXd5Wnk5R3NwWC81T3Z0MW4vYnd5S1ZZClV1NlU1Tzd6UkxPQ2M4MmhLV01G + d3VIb0RhR1RiNTZqNjlQcmg2YjdPeGsKLS0tIDQ1RTFTWGN4MnEvWkRUR3VnN204 + WVdFOXdmNC9FVFhBSGNEUUgyYWpYYzAKfdpeaFL/RrIbqpD9hNj8L7UxpmiBjE2I + go/dR2E1LLXsDPtnSuJb2EZYoFvSsjsIQQQDt+YwRv0fplRtssKdxQ== -----END AGE ENCRYPTED FILE----- recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzbXdrajlVQU5RditMckh6 - ZGhvcU01UEdMbURlcG9mZzg0RTJhcHJ4QlQwCjgyRGVXYytxUFB0WXhadHQ1YVpF - VUJpVzE2SzFPc3FHV25FQmVSQzNJTm8KLS0tIG9DazRkUkwyNHNQMnp5R3RySDBo - aGFLUFBjZjVxeWFzTXFldHVRNkNyYTAKd6eS5lks4+3SV1bFBQWyPi7OZcvRMDIc - kYD5C/vMkKlvBYyODCWJyim2xgM/nyQNcf/q4BUS3HF/RidRdJh9NQ== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB6TjhmOTJ0bUpMQWowb0hB + T3Era1loU1pMdmkxdnAvRkViekpqZWZjaWgwCkFRQXhFUy9PRVBma2JMUDhqY2F1 + VVFDRFNVbWpNaEczY1JVQUMyck9XdEkKLS0tIGpxc0tGdVFKK3FteVJKM1Fxa2ky + a21WLy9qV05hUURCTVBvcVh3cE45Z3cKXCYfXSjhApBoLbHDu2OOd57Y1zN54yy+ + WDQvz8PpMxhc1nU5Kw/cI+WmL1KvN0qQZfOx/7D4W+dy/ZDWX27TpA== -----END AGE ENCRYPTED FILE----- recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFZEhIT1REUTFtdkIzaDFY - dGU4VjU3QlpPU1Rod1dTUWhBbjc0WTR5bUF3CkNGY3F2Q251aXlGNCt5ZjFWMTE1 - RjY4bHZ1T29aK3N0SktjdUpZbSs4QVEKLS0tIDZrMWd3aFVRMEZMU1N1dlZxYTZS - U3gxWU5EVVRXUGxJVlZQZmpZRmdhbW8KJ8yD9laK2T1qn0z2uYNeI80rtMlOVi2M - qIPn/FSoWnKQ7NotSTGUtEhC+f/nXfUQrBd5fsfmja++hrevTmkqYw== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZK0FYQnBHMHZ6dlpMYTlC + WFdOWDFkRVBuY1pmdTFiUndLV3JXcndZa3pNCnJsd0tHN0FveWV6UUNQSEdpdWw5 + dWZITkxWelNIRlpKS1pnN0ZmVlQvZjAKLS0tIEUwMXdtNFdkUWdIRjlxc0owdTRr + c1o1TmptWWd1ZGxzcWJJNzJ0K25PTTAKoos5rnkyQBCm+ZuhCCaMJwqJBo1fpnsl + G74wu5vbTBG4VjVhI5KqyiuiTRU4jPcGxysECqe7AyZUBGp7ndewgw== -----END AGE ENCRYPTED FILE----- recipient: age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0N1owUGpuWWxWb2t2Mk5Z - QlZ6Nm5lcnNBY0d1NldBQXY3OThLM2UxbkY4CkZyRUl4S1JRaDhnemRIYWR1OHZx - aWliVWh3V1RBL0R3Q0RqOE1SM0VBblUKLS0tIGpaUjJxdnV0UW55T01HcFBsdHR2 - aEJ3UzJ6RklZVFRrNGMraUtGYXVZd0kKaQmzjnI6xgV7H1YcoI+gRn2IVfOLTJBY - rO43fvVtZ79c1as6CL+GoEIJY/uVA/3M4gQ0vYbDDn57vm5OOJW8jg== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTVkVXaCtGTXVkRXRZMFZ1 + ZUdXM1hRcDhaSURqcGh2eksxTUNCckk5SVJjCnhkSDRKdDFSckxUWXd4SmVxWG5p + bU5OV0hzaWR6VDFwcDY2WlY4WnN3cFEKLS0tIElOVzRCcXR4U0dhajJySUhaZGps + MW9rQk1JVDFWRnFxVzhCUkRIS09EamsK1rVidD48PqwlEWQyjF7iQWU7aBdPqQHy + z5LaSi3LvJX3rNE/+q0E8/gbZyjGpbEn3AUI5mBF64GY3IZkRxZSXQ== -----END AGE ENCRYPTED FILE----- recipient: age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMWTcwd0JLS0lybWREaU5h - NHczcEQxeUx4U09oV2k5ZXpKTGxxR3ZYd2tnCkVXR3kyTHZSSWNaUCszL3Rvakhm - bVl5bnFRVkhJRFVaY1NQYTNrd2JKMTAKLS0tIDJTT0YvUE5HUWVKRnhXY2tvMjI1 - VXNlcnJqcDM5d3poVU1SKzQyNWdpRzQKTht/ko7cy7OY0wGfza4eierYS7q/nCFH - YsG74ez3piUQ1rdJRi0e29QWz4xL3JeU9oE+tr0rMB6WvgTmuX5gwQ== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvbnJCWU9UMXJXMG92b3hm + WWNBTFpQamVWQThITmt5QUVwR3h2OHI2SlVRCjcxOEJTVkFjN0NhamFZQ1plK29w + dk5XYkYxOXQ0YkVzcVc3VnhCQWlsV0UKLS0tIHZ5cWtFZUhDKzZkOE1BK2Y5TStR + SFlDRjE4ZHpiVEJOQk5TUGNEN1B4amMKUCJ8CL8QpmRpFs83HD9TUn7NrPguuP8S + JQH/bzPorXTXJuyOKuKAZq1hK8BmiMUFksaZ03yN6YaFVIOeelEEMg== -----END AGE ENCRYPTED FILE----- recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLWmtNOVVvRW9QbU50OUxv - TUYxWXJ4cFVaMjVnVHJVK1VTZzhDd2tCMWpnCllPWnl2bjcxemF6cFp1blZnMEMy - clhqdlVzdEZTYnY3b0ZhR2hsMXRlQTgKLS0tIHEzNU9yNmhackt6R2dLT2dZSEFJ - UnJXeWVyQld6NFhXUWowUlVXaEtKVUkK2Saa8w9/SOWw1VXWkUg62ay31bSIqHWB - 8QmNafddV0WQ8oYcD8ZMf8Bm8jP9yLMUDOiVzVZiiTGWEx7EbUQV3Q== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxWHV0S2dxeHZEMDNRSkhO + ZFJHaiszd0tzYzlzd3gvbXNSQmlMVlJUNGcwCnVrM29MdFZCR1NBYnpkQ1k5VFZQ + b3Z2Q3ZGekVQZkZKWGlka3NDOHJ0R1EKLS0tIDlXTmNzUk0wVXo0UWhkd0ZvK3FI + UzJxU3RkdWs4aTZYVVkrS056bTN1ek0KgKJNz8GvynX5pK33aW9x3v6yr2Ox0LCT + GGrt+ddbKLcwpBpYjfWkFhffO330EKui73S+c/qMf8N9j6wzalOTpQ== -----END AGE ENCRYPTED FILE----- recipient: age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2OEs4dWJ5UTBoY2hHanZY - N3VNMWdGVkJHMlVublBmN2NiMUNtRnVnaXdrCnYzV3EvRzV1OXNyOFVKc2lXMFov - Mkgvc2dFdyt0MGJIMlFPckpBeDZ5c0UKLS0tIERrOGc4aWtFSTJWN1M2V0hyQ3hS - TC9HYVMyVWVBc1lHbUlqbnpsOWJobU0KkZZph2dnKwi4w+RXf7RXoHCYkXxmuLNj - rinQQlQeOwQptBn7+kRJubNjyHOlSHXhpsbV3/IkJRNMhw69knzasQ== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3ZWJlbDl1TEkvWEpBMC9Y + ejY1MnFMUExBeDhITndxNy9YUi9hU0tXM1JJCjcwNEppcDdxYzlCSVMrMExWa3A4 + cmFPSjV3LzkyMXZCUDU2QmtHRmpHRmsKLS0tIFFTdXBOaDJJTERseXlGbmdrQzhD + TWtnRFdIRXpsNkY0U1BiczNsdUk1V1kKGpndKmT8kj/oIxQuxQALfzscw+CsVmnj + cyPC3bF+tG6LcqqoKLjPSJfcIgzhnX7cAr/wwESavemLn8L/zQMe4w== -----END AGE ENCRYPTED FILE----- recipient: age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoWkljeFFBS3hhOG0relVz - V0JUK2hWaklxTGVhMWR4bytPczhjcldheno4CmZUTWszREVUdVVQazNOdXAvY0FN - bEtQSDJveHplNTE0a0FvWEFxVlEreDgKLS0tIGNKUkNVQ3BDQ3d0V1lyY0VsaHda - cDNHa3lYTkxBeVFlSGtIMVFScmxsazAK5ZyJ35/jjFRhQdG1PMOcEJ3MJf96i8DZ - AvCw9jc7Hsbs6+LoZ45K1QKoWoZmBkmAatbksXo3jnAzEHmcQqODKQ== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpQVVyY3UrSHFWSmpDdmRE + dlh5akFqVFdDVFNMSHE1eVJnZzR6YzFHSVFRCko4UE9EdXNxZzF2MW5PTTN6dEdU + ZHM1MGowcVB2Y1ZlOTVHdnNtY3diM2cKLS0tIDBYSmh5dVVPaTM3d0ErcC8wMDNB + eUpHWnZlYnJsbHZuS3pwbG15UGtwN2MKVPQA1MpjIfYAsNacoAbpvZNuAIkvx7ER + CvWBKEHUVm6m8905BXzv8MdGTAk0EyCIP3aMmYqTIYfv2k9pP0T08A== -----END AGE ENCRYPTED FILE----- recipient: age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtWGRLTCt5dXZMR1d6a3lD - Z0t1SDAwY3V3MXI1MGt5RGkrc1JHSVRUUzNZCi9UOTBTcVY3NkhFb1FFOS9vbTY2 - S1BqNmd2eWVXbnliOFJWRDY0UmhYV3MKLS0tIExYNlJxcTBidnpObHgweHlqNHhz - OXpwRHhyOGhNRXNTWDBIUVJCR0VPTkkKQZh9e7lOINL0khHS6tBehCrm+SX5Q6XE - XL4FpBsJ5+MnxTx1O/bu6/f7OvZqfnxhQv0lkKOMJnkZfPZx5D+meg== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0MTQ4MzN0bDJNL1l1bmpX + ekxDUXRNa0JHWWltZFNGTVltTFdSSE82SVZvCmdUWUdja3JIajMzY09IMUE5elox + MDdEakFJTmtkRWF2R1BGNkQ4U3grNWsKLS0tIE9hZUhkVGI1ZEpzdDhRU21EZm91 + VnJNb1kyQ05MM0RJa1lLUEtjWWxkSTAKHVAKcGcWl6LncJALRBU9RKP7ot6C6GSE + 1iZtj1SNX6wzEWrhOEnV37aQ8bKZj6u+Y/q6/vJ4qiBs78y/drdIzA== -----END AGE ENCRYPTED FILE----- recipient: age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05 + - enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBNUEVWY2NpVEU4OTZPZTZR + UzdKcktpUGJOcnJ0aEhkQlhvUWdUYkV2SkdzCm51eGJVeHJMcVRRRld0dFRCYUxr + TTN2WEhOVjRqV0FtQXowZWNTbkJneEUKLS0tIFpUazZpTUNWZUZBSFE0VDZZbkJu + SFVlUVhySnNqUENYOG9qUm5ZMDc1ZW8Kv0lY5dhnCEheM0sttfr4p7IL+EVog16T + OapUdbuXL2l7t7URzHnvfG/nbOtJIjH8a0XFsWyJChtNXpF2d/vf2g== + -----END AGE ENCRYPTED FILE----- + recipient: age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk + - enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvTzc2aExGTHFrc0ZtL3pq + YlZLbzd3MVZHZTB0VUQvSXZ4NGVsZk42c2hBCk1vbTk0Tnl3b01vbVZaMkJ5aldE + WmZBMGFjb2pjQXpYcnBxWmp0UUsrdXcKLS0tIG16SG9JbFdkbmVidCsxUnpBR3V3 + RzNOY3hIRWk4UXh6N3NrcjNSU3ZwWTgKaExY4U2s8E6ojljJ+4TU+YJhcLXyuVA1 + ROB70jQCjFvQOeo6thjQohSSUoPKhxSl1/nr4ZiGBO3/VskzihckKg== + -----END AGE ENCRYPTED FILE----- + recipient: age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh lastmodified: "2026-07-19T02:30:40Z" mac: ENC[AES256_GCM,data:UiL3VMDF6rq4Nr87KspcDx434q3tfNXeb5pwH2O+4ssNQ6xzcYDdzXBnhAY3zLBsqPMKrvHBd4Ot/gEMcq3FMIVe7Q6p9yWKpep66KZ/yWEhAlwIVhD79Oj8VS+1CHKjf25zpRdhZorp04oeFQQd9VfjJB4EE/Q1aVbwTGlpIic=,iv:i/0conaFgFia+wzNTdUL6tlSTw35HTK3Ap1Sr5RGHf8=,tag:ULbz5FllShA/JjlSRdxA0g==,type:str] unencrypted_suffix: _unencrypted diff --git a/secrets/server.yaml b/secrets/server.yaml index ec1f414..5365d88 100644 --- a/secrets/server.yaml +++ b/secrets/server.yaml @@ -3,22 +3,40 @@ sops: age: - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBURmMzN3hrSlNrUkkvVWNl - L3M1dEhWeW14N0RFNVRPci9QK1YyTFdqRVVJCk5WaWswT2NicldkYzZjbVhYU2xu - MGFsNmUzeTN2TS9wOEdvRURpVUVYZXMKLS0tIDZ6MEdPTVhCaTQ2UXFWTUFtc0pm - MFlJb0c2WXJtMGRLZEZYY0pZWWpFWm8K/mlYZIe8UC0QU+1mq3NtrtTF5b2m5hCK - +K0QiZLTKmmDcr4bRhZ32VE7R7GRwtMNnOP/mElZvPAyWyHHhRiOHg== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaYU9HR0lETDhkYXR5NWlj + L0diSE5JQU9KWjRoU3hrVjdqZ2tPNUtOdFZvCjdUVGNFbDVYa3pVSC9ZWVNORER4 + QWozUlhoSEtjTk9IRlM3VkZoYlc1RTAKLS0tIDlHS01WVWlOMWFEQU5GTVRLZVhV + VTcwekhrRHB3SlVYT2MzOW5GbE52dkkKKCWehPhpdGapdyzpll20NJUcZwvW/7X8 + KQ1EqAgI2fewnbwuIDYCleN0b0SLJNUeSV/tFKDDoTMnHWCdeD4ECg== -----END AGE ENCRYPTED FILE----- recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad - enc: | -----BEGIN AGE ENCRYPTED FILE----- - YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBURHlUL0RNMUtNallEcG5p - eG4xby8yVzdQUTBaNkl4ano0YjBMcDd0Wm5NCkZvaXNPZm9wemJkMmNSdGdOaTI4 - Z1RwUnhiRUpCMWZaeWtlSVBmNW5KOXMKLS0tIDk0R0k3ZHczTFNCWUZxSWF0M0FJ - MGlZMmtuSFYrcG1meDNMWDNqSjFxcE0KDu2dAc0gqmmPkpbpBe4YohM7rYmUwEkI - V2FUQwjlvh50svtjCVdYbx2xuq4sQLnKelk/q1onLw60FwsVfzD8sQ== + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4YndybFBTQ2p4SGZ4SDJs + a2p0eFRQOVVWcGd2a25ESW9ESGx1RWw0Zno4CnhpVVh3cGI5UjY0YmFINFFPMTh4 + b1B5SjJ3NTNvUE1QUmJjVFozY1dYS2MKLS0tIC9HZGNpOFZhZGNFZGt5blJuZXVV + SXpkRzV4d2ppV3ZQZSt1dmxYNGVFMUEKmSe9dkrmkND81Hw2/ATAmFvcmhk1tUC1 + LxxBw54IVHUqwYKgRYUYRNu+pykDT5OnFDPiskd49Xso99LY87PyiA== -----END AGE ENCRYPTED FILE----- recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf + - enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLd1FSSHVTSGdHWm8wMVQz + d0dlOEsxeGwxdHU0eUlFSUxka0ZmcVpnOTM0CndYNUVjVy92QVhNY2orQmpSQjYy + VS9KM0NUTXhuM0lCSDBZMWtISEdtWTgKLS0tIExTL2wvS3FEdVViUmRYZEFsR3R0 + YTJFM200RjF6MjNxOFA4eHRpWVhtRkUKOqBIT445HnPXrrH/qV6FIuAhAuJmSL6V + +PQopM/m3PAnK5m5Mu3cfjYfDiB8+GWTABhljfT+GbcoK7CqWLehrw== + -----END AGE ENCRYPTED FILE----- + recipient: age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk + - enc: | + -----BEGIN AGE ENCRYPTED FILE----- + YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvVXF4a2NmaW15R2VXS3FP + SDVPUGpUWTlIWjl2N2N2SXR1UlRkdllWNVRJClZDbVo4dUhrZytqZkMrYWtpOEZx + c0dSZGViN04zQ1B1WEZEWm1QM1lsejQKLS0tIE0wa0k5Rm1xZmw3OHFESkVXc25j + NlFpYTJSckQ4MVlZQ01reDlkaWY3TkEKsmQlreRhRAjVZ/q5x52FNATDF2sLhbHo + djOZigZx2rs6shqQ6It/XRJ8CiPaXZBPOt529Gwmplu+hWlAU1+l2Q== + -----END AGE ENCRYPTED FILE----- + recipient: age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh lastmodified: "2026-07-19T02:30:40Z" mac: ENC[AES256_GCM,data:rKHZjU/MH08ASTlu32HZO9uWmsBYuMCEC6M8gwVhzuWvmablnP05tS2z13XfaWaCEUXk6kmGJKuU0zu5+IKVZgamCF6DAMtxQb6bVCaLsoAm/GSqWQ5VI9eHqgnSSdN/o3ul/33Rf8iBQo4aw8FFAmDVuNz8bfAn0QefFTj0ByI=,iv:JD2gtqRinOY77etg6PUmZNovkYl1Q3F6ZvRi4x7RznQ=,tag:/5IMpWKRVt+l1luCTQE0BA==,type:str] unencrypted_suffix: _unencrypted From 12f915395791998959b59d0f461f0faebb62bce1 Mon Sep 17 00:00:00 2001 From: beatzaplenty Date: Tue, 21 Jul 2026 20:07:30 +0000 Subject: [PATCH 3/3] claude audit report --- AUDIT_REPORT.md | 150 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 150 insertions(+) create mode 100644 AUDIT_REPORT.md diff --git a/AUDIT_REPORT.md b/AUDIT_REPORT.md new file mode 100644 index 0000000..0bce167 --- /dev/null +++ b/AUDIT_REPORT.md @@ -0,0 +1,150 @@ +# Flake End-to-End Audit Report + +**Date:** 2026-07-21 +**Scope:** Full static lint/eval sweep + live build/deploy/interrogate/destroy testing of every `lxc-*` and `proxmox-*` flake target against `pve.sweet.home`, plus an audit of the operator's ability to manage the flake/secrets tooling. +**Branch:** `worktree-flake-e2e-audit` (this session's isolated worktree) + +## Executive Summary + +The flake itself is in good shape: `nixpkgs-fmt`, `statix`, and a full eval + dry-run build of every host and package are all clean. Every `lxc-*`/`proxmox-*` target's NixOS configuration builds successfully — no target has a broken derivation graph. + +The issues found are **operational, not code-level**: + +1. **pve.sweet.home is critically low on disk space** (91-95% full during this session) and cannot currently build the two largest closures (`gui`, `pxe-boot`) to completion — this actively blocks deploying/redeploying those hosts via the documented workflow. +2. **A real, reproducible secrets-decryption failure** was caught live: a stale cached container image (built before a same-day sops-key fix) boots with sshd never starting and every secret failing to decrypt. This is a **general hazard in `create-proxmox-resource.sh`'s "reuse the cached image if present" default**, not a one-off. +3. **sops key/anchor drift**: `proxmox-minimal` has a `.sops.yaml` recipient anchor with no corresponding private key anywhere in this environment; several `lxc-*`/`proxmox-*` targets have no sops registration at all yet. +4. One concrete script bug was found and **fixed in this session**: `create-proxmox-resource.sh` never enabled the QEMU guest agent channel on VMs it creates, despite the guest OS already running it. +5. A management-surface audit (of the operator's ability to run this repo day to day) found 5 process gaps, detailed below. + +Nothing here required or received a `nixos-rebuild switch/boot/test`, `nixos-install`, or any disk-formatting command — all validation was `nix build`/`nix eval`, plus disposable `pct`/`qm` create-then-destroy cycles via the repo's own `create-proxmox-resource.sh`. + +--- + +## 1. Static Analysis Results — all clean + +`bash scripts/codex-maintenance.sh --full-check --dry-run` (whole-tree sweep, not just changed files): + +| Check | Result | +|---|---| +| Secret grep | Clean — only the documented exceptions (installer's own hashed passwords, `access-tokens` comment references) | +| `nixpkgs-fmt --check` | 0/53 files would be reformatted | +| `statix` | No lint warnings | +| nix-cache host key drift check | Up to date | +| Full eval of every host's `system.build.toplevel` | All 19 `nixosConfigurations` targets evaluate cleanly | +| Dry-run build of every host + package | All succeed, no derivation errors | + +No drift, no formatting issues, no lint findings anywhere in the tree. + +--- + +## 2. Per-Target Test Results + +Legend: **LIVE** = built on pve, `pct`/`qm` create → interrogated → destroyed. **BUILD-ONLY** = `nix build` validated the config (mostly `.config.system.build.toplevel`, occasionally `.tarball`), no resource created on pve. + +| Target | Test type | Result | Notes | +|---|---|---|---| +| `lxc-docker` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT105 is already running this identity in production; `--allow-duplicate-host` would have destroyed it. | +| `lxc-minimal` | **LIVE** | ✅ PASS (after retry) | First attempt reused a stale cached tarball predating a same-day sops-key commit → activation failed, sshd never started (see Finding #2). Redeployed with `--force-rebuild`: clean boot, `systemctl is-system-running` = `running`, secrets decrypted, sshd listening, users correct. | +| `lxc-nix-cache` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — CT101 is already running this identity. First local build attempt appeared to hang on a remote-builder handoff to nix-cache; killed and retried with `--builders ""` (local-only), succeeded. | +| `lxc-gui` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Registered a fresh sops key (no prior registration existed), built successfully through the full NixOS system closure, then **failed packaging the tarball**: `No space left on device` on pve's root filesystem. Not a flake defect. | +| `lxc-pxe-boot` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Same failure as `lxc-gui` — this target additionally builds a full nested installer/netboot image (`stage-installer-artifacts.nix`), making it similarly large. Failed with the same `No space left on device` error, immediately after the gui attempt had already consumed pve's remaining headroom. | +| `lxc-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; live deploy also would have hit `boot.zfs.extraPools` trying to import a real ZFS pool that doesn't exist in an isolated test container — an expected limitation of testing this build type outside its real hardware, not a bug. | +| `lxc-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. | +| `lxc-tor-relay` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT106 already holds this identity in production. | +| `proxmox-docker` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — both CT105 *and* VM103 already hold `docker` identities. Combined `toplevel` + `diskoImagesScript` build crashed with a **Nix-internal assertion failure** (`worker.cc:360`) under this session's memory pressure (see Finding #6) — not a flake bug. Retried with `toplevel` alone: clean. | +| `proxmox-minimal` | **LIVE (attempted)** | ⚠️ BLOCKED by key drift → BUILD-ONLY | `.sops.yaml` has a registered `&proxmox-minimal` anchor but **no corresponding private key exists anywhere in this environment** — the script correctly refused to generate a mismatched replacement. Fell back to `toplevel` build: ✅ PASS. | +| `proxmox-nix-cache` | BUILD-ONLY | ✅ PASS | No sops key registered yet. | +| `proxmox-gui` | BUILD-ONLY | ⚠️ Killed after ~40min (resource-limited) | This session's local build machine has only 2GB RAM; swap filled completely (2.0/2.0GB) and the build stalled, so it was killed rather than risk destabilizing the session further. **Not a flake defect** — the equivalent `gui` NixOS configuration already proved fully buildable during the `lxc-gui` live attempt above (it built the entire system closure successfully and only failed at the pve-side tarball-packaging step due to disk space, not the config). | +| `proxmox-pxe-boot` | BUILD-ONLY | ⚠️ Killed after ~35min (resource-limited) | Was deep into building the nested installer's kernel initrd (this build type bundles a full netboot installer image via `stage-installer-artifacts.nix`) when killed to keep the audit moving. **Not a flake defect** — this target's own module logic was already effectively validated via the earlier *live* pve deploy attempt (`lxc-pxe-boot` above), which built the complete image and only failed at the final tarball-packaging step due to pve's disk space (Finding 1). | +| `proxmox-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; same ZFS-pool caveat as `lxc-server` would apply to a live deploy. | +| `proxmox-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. | + +**Not tested at all:** `linode-*` targets (not deployable to Proxmox) and `installer` (not a normal host) — both were still covered by the static eval/dry-run-build sweep above. + +--- + +## 3. Findings, Ranked by Severity + +### Finding 1 — pve.sweet.home is critically low on disk space (blocks real deployments) + +At session start: `/dev/mapper/pve-root` was **95% full, 5.3GB free** (of 94GB). After two failed large builds it recovered slightly to **91% full, 8.2GB free** (nix cleans up its own failed-build scratch space). `/nix/store` alone is 26GB; `nix-store --gc --print-dead` reports **zero** reclaimable garbage — everything currently in the store is a live GC root, so `nix-collect-garbage` won't help without first removing old roots. + +**Why it matters:** `create-proxmox-resource.sh` builds every VM/CT image **directly on pve**, not on a build machine and transferred over. With <10GB headroom, any closure approaching a few GB (the `gui` build type: full Cinnamon desktop + Firefox + LibreOffice + GIMP + VS Code + xrdp; the `pxe-boot` build type: nginx/atftpd *plus* an entire nested installer/netboot image) cannot currently be built there at all. Both `lxc-gui` and `lxc-pxe-boot` failed live with `No space left on device` during this audit. + +**Recommended action:** Expand `pve-root`'s LV, or free space by pruning old container templates in `/var/lib/vz/template/cache` (1.5GB) / old backups in `/var/lib/vz/dump` (306MB) / auditing what's pinning 26GB of `/nix/store` as live GC roots (likely `result-*` symlinks — see below). This is real production disk state; **not something this session touched or fixed** — it needs the operator's judgment on what's safe to remove. + +**Secondary, smaller finding:** every `create-proxmox-resource.sh` run leaves a `result-` symlink in the node's repo checkout as a permanent GC root (`ls /root/nixos/result-*` on pve showed 3 from this session alone: `lxc-docker`, `lxc-minimal`, `lxc-nix-cache`). These accumulate forever and pin their entire closures in the store. Consider having the script clean up its own `result-*` link after staging the built artifact (or use a temp `--out-link` under `/tmp`), so `nix-collect-garbage` can actually reclaim old build outputs. + +### Finding 2 — Stale cached images can silently ship broken secrets (reproduced live) + +`create-proxmox-resource.sh`'s default behavior is: if the node already has `.tar.xz`/`.raw` staged, **reuse it** — only `--force-rebuild` forces a fresh build. This session hit exactly the failure mode `docs/auto-installer.md` already warns about: `lxc-minimal`'s cached tarball (built 2026-07-20T15:57Z) predated a same-day sops-key fix commit (2026-07-20T17:49Z, "clean up in ailse 3"). The deployed container booted with: + +``` +sops-install-secrets: failed to decrypt '.../common.yaml': Error getting data key: 0 successful groups required, got 0 +Activation script snippet 'setupSecrets' failed (1) +``` + +— every secret permanently failed to decrypt, `sshd` never started (though the container otherwise looked "running"). This was **not a code bug**: the currently-committed `secrets/common.yaml` decrypts fine for that host's key when checked independently; the *cached artifact on pve* simply reflected an older commit's ciphertext. Redeploying with `--force-rebuild` fixed it immediately. + +**Why it matters:** this is silent and easy to trigger by accident — any operator who redeploys a host without remembering `--force-rebuild` after a secrets change gets a container that looks like it started (`pct start` succeeds, `pct status` = running) but is completely inaccessible. + +**Recommended action:** Have `create-proxmox-resource.sh` compare the cached image's build timestamp (or embed the source commit hash in the staged filename) against current HEAD, and warn (or refuse without `--force-rebuild`) if they differ — rather than silently trusting presence alone. + +### Finding 3 — sops key/anchor drift + +Two concrete instances hit live during this session: + +- **`proxmox-minimal`**: `.sops.yaml` already has a registered `&proxmox-minimal` age recipient, but this environment's `host-keys/` directory has no corresponding private key file. `sync-host-keys.sh` correctly refused to generate a replacement (it would silently mismatch whatever's already registered/deployed) — but this means **no environment currently has this host's private key**, unless it exists on some other machine that was never backed up here. +- **`lxc-gui`**, and by the same logic `lxc-server`/`lxc-tailscale-exit-node`/most `proxmox-*` targets, have **no sops registration at all yet** — expected for undeployed hosts per `docs/auto-installer.md`, but this session's live-testing needed to register `lxc-gui`'s key on the fly, which immediately hit **Finding 3b**: registering a key locally does nothing for pve's build until it's pushed to `origin/main` (pve builds via `git pull`, not from this uncommitted worktree). This is exactly gap #4 the management-surface audit (below) already flagged in the abstract — this session hit it concretely. + +**Recommended action:** for `proxmox-minimal`, decide whether to regenerate its key (destroying old-key decrypt access, if anything still holds it) or track down wherever the original private key lives and back it up here. For the general pattern, see the management-surface audit's recommendation to pre-flight-check key registration before building. + +### Finding 4 — QEMU guest agent never wired up (found and fixed this session) + +`modules/common/configuration.nix:44` sets `services.qemuGuest.enable = true` on every host — the guest-side agent daemon is correctly enabled everywhere. But `scripts/proxmox/create-proxmox-resource.sh`'s `qm create` call never passed `--agent 1`, so **Proxmox never created the virtio-serial channel** the agent needs. Every `proxmox-*` VM this script ever created was silently missing `qm guest exec`/IP-address reporting in the Proxmox UI, despite the guest daemon actually running. + +**Status: fixed in this session's worktree** (`scripts/proxmox/create-proxmox-resource.sh`, `qm create` now includes `--agent enabled=1`) — see the diff, included in the PR from this session. + +### Finding 5 — Orphaned container on pve (CT102) + +`pve.sweet.home` has a stopped LXC container, **VMID 102**, with an essentially empty config (`lock: create` and nothing else — no hostname, no rootfs, no network) — the leftover of a `pct create` that started and never finished. It predates this session (not created by any of this audit's activity) and wasn't touched. **Recommend the operator confirm it's abandoned and remove it** (`pct destroy 102 --purge 1`) — left as-is it may be someone's genuine in-progress work, so it wasn't assumed safe to delete autonomously. + +### Finding 6 — Nix-internal crash under memory pressure (tooling, not flake) + +Building `proxmox-docker`'s `toplevel` and `diskoImagesScript` together crashed with a Nix-internal assertion failure (`Assertion '!awake.empty()' failed ... worker.cc:360`, a known class of bug in Nix's multi-goal build scheduler) while this session's 2GB-RAM build container was under heavy swap pressure (1.8-2.0/2GB swap in use) from a separate concurrent build. Retrying the same target alone (no concurrency) succeeded cleanly. **Not a flake defect** — purely an artifact of this session's constrained build environment; noted for completeness since it looked alarming in isolation. + +### Finding 7 — Management-surface audit: 5 operability gaps + +A focused audit of "can the operator actually run this repo day to day" (flake, home-manager, sops, related scripts) found: + +1. **No documented recovery path if the `&admin` sops age key is lost without a backup.** `scripts/secrets/backup-admin-key.sh` exists and works but is referenced nowhere in `README.md`/`docs/` — no forcing function ensures a backup was ever taken. `rotate-admin-key.sh` requires the *old* key to re-key; there's no bootstrap-from-nothing path documented (the real fallback — deriving an age identity from any still-live host's own SSH key — isn't written down anywhere). +2. **home-manager has no standalone iteration path.** It's wired only inside `nixosConfigurations` (`flake.nix`) — no `homeConfigurations` output. The fastest real shortcut (`nix build .#nixosConfigurations..config.home-manager.users.nixos.home.activationPackage`) isn't documented anywhere, so the practical workflow is a full host rebuild to test one HM tweak. +3. **Gitea's flake-lock-update workflow pushes straight to `main` with no pre-merge validation.** `.gitea/workflows/update-flake-lock.yml` commits and pushes `nix flake update`'s result directly; `codex-maintenance.sh` only runs *after*, on the resulting push — a genuinely broken lockfile bump lands on `main` before anything catches it. (The GitHub-side workflow is safer — PR-based — but has the opposite gap: nothing alerts if the PR sits unmerged.) +4. **No pre-flight check that a build target has a registered sops key before building it.** `docs/auto-installer.md` documents the failure mode (silent, total secrets-decrypt failure) but nothing in `create-proxmox-resource.sh` refuses to proceed when it's about to build a target with no `.sops.yaml` anchor — it's on the operator to remember. This session's `lxc-gui` test hit close to this exact gap (needed the key added on the fly, mid-session). +5. **`vars.remoteBuilderAuthorizedKeys` has the same drift risk as `vars.nixCacheHostKey`, but no checker script.** `sync-nix-cache-host-key.sh --check` guards the latter; the former (and `vars.pxeServerIp`/`vars.pbsIp`) has no equivalent — a rotated/revoked client key just silently stops working with no diagnostic pointing back here. + +--- + +## 4. Action Plan (priority order) + +1. **Free up disk space on pve.sweet.home** (or expand `pve-root`). Blocking: `lxc-gui`, `proxmox-gui`, `lxc-pxe-boot`, `proxmox-pxe-boot` cannot currently be built/redeployed on this node at all. +2. **Decide on `proxmox-minimal`'s orphaned sops key**: locate the original private key and back it up here, or accept regenerating it (breaks decrypt access for whoever/whatever currently holds the old one). +3. **Merge this session's PR** (see below) to get the `--agent 1` fix and `lxc-gui`'s new sops registration onto `main` — required before `lxc-gui` can be live-redeployed with working secrets. +4. **Add a staleness guard to `create-proxmox-resource.sh`'s cache-reuse path** (Finding 2) — highest-leverage fix, since it silently produces a broken-but-"running" host. +5. **Add a pre-flight sops-anchor check to `create-proxmox-resource.sh`** (management-surface gap #4) — same root cause class as #4 above, catch it before building instead of at first boot. +6. Investigate/clean up **CT102** on pve (Finding 5) — confirm abandoned, then remove. +7. Document `backup-admin-key.sh` in `README.md`'s Security Notes and add the live-host-key bootstrap-recovery procedure to `docs/` (management-surface gap #1). +8. Add pre-push validation to the Gitea flake-lock-update workflow (management-surface gap #3). +9. Lower-priority: document the home-manager `activationPackage` shortcut (gap #2); extend `sync-nix-cache-host-key.sh`'s drift-check pattern to `remoteBuilderAuthorizedKeys` (gap #5). +10. Follow-up session: finish build-validating `proxmox-gui` and `proxmox-pxe-boot` (both killed here after 35-40min on this session's 2GB-RAM machine — not failures, just unfinished) once pve has headroom (item 1) — ideally from a machine with more RAM. `proxmox-server` and `proxmox-tailscale-exit-node` already passed build-only validation in this session, no follow-up needed. + +--- + +## 5. Uncommitted Changes From This Session + +This worktree (`worktree-flake-e2e-audit`) currently has: + +- `scripts/proxmox/create-proxmox-resource.sh` — the `--agent enabled=1` fix (Finding 4). +- `.sops.yaml` / `secrets/common.yaml` — `lxc-gui`'s new age key registered as a recipient (generated live during this session's testing). + +Per this session's standard workflow, these will be committed, pushed, and opened as a draft PR rather than pushed to `main` directly — merging it is the operator's call, and is also **prerequisite to live-redeploying `lxc-gui` successfully** (its build will keep hitting the sops-staleness failure from Finding 2 on pve until this registration is on `origin/main`).