Compare commits

...
Author SHA1 Message Date
beatzaplenty 60fd64f0f7 Merge branch 'main' into worktree-migrate-vm-script
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m20s
2026-07-21 23:02:07 +00:00
beatzaplentyandClaude Sonnet 5 d35b7c4b57 Add pve1->pve-test VM/CT migration script (vzdump + qmrestore pattern)
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m29s
Reuses the clone script's vzdump/relay/restore flow but as a move rather
than a copy: preserves the original MAC by default, adds a stop-source/
start-target cutover step, and an opt-in --remove-source to destroy the
source guest afterward. Not a true qm migrate (nodes aren't clustered),
so the default --mode snapshot has a documented write gap between
snapshot and cutover; --mode stop avoids it at the cost of downtime.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Lkv8k24Wuua3GRxrfMBLz
2026-07-21 23:00:05 +00:00
beatzaplenty cb141f0a41 Merge pull request 'Rename PXE installer menu entry, add vanilla NixOS minimal netboot entry' (#31) from worktree-pxe-menu-rename-and-minimal into main
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
Reviewed-on: #31
2026-07-21 22:48:49 +00:00
beatzaplenty e92aab617f Rename PXE installer menu entry, add vanilla NixOS minimal netboot entry
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m42s
The iPXE menu's "nixos" entry actually chain-loads this flake's own
custom auto-installer image, not a stock NixOS image — rename it to
"auto-installer" (label "NixOS Auto-Installer") so the menu says what it
boots, and set networking.hostName on netbootSystem to match, so the
generated system name (nixos-system-auto-installer-*) and staged
directory (/srv/pxe/http/auto-installer) agree with the menu entry too.

Add a second, genuinely vanilla NixOS minimal netboot image
(netbootMinimalSystem in flake.nix — nixpkgs' netboot-minimal.nix on its
own, none of modules/installer/common.nix's auto-installer wiring),
built from source the same way as the auto-installer image and exposed
as packages.x86_64-linux.pxe-minimal. Staged and menu-wired the same
way, as "nixos-minimal" (item, hostname, and directory all matching).

modules/pxe-boot/stage-installer-artifacts.nix is generalized to stage
both images via a shared rule-builder instead of one hardcoded set of
paths.

Verified: nix eval confirms both images' config.system.name matches
their menu entry/directory names, the pxe-boot host itself builds
clean with the new menu.ipxe, and the new pxe-minimal image was booted
directly under QEMU (kernel+initrd, no KVM) to a working login shell
with hostname nixos-minimal, no hang.
2026-07-21 22:45:25 +00:00
beatzaplenty b4474cf1e1 Merge pull request 'Fix PXE netboot installer hanging at boot (ISO/netboot module conflict)' (#30) from worktree-fix-pxe-netboot-hang into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #30
2026-07-21 22:20:03 +00:00
beatzaplenty 453c7b5513 Fix PXE netboot installer hanging at boot (ISO/netboot module conflict)
Check NixOS configurations / eval-hosts (pull_request) Successful in 11m1s
The netboot build composed ./modules/installer/iso.nix (which pulls in
nixpkgs' installation-cd-minimal.nix) together with nixpkgs'
netboot-minimal.nix. Both installation-cd-base.nix and netboot.nix set
fileSystems."/" via the identical lib.mkImageMediaOverride (mkOverride
60) priority - genuinely conflicting root-filesystem strategies
(ISO-by-label vs. netboot-tmpfs) at the same priority, and the ISO one
was winning. Every netboot boot hung waiting for a device that can
never exist outside a real CD/USB:

  A start job is running for /dev/disk/by-label/nixos-minimal-...

Reproduced live: deployed a scratch lxc-pxe-boot on pve-test, pulled its
built kernel/initrd, and booted them directly with QEMU to confirm the
hang and capture full console output. netboot-minimal.nix's own chain
(netboot-base.nix) already imports profiles/installation-device.nix
independently, so common.nix's initialHashedPassword override still
applies correctly with iso.nix removed from this composition. Rebuilt
and re-booted the same way after the fix - full boot to a working shell
with SSH up, no hang.
2026-07-21 22:15:44 +00:00
beatzaplenty b3463e4b33 Merge pull request 'Add ad hoc pve1 -> pve-test clone script (vzdump + qmrestore/pct restore)' (#29) from worktree-clone-pve1-to-pve-test into main
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
Reviewed-on: #29
2026-07-21 21:55:25 +00:00
beatzaplentyandClaude Sonnet 5 013b2c7009 Add ad hoc pve1 -> pve-test clone script (vzdump + qmrestore/pct restore)
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m21s
Backs up a VM/CT on pve1 (snapshot mode by default, so the source stays
online), relays the archive to pve-test, restores it there with fresh
MAC addresses (--unique), and deletes both the source and relayed
backup copies afterward -- no ad hoc backup files left behind on
either node.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-21 20:22:56 +00:00
beatzaplenty 12f9153957 claude audit report
Check NixOS configurations / eval-hosts (push) Successful in 10m36s
2026-07-21 20:07:30 +00:00
beatzaplenty 1004538f00 updated sops secrets 2026-07-21 20:07:14 +00:00
beatzaplentyandClaude Sonnet 5 87873300e1 Add pve-test.sweet.home as a second Proxmox target
pve1.sweet.home is production; scripts/env.sh now also defines
PVE_TEST_HOST for a separate sandbox node, individually targetable via
--node/PROXMOX_HOST. Tooling defaults are unchanged (still pve1) -- the
new restriction (Claude defaults to pve-test unless explicitly told to
use pve1) is documented as policy in CLAUDE.md, not enforced in the
scripts.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-21 19:57:15 +00:00
beatzaplenty e9e2312163 Merge pull request 'Make lxc-docker a privileged container: unprivileged can't NFS-mount at all' (#28) from worktree-fix-container-dns-search-domain into main
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
Reviewed-on: #28
2026-07-21 08:50:14 +00:00
beatzaplentyandClaude Sonnet 5 6b09a808ed Make lxc-docker a privileged container: unprivileged can't NFS-mount at all
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
The kernel's NFS client filesystem doesn't set FS_USERNS_MOUNT, so mounting
NFS from inside any non-init user namespace -- exactly what an unprivileged
LXC container's UID-mapped root runs in -- is rejected at the VFS layer
with EPERM, regardless of Proxmox's mount=nfs;nfs4 container feature (which
only patches the AppArmor layer). Confirmed live on the redeployed lxc-docker
container: TCP to the NFS server's port 2049 succeeds, the server's export
table matches the container's IP, and mount.nfs: Operation not permitted
still fires immediately with no corresponding denial anywhere in the
server's own logs -- a kernel-level rejection that no amount of DNS/
automount/export tweaking (this branch's earlier commits) could ever fix.

modules/platforms/lxc.nix now keys proxmoxLXC.privileged off hostName
("docker" -> true) rather than a blanket false, since build-types/docker.nix
is also composed for linode-docker/proxmox-docker, which don't import
proxmox-lxc.nix at all -- setting this option there would break their eval.
create-proxmox-resource.sh reads the value back via a new
flake_target_lxc_privileged helper instead of hardcoding --unprivileged 1,
so the two stay in sync automatically for every lxc-* target.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
2026-07-21 05:57:06 +00:00
beatzaplenty 9496efdd22 Merge pull request 'Add pve.sweet.home guard rails to CLAUDE.md' (#26) from worktree-claude-md-pve-guardrails into main
Check NixOS configurations / eval-hosts (push) Successful in 10m13s
Reviewed-on: #26
2026-07-21 04:28:14 +00:00
beatzaplenty 33c9506c7d Merge pull request 'Fix NFS mount device strings on lxc-docker: use FQDN, not search domain' (#25) from worktree-fix-container-dns-search-domain into main
Check NixOS configurations / eval-hosts (push) Failing after 7m3s
Reviewed-on: #25
2026-07-21 04:26:32 +00:00
beatzaplentyandClaude Sonnet 5 abe3763cb3 Add pve.sweet.home guard rails to CLAUDE.md
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m24s
Codifies read-only access to existing Proxmox config/VMs/containers,
allows scratch test VMs/containers as long as they're torn down again,
and forbids any change to production on the node.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-21 02:01:15 +00:00
beatzaplenty 744904b19f Merge pull request 'Enable QEMU guest agent on Proxmox VMs; register lxc-gui sops key' (#24) from worktree-flake-e2e-audit into main
Check NixOS configurations / eval-hosts (push) Successful in 16m3s
Reviewed-on: #24
2026-07-21 01:46:06 +00:00
beatzaplentyandClaude Sonnet 5 42da626397 Fix NFS mount device strings on lxc-docker: use FQDN, not search domain
Check NixOS configurations / eval-hosts (pull_request) Successful in 17m29s
The previous commit's networking.search fix was wrong. Confirmed live
on lxc-docker (vmid 105) after redeploying with it: `resolvectl query
server.sweet.home` started failing again, even though
`resolvectl query --interface=eth0 server.sweet.home` still resolved
correctly to the right IP via the LAN's real DNS server. The debug log
showed why -- adding a *global* search domain via networking.search
gave systemd-resolved a domain-matched but server-less "global" scope,
which it now prioritizes over eth0's correctly-configured scope for
every "*.sweet.home" query, silently sending them to public fallback
DNS (1.1.1.1 et al) instead, which of course returns NXDOMAIN for an
internal-only name. Bare single-label names (e.g. "server") were never
going to work either way -- systemd-resolved only ever tries LLMNR for
those, never DNS search-suffixing, regardless of configuration.

Reverts the networking.search addition and instead has
modules/docker/mount-data.nix build each NFS device string from
"${vars.nfsServerHost}.${vars.homeDomain}" (a plain FQDN, no dependency
on search-domain behavior at all) -- the same pattern
modules/raspi/mount-data.nix already uses for the Raspberry Pi's share
and for the identical reason.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
2026-07-21 01:19:42 +00:00
19 changed files with 1162 additions and 148 deletions
+6
View File
@@ -10,6 +10,8 @@ keys:
- &lxc-minimal age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep - &lxc-minimal age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep
- &lxc-pxe-boot age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp - &lxc-pxe-boot age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp
- &lxc-gui age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05 - &lxc-gui age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05
- &proxmox-server age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
- &vm-server age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
creation_rules: creation_rules:
# Shared across every currently-deployed host: root/nixos password hash, # Shared across every currently-deployed host: root/nixos password hash,
@@ -30,6 +32,8 @@ creation_rules:
- *lxc-docker - *lxc-docker
- *lxc-pxe-boot - *lxc-pxe-boot
- *lxc-gui - *lxc-gui
- *proxmox-server
- *vm-server
- path_regex: secrets/nix-cache\.yaml$ - path_regex: secrets/nix-cache\.yaml$
key_groups: key_groups:
@@ -43,6 +47,8 @@ creation_rules:
- age: - age:
- *admin - *admin
- *server - *server
- *proxmox-server
- *vm-server
- path_regex: secrets/docker\.yaml$ - path_regex: secrets/docker\.yaml$
key_groups: key_groups:
+150
View File
@@ -0,0 +1,150 @@
# Flake End-to-End Audit Report
**Date:** 2026-07-21
**Scope:** Full static lint/eval sweep + live build/deploy/interrogate/destroy testing of every `lxc-*` and `proxmox-*` flake target against `pve.sweet.home`, plus an audit of the operator's ability to manage the flake/secrets tooling.
**Branch:** `worktree-flake-e2e-audit` (this session's isolated worktree)
## Executive Summary
The flake itself is in good shape: `nixpkgs-fmt`, `statix`, and a full eval + dry-run build of every host and package are all clean. Every `lxc-*`/`proxmox-*` target's NixOS configuration builds successfully — no target has a broken derivation graph.
The issues found are **operational, not code-level**:
1. **pve.sweet.home is critically low on disk space** (91-95% full during this session) and cannot currently build the two largest closures (`gui`, `pxe-boot`) to completion — this actively blocks deploying/redeploying those hosts via the documented workflow.
2. **A real, reproducible secrets-decryption failure** was caught live: a stale cached container image (built before a same-day sops-key fix) boots with sshd never starting and every secret failing to decrypt. This is a **general hazard in `create-proxmox-resource.sh`'s "reuse the cached image if present" default**, not a one-off.
3. **sops key/anchor drift**: `proxmox-minimal` has a `.sops.yaml` recipient anchor with no corresponding private key anywhere in this environment; several `lxc-*`/`proxmox-*` targets have no sops registration at all yet.
4. One concrete script bug was found and **fixed in this session**: `create-proxmox-resource.sh` never enabled the QEMU guest agent channel on VMs it creates, despite the guest OS already running it.
5. A management-surface audit (of the operator's ability to run this repo day to day) found 5 process gaps, detailed below.
Nothing here required or received a `nixos-rebuild switch/boot/test`, `nixos-install`, or any disk-formatting command — all validation was `nix build`/`nix eval`, plus disposable `pct`/`qm` create-then-destroy cycles via the repo's own `create-proxmox-resource.sh`.
---
## 1. Static Analysis Results — all clean
`bash scripts/codex-maintenance.sh --full-check --dry-run` (whole-tree sweep, not just changed files):
| Check | Result |
|---|---|
| Secret grep | Clean — only the documented exceptions (installer's own hashed passwords, `access-tokens` comment references) |
| `nixpkgs-fmt --check` | 0/53 files would be reformatted |
| `statix` | No lint warnings |
| nix-cache host key drift check | Up to date |
| Full eval of every host's `system.build.toplevel` | All 19 `nixosConfigurations` targets evaluate cleanly |
| Dry-run build of every host + package | All succeed, no derivation errors |
No drift, no formatting issues, no lint findings anywhere in the tree.
---
## 2. Per-Target Test Results
Legend: **LIVE** = built on pve, `pct`/`qm` create → interrogated → destroyed. **BUILD-ONLY** = `nix build` validated the config (mostly `.config.system.build.toplevel`, occasionally `.tarball`), no resource created on pve.
| Target | Test type | Result | Notes |
|---|---|---|---|
| `lxc-docker` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT105 is already running this identity in production; `--allow-duplicate-host` would have destroyed it. |
| `lxc-minimal` | **LIVE** | ✅ PASS (after retry) | First attempt reused a stale cached tarball predating a same-day sops-key commit → activation failed, sshd never started (see Finding #2). Redeployed with `--force-rebuild`: clean boot, `systemctl is-system-running` = `running`, secrets decrypted, sshd listening, users correct. |
| `lxc-nix-cache` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — CT101 is already running this identity. First local build attempt appeared to hang on a remote-builder handoff to nix-cache; killed and retried with `--builders ""` (local-only), succeeded. |
| `lxc-gui` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Registered a fresh sops key (no prior registration existed), built successfully through the full NixOS system closure, then **failed packaging the tarball**: `No space left on device` on pve's root filesystem. Not a flake defect. |
| `lxc-pxe-boot` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Same failure as `lxc-gui` — this target additionally builds a full nested installer/netboot image (`stage-installer-artifacts.nix`), making it similarly large. Failed with the same `No space left on device` error, immediately after the gui attempt had already consumed pve's remaining headroom. |
| `lxc-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; live deploy also would have hit `boot.zfs.extraPools` trying to import a real ZFS pool that doesn't exist in an isolated test container — an expected limitation of testing this build type outside its real hardware, not a bug. |
| `lxc-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
| `lxc-tor-relay` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT106 already holds this identity in production. |
| `proxmox-docker` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — both CT105 *and* VM103 already hold `docker` identities. Combined `toplevel` + `diskoImagesScript` build crashed with a **Nix-internal assertion failure** (`worker.cc:360`) under this session's memory pressure (see Finding #6) — not a flake bug. Retried with `toplevel` alone: clean. |
| `proxmox-minimal` | **LIVE (attempted)** | ⚠️ BLOCKED by key drift → BUILD-ONLY | `.sops.yaml` has a registered `&proxmox-minimal` anchor but **no corresponding private key exists anywhere in this environment** — the script correctly refused to generate a mismatched replacement. Fell back to `toplevel` build: ✅ PASS. |
| `proxmox-nix-cache` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
| `proxmox-gui` | BUILD-ONLY | ⚠️ Killed after ~40min (resource-limited) | This session's local build machine has only 2GB RAM; swap filled completely (2.0/2.0GB) and the build stalled, so it was killed rather than risk destabilizing the session further. **Not a flake defect** — the equivalent `gui` NixOS configuration already proved fully buildable during the `lxc-gui` live attempt above (it built the entire system closure successfully and only failed at the pve-side tarball-packaging step due to disk space, not the config). |
| `proxmox-pxe-boot` | BUILD-ONLY | ⚠️ Killed after ~35min (resource-limited) | Was deep into building the nested installer's kernel initrd (this build type bundles a full netboot installer image via `stage-installer-artifacts.nix`) when killed to keep the audit moving. **Not a flake defect** — this target's own module logic was already effectively validated via the earlier *live* pve deploy attempt (`lxc-pxe-boot` above), which built the complete image and only failed at the final tarball-packaging step due to pve's disk space (Finding 1). |
| `proxmox-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; same ZFS-pool caveat as `lxc-server` would apply to a live deploy. |
| `proxmox-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
**Not tested at all:** `linode-*` targets (not deployable to Proxmox) and `installer` (not a normal host) — both were still covered by the static eval/dry-run-build sweep above.
---
## 3. Findings, Ranked by Severity
### Finding 1 — pve.sweet.home is critically low on disk space (blocks real deployments)
At session start: `/dev/mapper/pve-root` was **95% full, 5.3GB free** (of 94GB). After two failed large builds it recovered slightly to **91% full, 8.2GB free** (nix cleans up its own failed-build scratch space). `/nix/store` alone is 26GB; `nix-store --gc --print-dead` reports **zero** reclaimable garbage — everything currently in the store is a live GC root, so `nix-collect-garbage` won't help without first removing old roots.
**Why it matters:** `create-proxmox-resource.sh` builds every VM/CT image **directly on pve**, not on a build machine and transferred over. With <10GB headroom, any closure approaching a few GB (the `gui` build type: full Cinnamon desktop + Firefox + LibreOffice + GIMP + VS Code + xrdp; the `pxe-boot` build type: nginx/atftpd *plus* an entire nested installer/netboot image) cannot currently be built there at all. Both `lxc-gui` and `lxc-pxe-boot` failed live with `No space left on device` during this audit.
**Recommended action:** Expand `pve-root`'s LV, or free space by pruning old container templates in `/var/lib/vz/template/cache` (1.5GB) / old backups in `/var/lib/vz/dump` (306MB) / auditing what's pinning 26GB of `/nix/store` as live GC roots (likely `result-*` symlinks — see below). This is real production disk state; **not something this session touched or fixed** — it needs the operator's judgment on what's safe to remove.
**Secondary, smaller finding:** every `create-proxmox-resource.sh` run leaves a `result-<target>` symlink in the node's repo checkout as a permanent GC root (`ls /root/nixos/result-*` on pve showed 3 from this session alone: `lxc-docker`, `lxc-minimal`, `lxc-nix-cache`). These accumulate forever and pin their entire closures in the store. Consider having the script clean up its own `result-*` link after staging the built artifact (or use a temp `--out-link` under `/tmp`), so `nix-collect-garbage` can actually reclaim old build outputs.
### Finding 2 — Stale cached images can silently ship broken secrets (reproduced live)
`create-proxmox-resource.sh`'s default behavior is: if the node already has `<target>.tar.xz`/`.raw` staged, **reuse it** — only `--force-rebuild` forces a fresh build. This session hit exactly the failure mode `docs/auto-installer.md` already warns about: `lxc-minimal`'s cached tarball (built 2026-07-20T15:57Z) predated a same-day sops-key fix commit (2026-07-20T17:49Z, "clean up in ailse 3"). The deployed container booted with:
```
sops-install-secrets: failed to decrypt '.../common.yaml': Error getting data key: 0 successful groups required, got 0
Activation script snippet 'setupSecrets' failed (1)
```
— every secret permanently failed to decrypt, `sshd` never started (though the container otherwise looked "running"). This was **not a code bug**: the currently-committed `secrets/common.yaml` decrypts fine for that host's key when checked independently; the *cached artifact on pve* simply reflected an older commit's ciphertext. Redeploying with `--force-rebuild` fixed it immediately.
**Why it matters:** this is silent and easy to trigger by accident — any operator who redeploys a host without remembering `--force-rebuild` after a secrets change gets a container that looks like it started (`pct start` succeeds, `pct status` = running) but is completely inaccessible.
**Recommended action:** Have `create-proxmox-resource.sh` compare the cached image's build timestamp (or embed the source commit hash in the staged filename) against current HEAD, and warn (or refuse without `--force-rebuild`) if they differ — rather than silently trusting presence alone.
### Finding 3 — sops key/anchor drift
Two concrete instances hit live during this session:
- **`proxmox-minimal`**: `.sops.yaml` already has a registered `&proxmox-minimal` age recipient, but this environment's `host-keys/` directory has no corresponding private key file. `sync-host-keys.sh` correctly refused to generate a replacement (it would silently mismatch whatever's already registered/deployed) — but this means **no environment currently has this host's private key**, unless it exists on some other machine that was never backed up here.
- **`lxc-gui`**, and by the same logic `lxc-server`/`lxc-tailscale-exit-node`/most `proxmox-*` targets, have **no sops registration at all yet** — expected for undeployed hosts per `docs/auto-installer.md`, but this session's live-testing needed to register `lxc-gui`'s key on the fly, which immediately hit **Finding 3b**: registering a key locally does nothing for pve's build until it's pushed to `origin/main` (pve builds via `git pull`, not from this uncommitted worktree). This is exactly gap #4 the management-surface audit (below) already flagged in the abstract — this session hit it concretely.
**Recommended action:** for `proxmox-minimal`, decide whether to regenerate its key (destroying old-key decrypt access, if anything still holds it) or track down wherever the original private key lives and back it up here. For the general pattern, see the management-surface audit's recommendation to pre-flight-check key registration before building.
### Finding 4 — QEMU guest agent never wired up (found and fixed this session)
`modules/common/configuration.nix:44` sets `services.qemuGuest.enable = true` on every host — the guest-side agent daemon is correctly enabled everywhere. But `scripts/proxmox/create-proxmox-resource.sh`'s `qm create` call never passed `--agent 1`, so **Proxmox never created the virtio-serial channel** the agent needs. Every `proxmox-*` VM this script ever created was silently missing `qm guest exec`/IP-address reporting in the Proxmox UI, despite the guest daemon actually running.
**Status: fixed in this session's worktree** (`scripts/proxmox/create-proxmox-resource.sh`, `qm create` now includes `--agent enabled=1`) — see the diff, included in the PR from this session.
### Finding 5 — Orphaned container on pve (CT102)
`pve.sweet.home` has a stopped LXC container, **VMID 102**, with an essentially empty config (`lock: create` and nothing else — no hostname, no rootfs, no network) — the leftover of a `pct create` that started and never finished. It predates this session (not created by any of this audit's activity) and wasn't touched. **Recommend the operator confirm it's abandoned and remove it** (`pct destroy 102 --purge 1`) — left as-is it may be someone's genuine in-progress work, so it wasn't assumed safe to delete autonomously.
### Finding 6 — Nix-internal crash under memory pressure (tooling, not flake)
Building `proxmox-docker`'s `toplevel` and `diskoImagesScript` together crashed with a Nix-internal assertion failure (`Assertion '!awake.empty()' failed ... worker.cc:360`, a known class of bug in Nix's multi-goal build scheduler) while this session's 2GB-RAM build container was under heavy swap pressure (1.8-2.0/2GB swap in use) from a separate concurrent build. Retrying the same target alone (no concurrency) succeeded cleanly. **Not a flake defect** — purely an artifact of this session's constrained build environment; noted for completeness since it looked alarming in isolation.
### Finding 7 — Management-surface audit: 5 operability gaps
A focused audit of "can the operator actually run this repo day to day" (flake, home-manager, sops, related scripts) found:
1. **No documented recovery path if the `&admin` sops age key is lost without a backup.** `scripts/secrets/backup-admin-key.sh` exists and works but is referenced nowhere in `README.md`/`docs/` — no forcing function ensures a backup was ever taken. `rotate-admin-key.sh` requires the *old* key to re-key; there's no bootstrap-from-nothing path documented (the real fallback — deriving an age identity from any still-live host's own SSH key — isn't written down anywhere).
2. **home-manager has no standalone iteration path.** It's wired only inside `nixosConfigurations` (`flake.nix`) — no `homeConfigurations` output. The fastest real shortcut (`nix build .#nixosConfigurations.<target>.config.home-manager.users.nixos.home.activationPackage`) isn't documented anywhere, so the practical workflow is a full host rebuild to test one HM tweak.
3. **Gitea's flake-lock-update workflow pushes straight to `main` with no pre-merge validation.** `.gitea/workflows/update-flake-lock.yml` commits and pushes `nix flake update`'s result directly; `codex-maintenance.sh` only runs *after*, on the resulting push — a genuinely broken lockfile bump lands on `main` before anything catches it. (The GitHub-side workflow is safer — PR-based — but has the opposite gap: nothing alerts if the PR sits unmerged.)
4. **No pre-flight check that a build target has a registered sops key before building it.** `docs/auto-installer.md` documents the failure mode (silent, total secrets-decrypt failure) but nothing in `create-proxmox-resource.sh` refuses to proceed when it's about to build a target with no `.sops.yaml` anchor — it's on the operator to remember. This session's `lxc-gui` test hit close to this exact gap (needed the key added on the fly, mid-session).
5. **`vars.remoteBuilderAuthorizedKeys` has the same drift risk as `vars.nixCacheHostKey`, but no checker script.** `sync-nix-cache-host-key.sh --check` guards the latter; the former (and `vars.pxeServerIp`/`vars.pbsIp`) has no equivalent — a rotated/revoked client key just silently stops working with no diagnostic pointing back here.
---
## 4. Action Plan (priority order)
1. **Free up disk space on pve.sweet.home** (or expand `pve-root`). Blocking: `lxc-gui`, `proxmox-gui`, `lxc-pxe-boot`, `proxmox-pxe-boot` cannot currently be built/redeployed on this node at all.
2. **Decide on `proxmox-minimal`'s orphaned sops key**: locate the original private key and back it up here, or accept regenerating it (breaks decrypt access for whoever/whatever currently holds the old one).
3. **Merge this session's PR** (see below) to get the `--agent 1` fix and `lxc-gui`'s new sops registration onto `main` — required before `lxc-gui` can be live-redeployed with working secrets.
4. **Add a staleness guard to `create-proxmox-resource.sh`'s cache-reuse path** (Finding 2) — highest-leverage fix, since it silently produces a broken-but-"running" host.
5. **Add a pre-flight sops-anchor check to `create-proxmox-resource.sh`** (management-surface gap #4) — same root cause class as #4 above, catch it before building instead of at first boot.
6. Investigate/clean up **CT102** on pve (Finding 5) — confirm abandoned, then remove.
7. Document `backup-admin-key.sh` in `README.md`'s Security Notes and add the live-host-key bootstrap-recovery procedure to `docs/` (management-surface gap #1).
8. Add pre-push validation to the Gitea flake-lock-update workflow (management-surface gap #3).
9. Lower-priority: document the home-manager `activationPackage` shortcut (gap #2); extend `sync-nix-cache-host-key.sh`'s drift-check pattern to `remoteBuilderAuthorizedKeys` (gap #5).
10. Follow-up session: finish build-validating `proxmox-gui` and `proxmox-pxe-boot` (both killed here after 35-40min on this session's 2GB-RAM machine — not failures, just unfinished) once pve has headroom (item 1) — ideally from a machine with more RAM. `proxmox-server` and `proxmox-tailscale-exit-node` already passed build-only validation in this session, no follow-up needed.
---
## 5. Uncommitted Changes From This Session
This worktree (`worktree-flake-e2e-audit`) currently has:
- `scripts/proxmox/create-proxmox-resource.sh` — the `--agent enabled=1` fix (Finding 4).
- `.sops.yaml` / `secrets/common.yaml``lxc-gui`'s new age key registered as a recipient (generated live during this session's testing).
Per this session's standard workflow, these will be committed, pushed, and opened as a draft PR rather than pushed to `main` directly — merging it is the operator's call, and is also **prerequisite to live-redeploying `lxc-gui` successfully** (its build will keep hitting the sops-staleness failure from Finding 2 on pve until this registration is on `origin/main`).
+95
View File
@@ -31,6 +31,74 @@ machines when deployed.
the auto-installer (see `docs/auto-installer.md`). Never commit its the auto-installer (see `docs/auto-installer.md`). Never commit its
contents; if `git status` ever shows it as trackable, something is wrong. contents; if `git status` ever shows it as trackable, something is wrong.
### Two Proxmox nodes: `pve1.sweet.home` (production) and `pve-test.sweet.home` (sandbox)
There are two SSH-reachable Proxmox nodes on the LAN, both defined in
`scripts/env.sh` (`PVE1_HOST` / `PVE_TEST_HOST`), individually targetable
via `scripts/proxmox/create-proxmox-resource.sh --node <host>` or by
overriding `PROXMOX_HOST`. `PROXMOX_HOST` itself still defaults to
`PVE1_HOST` (production) — that default, and every other script behavior,
is unchanged from before `pve-test` existed; the only thing new is that
`pve-test` can now be reached at all. They are **not interchangeable**
one is real production infrastructure, the other exists specifically so
there's somewhere safe to test. The restriction below is a policy for
Claude specifically, not a change to the tooling's own default or
anything the operator needs to opt into.
#### `pve1.sweet.home` (production — off-limits to Claude)
A real, live Proxmox node hosting production VMs/containers — not a
sandbox, and not Claude's to touch by default.
- **Off-limits at all times unless the operator has given explicit,
same-session instructions to act on this specific host.** That
authorization is scoped to the task it was given for — don't carry it
forward to unrelated later work in the same conversation, and never
assume it from a previous session.
- **Read-only for existing state is always fine, authorization or not.**
You may SSH in (or use `pvesm`, `qm list`, `pct list`, `qm config`, `pct
config`, the Proxmox API, etc.) to inspect the node's config, storage,
and any existing VM/container — including ones this repo didn't create.
- **Never** modify, stop, restart, delete, reconfigure, or create anything
on this node (`qm set`, `pct set`, `qm destroy`, `pct destroy`, `qm
stop`, `pct stop`, `qm create`, `pct create`, snapshot operations,
storage changes, etc.) — including scratch/test resources — without
that explicit go-ahead. Use `pve-test.sweet.home` for anything
exploratory instead; it exists precisely so `pve1` never has to be the
answer to "where do I test this."
- **This is a Claude-specific policy, not something the scripts enforce.**
`scripts/env.sh`/`create-proxmox-resource.sh` default to `pve1` exactly
as they did before `pve-test` existed, with no extra flag or prompt
required — that's deliberate, so the operator's own existing workflows
don't change. Claude, however, must never rely on that default: every
Proxmox action Claude takes on its own initiative — not explicitly
pointed at `pve1` by the operator this session — targets `pve-test`
instead (e.g. `--node "$PVE_TEST_HOST"`, or `PROXMOX_HOST=$PVE_TEST_HOST`).
Claude's own default is `pve-test`, full stop, regardless of what the
tooling's own unqualified default happens to be.
#### `pve-test.sweet.home` (sandbox — Claude's default target)
A separate Proxmox node set aside for testing. The *tooling's* default is
still production (`PROXMOX_HOST``PVE1_HOST`, see above) — but
**Claude's own default is this node**: absent an explicit, same-session
instruction to use `pve1`, every Proxmox action Claude initiates targets
`pve-test`. Once targeted, it's safe to create, interrogate, and destroy
resources on without asking first.
- **Test VMs/containers are allowed, but must be torn down.** Create a
scratch VM or container here (e.g. via
`scripts/proxmox/create-proxmox-resource.sh` or raw `qm`/`pct create`)
to validate something. Anything created this way must be destroyed
again in the same session, before ending the task — never leave a test
resource running. Use a VMID/name that's obviously scratch (and doesn't
collide with a real flake target) so it's unambiguous what's safe to
remove.
- **Node-level config is still not yours to change.** Creating/destroying
your own scratch guests is fine; Proxmox host config, storage pools, and
networking on `pve-test` itself are still the operator's call to make
manually, same as on `pve1`.
## Commands ## Commands
```bash ```bash
@@ -188,6 +256,33 @@ lift the shared part into `scripts/lib/` instead of copying it.
`/etc/ssh/ssh_known_hosts`. Idempotent (re-running replaces its own `/etc/ssh/ssh_known_hosts`. Idempotent (re-running replaces its own
marked block rather than duplicating it); restarts `nix-daemon` by marked block rather than duplicating it); restarts `nix-daemon` by
default so the change takes effect immediately. default so the change takes effect immediately.
- `scripts/proxmox/clone-pve1-to-pve-test.sh --vmid <n> [options]` — ad hoc
copy of a single VM/CT from one node to another via `vzdump` +
`qmrestore`/`pct restore`, relayed through this machine
(`ssh src cat ... | ssh dst cat > ...`, no on-disk staging copy).
Defaults `pve1` → `pve-test` (see "Two Proxmox nodes" above); source is
left running throughout (`--mode snapshot` default) and untouched
beyond the vzdump + a typed-VMID confirm, restored on the target with a
fresh MAC (`--unique 1`, always) since the source is expected to still
be live on the same LAN. Backup archives are deleted from both nodes
afterward unless `--keep-backup`. `--dry-run` throughout.
- `scripts/proxmox/migrate-vm.sh --vmid <n> [options]` — moves (rather
than clones) a single VM/CT between nodes, built on the same
vzdump/relay/restore pattern as `clone-pve1-to-pve-test.sh` above, plus
a cutover step (stop the guest on the source node, start it on the
target) and, optionally, `--remove-source` to destroy it on the source
node afterward (its own separate typed confirmation, since that step is
irreversible) — the default instead leaves the stopped source guest
intact as a safety net. Restores with the *original* MAC (no
`--unique`) unless `--unique` is passed, since the source is expected to
go away rather than keep running alongside the target. **Not** a true
`qm migrate`-style live migration — pve1/pve-test aren't a clustered
pair with shared storage, so this is vzdump-based and, with the default
`--mode snapshot`, any writes on the source between the snapshot and
cutover are lost (use `--mode stop` if that's not acceptable). Generic
node-to-node (not pve1/pve-test-specific) via
`--source-node`/`--target-node`, defaulting to pve1 → pve-test like the
clone script. `--dry-run` throughout.
### `scripts/lib/` ### `scripts/lib/`
+5
View File
@@ -36,6 +36,11 @@ Check the Proxmox node itself, or `/etc/flake-target` on a running host (see
below), if you need to know what's really out there right now. below), if you need to know what's really out there right now.
`scripts/proxmox/create-proxmox-resource.sh`'s duplicate-host guard works the same `scripts/proxmox/create-proxmox-resource.sh`'s duplicate-host guard works the same
way: it checks the Proxmox node directly rather than any file here. way: it checks the Proxmox node directly rather than any file here.
Real, production deployments live on `pve1.sweet.home`; there's a second
node, `pve-test.sweet.home`, set aside purely for scratch/test resources —
see `scripts/env.sh` (`PVE1_HOST` / `PVE_TEST_HOST`, and the
`--node`/`PROXMOX_HOST` targeting they feed into) and CLAUDE.md's Proxmox
section for which is which.
Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
(hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay (hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay
+5 -1
View File
@@ -150,7 +150,11 @@ use case.
The `pxe` variant is also built automatically as part of the `pxe-boot` host The `pxe` variant is also built automatically as part of the `pxe-boot` host
itself (`modules/pxe-boot/stage-installer-artifacts.nix`) and served over itself (`modules/pxe-boot/stage-installer-artifacts.nix`) and served over
iPXE — see `docs/pxe-boot.md`. iPXE as the menu's "NixOS Auto-Installer" entry — see `docs/pxe-boot.md`.
That same host also builds and serves `packages.x86_64-linux.pxe-minimal`,
a vanilla NixOS minimal netboot image with none of this auto-installer's
wiring, as a separate "NixOS Minimal" menu entry — also documented in
`docs/pxe-boot.md`, not covered further here since it's not this installer.
## Host keys ## Host keys
+35 -16
View File
@@ -1,9 +1,10 @@
# pxe-boot # pxe-boot
The `pxe-boot` host serves HTTP boot assets for iPXE clients — including a The `pxe-boot` host serves HTTP boot assets for iPXE clients — including
self-staged copy of this flake's own auto-installer netboot image, see self-staged copies of both this flake's own auto-installer netboot image
`docs/auto-installer.md` for what that image actually is and does once (see `docs/auto-installer.md` for what that image actually is and does once
booted. booted) and a vanilla, unmodified NixOS minimal netboot image for plain
rescue/inspection use.
## Host Role ## Host Role
@@ -28,7 +29,8 @@ The host creates these directories with systemd tmpfiles:
/srv/pxe /srv/pxe
/srv/pxe/http /srv/pxe/http
/srv/pxe/http/images /srv/pxe/http/images
/srv/pxe/http/nixos /srv/pxe/http/auto-installer
/srv/pxe/http/nixos-minimal
/srv/pxe/http/systemrescue /srv/pxe/http/systemrescue
/srv/pxe/http/ubuntu /srv/pxe/http/ubuntu
/srv/pxe/http/rescue /srv/pxe/http/rescue
@@ -37,7 +39,7 @@ The host creates these directories with systemd tmpfiles:
Mount shared image storage under `/srv/pxe/http`, preferably Mount shared image storage under `/srv/pxe/http`, preferably
`/srv/pxe/http/images` unless a menu entry expects files in a specific `/srv/pxe/http/images` unless a menu entry expects files in a specific
directory such as `/srv/pxe/http/nixos`. directory such as `/srv/pxe/http/auto-installer`.
The HTTP iPXE chain is: The HTTP iPXE chain is:
@@ -50,20 +52,37 @@ undionly.kpxe or ipxe.efi
The generated menu currently exposes entries for: The generated menu currently exposes entries for:
- NixOS installer - NixOS Auto-Installer
- NixOS Minimal
- SystemRescue environment - SystemRescue environment
- iPXE shell - iPXE shell
- Reboot - Reboot
The NixOS installer entry chain-loads `/srv/pxe/http/nixos/netboot.ipxe`, Both NixOS entries chain-load a `netboot.ipxe` staged into their own
which is nixpkgs' own generated netboot iPXE script (correct `init=`/`initrd=` directory (`/srv/pxe/http/auto-installer/netboot.ipxe` and
kernel parameters included) rather than a hand-rolled boot line — that script `/srv/pxe/http/nixos-minimal/netboot.ipxe`), each nixpkgs' own generated
in turn expects its kernel/initrd siblings in the same directory. All three netboot iPXE script (correct `init=`/`initrd=` kernel parameters included)
files (`bzImage`, `initrd`, `netboot.ipxe`) are built from this flake's own rather than a hand-rolled boot line — that script in turn expects its
`modules/installer/iso.nix` netboot image (the same one `nix build .#pxe` kernel/initrd siblings in the same directory. Each directory's three files
produces) and staged automatically by (`bzImage`, `initrd`, `netboot.ipxe`) are built from source and staged
`modules/pxe-boot/stage-installer-artifacts.nix` via `systemd.tmpfiles.rules` automatically by `modules/pxe-boot/stage-installer-artifacts.nix` via
— no manual operator step required. `systemd.tmpfiles.rules` — no manual operator step required:
- `auto-installer` is this flake's own `netbootSystem` (`flake.nix`) — the
same auto-installer image `nix build .#pxe` produces. See
`docs/auto-installer.md`.
- `nixos-minimal` is `netbootMinimalSystem` (`flake.nix`) — nixpkgs'
`netboot-minimal.nix` composed on its own, with none of this flake's
auto-installer wiring (no `common.nix`, no `auto-install.sh`, no baked
host keys or custom users). Same `nix build .#pxe-minimal` mechanism as
the auto-installer image, just a different module composition. Useful
as a plain rescue/inspection shell that doesn't assume anything about
this flake.
Both images set `networking.hostName` to match their menu entry/staged
directory name (`auto-installer` / `nixos-minimal`), so each one's
generated system name (`nixos-system-<name>-*`) is self-describing rather
than the nixpkgs default of `nixos-system-nixos-*` for both.
The SystemRescue entry expects the source ISO at: The SystemRescue entry expects the source ISO at:
+53 -2
View File
@@ -65,7 +65,7 @@
# file without a same-option circular dependency (a module # file without a same-option circular dependency (a module
# contributing to environment.etc can't read the merged # contributing to environment.etc can't read the merged
# environment.etc it's itself contributing to). # environment.etc it's itself contributing to).
specialArgs = { inherit inputs vars netbootSystem flakeTarget; }; specialArgs = { inherit inputs vars netbootSystem netbootMinimalSystem flakeTarget; };
}; };
# Generated platform x build-type matrix. pxe-boot has no linode # Generated platform x build-type matrix. pxe-boot has no linode
@@ -119,19 +119,61 @@
# Same installer environment, built as netboot (kernel + initrd + # Same installer environment, built as netboot (kernel + initrd +
# iPXE script) instead of an ISO — this is what packages.pxe bundles. # iPXE script) instead of an ISO — this is what packages.pxe bundles.
#
# Deliberately imports common.nix directly, NOT ./modules/installer/iso.nix
# (which pulls in nixpkgs' installation-cd-minimal.nix) -- confirmed live
# that composing the ISO module together with netboot-minimal.nix hangs
# every boot waiting for a device that can never exist on a netboot
# client ("A start job is running for /dev/disk/by-label/nixos-minimal-...").
# Both installation-cd-base.nix and netboot.nix set fileSystems."/" via
# the identical lib.mkImageMediaOverride (mkOverride 60) priority --
# genuinely conflicting root-filesystem strategies (ISO-by-label vs.
# netboot-tmpfs) at the same priority, and the ISO one was winning.
# netboot-minimal.nix's own chain (netboot-base.nix) already imports
# profiles/installation-device.nix independently, so common.nix's
# initialHashedPassword override (which assumes that profile is
# present) still applies correctly without iso.nix in the mix.
#
# networking.hostName is set explicitly (rather than left at nixpkgs'
# own "nixos" default) so this image's generated system name
# (nixos-system-auto-installer-*) matches its iPXE menu entry —
# see modules/build-types/pxe-boot.nix's :auto-installer item — and
# its staged directory, /srv/pxe/http/auto-installer.
netbootSystem = nixpkgs.lib.nixosSystem { netbootSystem = nixpkgs.lib.nixosSystem {
inherit system; inherit system;
modules = [ modules = [
./modules/installer/iso.nix ./modules/installer/common.nix
({ modulesPath, ... }: { ({ modulesPath, ... }: {
imports = [ imports = [
(modulesPath + "/installer/netboot/netboot-minimal.nix") (modulesPath + "/installer/netboot/netboot-minimal.nix")
]; ];
}) })
{ networking.hostName = "auto-installer"; }
]; ];
specialArgs = { inherit vars; }; specialArgs = { inherit vars; };
}; };
# A genuinely vanilla NixOS minimal netboot image: nixpkgs'
# netboot-minimal.nix on its own, with none of this flake's
# auto-installer wiring (no common.nix — no auto-install.sh, no
# baked host keys, no custom users/passwords). Built from source via
# the same nixosSystem + netboot-minimal.nix path as netbootSystem
# above, so both go through an identical build mechanism; the only
# difference is what's composed in. hostName again matches this
# image's iPXE menu entry (:nixos-minimal) and staged directory
# (/srv/pxe/http/nixos-minimal).
netbootMinimalSystem = nixpkgs.lib.nixosSystem {
inherit system;
modules = [
({ modulesPath, ... }: {
imports = [
(modulesPath + "/installer/netboot/netboot-minimal.nix")
];
})
{ networking.hostName = "nixos-minimal"; }
];
};
in in
{ {
@@ -153,6 +195,15 @@
{ name = "initrd"; path = netbootSystem.config.system.build.netbootRamdisk; } { name = "initrd"; path = netbootSystem.config.system.build.netbootRamdisk; }
{ name = "kernel"; path = netbootSystem.config.system.build.kernel; } { name = "kernel"; path = netbootSystem.config.system.build.kernel; }
]; ];
# Vanilla NixOS minimal netboot bundle — see netbootMinimalSystem
# above. Staged onto the pxe-boot host alongside packages.pxe by
# modules/pxe-boot/stage-installer-artifacts.nix.
pxe-minimal = pkgs.linkFarm "pxe-minimal" [
{ name = "netboot.ipxe"; path = netbootMinimalSystem.config.system.build.netbootIpxeScript; }
{ name = "initrd"; path = netbootMinimalSystem.config.system.build.netbootRamdisk; }
{ name = "kernel"; path = netbootMinimalSystem.config.system.build.kernel; }
];
}; };
}; };
} }
+9 -4
View File
@@ -68,15 +68,19 @@ let
set base ${pxeBaseUrl} set base ${pxeBaseUrl}
menu PXE Boot Menu menu PXE Boot Menu
item nixos NixOS Installer item auto-installer NixOS Auto-Installer
item nixos-minimal NixOS Minimal
item rescue Rescue Environment item rescue Rescue Environment
item shell iPXE Shell item shell iPXE Shell
item reboot Reboot item reboot Reboot
choose target && goto ''${target} choose target && goto ''${target}
:nixos :auto-installer
chain ''${base}/nixos/netboot.ipxe chain ''${base}/auto-installer/netboot.ipxe
:nixos-minimal
chain ''${base}/nixos-minimal/netboot.ipxe
:rescue :rescue
chain ''${base}/systemrescue.ipxe chain ''${base}/systemrescue.ipxe
@@ -129,7 +133,8 @@ in
"d ${pxeRoot} 0755 root root -" "d ${pxeRoot} 0755 root root -"
"d ${httpRoot} 0755 root root -" "d ${httpRoot} 0755 root root -"
"d ${httpRoot}/images 0755 root root -" "d ${httpRoot}/images 0755 root root -"
"d ${httpRoot}/nixos 0755 root root -" "d ${httpRoot}/auto-installer 0755 root root -"
"d ${httpRoot}/nixos-minimal 0755 root root -"
"d ${httpRoot}/systemrescue 0755 root root -" "d ${httpRoot}/systemrescue 0755 root root -"
"d ${httpRoot}/ubuntu 0755 root root -" "d ${httpRoot}/ubuntu 0755 root root -"
"d ${httpRoot}/rescue 0755 root root -" "d ${httpRoot}/rescue 0755 root root -"
-18
View File
@@ -13,24 +13,6 @@
networking.networkmanager.enable = true; # Easiest to use and most distros use this by default. networking.networkmanager.enable = true; # Easiest to use and most distros use this by default.
# No host declares a DNS search domain anywhere else, and cross-host
# references throughout this repo (vars.nfsServerHost, vars.nixCacheHost,
# vars.dockerHost, ...) are bare short names, not FQDNs -- resolving them
# depends entirely on whatever network stack happens to be in play
# picking up the DHCP-advertised domain as a search suffix. NetworkManager
# does that by default, which is why this went unnoticed on
# NetworkManager-managed hosts, but LXC containers (modules/platforms/lxc.nix
# force-disables NetworkManager and Proxmox writes their systemd-networkd
# config itself) never get one. Confirmed live on lxc-docker: systemd-resolved
# had no search domain for eth0, "server" failed to resolve
# ("Name or service not known") while "server.sweet.home" resolved fine via
# the same DNS server, so every NFS mount in modules/docker/mount-data.nix
# failed even after fixing the automount/mount=nfs bugs. This applies the
# search domain globally via systemd-resolved's own config rather than the
# per-link DHCP path, so it isn't at the mercy of whichever component owns
# a given host's interface file.
networking.search = [ vars.homeDomain ];
# Recommended over the true default (bypasses ZFS's own import safeguards) # Recommended over the true default (bypasses ZFS's own import safeguards)
# per the option's own docs; matches hosts/docker/host.nix and # per the option's own docs; matches hosts/docker/host.nix and
# modules/services/zfs/enable-service.nix, which already set this # modules/services/zfs/enable-service.nix, which already set this
+19 -5
View File
@@ -9,11 +9,25 @@ let
# (the VM platforms rely on automount itself to get that same # (the VM platforms rely on automount itself to get that same
# non-blocking behavior, so they don't need `nofail` too). # non-blocking behavior, so they don't need `nofail` too).
automountOpts = if config.boot.isContainer then [ "nofail" ] else [ "x-systemd.automount" ]; automountOpts = if config.boot.isContainer then [ "nofail" ] else [ "x-systemd.automount" ];
# A bare hostname here never resolves reliably: systemd-resolved only
# ever tries LLMNR for single-label names (never DNS, regardless of any
# configured search domain), and a *global* search domain (the first fix
# attempted here) backfires worse -- confirmed live on lxc-docker, adding
# `networking.search` made systemd-resolved prioritize its domain-matched
# but server-less global scope over eth0's correctly-configured one for
# every "*.sweet.home" query, silently sending them to public fallback
# DNS instead. `resolvectl query --interface=eth0 server.sweet.home`
# resolved fine throughout, proving the LAN DNS server was never the
# problem -- only the ambient, unqualified device string was. Using the
# FQDN directly sidesteps all of that, matching the pattern
# ../raspi/mount-data.nix already uses for the same reason.
nfsServer = "${vars.nfsServerHost}.${vars.homeDomain}";
in in
{ {
fileSystems = { fileSystems = {
${vars.nfsShares.dockerConfig.mountpoint} = { ${vars.nfsShares.dockerConfig.mountpoint} = {
device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath}"; device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
@@ -24,7 +38,7 @@ in
}; };
${vars.nfsShares.dockerDatabases.mountpoint} = { ${vars.nfsShares.dockerDatabases.mountpoint} = {
device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath}"; device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
@@ -35,7 +49,7 @@ in
}; };
${vars.nfsShares.dockerVolumes.mountpoint} = { ${vars.nfsShares.dockerVolumes.mountpoint} = {
device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}"; device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
@@ -46,7 +60,7 @@ in
}; };
${vars.nfsShares.nextcloudData.mountpoint} = { ${vars.nfsShares.nextcloudData.mountpoint} = {
device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath}"; device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
@@ -57,7 +71,7 @@ in
}; };
${vars.nfsShares.raspiVolumes.mountpoint} = { ${vars.nfsShares.raspiVolumes.mountpoint} = {
device = "${vars.nfsServerHost}:${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath}"; device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
+23 -3
View File
@@ -1,4 +1,4 @@
{ lib, modulesPath, flakeTarget, ... }: { config, lib, modulesPath, flakeTarget, ... }:
let let
# Bakes this exact flake target's pre-generated SSH host key straight # Bakes this exact flake target's pre-generated SSH host key straight
@@ -58,8 +58,28 @@ in
# host.nix declares each host's real hostname (networking.hostName); # host.nix declares each host's real hostname (networking.hostName);
# keep that instead of letting Proxmox's ambient container config win. # keep that instead of letting Proxmox's ambient container config win.
manageHostName = true; manageHostName = true;
# Unprivileged matches how these containers are actually created. # Unprivileged by default -- matches how these containers are actually
privileged = false; # created (scripts/proxmox/create-proxmox-resource.sh reads this value
# back to decide `pct create`'s --unprivileged flag, so the two stay
# in sync).
#
# lxc-docker is the one exception: the kernel's NFS client doesn't set
# FS_USERNS_MOUNT, so mounting NFS from inside *any* non-init user
# namespace -- which is exactly what an unprivileged container's
# UID-mapped root runs in -- is rejected at the VFS layer with EPERM,
# no matter what Proxmox's own `mount=nfs;nfs4` container feature
# allows at the AppArmor layer (confirmed live: TCP to the NFS server
# succeeds, the server's export table matches the container's IP, and
# `mount.nfs: Operation not permitted` still fires immediately with no
# corresponding denial anywhere in the server's logs -- a kernel-level
# rejection, not a network or export-permission one). Keying off
# hostName rather than something docker-build-type-specific because
# modules/build-types/docker.nix is also composed for linode-docker/
# proxmox-docker, which don't import proxmox-lxc.nix at all --setting
# this option there would break their eval with "option does not
# exist" regardless of any mkIf guard, since mkIf only makes a value
# conditional, not whether the option needs to exist somewhere.
privileged = config.networking.hostName == "docker";
}; };
boot.loader = { boot.loader = {
+23 -14
View File
@@ -1,23 +1,32 @@
{ netbootSystem, ... }: { netbootSystem, netbootMinimalSystem, ... }:
let let
# config.system.build.kernel and .netbootRamdisk are directories, not the # config.system.build.kernel and .netbootRamdisk are directories, not the
# files themselves — nixpkgs' own system.build.kexecTree does the same # files themselves — nixpkgs' own system.build.kexecTree does the same
# ${...}/<file> dereference for the same reason. # ${...}/<file> dereference for the same reason.
inherit (netbootSystem.config.system.boot.loader) kernelFile; mkStageRules = { dirName, system }:
let
inherit (system.config.system.boot.loader) kernelFile;
dir = "/srv/pxe/http/${dirName}";
in
[
# Declared here too (not just in build-types/pxe-boot.nix) so this
# module's C+ rules don't depend on cross-module list-merge ordering —
# tmpfiles' C type needs the target directory to already exist.
"d ${dir} 0755 root root -"
"C+ ${dir}/${kernelFile} 0644 root root - ${system.config.system.build.kernel}/${kernelFile}"
"C+ ${dir}/initrd 0644 root root - ${system.config.system.build.netbootRamdisk}/initrd"
"C+ ${dir}/netboot.ipxe 0644 root root - ${system.config.system.build.netbootIpxeScript}/netboot.ipxe"
];
in in
{ {
# Builds this flake's own installer netboot image (the same one # Builds this flake's own installer netboot image (the same one
# `nix build .#pxe` produces) and stages it where menu.ipxe's :nixos # `nix build .#pxe` produces) plus the vanilla NixOS minimal netboot image
# entry expects it, so the pxe-boot host is self-contained — no manual # (`nix build .#pxe-minimal`), and stages both where menu.ipxe's
# operator step to populate /srv/pxe/http/nixos after deploy. # :auto-installer / :nixos-minimal entries expect them, so the pxe-boot
systemd.tmpfiles.rules = [ # host is self-contained — no manual operator step to populate
# Declared here too (not just in build-types/pxe-boot.nix) so this # /srv/pxe/http after deploy.
# module's C+ rules don't depend on cross-module list-merge ordering — systemd.tmpfiles.rules =
# tmpfiles' C type needs the target directory to already exist. mkStageRules { dirName = "auto-installer"; system = netbootSystem; }
"d /srv/pxe/http/nixos 0755 root root -" ++ mkStageRules { dirName = "nixos-minimal"; system = netbootMinimalSystem; };
"C+ /srv/pxe/http/nixos/${kernelFile} 0644 root root - ${netbootSystem.config.system.build.kernel}/${kernelFile}"
"C+ /srv/pxe/http/nixos/initrd 0644 root root - ${netbootSystem.config.system.build.netbootRamdisk}/initrd"
"C+ /srv/pxe/http/nixos/netboot.ipxe 0644 root root - ${netbootSystem.config.system.build.netbootIpxeScript}/netboot.ipxe"
];
} }
+21 -11
View File
@@ -6,12 +6,22 @@
# environment (e.g. PROXMOX_STORAGE=tank-nvme ./scripts/proxmox/create-proxmox-resource.sh ...) # environment (e.g. PROXMOX_STORAGE=tank-nvme ./scripts/proxmox/create-proxmox-resource.sh ...)
# since each one only sets a default if unset. # since each one only sets a default if unset.
# SSH-reachable Proxmox node that scripts/proxmox/create-proxmox-resource.sh runs # Two SSH-reachable Proxmox nodes exist on the LAN:
# pct/qm on. Matches the Proxmox web UI hostname already used in # - pve1.sweet.home -- production. Real, live VMs/containers.
# hosts/nixos/home.nix's desktop shortcuts (pve.<homeDomain> from # - pve-test.sweet.home -- sandbox/test node, for scratch VMs/containers
# variables.nix) -- change this if that's not actually reachable over SSH, # that don't belong on production.
# or if you're targeting a different node in a multi-node cluster. #
: "${PROXMOX_HOST:=pve.sweet.home}" # PROXMOX_HOST is what scripts/proxmox/create-proxmox-resource.sh actually
# targets by default -- overridable per-invocation with --node <hostname>,
# or per-variable as usual (e.g. PROXMOX_HOST=$PVE_TEST_HOST). It defaults
# to production, matching this repo's behavior before pve-test existed --
# see CLAUDE.md's "Two Proxmox nodes" section for the policy on which
# situations should target which node (in particular: Claude defaults to
# pve-test, not this variable's own default, unless explicitly told
# otherwise).
: "${PVE1_HOST:=pve1.sweet.home}"
: "${PVE_TEST_HOST:=pve-test.sweet.home}"
: "${PROXMOX_HOST:=$PVE1_HOST}"
: "${PROXMOX_SSH_USER:=root}" : "${PROXMOX_SSH_USER:=root}"
# Where this flake repo lives on the Proxmox node itself. # Where this flake repo lives on the Proxmox node itself.
@@ -25,7 +35,7 @@
# Storage pool names -- Proxmox's own stock-install defaults, but this # Storage pool names -- Proxmox's own stock-install defaults, but this
# varies a lot by setup (ZFS pool name, custom LVM-thin volume, etc.). # varies a lot by setup (ZFS pool name, custom LVM-thin volume, etc.).
# Verify with `pvesm status` on the node and correct these if wrong. # Verify with `pvesm status` on the node and correct these if wrong.
: "${PROXMOX_STORAGE:=local-lvm}" # VM disks / CT rootfs : "${PROXMOX_STORAGE:=local-zfs}" # VM disks / CT rootfs
: "${PROXMOX_ISO_STORAGE:=local}" # uploaded images/ISOs/CT templates : "${PROXMOX_ISO_STORAGE:=local}" # uploaded images/ISOs/CT templates
: "${PROXMOX_BRIDGE:=vmbr0}" : "${PROXMOX_BRIDGE:=vmbr0}"
@@ -63,10 +73,10 @@
# nothing here forces a mount to happen. # nothing here forces a mount to happen.
: "${PROXMOX_DEFAULT_LXC_FEATURES:=nesting=1,keyctl=1,mount=nfs;nfs4}" : "${PROXMOX_DEFAULT_LXC_FEATURES:=nesting=1,keyctl=1,mount=nfs;nfs4}"
export PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE PROXMOX_ISO_STORAGE \ export PVE1_HOST PVE_TEST_HOST PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE \
PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES PROXMOX_DEFAULT_MEMORY_MB \ PROXMOX_ISO_STORAGE PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES \
PROXMOX_DEFAULT_LXC_DISK_GB PROXMOX_DEFAULT_LXC_FEATURES \ PROXMOX_DEFAULT_MEMORY_MB PROXMOX_DEFAULT_LXC_DISK_GB \
PROXMOX_REMOTE_REPO_DIR PROXMOX_DEFAULT_LXC_FEATURES PROXMOX_REMOTE_REPO_DIR
# Matches variables.nix's nixCacheHost -- update both if it ever changes. # Matches variables.nix's nixCacheHost -- update both if it ever changes.
: "${NIX_CACHE_HOST:=nix-cache}" : "${NIX_CACHE_HOST:=nix-cache}"
+18
View File
@@ -34,3 +34,21 @@ flake_target_hostname() {
nix eval --raw "${NIX_EVAL_FLAGS[@]}" \ nix eval --raw "${NIX_EVAL_FLAGS[@]}" \
"${flake_ref}#nixosConfigurations.${target}.config.networking.hostName" 2>/dev/null "${flake_ref}#nixosConfigurations.${target}.config.networking.hostName" 2>/dev/null
} }
# flake_target_lxc_privileged <flake_ref> <target>
# Prints "true" or "false" for one lxc-* target's config.proxmoxLXC.privileged
# (modules/platforms/lxc.nix is the single source of truth -- e.g.
# lxc-docker sets this true so it can NFS-mount; every other lxc-* host
# stays unprivileged). Only meaningful for lxc-* targets -- the option
# doesn't exist for linode-*/proxmox-* (nixpkgs' proxmox-lxc.nix, which
# declares it, is only ever imported by modules/platforms/lxc.nix). Empty
# (not an error under set -e) if the eval fails.
flake_target_lxc_privileged() {
local flake_ref="$1" target="$2"
# Not --raw: the option is a Nix boolean, and --raw can only coerce
# strings ("cannot coerce a Boolean to a string"). Plain `nix eval`
# prints a bare `true`/`false` for a boolean, which is exactly the
# string this needs.
nix eval "${NIX_EVAL_FLAGS[@]}" \
"${flake_ref}#nixosConfigurations.${target}.config.proxmoxLXC.privileged" 2>/dev/null
}
+221
View File
@@ -0,0 +1,221 @@
#!/usr/bin/env bash
# Ad hoc clone of a single VM/CT from pve1 (production) to pve-test
# (sandbox), via vzdump + qmrestore/pct restore -- not a general-purpose
# backup tool, just a quick "give me a disposable copy of this thing on
# pve-test" for testing against real-ish data without touching prod.
#
# Flow:
# 1. vzdump the resource on pve1 into its "local" storage (--mode
# snapshot by default, so the source keeps running throughout --
# see --mode below for when that's not possible).
# 2. Stream the resulting archive straight from pve1 to pve-test
# (ssh pve1 cat ... | ssh pve-test cat > ...) -- this machine is
# just the relay, no separate on-disk staging copy here.
# 3. qmrestore / pct restore it on pve-test under --new-vmid (default:
# same VMID as the source -- pve-test is a separate node/cluster, so
# no collision unless that VMID is already in use there too).
# Always restored with --unique 1 (fresh MAC addresses) since the
# source is typically still running on the same LAN -- restoring
# with the *same* MAC would put two live guests on the wire with
# identical hardware addresses.
# 4. Delete the vzdump archive from pve1's local storage and the
# relayed copy on pve-test, so neither node accumulates ad hoc
# backup files from this script. Only the pve1 original is
# preserved on any failure after step 1, so a failed
# transfer/restore can be retried without re-running the backup.
#
# This script's own defaults are pve1 -> pve-test, unlike
# create-proxmox-resource.sh's --node (which defaults to production) --
# see CLAUDE.md's "Two Proxmox nodes" section. pve1 is only ever touched
# here after typing the source VMID back to confirm; pve-test is treated
# as disposable, matching this repo's usual policy for that node.
#
# See --help for the full option list.
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=../lib/confirm.sh
source "${repo_root}/scripts/lib/confirm.sh"
usage() {
cat <<EOF
Usage: $0 --vmid <n> [options]
--vmid <n> Required: VMID on the source node to clone.
Kind (qemu VM vs LXC CT) is auto-detected.
--new-vmid <n> VMID to restore as on the target node
(default: same as --vmid).
--mode snapshot|suspend|stop
vzdump backup mode (default: snapshot -- the
source resource keeps running throughout;
requires snapshot-capable storage, e.g.
ZFS/LVM-thin/Ceph/qcow2). Fall back to
"suspend" (brief pause) or "stop" (source
goes down for the duration) if the source's
storage doesn't support live snapshots --
vzdump's own error will say so.
--source-node <host> (default: \$PVE1_HOST, ${PVE1_HOST})
--target-node <host> (default: \$PVE_TEST_HOST, ${PVE_TEST_HOST})
--source-storage <pool> Where vzdump writes the backup on the
source node (default: local).
--target-storage <pool> Where the restored disk/rootfs lands on
the target node (default: \$PROXMOX_STORAGE, ${PROXMOX_STORAGE}).
--keep-backup Don't delete the vzdump archive from
either node afterward (debugging aid).
--yes Skip the typed VMID confirmation
before touching the source node.
--dry-run Print the full plan and skip every
mutating step (vzdump, transfer,
restore, delete) and the confirm
prompt. Still makes read-only SSH
calls to look up the source kind
and check the target VMID is free
-- harmless on either node.
-h, --help
EOF
}
vmid=""
new_vmid=""
mode="snapshot"
source_node="$PVE1_HOST"
target_node="$PVE_TEST_HOST"
source_storage="local"
target_storage="$PROXMOX_STORAGE"
keep_backup=0
skip_confirm=0
dry_run=0
while [[ $# -gt 0 ]]; do
case "$1" in
--vmid) vmid="$2"; shift 2 ;;
--new-vmid) new_vmid="$2"; shift 2 ;;
--mode) mode="$2"; shift 2 ;;
--source-node) source_node="$2"; shift 2 ;;
--target-node) target_node="$2"; shift 2 ;;
--source-storage) source_storage="$2"; shift 2 ;;
--target-storage) target_storage="$2"; shift 2 ;;
--keep-backup) keep_backup=1; shift ;;
--yes) skip_confirm=1; shift ;;
--dry-run) dry_run=1; shift ;;
-h | --help) usage; exit 0 ;;
*) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;;
esac
done
if [[ -z "$vmid" ]]; then
echo "ERROR: --vmid is required." >&2
usage >&2
exit 1
fi
if [[ "$mode" != "snapshot" && "$mode" != "suspend" && "$mode" != "stop" ]]; then
echo "ERROR: --mode must be snapshot, suspend, or stop." >&2
exit 1
fi
[[ -z "$new_vmid" ]] && new_vmid="$vmid"
source_target="${PROXMOX_SSH_USER}@${source_node}"
target_target="${PROXMOX_SSH_USER}@${target_node}"
# No dry-run wrapper needed for the calls below: every mutating step
# (vzdump, transfer, restore, delete) is reached only after the --dry-run
# early-exit further down, so a plain `ssh` call is never in the dry-run
# path.
# --- identify the resource kind on the source node -----------------------
echo "==> Looking up VMID ${vmid} on ${source_node}..."
kind=""
if ssh "$source_target" "qm status ${vmid}" >/dev/null 2>&1; then
kind="vm"
elif ssh "$source_target" "pct status ${vmid}" >/dev/null 2>&1; then
kind="lxc"
else
echo "ERROR: VMID ${vmid} doesn't exist on ${source_node} as either a VM or CT." >&2
exit 1
fi
echo "VMID ${vmid} on ${source_node} is a ${kind}."
# --- refuse to clobber an existing resource on the target node -----------
if ssh "$target_target" "qm status ${new_vmid}" >/dev/null 2>&1 \
|| ssh "$target_target" "pct status ${new_vmid}" >/dev/null 2>&1; then
echo "ERROR: VMID ${new_vmid} already exists on ${target_node}. Pass --new-vmid" >&2
echo "with a free ID, or remove the existing resource there first." >&2
exit 1
fi
echo
echo "Plan:"
echo " source: ${kind} VMID ${vmid} on ${source_node} (storage: ${source_storage}, mode: ${mode})"
echo " target: VMID ${new_vmid} on ${target_node} (storage: ${target_storage}, fresh MAC via --unique)"
[[ "$keep_backup" -eq 1 ]] && echo " backup archives are kept on both nodes afterward (--keep-backup)"
if [[ "$dry_run" -eq 1 ]]; then
echo
echo "[dry-run] No backup, transfer, restore, or delete was performed."
exit 0
fi
if [[ "$skip_confirm" -ne 1 ]]; then
echo
if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) to confirm backing it up from ${source_node}: "; then
echo "Cancelled -- input didn't match ${vmid}." >&2
exit 1
fi
fi
# --- vzdump on the source node --------------------------------------------
echo
echo "==> Backing up VMID ${vmid} on ${source_node} (mode=${mode}, storage=${source_storage})..."
vzdump_log="$(ssh "$source_target" \
"vzdump ${vmid} --mode ${mode} --storage ${source_storage} --compress zstd" 2>&1)" \
|| {
echo "$vzdump_log" >&2
echo "ERROR: vzdump failed on ${source_node}." >&2
exit 1
}
echo "$vzdump_log"
archive="$(echo "$vzdump_log" | grep -oP "creating vzdump archive '\K[^']+" | tail -n1)"
if [[ -z "$archive" ]]; then
echo "ERROR: couldn't find the archive path in vzdump's output above." >&2
exit 1
fi
archive_basename="$(basename "$archive")"
target_tmp_archive="/var/tmp/${archive_basename}"
echo "Archive: ${archive}"
# Always clean up the relayed copy on the target node, success or failure
# -- it's only ever a working copy, restored or not.
cleanup_target_tmp() {
if [[ "$keep_backup" -ne 1 ]]; then
ssh "$target_target" "rm -f '${target_tmp_archive}'" >/dev/null 2>&1 || true
fi
}
trap cleanup_target_tmp EXIT
# --- relay the archive from source to target ------------------------------
echo
echo "==> Transferring archive to ${target_node}..."
ssh "$source_target" "cat '${archive}'" | ssh "$target_target" "cat > '${target_tmp_archive}'"
# --- restore on the target node --------------------------------------------
echo
echo "==> Restoring as VMID ${new_vmid} on ${target_node} (storage=${target_storage})..."
if [[ "$kind" == "vm" ]]; then
ssh "$target_target" "qmrestore '${target_tmp_archive}' ${new_vmid} --storage ${target_storage} --unique 1"
else
ssh "$target_target" "pct restore ${new_vmid} '${target_tmp_archive}' --storage ${target_storage} --unique 1"
fi
# --- clean up the source backup now that the restore succeeded -----------
if [[ "$keep_backup" -ne 1 ]]; then
echo
echo "==> Deleting backup archive from ${source_node}'s ${source_storage} storage..."
ssh "$source_target" "rm -f '${archive}' '${archive}.notes' '${archive}.log'" >/dev/null 2>&1 || true
fi
echo
echo "Done. VMID ${new_vmid} (${kind}) is now on ${target_node}, cloned from" \
"VMID ${vmid} on ${source_node}."
+31 -9
View File
@@ -12,6 +12,17 @@
# copies over the locally-managed host-keys/ (gitignored, so a git pull # copies over the locally-managed host-keys/ (gitignored, so a git pull
# alone wouldn't carry it) before building. # alone wouldn't carry it) before building.
# #
# --node (default: $PROXMOX_HOST, see scripts/env.sh) picks which of the two
# LAN Proxmox nodes this runs against: production, pve1.sweet.home
# ($PVE1_HOST, PROXMOX_HOST's own default), or the sandbox node,
# pve-test.sweet.home ($PVE_TEST_HOST) -- pass --node "$PVE_TEST_HOST" (or
# set PROXMOX_HOST=$PVE_TEST_HOST) to target the sandbox instead. See
# CLAUDE.md's "Two Proxmox nodes" section: an agent session should default
# to pve-test and only touch pve1 when the operator has explicitly said so
# for the current task -- this script itself doesn't enforce that (its own
# default is production, matching this repo's behavior before pve-test
# existed), it's a policy for whoever/whatever is driving it.
#
# Usage: # Usage:
# scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --host <name> [options] # scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --host <name> [options]
# scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --list # scripts/proxmox/create-proxmox-resource.sh --type lxc|vm --list
@@ -120,7 +131,9 @@ Shared:
--iso-storage <pool> (default: \$PROXMOX_ISO_STORAGE, ${PROXMOX_ISO_STORAGE}) --iso-storage <pool> (default: \$PROXMOX_ISO_STORAGE, ${PROXMOX_ISO_STORAGE})
--bridge <bridge> (default: \$PROXMOX_BRIDGE, ${PROXMOX_BRIDGE}) --bridge <bridge> (default: \$PROXMOX_BRIDGE, ${PROXMOX_BRIDGE})
--node <host> Proxmox node to SSH into (default: --node <host> Proxmox node to SSH into (default:
\$PROXMOX_HOST, ${PROXMOX_HOST}) \$PROXMOX_HOST, ${PROXMOX_HOST} --
production; the sandbox node is
\$PVE_TEST_HOST, ${PVE_TEST_HOST}).
--dry-run Print the full plan; touch nothing --dry-run Print the full plan; touch nothing
local or remote, no prompts. local or remote, no prompts.
-h, --help -h, --help
@@ -766,13 +779,22 @@ if [[ "$type" == "lxc" ]]; then
# 512M default otherwise (confirmed live: --memory 2048 left swap at # 512M default otherwise (confirmed live: --memory 2048 left swap at
# 512). Default to matching whatever --memory resolved to above. # 512). Default to matching whatever --memory resolved to above.
local_swap="${swap:-$memory}" local_swap="${swap:-$memory}"
# --unprivileged 1: modules/platforms/lxc.nix sets proxmoxLXC.privileged # --unprivileged: read back from modules/platforms/lxc.nix's own
# = false, so the NixOS config inside the image assumes it's running as # proxmoxLXC.privileged (via flake_target_lxc_privileged) rather than
# an unprivileged container (cgroup/capability/mount expectations baked # hardcoded, since that's no longer the same for every lxc-* target --
# in at boot). `pct create`'s own CLI default for this flag is # lxc-docker sets it true so the container's NFS mounts work at all (the
# privileged (unlike the web UI, which defaults its checkbox the other # kernel's NFS client can't mount from inside any unprivileged
# way) -- leaving it unset creates a privileged container running a # container's user namespace, no matter what AppArmor allows -- see that
# NixOS config that assumes unprivileged, a real mismatch. # option's own comment). The NixOS config inside the image bakes in
# cgroup/capability/mount expectations matching whichever value it was
# built with, so this must stay in sync with it -- `pct create`'s own
# CLI default for this flag is privileged (unlike the web UI, which
# defaults its checkbox the other way), so leaving it unset would create
# a privileged container running a NixOS config that assumes
# unprivileged for every target except lxc-docker, a real mismatch.
privileged_eval="$(flake_target_lxc_privileged "$repo_root" "$flake_target")"
unprivileged_flag=1
[[ "$privileged_eval" == "true" ]] && unprivileged_flag=0
# #
# --features nesting=1,keyctl=1: required for a modern (v247+) systemd # --features nesting=1,keyctl=1: required for a modern (v247+) systemd
# guest to actually boot unprivileged -- confirmed live: without this, # guest to actually boot unprivileged -- confirmed live: without this,
@@ -790,7 +812,7 @@ if [[ "$type" == "lxc" ]]; then
# hands the whole string to `ssh` as a single command for the *remote* # hands the whole string to `ssh` as a single command for the *remote*
# shell to parse -- unquoted, that `;` would be read as a remote # shell to parse -- unquoted, that `;` would be read as a remote
# command separator and silently truncate this into two commands. # command separator and silently truncate this into two commands.
create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --unprivileged 1 --features '${PROXMOX_DEFAULT_LXC_FEATURES}' --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --swap ${local_swap} --net0 name=eth0,bridge=${bridge},ip=dhcp" create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --unprivileged ${unprivileged_flag} --features '${PROXMOX_DEFAULT_LXC_FEATURES}' --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --swap ${local_swap} --net0 name=eth0,bridge=${bridge},ip=dhcp"
remote "$create_cmd" remote "$create_cmd"
remote "pct start ${vmid}" remote "pct start ${vmid}"
else else
+347
View File
@@ -0,0 +1,347 @@
#!/usr/bin/env bash
# Move a single VM/CT from one Proxmox node to another via vzdump +
# qmrestore/pct restore -- the same relay pattern as
# clone-pve1-to-pve-test.sh, but for a *migration* (the guest ends up
# living on the target node only) rather than a *clone* (a disposable
# copy, source untouched, fresh MAC).
#
# There is no real `qm migrate` here: that command only works between
# nodes in the same Proxmox cluster with shared/replicated storage, which
# pve1 and pve-test are not (see CLAUDE.md's "Two Proxmox nodes" section --
# they're deliberately separate, unclustered nodes). This script is the
# closest equivalent across two independent nodes, built out of the same
# primitives as the clone script. It is NOT a true live migration -- read
# the caveat below before using it for anything where losing a few
# seconds/minutes of writes is unacceptable.
#
# Flow:
# 1. vzdump the resource on the source node (--mode snapshot by
# default, so the source keeps running throughout the bulk copy --
# this is the part that's "live": the guest stays reachable while
# its disk data is captured and shipped to the target).
# 2. Stream the archive straight from the source node to the target
# node (ssh source cat ... | ssh target cat > ...), same relay this
# machine does in the clone script -- no separate on-disk staging
# copy here either.
# 3. qmrestore / pct restore it on the target node under the same VMID
# by default (--new-vmid to pick a different one). Restored WITHOUT
# --unique (i.e. original MAC preserved) unless --unique is passed
# explicitly -- unlike the clone script, the source is being retired,
# not left running alongside a copy, so there's no address collision
# to avoid. qmrestore/pct restore don't start the guest, so nothing
# is live on the target node yet.
# 4. Cutover: stop the guest on the source node, then (unless --no-start)
# start it on the target node. This is the only real downtime window
# -- everything before this point runs with the source guest still
# up.
# 5. Unless --remove-source is passed, the now-stopped guest is LEFT ON
# the source node as a safety net (config + disk intact, just
# powered off) -- easy to start back up if the target copy turns out
# to be broken. Pass --remove-source to actually destroy it there
# once you've verified the target is good; this step gets its own
# typed confirmation since qm destroy/pct destroy is irreversible.
# 6. Delete the vzdump archive from both nodes' storage (unless
# --keep-backup), same as the clone script -- neither node
# accumulates ad hoc backup files from this script.
#
# --- IMPORTANT CAVEAT: this is not byte-perfect live migration ---
# vzdump is not incremental. With the default --mode snapshot, the source
# guest keeps running (and can keep writing to disk) for the entire time
# between when the snapshot is taken and when this script stops it at
# cutover. Any writes in that window are NOT captured in the archive and
# will NOT exist on the target. For a large disk this window can be
# minutes. If the workload can't tolerate that:
# - use --mode stop (or --mode suspend) instead, which makes the source
# guest go down *before* vzdump reads its disk, so the archive is
# exactly the state being migrated and cutover has nothing left to
# lose -- at the cost of the guest being down for the whole backup
# duration instead of just the final cutover.
# - or don't use this script -- put both nodes in an actual Proxmox
# cluster with shared storage and use `qm migrate --online` instead,
# which is the real thing this script is only approximating.
#
# This script's own defaults are pve1 -> pve-test, matching
# clone-pve1-to-pve-test.sh and CLAUDE.md's "Two Proxmox nodes" section --
# but unlike that script, this one is a generic node-to-node mover: pass
# --source-node/--target-node for any other pair. Regardless of node
# names, the source resource is only ever touched here after typing the
# source VMID back to confirm, and destroying it on the source node
# (--remove-source) requires a second, separate typed confirmation.
#
# See --help for the full option list.
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=../lib/confirm.sh
source "${repo_root}/scripts/lib/confirm.sh"
usage() {
cat <<EOF
Usage: $0 --vmid <n> [options]
--vmid <n> Required: VMID on the source node to migrate.
Kind (qemu VM vs LXC CT) is auto-detected.
--new-vmid <n> VMID to restore as on the target node
(default: same as --vmid).
--mode snapshot|suspend|stop
vzdump backup mode (default: snapshot -- the
source resource keeps running until cutover;
requires snapshot-capable storage, e.g.
ZFS/LVM-thin/Ceph/qcow2). Use "stop" for a
byte-perfect migration with no post-snapshot
write gap (source goes down for the whole
backup duration instead of just cutover) --
see the caveat at the top of this script.
--source-node <host> (default: \$PVE1_HOST, ${PVE1_HOST})
--target-node <host> (default: \$PVE_TEST_HOST, ${PVE_TEST_HOST})
--source-storage <pool> Where vzdump writes the backup on the
source node (default: local).
--target-storage <pool> Where the restored disk/rootfs lands on
the target node (default: \$PROXMOX_STORAGE, ${PROXMOX_STORAGE}).
--unique Restore with a fresh MAC address
(--unique 1), as the clone script always
does. Off by default here since the
source is being retired, not left
running alongside the target.
--no-start Don't start the guest on the target
node after cutover (default: start
it).
--remove-source Destroy the guest on the source
node (qm destroy/pct destroy) after
a successful cutover, instead of
just leaving it stopped there.
Irreversible -- prompts for its own
typed confirmation unless --yes.
--keep-backup Don't delete the vzdump archive from
either node afterward (debugging aid).
--yes Skip all typed confirmations (initial
backup, cutover, and --remove-source
if passed).
--dry-run Print the full plan and skip every
mutating step (vzdump, transfer,
restore, cutover, destroy, delete)
and every confirm prompt. Still makes
read-only SSH calls to look up the
source kind and check the target
VMID is free -- harmless on either
node.
-h, --help
EOF
}
vmid=""
new_vmid=""
mode="snapshot"
source_node="$PVE1_HOST"
target_node="$PVE_TEST_HOST"
source_storage="local"
target_storage="$PROXMOX_STORAGE"
unique=0
start_target=1
remove_source=0
keep_backup=0
skip_confirm=0
dry_run=0
while [[ $# -gt 0 ]]; do
case "$1" in
--vmid) vmid="$2"; shift 2 ;;
--new-vmid) new_vmid="$2"; shift 2 ;;
--mode) mode="$2"; shift 2 ;;
--source-node) source_node="$2"; shift 2 ;;
--target-node) target_node="$2"; shift 2 ;;
--source-storage) source_storage="$2"; shift 2 ;;
--target-storage) target_storage="$2"; shift 2 ;;
--unique) unique=1; shift ;;
--no-start) start_target=0; shift ;;
--remove-source) remove_source=1; shift ;;
--keep-backup) keep_backup=1; shift ;;
--yes) skip_confirm=1; shift ;;
--dry-run) dry_run=1; shift ;;
-h | --help) usage; exit 0 ;;
*) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;;
esac
done
if [[ -z "$vmid" ]]; then
echo "ERROR: --vmid is required." >&2
usage >&2
exit 1
fi
if [[ "$mode" != "snapshot" && "$mode" != "suspend" && "$mode" != "stop" ]]; then
echo "ERROR: --mode must be snapshot, suspend, or stop." >&2
exit 1
fi
[[ -z "$new_vmid" ]] && new_vmid="$vmid"
source_target="${PROXMOX_SSH_USER}@${source_node}"
target_target="${PROXMOX_SSH_USER}@${target_node}"
# No dry-run wrapper needed for the calls below: every mutating step
# (vzdump, transfer, restore, cutover, destroy, delete) is reached only
# after the --dry-run early-exit further down, so a plain `ssh` call is
# never in the dry-run path.
# --- identify the resource kind on the source node -----------------------
echo "==> Looking up VMID ${vmid} on ${source_node}..."
kind=""
if ssh "$source_target" "qm status ${vmid}" >/dev/null 2>&1; then
kind="vm"
elif ssh "$source_target" "pct status ${vmid}" >/dev/null 2>&1; then
kind="lxc"
else
echo "ERROR: VMID ${vmid} doesn't exist on ${source_node} as either a VM or CT." >&2
exit 1
fi
echo "VMID ${vmid} on ${source_node} is a ${kind}."
# --- refuse to clobber an existing resource on the target node -----------
if ssh "$target_target" "qm status ${new_vmid}" >/dev/null 2>&1 \
|| ssh "$target_target" "pct status ${new_vmid}" >/dev/null 2>&1; then
echo "ERROR: VMID ${new_vmid} already exists on ${target_node}. Pass --new-vmid" >&2
echo "with a free ID, or remove the existing resource there first." >&2
exit 1
fi
echo
echo "Plan:"
echo " source: ${kind} VMID ${vmid} on ${source_node} (storage: ${source_storage}, mode: ${mode})"
echo " target: VMID ${new_vmid} on ${target_node} (storage: ${target_storage}," \
"$([[ "$unique" -eq 1 ]] && echo "fresh MAC via --unique" || echo "original MAC preserved"))"
echo " cutover: stop VMID ${vmid} on ${source_node}," \
"$([[ "$start_target" -eq 1 ]] && echo "then start VMID ${new_vmid} on ${target_node}" || echo "target left stopped (--no-start)")"
if [[ "$remove_source" -eq 1 ]]; then
echo " after cutover: DESTROY VMID ${vmid} on ${source_node} (--remove-source, irreversible)"
else
echo " after cutover: source VMID ${vmid} left stopped (but intact) on ${source_node}"
fi
[[ "$keep_backup" -eq 1 ]] && echo " backup archives are kept on both nodes afterward (--keep-backup)"
if [[ "$dry_run" -eq 1 ]]; then
echo
echo "[dry-run] No backup, transfer, restore, cutover, destroy, or delete was performed."
exit 0
fi
if [[ "$skip_confirm" -ne 1 ]]; then
echo
if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) to confirm backing it up from ${source_node} for migration: "; then
echo "Cancelled -- input didn't match ${vmid}." >&2
exit 1
fi
fi
# --- vzdump on the source node --------------------------------------------
echo
echo "==> Backing up VMID ${vmid} on ${source_node} (mode=${mode}, storage=${source_storage})..."
vzdump_log="$(ssh "$source_target" \
"vzdump ${vmid} --mode ${mode} --storage ${source_storage} --compress zstd" 2>&1)" \
|| {
echo "$vzdump_log" >&2
echo "ERROR: vzdump failed on ${source_node}." >&2
exit 1
}
echo "$vzdump_log"
archive="$(echo "$vzdump_log" | grep -oP "creating vzdump archive '\K[^']+" | tail -n1)"
if [[ -z "$archive" ]]; then
echo "ERROR: couldn't find the archive path in vzdump's output above." >&2
exit 1
fi
archive_basename="$(basename "$archive")"
target_tmp_archive="/var/tmp/${archive_basename}"
echo "Archive: ${archive}"
# Always clean up the relayed copy on the target node, success or failure
# -- it's only ever a working copy, restored or not.
cleanup_target_tmp() {
if [[ "$keep_backup" -ne 1 ]]; then
ssh "$target_target" "rm -f '${target_tmp_archive}'" >/dev/null 2>&1 || true
fi
}
trap cleanup_target_tmp EXIT
# --- relay the archive from source to target ------------------------------
echo
echo "==> Transferring archive to ${target_node}..."
ssh "$source_target" "cat '${archive}'" | ssh "$target_target" "cat > '${target_tmp_archive}'"
# --- restore on the target node --------------------------------------------
echo
echo "==> Restoring as VMID ${new_vmid} on ${target_node} (storage=${target_storage})..."
restore_unique_flag=0
[[ "$unique" -eq 1 ]] && restore_unique_flag=1
if [[ "$kind" == "vm" ]]; then
ssh "$target_target" "qmrestore '${target_tmp_archive}' ${new_vmid} --storage ${target_storage} --unique ${restore_unique_flag}"
else
ssh "$target_target" "pct restore ${new_vmid} '${target_tmp_archive}' --storage ${target_storage} --unique ${restore_unique_flag}"
fi
echo "Restored on ${target_node}. Source VMID ${vmid} on ${source_node} is still up --" \
"the guest hasn't moved yet."
# --- cutover: stop source, start target -----------------------------------
if [[ "$skip_confirm" -ne 1 ]]; then
echo
if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) again to confirm CUTOVER (stop it on ${source_node}$([[ "$start_target" -eq 1 ]] && echo ", start VMID ${new_vmid} on ${target_node}")): "; then
echo "Cancelled before cutover -- input didn't match ${vmid}." >&2
echo "The restored (but not started) copy remains on ${target_node} as VMID ${new_vmid};" >&2
echo "the source on ${source_node} is untouched. Re-run cutover manually, or clean up" >&2
echo "the target copy if you no longer want it." >&2
exit 1
fi
fi
echo
echo "==> Stopping VMID ${vmid} on ${source_node}..."
if [[ "$kind" == "vm" ]]; then
ssh "$source_target" "qm stop ${vmid}"
else
ssh "$source_target" "pct stop ${vmid}"
fi
if [[ "$start_target" -eq 1 ]]; then
echo
echo "==> Starting VMID ${new_vmid} on ${target_node}..."
if [[ "$kind" == "vm" ]]; then
ssh "$target_target" "qm start ${new_vmid}"
else
ssh "$target_target" "pct start ${new_vmid}"
fi
fi
# --- optionally destroy the now-stopped source resource -------------------
if [[ "$remove_source" -eq 1 ]]; then
if [[ "$skip_confirm" -ne 1 ]]; then
echo
if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) one more time to permanently DESTROY it on ${source_node}: "; then
echo "Cancelled -- input didn't match ${vmid}." >&2
echo "VMID ${vmid} on ${source_node} is left stopped (not destroyed). VMID ${new_vmid}" >&2
echo "on ${target_node} is up and running." >&2
exit 1
fi
fi
echo
echo "==> Destroying VMID ${vmid} on ${source_node}..."
if [[ "$kind" == "vm" ]]; then
ssh "$source_target" "qm destroy ${vmid} --purge"
else
ssh "$source_target" "pct destroy ${vmid} --purge"
fi
fi
# --- clean up backup archives now that the migration succeeded -----------
if [[ "$keep_backup" -ne 1 ]]; then
echo
echo "==> Deleting backup archive from ${source_node}'s ${source_storage} storage..."
ssh "$source_target" "rm -f '${archive}' '${archive}.notes' '${archive}.log'" >/dev/null 2>&1 || true
fi
echo
echo "Done. VMID ${new_vmid} (${kind}) is now on ${target_node}, migrated from" \
"VMID ${vmid} on ${source_node}."
if [[ "$remove_source" -ne 1 ]]; then
echo "The source copy is stopped but still present on ${source_node} -- re-run with" \
"--remove-source once you've verified the target, or remove it manually."
fi
+73 -55
View File
@@ -5,103 +5,121 @@ sops:
age: age:
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAzdUlybVpDamVxYk94MDEv YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQUFhWVFVlVlBnNE5FTnMz
VjNiVjJkWjZvYjN1SUhMSG5ZOS91a3FrNDJRCi9IQWh4RVdwdkhwQjd6TTBhcW9J VkxkTmxpRXlzZ3pSNTVZWFUrSllsYWo0alRnCkJSc25TYktSTFFJdkQydHcxOUlj
alhLMlArM1VvVkE5VUxOQ1ZWNmw2SmcKLS0tIFRvWklUN0xxSFdKSk1vakExQ3Rv ajhQU1ZIb1lodEpHTnVhQjJ6WEthaDQKLS0tIDJCY1E2UVBaU3BoMzhXUXlIdnMv
OWpmSGRTUHlvVysvR2N4UHRYWHgvMHMKqapmFB4ct1FTPa1hMWyylvLycUvOEFop djZTcE1rcWNTOXFPMmFDYTVoRGo4ZTQKYy8g6pqP3VpTKDIBPbnC8NzCdDvOCKnL
enZI5SV1F86HOTEhK0QbnEW3jMl4ZXOtAlwcys2oE9a783MSgVPxeg== 14kSrKmKlzefTrbkVyriz2Jdl2s0F374yfQQFreZ3m4AffSACCxziQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEc0lHazMrVUR2UUlaQ21O YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBVQnpVWC9wcnIreG9GeE9U
T2Jha0EvUmJzVC96Wksxay9GVlEwU2lZWVRRClBQdWVqTjNTYnBrQldMeWR0dWNz UWhuRytkc2Flc0hyQm5yMjZnelNwaXhlWWc4CjBkWnd4cHNRRXQ0UXFkZGp4QlR1
QTN6T0VtQzhpM3UwKzVjUjl0Rkp4VzQKLS0tIEg5VzhUT1VTamQzZjNYT2orMnY0 eW5NNnE1WFhnb054M1pac2ZidFg4Y3MKLS0tIFdTNmk2V1l2WC9rUk8yd0ZnOEJS
Q1BpNzI2cVk3by9Ic3lrQ25sM05YMlkKXVofmCu7iI/my1o47p2eUzhXuP/V0NS8 VkNnejVGVUZPZkorQkltVEplN2FmdTAKRY7DPP5HeFQntn2f/fXLjU6M1V6iug86
mmIerz2v2uUwoS252qSU4a2vmxPOgWM0Os1vWsYamsapofQc6TaU2Q== BD09PI+T2DbIBQPotRZisw8IzHu9gY/O3+h0TccyIsXjI9wy/XPCAQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e recipient: age19gfn2yedg76dmztm4hncr7vf3r3c9j0qpt4rap7y7gersjk4m3ks2lhd0e
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYZHVnTEk2SjUwdTdKdWNV YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkczFSTVhxWHlIWjhRcWlV
dVFPNDFvdzY2VUlzMU5nK0t0UlUxU3Ewa0g0CjJCd0ZrMjhRZldaUDNIS1pGZTZn V2JPQXd5Wnk5R3NwWC81T3Z0MW4vYnd5S1ZZClV1NlU1Tzd6UkxPQ2M4MmhLV01G
Z2h2MEJ6TUpna1NNdXl4R3d1S1djVDQKLS0tIFJqMlZ3b0Zzalp2N0hIc0NHMzU4 d3VIb0RhR1RiNTZqNjlQcmg2YjdPeGsKLS0tIDQ1RTFTWGN4MnEvWkRUR3VnN204
RGRoVjdvMEkxam9DRzdYaW16enAyQ3cKUZTDqvWnmEMKHhI430coKHw3raIiD/o1 WVdFOXdmNC9FVFhBSGNEUUgyYWpYYzAKfdpeaFL/RrIbqpD9hNj8L7UxpmiBjE2I
1BtDWOmKFFcuaF5mRx/qTUjEwU5OZWOujqLSPoVbmsbhyxsG3XQSHg== go/dR2E1LLXsDPtnSuJb2EZYoFvSsjsIQQQDt+YwRv0fplRtssKdxQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzbXdrajlVQU5RditMckh6 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB6TjhmOTJ0bUpMQWowb0hB
ZGhvcU01UEdMbURlcG9mZzg0RTJhcHJ4QlQwCjgyRGVXYytxUFB0WXhadHQ1YVpF T3Era1loU1pMdmkxdnAvRkViekpqZWZjaWgwCkFRQXhFUy9PRVBma2JMUDhqY2F1
VUJpVzE2SzFPc3FHV25FQmVSQzNJTm8KLS0tIG9DazRkUkwyNHNQMnp5R3RySDBo VVFDRFNVbWpNaEczY1JVQUMyck9XdEkKLS0tIGpxc0tGdVFKK3FteVJKM1Fxa2ky
aGFLUFBjZjVxeWFzTXFldHVRNkNyYTAKd6eS5lks4+3SV1bFBQWyPi7OZcvRMDIc a21WLy9qV05hUURCTVBvcVh3cE45Z3cKXCYfXSjhApBoLbHDu2OOd57Y1zN54yy+
kYD5C/vMkKlvBYyODCWJyim2xgM/nyQNcf/q4BUS3HF/RidRdJh9NQ== WDQvz8PpMxhc1nU5Kw/cI+WmL1KvN0qQZfOx/7D4W+dy/ZDWX27TpA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu recipient: age120le4a5l8dh3lyfgvmj3d9ksmej6ajs5mer5y7r0vfg3x9fn69dqf8xgzu
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBFZEhIT1REUTFtdkIzaDFY YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZK0FYQnBHMHZ6dlpMYTlC
dGU4VjU3QlpPU1Rod1dTUWhBbjc0WTR5bUF3CkNGY3F2Q251aXlGNCt5ZjFWMTE1 WFdOWDFkRVBuY1pmdTFiUndLV3JXcndZa3pNCnJsd0tHN0FveWV6UUNQSEdpdWw5
RjY4bHZ1T29aK3N0SktjdUpZbSs4QVEKLS0tIDZrMWd3aFVRMEZMU1N1dlZxYTZS dWZITkxWelNIRlpKS1pnN0ZmVlQvZjAKLS0tIEUwMXdtNFdkUWdIRjlxc0owdTRr
U3gxWU5EVVRXUGxJVlZQZmpZRmdhbW8KJ8yD9laK2T1qn0z2uYNeI80rtMlOVi2M c1o1TmptWWd1ZGxzcWJJNzJ0K25PTTAKoos5rnkyQBCm+ZuhCCaMJwqJBo1fpnsl
qIPn/FSoWnKQ7NotSTGUtEhC+f/nXfUQrBd5fsfmja++hrevTmkqYw== G74wu5vbTBG4VjVhI5KqyiuiTRU4jPcGxysECqe7AyZUBGp7ndewgw==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep recipient: age1jy444f9d9stygj4p3w9kh54cqcfr654tvr75tdvee5cxsgtdtc9q3v60ep
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0N1owUGpuWWxWb2t2Mk5Z YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTVkVXaCtGTXVkRXRZMFZ1
QlZ6Nm5lcnNBY0d1NldBQXY3OThLM2UxbkY4CkZyRUl4S1JRaDhnemRIYWR1OHZx ZUdXM1hRcDhaSURqcGh2eksxTUNCckk5SVJjCnhkSDRKdDFSckxUWXd4SmVxWG5p
aWliVWh3V1RBL0R3Q0RqOE1SM0VBblUKLS0tIGpaUjJxdnV0UW55T01HcFBsdHR2 bU5OV0hzaWR6VDFwcDY2WlY4WnN3cFEKLS0tIElOVzRCcXR4U0dhajJySUhaZGps
aEJ3UzJ6RklZVFRrNGMraUtGYXVZd0kKaQmzjnI6xgV7H1YcoI+gRn2IVfOLTJBY MW9rQk1JVDFWRnFxVzhCUkRIS09EamsK1rVidD48PqwlEWQyjF7iQWU7aBdPqQHy
rO43fvVtZ79c1as6CL+GoEIJY/uVA/3M4gQ0vYbDDn57vm5OOJW8jg== z5LaSi3LvJX3rNE/+q0E8/gbZyjGpbEn3AUI5mBF64GY3IZkRxZSXQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu recipient: age120whqj96g26lsgy4udvgsn8dc9lumh8jeu3a564fx79rjr5lxffqmrljuu
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMWTcwd0JLS0lybWREaU5h YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvbnJCWU9UMXJXMG92b3hm
NHczcEQxeUx4U09oV2k5ZXpKTGxxR3ZYd2tnCkVXR3kyTHZSSWNaUCszL3Rvakhm WWNBTFpQamVWQThITmt5QUVwR3h2OHI2SlVRCjcxOEJTVkFjN0NhamFZQ1plK29w
bVl5bnFRVkhJRFVaY1NQYTNrd2JKMTAKLS0tIDJTT0YvUE5HUWVKRnhXY2tvMjI1 dk5XYkYxOXQ0YkVzcVc3VnhCQWlsV0UKLS0tIHZ5cWtFZUhDKzZkOE1BK2Y5TStR
VXNlcnJqcDM5d3poVU1SKzQyNWdpRzQKTht/ko7cy7OY0wGfza4eierYS7q/nCFH SFlDRjE4ZHpiVEJOQk5TUGNEN1B4amMKUCJ8CL8QpmRpFs83HD9TUn7NrPguuP8S
YsG74ez3piUQ1rdJRi0e29QWz4xL3JeU9oE+tr0rMB6WvgTmuX5gwQ== JQH/bzPorXTXJuyOKuKAZq1hK8BmiMUFksaZ03yN6YaFVIOeelEEMg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j recipient: age1xjst4frdh0th6q8m7p7u9g5af7ty5jqeum0p6z8a52a9q7st7ewqw8yl9j
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLWmtNOVVvRW9QbU50OUxv YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxWHV0S2dxeHZEMDNRSkhO
TUYxWXJ4cFVaMjVnVHJVK1VTZzhDd2tCMWpnCllPWnl2bjcxemF6cFp1blZnMEMy ZFJHaiszd0tzYzlzd3gvbXNSQmlMVlJUNGcwCnVrM29MdFZCR1NBYnpkQ1k5VFZQ
clhqdlVzdEZTYnY3b0ZhR2hsMXRlQTgKLS0tIHEzNU9yNmhackt6R2dLT2dZSEFJ b3Z2Q3ZGekVQZkZKWGlka3NDOHJ0R1EKLS0tIDlXTmNzUk0wVXo0UWhkd0ZvK3FI
UnJXeWVyQld6NFhXUWowUlVXaEtKVUkK2Saa8w9/SOWw1VXWkUg62ay31bSIqHWB UzJxU3RkdWs4aTZYVVkrS056bTN1ek0KgKJNz8GvynX5pK33aW9x3v6yr2Ox0LCT
8QmNafddV0WQ8oYcD8ZMf8Bm8jP9yLMUDOiVzVZiiTGWEx7EbUQV3Q== GGrt+ddbKLcwpBpYjfWkFhffO330EKui73S+c/qMf8N9j6wzalOTpQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg recipient: age10at8862478urh0eeuwh8hzln6ck78jgwtztgxatwqlzwagg77y5snm4xzg
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2OEs4dWJ5UTBoY2hHanZY YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3ZWJlbDl1TEkvWEpBMC9Y
N3VNMWdGVkJHMlVublBmN2NiMUNtRnVnaXdrCnYzV3EvRzV1OXNyOFVKc2lXMFov ejY1MnFMUExBeDhITndxNy9YUi9hU0tXM1JJCjcwNEppcDdxYzlCSVMrMExWa3A4
Mkgvc2dFdyt0MGJIMlFPckpBeDZ5c0UKLS0tIERrOGc4aWtFSTJWN1M2V0hyQ3hS cmFPSjV3LzkyMXZCUDU2QmtHRmpHRmsKLS0tIFFTdXBOaDJJTERseXlGbmdrQzhD
TC9HYVMyVWVBc1lHbUlqbnpsOWJobU0KkZZph2dnKwi4w+RXf7RXoHCYkXxmuLNj TWtnRFdIRXpsNkY0U1BiczNsdUk1V1kKGpndKmT8kj/oIxQuxQALfzscw+CsVmnj
rinQQlQeOwQptBn7+kRJubNjyHOlSHXhpsbV3/IkJRNMhw69knzasQ== cyPC3bF+tG6LcqqoKLjPSJfcIgzhnX7cAr/wwESavemLn8L/zQMe4w==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs recipient: age1ezk9x53zt8kcnscdm80jcyf0xq97vndv7jsn3rl8cc0cwm2jmpmq372dzs
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoWkljeFFBS3hhOG0relVz YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpQVVyY3UrSHFWSmpDdmRE
V0JUK2hWaklxTGVhMWR4bytPczhjcldheno4CmZUTWszREVUdVVQazNOdXAvY0FN dlh5akFqVFdDVFNMSHE1eVJnZzR6YzFHSVFRCko4UE9EdXNxZzF2MW5PTTN6dEdU
bEtQSDJveHplNTE0a0FvWEFxVlEreDgKLS0tIGNKUkNVQ3BDQ3d0V1lyY0VsaHda ZHM1MGowcVB2Y1ZlOTVHdnNtY3diM2cKLS0tIDBYSmh5dVVPaTM3d0ErcC8wMDNB
cDNHa3lYTkxBeVFlSGtIMVFScmxsazAK5ZyJ35/jjFRhQdG1PMOcEJ3MJf96i8DZ eUpHWnZlYnJsbHZuS3pwbG15UGtwN2MKVPQA1MpjIfYAsNacoAbpvZNuAIkvx7ER
AvCw9jc7Hsbs6+LoZ45K1QKoWoZmBkmAatbksXo3jnAzEHmcQqODKQ== CvWBKEHUVm6m8905BXzv8MdGTAk0EyCIP3aMmYqTIYfv2k9pP0T08A==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp recipient: age1fxxzpnfse8nd9wz78ht3m0plrmraacf4cpga0pe8fm2tdnqcgy8q7qsyvp
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtWGRLTCt5dXZMR1d6a3lD YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA0MTQ4MzN0bDJNL1l1bmpX
Z0t1SDAwY3V3MXI1MGt5RGkrc1JHSVRUUzNZCi9UOTBTcVY3NkhFb1FFOS9vbTY2 ekxDUXRNa0JHWWltZFNGTVltTFdSSE82SVZvCmdUWUdja3JIajMzY09IMUE5elox
S1BqNmd2eWVXbnliOFJWRDY0UmhYV3MKLS0tIExYNlJxcTBidnpObHgweHlqNHhz MDdEakFJTmtkRWF2R1BGNkQ4U3grNWsKLS0tIE9hZUhkVGI1ZEpzdDhRU21EZm91
OXpwRHhyOGhNRXNTWDBIUVJCR0VPTkkKQZh9e7lOINL0khHS6tBehCrm+SX5Q6XE VnJNb1kyQ05MM0RJa1lLUEtjWWxkSTAKHVAKcGcWl6LncJALRBU9RKP7ot6C6GSE
XL4FpBsJ5+MnxTx1O/bu6/f7OvZqfnxhQv0lkKOMJnkZfPZx5D+meg== 1iZtj1SNX6wzEWrhOEnV37aQ8bKZj6u+Y/q6/vJ4qiBs78y/drdIzA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05 recipient: age190htw7prp4vln076dxjx3gxxaq06h0zl0te7cqgpx79vl3lhkaes8suy05
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBNUEVWY2NpVEU4OTZPZTZR
UzdKcktpUGJOcnJ0aEhkQlhvUWdUYkV2SkdzCm51eGJVeHJMcVRRRld0dFRCYUxr
TTN2WEhOVjRqV0FtQXowZWNTbkJneEUKLS0tIFpUazZpTUNWZUZBSFE0VDZZbkJu
SFVlUVhySnNqUENYOG9qUm5ZMDc1ZW8Kv0lY5dhnCEheM0sttfr4p7IL+EVog16T
OapUdbuXL2l7t7URzHnvfG/nbOtJIjH8a0XFsWyJChtNXpF2d/vf2g==
-----END AGE ENCRYPTED FILE-----
recipient: age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvTzc2aExGTHFrc0ZtL3pq
YlZLbzd3MVZHZTB0VUQvSXZ4NGVsZk42c2hBCk1vbTk0Tnl3b01vbVZaMkJ5aldE
WmZBMGFjb2pjQXpYcnBxWmp0UUsrdXcKLS0tIG16SG9JbFdkbmVidCsxUnpBR3V3
RzNOY3hIRWk4UXh6N3NrcjNSU3ZwWTgKaExY4U2s8E6ojljJ+4TU+YJhcLXyuVA1
ROB70jQCjFvQOeo6thjQohSSUoPKhxSl1/nr4ZiGBO3/VskzihckKg==
-----END AGE ENCRYPTED FILE-----
recipient: age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
lastmodified: "2026-07-19T02:30:40Z" lastmodified: "2026-07-19T02:30:40Z"
mac: ENC[AES256_GCM,data:UiL3VMDF6rq4Nr87KspcDx434q3tfNXeb5pwH2O+4ssNQ6xzcYDdzXBnhAY3zLBsqPMKrvHBd4Ot/gEMcq3FMIVe7Q6p9yWKpep66KZ/yWEhAlwIVhD79Oj8VS+1CHKjf25zpRdhZorp04oeFQQd9VfjJB4EE/Q1aVbwTGlpIic=,iv:i/0conaFgFia+wzNTdUL6tlSTw35HTK3Ap1Sr5RGHf8=,tag:ULbz5FllShA/JjlSRdxA0g==,type:str] mac: ENC[AES256_GCM,data:UiL3VMDF6rq4Nr87KspcDx434q3tfNXeb5pwH2O+4ssNQ6xzcYDdzXBnhAY3zLBsqPMKrvHBd4Ot/gEMcq3FMIVe7Q6p9yWKpep66KZ/yWEhAlwIVhD79Oj8VS+1CHKjf25zpRdhZorp04oeFQQd9VfjJB4EE/Q1aVbwTGlpIic=,iv:i/0conaFgFia+wzNTdUL6tlSTw35HTK3Ap1Sr5RGHf8=,tag:ULbz5FllShA/JjlSRdxA0g==,type:str]
unencrypted_suffix: _unencrypted unencrypted_suffix: _unencrypted
+28 -10
View File
@@ -3,22 +3,40 @@ sops:
age: age:
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBURmMzN3hrSlNrUkkvVWNl YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaYU9HR0lETDhkYXR5NWlj
L3M1dEhWeW14N0RFNVRPci9QK1YyTFdqRVVJCk5WaWswT2NicldkYzZjbVhYU2xu L0diSE5JQU9KWjRoU3hrVjdqZ2tPNUtOdFZvCjdUVGNFbDVYa3pVSC9ZWVNORER4
MGFsNmUzeTN2TS9wOEdvRURpVUVYZXMKLS0tIDZ6MEdPTVhCaTQ2UXFWTUFtc0pm QWozUlhoSEtjTk9IRlM3VkZoYlc1RTAKLS0tIDlHS01WVWlOMWFEQU5GTVRLZVhV
MFlJb0c2WXJtMGRLZEZYY0pZWWpFWm8K/mlYZIe8UC0QU+1mq3NtrtTF5b2m5hCK VTcwekhrRHB3SlVYT2MzOW5GbE52dkkKKCWehPhpdGapdyzpll20NJUcZwvW/7X8
+K0QiZLTKmmDcr4bRhZ32VE7R7GRwtMNnOP/mElZvPAyWyHHhRiOHg== KQ1EqAgI2fewnbwuIDYCleN0b0SLJNUeSV/tFKDDoTMnHWCdeD4ECg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBURHlUL0RNMUtNallEcG5p YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB4YndybFBTQ2p4SGZ4SDJs
eG4xby8yVzdQUTBaNkl4ano0YjBMcDd0Wm5NCkZvaXNPZm9wemJkMmNSdGdOaTI4 a2p0eFRQOVVWcGd2a25ESW9ESGx1RWw0Zno4CnhpVVh3cGI5UjY0YmFINFFPMTh4
Z1RwUnhiRUpCMWZaeWtlSVBmNW5KOXMKLS0tIDk0R0k3ZHczTFNCWUZxSWF0M0FJ b1B5SjJ3NTNvUE1QUmJjVFozY1dYS2MKLS0tIC9HZGNpOFZhZGNFZGt5blJuZXVV
MGlZMmtuSFYrcG1meDNMWDNqSjFxcE0KDu2dAc0gqmmPkpbpBe4YohM7rYmUwEkI SXpkRzV4d2ppV3ZQZSt1dmxYNGVFMUEKmSe9dkrmkND81Hw2/ATAmFvcmhk1tUC1
V2FUQwjlvh50svtjCVdYbx2xuq4sQLnKelk/q1onLw60FwsVfzD8sQ== LxxBw54IVHUqwYKgRYUYRNu+pykDT5OnFDPiskd49Xso99LY87PyiA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf recipient: age1ll6hj5ggruetgjwjfnplpn5xtq35uhlcdflksx3xmnjm6s3uad9sz70jkf
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLd1FSSHVTSGdHWm8wMVQz
d0dlOEsxeGwxdHU0eUlFSUxka0ZmcVpnOTM0CndYNUVjVy92QVhNY2orQmpSQjYy
VS9KM0NUTXhuM0lCSDBZMWtISEdtWTgKLS0tIExTL2wvS3FEdVViUmRYZEFsR3R0
YTJFM200RjF6MjNxOFA4eHRpWVhtRkUKOqBIT445HnPXrrH/qV6FIuAhAuJmSL6V
+PQopM/m3PAnK5m5Mu3cfjYfDiB8+GWTABhljfT+GbcoK7CqWLehrw==
-----END AGE ENCRYPTED FILE-----
recipient: age1ukpqxzl44mnjpy5r96sfuc5sqzm47u4k8ujjh5qdgy6jvl9uqgpspymqfk
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAvVXF4a2NmaW15R2VXS3FP
SDVPUGpUWTlIWjl2N2N2SXR1UlRkdllWNVRJClZDbVo4dUhrZytqZkMrYWtpOEZx
c0dSZGViN04zQ1B1WEZEWm1QM1lsejQKLS0tIE0wa0k5Rm1xZmw3OHFESkVXc25j
NlFpYTJSckQ4MVlZQ01reDlkaWY3TkEKsmQlreRhRAjVZ/q5x52FNATDF2sLhbHo
djOZigZx2rs6shqQ6It/XRJ8CiPaXZBPOt529Gwmplu+hWlAU1+l2Q==
-----END AGE ENCRYPTED FILE-----
recipient: age15kh7akxlx7zn00tey79rq2g8lgs4j5y77rcnyfxrxap8ckfu0a9sqvtdhh
lastmodified: "2026-07-19T02:30:40Z" lastmodified: "2026-07-19T02:30:40Z"
mac: ENC[AES256_GCM,data:rKHZjU/MH08ASTlu32HZO9uWmsBYuMCEC6M8gwVhzuWvmablnP05tS2z13XfaWaCEUXk6kmGJKuU0zu5+IKVZgamCF6DAMtxQb6bVCaLsoAm/GSqWQ5VI9eHqgnSSdN/o3ul/33Rf8iBQo4aw8FFAmDVuNz8bfAn0QefFTj0ByI=,iv:JD2gtqRinOY77etg6PUmZNovkYl1Q3F6ZvRi4x7RznQ=,tag:/5IMpWKRVt+l1luCTQE0BA==,type:str] mac: ENC[AES256_GCM,data:rKHZjU/MH08ASTlu32HZO9uWmsBYuMCEC6M8gwVhzuWvmablnP05tS2z13XfaWaCEUXk6kmGJKuU0zu5+IKVZgamCF6DAMtxQb6bVCaLsoAm/GSqWQ5VI9eHqgnSSdN/o3ul/33Rf8iBQo4aw8FFAmDVuNz8bfAn0QefFTj0ByI=,iv:JD2gtqRinOY77etg6PUmZNovkYl1Q3F6ZvRi4x7RznQ=,tag:/5IMpWKRVt+l1luCTQE0BA==,type:str]
unencrypted_suffix: _unencrypted unencrypted_suffix: _unencrypted