Archived
Compare commits
18
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
  beatzaplentyandClaude Sonnet 4.6
|
2741451642 | ||
|
beatzaplenty
|
9f1fed26c2 | ||
|
beatzaplentyandClaude Sonnet 4.6
|
3b6ac30946 | ||
|
beatzaplentyandClaude Sonnet 4.6
|
cee33aa2d8 | ||
|
beatzaplenty
|
ac9099a1f2 | ||
|
beatzaplentyandClaude Sonnet 4.6
|
c96752e5d0 | ||
|
beatzaplenty
|
43314b6aa3 | ||
|
beatzaplenty
|
db84a0bcc1 | ||
|
beatzaplenty
|
419b0f706b | ||
|
beatzaplenty
|
89e1c5dc9a | ||
|
beatzaplenty
|
77db51a88f | ||
|
beatzaplenty
|
ad0b4a14e6 | ||
|
beatzaplenty
|
c9dbb03e31 | ||
|
beatzaplenty
|
b6ed0c6007 | ||
|
beatzaplentyandClaude Sonnet 4.6
|
78fdf3d94c | ||
|
beatzaplentyandClaude Sonnet 4.6
|
f85c65870f | ||
|
beatzaplenty
|
97c9353fa8 | ||
|
beatzaplentyandClaude Sonnet 4.6
|
cded77919d |
+20
@@ -21,6 +21,8 @@ keys:
|
||||
- &proxmox-tailscale-router age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
|
||||
- &proxmox-ha-server-1 age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
|
||||
- &proxmox-ha-server-2 age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
|
||||
- &proxmox-ha-docker-1 age16y0vfts5v2k20e2rj23qa5lc5gm96gm64uwsqsr0y688xl0ne46qcfr0sm
|
||||
- &proxmox-ha-docker-2 age1tm3zj5lp3elw2j832az4nj9xxmhqd66ag3cqcv3vskvmd7qdmqmsdpdcn0
|
||||
|
||||
creation_rules:
|
||||
# Shared across every currently-deployed host: root/nixos password hash,
|
||||
@@ -52,6 +54,8 @@ creation_rules:
|
||||
- *proxmox-tailscale-router
|
||||
- *proxmox-ha-server-1
|
||||
- *proxmox-ha-server-2
|
||||
- *proxmox-ha-docker-1
|
||||
- *proxmox-ha-docker-2
|
||||
|
||||
- path_regex: secrets/nix-cache\.yaml$
|
||||
key_groups:
|
||||
@@ -199,3 +203,19 @@ creation_rules:
|
||||
- *admin
|
||||
- *proxmox-ha-server-2
|
||||
# proxmox-ha-server-2 added by sync-host-keys.sh
|
||||
|
||||
# Host keytab for ha-docker-1 FreeIPA enrollment (binary sops file).
|
||||
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
|
||||
- path_regex: secrets/ha-docker-1\.keytab$
|
||||
key_groups:
|
||||
- age:
|
||||
- *admin
|
||||
- *proxmox-ha-docker-1
|
||||
|
||||
# Host keytab for ha-docker-2 FreeIPA enrollment (binary sops file).
|
||||
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
|
||||
- path_regex: secrets/ha-docker-2\.keytab$
|
||||
key_groups:
|
||||
- age:
|
||||
- *admin
|
||||
- *proxmox-ha-docker-2
|
||||
|
||||
@@ -7,7 +7,7 @@ servers and workstation.
|
||||
|
||||
The flake exposes NixOS configurations named `<platform>-<buildtype>`
|
||||
(platforms: `linode`, `proxmox`, `lxc`, `baremetal`; build types: `minimal`,
|
||||
`nix-cache`, `server`, `docker`, `gui`, `pxe-boot`, `tailscale-router`,
|
||||
`nix-cache`, `docker`, `gui`, `pxe-boot`, `tailscale-router`,
|
||||
`tor-relay`, `ha-server`), generated from `modules/platforms/*` and
|
||||
`modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not every
|
||||
combination is built — `pxe-boot` has no `linode` variant, `ha-server` only
|
||||
|
||||
-150
@@ -1,150 +0,0 @@
|
||||
# Flake End-to-End Audit Report
|
||||
|
||||
**Date:** 2026-07-21
|
||||
**Scope:** Full static lint/eval sweep + live build/deploy/interrogate/destroy testing of every `lxc-*` and `proxmox-*` flake target against `pve.sweet.home`, plus an audit of the operator's ability to manage the flake/secrets tooling.
|
||||
**Branch:** `worktree-flake-e2e-audit` (this session's isolated worktree)
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The flake itself is in good shape: `nixpkgs-fmt`, `statix`, and a full eval + dry-run build of every host and package are all clean. Every `lxc-*`/`proxmox-*` target's NixOS configuration builds successfully — no target has a broken derivation graph.
|
||||
|
||||
The issues found are **operational, not code-level**:
|
||||
|
||||
1. **pve.sweet.home is critically low on disk space** (91-95% full during this session) and cannot currently build the two largest closures (`gui`, `pxe-boot`) to completion — this actively blocks deploying/redeploying those hosts via the documented workflow.
|
||||
2. **A real, reproducible secrets-decryption failure** was caught live: a stale cached container image (built before a same-day sops-key fix) boots with sshd never starting and every secret failing to decrypt. This is a **general hazard in `create-proxmox-resource.sh`'s "reuse the cached image if present" default**, not a one-off.
|
||||
3. **sops key/anchor drift**: `proxmox-minimal` has a `.sops.yaml` recipient anchor with no corresponding private key anywhere in this environment; several `lxc-*`/`proxmox-*` targets have no sops registration at all yet.
|
||||
4. One concrete script bug was found and **fixed in this session**: `create-proxmox-resource.sh` never enabled the QEMU guest agent channel on VMs it creates, despite the guest OS already running it.
|
||||
5. A management-surface audit (of the operator's ability to run this repo day to day) found 5 process gaps, detailed below.
|
||||
|
||||
Nothing here required or received a `nixos-rebuild switch/boot/test`, `nixos-install`, or any disk-formatting command — all validation was `nix build`/`nix eval`, plus disposable `pct`/`qm` create-then-destroy cycles via the repo's own `create-proxmox-resource.sh`.
|
||||
|
||||
---
|
||||
|
||||
## 1. Static Analysis Results — all clean
|
||||
|
||||
`bash scripts/codex-maintenance.sh --full-check --dry-run` (whole-tree sweep, not just changed files):
|
||||
|
||||
| Check | Result |
|
||||
|---|---|
|
||||
| Secret grep | Clean — only the documented exceptions (installer's own hashed passwords, `access-tokens` comment references) |
|
||||
| `nixpkgs-fmt --check` | 0/53 files would be reformatted |
|
||||
| `statix` | No lint warnings |
|
||||
| nix-cache host key drift check | Up to date |
|
||||
| Full eval of every host's `system.build.toplevel` | All 19 `nixosConfigurations` targets evaluate cleanly |
|
||||
| Dry-run build of every host + package | All succeed, no derivation errors |
|
||||
|
||||
No drift, no formatting issues, no lint findings anywhere in the tree.
|
||||
|
||||
---
|
||||
|
||||
## 2. Per-Target Test Results
|
||||
|
||||
Legend: **LIVE** = built on pve, `pct`/`qm` create → interrogated → destroyed. **BUILD-ONLY** = `nix build` validated the config (mostly `.config.system.build.toplevel`, occasionally `.tarball`), no resource created on pve.
|
||||
|
||||
| Target | Test type | Result | Notes |
|
||||
|---|---|---|---|
|
||||
| `lxc-docker` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT105 is already running this identity in production; `--allow-duplicate-host` would have destroyed it. |
|
||||
| `lxc-minimal` | **LIVE** | ✅ PASS (after retry) | First attempt reused a stale cached tarball predating a same-day sops-key commit → activation failed, sshd never started (see Finding #2). Redeployed with `--force-rebuild`: clean boot, `systemctl is-system-running` = `running`, secrets decrypted, sshd listening, users correct. |
|
||||
| `lxc-nix-cache` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — CT101 is already running this identity. First local build attempt appeared to hang on a remote-builder handoff to nix-cache; killed and retried with `--builders ""` (local-only), succeeded. |
|
||||
| `lxc-gui` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Registered a fresh sops key (no prior registration existed), built successfully through the full NixOS system closure, then **failed packaging the tarball**: `No space left on device` on pve's root filesystem. Not a flake defect. |
|
||||
| `lxc-pxe-boot` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Same failure as `lxc-gui` — this target additionally builds a full nested installer/netboot image (`stage-installer-artifacts.nix`), making it similarly large. Failed with the same `No space left on device` error, immediately after the gui attempt had already consumed pve's remaining headroom. |
|
||||
| `lxc-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; live deploy also would have hit `boot.zfs.extraPools` trying to import a real ZFS pool that doesn't exist in an isolated test container — an expected limitation of testing this build type outside its real hardware, not a bug. |
|
||||
| `lxc-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
|
||||
| `lxc-tor-relay` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT106 already holds this identity in production. |
|
||||
| `proxmox-docker` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — both CT105 *and* VM103 already hold `docker` identities. Combined `toplevel` + `diskoImagesScript` build crashed with a **Nix-internal assertion failure** (`worker.cc:360`) under this session's memory pressure (see Finding #6) — not a flake bug. Retried with `toplevel` alone: clean. |
|
||||
| `proxmox-minimal` | **LIVE (attempted)** | ⚠️ BLOCKED by key drift → BUILD-ONLY | `.sops.yaml` has a registered `&proxmox-minimal` anchor but **no corresponding private key exists anywhere in this environment** — the script correctly refused to generate a mismatched replacement. Fell back to `toplevel` build: ✅ PASS. |
|
||||
| `proxmox-nix-cache` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
|
||||
| `proxmox-gui` | BUILD-ONLY | ⚠️ Killed after ~40min (resource-limited) | This session's local build machine has only 2GB RAM; swap filled completely (2.0/2.0GB) and the build stalled, so it was killed rather than risk destabilizing the session further. **Not a flake defect** — the equivalent `gui` NixOS configuration already proved fully buildable during the `lxc-gui` live attempt above (it built the entire system closure successfully and only failed at the pve-side tarball-packaging step due to disk space, not the config). |
|
||||
| `proxmox-pxe-boot` | BUILD-ONLY | ⚠️ Killed after ~35min (resource-limited) | Was deep into building the nested installer's kernel initrd (this build type bundles a full netboot installer image via `stage-installer-artifacts.nix`) when killed to keep the audit moving. **Not a flake defect** — this target's own module logic was already effectively validated via the earlier *live* pve deploy attempt (`lxc-pxe-boot` above), which built the complete image and only failed at the final tarball-packaging step due to pve's disk space (Finding 1). |
|
||||
| `proxmox-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; same ZFS-pool caveat as `lxc-server` would apply to a live deploy. |
|
||||
| `proxmox-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
|
||||
|
||||
**Not tested at all:** `linode-*` targets (not deployable to Proxmox) and `installer` (not a normal host) — both were still covered by the static eval/dry-run-build sweep above.
|
||||
|
||||
---
|
||||
|
||||
## 3. Findings, Ranked by Severity
|
||||
|
||||
### Finding 1 — pve.sweet.home is critically low on disk space (blocks real deployments)
|
||||
|
||||
At session start: `/dev/mapper/pve-root` was **95% full, 5.3GB free** (of 94GB). After two failed large builds it recovered slightly to **91% full, 8.2GB free** (nix cleans up its own failed-build scratch space). `/nix/store` alone is 26GB; `nix-store --gc --print-dead` reports **zero** reclaimable garbage — everything currently in the store is a live GC root, so `nix-collect-garbage` won't help without first removing old roots.
|
||||
|
||||
**Why it matters:** `create-proxmox-resource.sh` builds every VM/CT image **directly on pve**, not on a build machine and transferred over. With <10GB headroom, any closure approaching a few GB (the `gui` build type: full Cinnamon desktop + Firefox + LibreOffice + GIMP + VS Code + xrdp; the `pxe-boot` build type: nginx/atftpd *plus* an entire nested installer/netboot image) cannot currently be built there at all. Both `lxc-gui` and `lxc-pxe-boot` failed live with `No space left on device` during this audit.
|
||||
|
||||
**Recommended action:** Expand `pve-root`'s LV, or free space by pruning old container templates in `/var/lib/vz/template/cache` (1.5GB) / old backups in `/var/lib/vz/dump` (306MB) / auditing what's pinning 26GB of `/nix/store` as live GC roots (likely `result-*` symlinks — see below). This is real production disk state; **not something this session touched or fixed** — it needs the operator's judgment on what's safe to remove.
|
||||
|
||||
**Secondary, smaller finding:** every `create-proxmox-resource.sh` run leaves a `result-<target>` symlink in the node's repo checkout as a permanent GC root (`ls /root/nixos/result-*` on pve showed 3 from this session alone: `lxc-docker`, `lxc-minimal`, `lxc-nix-cache`). These accumulate forever and pin their entire closures in the store. Consider having the script clean up its own `result-*` link after staging the built artifact (or use a temp `--out-link` under `/tmp`), so `nix-collect-garbage` can actually reclaim old build outputs.
|
||||
|
||||
### Finding 2 — Stale cached images can silently ship broken secrets (reproduced live)
|
||||
|
||||
`create-proxmox-resource.sh`'s default behavior is: if the node already has `<target>.tar.xz`/`.raw` staged, **reuse it** — only `--force-rebuild` forces a fresh build. This session hit exactly the failure mode `docs/auto-installer.md` already warns about: `lxc-minimal`'s cached tarball (built 2026-07-20T15:57Z) predated a same-day sops-key fix commit (2026-07-20T17:49Z, "clean up in ailse 3"). The deployed container booted with:
|
||||
|
||||
```
|
||||
sops-install-secrets: failed to decrypt '.../common.yaml': Error getting data key: 0 successful groups required, got 0
|
||||
Activation script snippet 'setupSecrets' failed (1)
|
||||
```
|
||||
|
||||
— every secret permanently failed to decrypt, `sshd` never started (though the container otherwise looked "running"). This was **not a code bug**: the currently-committed `secrets/common.yaml` decrypts fine for that host's key when checked independently; the *cached artifact on pve* simply reflected an older commit's ciphertext. Redeploying with `--force-rebuild` fixed it immediately.
|
||||
|
||||
**Why it matters:** this is silent and easy to trigger by accident — any operator who redeploys a host without remembering `--force-rebuild` after a secrets change gets a container that looks like it started (`pct start` succeeds, `pct status` = running) but is completely inaccessible.
|
||||
|
||||
**Recommended action:** Have `create-proxmox-resource.sh` compare the cached image's build timestamp (or embed the source commit hash in the staged filename) against current HEAD, and warn (or refuse without `--force-rebuild`) if they differ — rather than silently trusting presence alone.
|
||||
|
||||
### Finding 3 — sops key/anchor drift
|
||||
|
||||
Two concrete instances hit live during this session:
|
||||
|
||||
- **`proxmox-minimal`**: `.sops.yaml` already has a registered `&proxmox-minimal` age recipient, but this environment's `host-keys/` directory has no corresponding private key file. `sync-host-keys.sh` correctly refused to generate a replacement (it would silently mismatch whatever's already registered/deployed) — but this means **no environment currently has this host's private key**, unless it exists on some other machine that was never backed up here.
|
||||
- **`lxc-gui`**, and by the same logic `lxc-server`/`lxc-tailscale-exit-node`/most `proxmox-*` targets, have **no sops registration at all yet** — expected for undeployed hosts per `docs/auto-installer.md`, but this session's live-testing needed to register `lxc-gui`'s key on the fly, which immediately hit **Finding 3b**: registering a key locally does nothing for pve's build until it's pushed to `origin/main` (pve builds via `git pull`, not from this uncommitted worktree). This is exactly gap #4 the management-surface audit (below) already flagged in the abstract — this session hit it concretely.
|
||||
|
||||
**Recommended action:** for `proxmox-minimal`, decide whether to regenerate its key (destroying old-key decrypt access, if anything still holds it) or track down wherever the original private key lives and back it up here. For the general pattern, see the management-surface audit's recommendation to pre-flight-check key registration before building.
|
||||
|
||||
### Finding 4 — QEMU guest agent never wired up (found and fixed this session)
|
||||
|
||||
`modules/common/configuration.nix:44` sets `services.qemuGuest.enable = true` on every host — the guest-side agent daemon is correctly enabled everywhere. But `scripts/proxmox/create-proxmox-resource.sh`'s `qm create` call never passed `--agent 1`, so **Proxmox never created the virtio-serial channel** the agent needs. Every `proxmox-*` VM this script ever created was silently missing `qm guest exec`/IP-address reporting in the Proxmox UI, despite the guest daemon actually running.
|
||||
|
||||
**Status: fixed in this session's worktree** (`scripts/proxmox/create-proxmox-resource.sh`, `qm create` now includes `--agent enabled=1`) — see the diff, included in the PR from this session.
|
||||
|
||||
### Finding 5 — Orphaned container on pve (CT102)
|
||||
|
||||
`pve.sweet.home` has a stopped LXC container, **VMID 102**, with an essentially empty config (`lock: create` and nothing else — no hostname, no rootfs, no network) — the leftover of a `pct create` that started and never finished. It predates this session (not created by any of this audit's activity) and wasn't touched. **Recommend the operator confirm it's abandoned and remove it** (`pct destroy 102 --purge 1`) — left as-is it may be someone's genuine in-progress work, so it wasn't assumed safe to delete autonomously.
|
||||
|
||||
### Finding 6 — Nix-internal crash under memory pressure (tooling, not flake)
|
||||
|
||||
Building `proxmox-docker`'s `toplevel` and `diskoImagesScript` together crashed with a Nix-internal assertion failure (`Assertion '!awake.empty()' failed ... worker.cc:360`, a known class of bug in Nix's multi-goal build scheduler) while this session's 2GB-RAM build container was under heavy swap pressure (1.8-2.0/2GB swap in use) from a separate concurrent build. Retrying the same target alone (no concurrency) succeeded cleanly. **Not a flake defect** — purely an artifact of this session's constrained build environment; noted for completeness since it looked alarming in isolation.
|
||||
|
||||
### Finding 7 — Management-surface audit: 5 operability gaps
|
||||
|
||||
A focused audit of "can the operator actually run this repo day to day" (flake, home-manager, sops, related scripts) found:
|
||||
|
||||
1. **No documented recovery path if the `&admin` sops age key is lost without a backup.** `scripts/secrets/backup-admin-key.sh` exists and works but is referenced nowhere in `README.md`/`docs/` — no forcing function ensures a backup was ever taken. `rotate-admin-key.sh` requires the *old* key to re-key; there's no bootstrap-from-nothing path documented (the real fallback — deriving an age identity from any still-live host's own SSH key — isn't written down anywhere).
|
||||
2. **home-manager has no standalone iteration path.** It's wired only inside `nixosConfigurations` (`flake.nix`) — no `homeConfigurations` output. The fastest real shortcut (`nix build .#nixosConfigurations.<target>.config.home-manager.users.nixos.home.activationPackage`) isn't documented anywhere, so the practical workflow is a full host rebuild to test one HM tweak.
|
||||
3. **Gitea's flake-lock-update workflow pushes straight to `main` with no pre-merge validation.** `.gitea/workflows/update-flake-lock.yml` commits and pushes `nix flake update`'s result directly; `codex-maintenance.sh` only runs *after*, on the resulting push — a genuinely broken lockfile bump lands on `main` before anything catches it. (The GitHub-side workflow is safer — PR-based — but has the opposite gap: nothing alerts if the PR sits unmerged.)
|
||||
4. **No pre-flight check that a build target has a registered sops key before building it.** `docs/auto-installer.md` documents the failure mode (silent, total secrets-decrypt failure) but nothing in `create-proxmox-resource.sh` refuses to proceed when it's about to build a target with no `.sops.yaml` anchor — it's on the operator to remember. This session's `lxc-gui` test hit close to this exact gap (needed the key added on the fly, mid-session).
|
||||
5. **`vars.remoteBuilderAuthorizedKeys` has the same drift risk as `vars.nixCacheHostKey`, but no checker script.** `sync-nix-cache-host-key.sh --check` guards the latter; the former (and `vars.pxeServerIp`/`vars.pbsIp`) has no equivalent — a rotated/revoked client key just silently stops working with no diagnostic pointing back here.
|
||||
|
||||
---
|
||||
|
||||
## 4. Action Plan (priority order)
|
||||
|
||||
1. **Free up disk space on pve.sweet.home** (or expand `pve-root`). Blocking: `lxc-gui`, `proxmox-gui`, `lxc-pxe-boot`, `proxmox-pxe-boot` cannot currently be built/redeployed on this node at all.
|
||||
2. **Decide on `proxmox-minimal`'s orphaned sops key**: locate the original private key and back it up here, or accept regenerating it (breaks decrypt access for whoever/whatever currently holds the old one).
|
||||
3. **Merge this session's PR** (see below) to get the `--agent 1` fix and `lxc-gui`'s new sops registration onto `main` — required before `lxc-gui` can be live-redeployed with working secrets.
|
||||
4. **Add a staleness guard to `create-proxmox-resource.sh`'s cache-reuse path** (Finding 2) — highest-leverage fix, since it silently produces a broken-but-"running" host.
|
||||
5. **Add a pre-flight sops-anchor check to `create-proxmox-resource.sh`** (management-surface gap #4) — same root cause class as #4 above, catch it before building instead of at first boot.
|
||||
6. Investigate/clean up **CT102** on pve (Finding 5) — confirm abandoned, then remove.
|
||||
7. Document `backup-admin-key.sh` in `README.md`'s Security Notes and add the live-host-key bootstrap-recovery procedure to `docs/` (management-surface gap #1).
|
||||
8. Add pre-push validation to the Gitea flake-lock-update workflow (management-surface gap #3).
|
||||
9. Lower-priority: document the home-manager `activationPackage` shortcut (gap #2); extend `sync-nix-cache-host-key.sh`'s drift-check pattern to `remoteBuilderAuthorizedKeys` (gap #5).
|
||||
10. Follow-up session: finish build-validating `proxmox-gui` and `proxmox-pxe-boot` (both killed here after 35-40min on this session's 2GB-RAM machine — not failures, just unfinished) once pve has headroom (item 1) — ideally from a machine with more RAM. `proxmox-server` and `proxmox-tailscale-exit-node` already passed build-only validation in this session, no follow-up needed.
|
||||
|
||||
---
|
||||
|
||||
## 5. Uncommitted Changes From This Session
|
||||
|
||||
This worktree (`worktree-flake-e2e-audit`) currently has:
|
||||
|
||||
- `scripts/proxmox/create-proxmox-resource.sh` — the `--agent enabled=1` fix (Finding 4).
|
||||
- `.sops.yaml` / `secrets/common.yaml` — `lxc-gui`'s new age key registered as a recipient (generated live during this session's testing).
|
||||
|
||||
Per this session's standard workflow, these will be committed, pushed, and opened as a draft PR rather than pushed to `main` directly — merging it is the operator's call, and is also **prerequisite to live-redeploying `lxc-gui` successfully** (its build will keep hitting the sops-staleness failure from Finding 2 on pve until this registration is on `origin/main`).
|
||||
@@ -252,6 +252,14 @@ instead of copying it.
|
||||
silent skip rather than a failure) only reports drift; the no-flags form
|
||||
updates both files in place. Declarative clients still need a rebuild to
|
||||
pick up the fix.
|
||||
- `scripts/secrets/push-host-keys.sh [--all | <target>] [--dry-run]
|
||||
[--skip-git-check]` — pushes newly-generated SSH host keys from
|
||||
`host-keys/` to already-running NixOS hosts, so they can decrypt sops
|
||||
secrets after a rebuild following `sync-host-keys.sh
|
||||
--regenerate-all-keys`. Verifies that `.sops.yaml` and `secrets/*.yaml`
|
||||
are committed and pushed to the remote first (hosts rebuild from the
|
||||
remote Gitea flake, so recipient changes must land there before any key
|
||||
push).
|
||||
|
||||
### `scripts/proxmox/`
|
||||
|
||||
@@ -273,6 +281,16 @@ instead of copying it.
|
||||
failure just falls back to building from source / `cache.nixos.org`) so
|
||||
the node substitutes from and can offload builds to nix-cache on every
|
||||
subsequent run, not just this one.
|
||||
- `scripts/proxmox/clone-pve1-to-pve-test.sh <vmid> [--new-vmid <id>]
|
||||
[--mode snapshot|suspend|stop] [--dry-run]` — ad-hoc clone of a single
|
||||
VM or CT from pve1 (production) to pve-test (sandbox) via vzdump +
|
||||
qmrestore/pct restore. Streams the archive directly between nodes (no
|
||||
local staging copy). Always restores with `--unique 1` (fresh MAC
|
||||
addresses) since the original is still running on the LAN. Cleans up
|
||||
the vzdump archive from both nodes after a successful restore. The
|
||||
script's own default is pve1 → pve-test, matching CLAUDE.md's policy
|
||||
(unlike `create-proxmox-resource.sh`, which defaults to production for
|
||||
the operator's own unqualified use).
|
||||
- `scripts/proxmox/configure-nix-cache-client.sh [--dry-run]
|
||||
[--no-remote-builder] [--no-restart]` — the non-NixOS equivalent of
|
||||
`modules/nix-cache/client.nix`/`remote-builder-client.nix`, for a plain
|
||||
@@ -288,6 +306,50 @@ instead of copying it.
|
||||
marked block rather than duplicating it); restarts `nix-daemon` by
|
||||
default so the change takes effect immediately.
|
||||
|
||||
### `scripts/ha/`
|
||||
|
||||
HA cluster lifecycle and operational scripts. All mutate real cluster state
|
||||
when run for real — always run against pve-test first unless the operator
|
||||
explicitly targets pve1.
|
||||
|
||||
- `scripts/ha/deploy.sh [--skip-*] [--destroy] [--dry-run]` — full
|
||||
lifecycle manager: phases through bridge creation, key sync, VM creation
|
||||
(via `create-proxmox-resource.sh`), NIC/disk attachment, and cluster
|
||||
initialisation. `--destroy` tears it back down. Safe to rerun
|
||||
idempotently; each phase can be individually skipped.
|
||||
- `scripts/ha/cluster-init.sh` — one-time cluster bootstrap run **as root
|
||||
on ha-server-1** after both VMs are booted. Generates/distributes the
|
||||
Corosync authkey, initialises DRBD metadata, creates XFS on `/dev/drbd0`,
|
||||
configures LIO iSCSI, and registers all Pacemaker resources (DRBD → XFS
|
||||
→ iSCSI → NFS → VIPs).
|
||||
- `scripts/ha/health.sh` — read-only cluster health snapshot: SSH
|
||||
reachability, quorum, DRBD state, Pacemaker resources, and VIP port
|
||||
reachability. Safe to run from the workstation at any time.
|
||||
- `scripts/ha/failover.sh [--to node1|node2] [--force] [--timeout <s>]
|
||||
[--dry-run]` — graceful failover by putting the active node into
|
||||
Pacemaker standby and waiting for resources to appear on the target.
|
||||
- `scripts/ha/acceptance-tests.sh` — T1–T7 acceptance tests (failover,
|
||||
NFS/iSCSI connectivity, DRBD sync, etc.) that must all pass before the
|
||||
cluster is considered production-ready.
|
||||
- `scripts/ha/resize-data-disk.sh --size +NNg [--force] [--dry-run]` —
|
||||
online data-disk resize: `qm resize` on both VMs, guest block-device
|
||||
rescan, `drbdadm resize`, `xfs_growfs`. No downtime required.
|
||||
- `scripts/ha/cluster-enable-stonith.sh` — enables the `fence_pve_ssh`
|
||||
STONITH resource after the fence SSH key is deployed to both nodes and
|
||||
authorised on the Proxmox host. Run once after `cluster-init.sh`.
|
||||
- `scripts/ha/fence-pve-ssh.py` — Python STONITH fence agent for Pacemaker.
|
||||
Deploy to `/etc/pacemaker/fence_pve_ssh` on both HA nodes (`chmod +x`).
|
||||
SSHes to the Proxmox host and runs `qm stop/start <vmid>`.
|
||||
|
||||
### `scripts/ipa/`
|
||||
|
||||
- `scripts/ipa/create-nixos-ipa-host-account.sh [options] <hostname>` —
|
||||
adds a NixOS host to the FreeIPA domain and produces a sops-encrypted
|
||||
keytab at `secrets/<hostname>.keytab`, ready for `modules/ipa/client.nix`.
|
||||
Replaces three error-prone manual steps: `ipa host-add`, `ipa-getkeytab`
|
||||
(run on the DC, SCP'd back), and `sops encrypt` in the correct location
|
||||
(must be at `secrets/<hostname>.keytab` for the creation rule to match).
|
||||
|
||||
### `scripts/lib/`
|
||||
|
||||
Sourced by the scripts above, never run directly:
|
||||
@@ -297,6 +359,15 @@ Sourced by the scripts above, never run directly:
|
||||
`create-proxmox-resource.sh` runs over SSH.
|
||||
- `nix-eval.sh` — `NIX_EVAL_FLAGS` plus `list_flake_targets`/
|
||||
`flake_target_hostname` flake-introspection helpers.
|
||||
- `nix-parallel.sh` — `run_nix_parallel`: fans out independent `nix eval`/
|
||||
`nix build --dry-run` calls across up to `NIX_PARALLEL_JOBS` processes,
|
||||
capped by available memory (~1 GB/job) rather than raw `nproc` to avoid
|
||||
OOM on constrained CI runners. Used by `codex-maintenance.sh`.
|
||||
- `clan-vars.sh` — helpers for reading/writing SSH host keys stored as clan
|
||||
vars (`vars/per-machine/<target>/openssh/`, sops-encrypted) instead of
|
||||
the gitignored `host-keys/` directory. Sourced by
|
||||
`create-proxmox-resource.sh` and `sync-host-keys.sh`; depends on
|
||||
`sops-age.sh` and `ssh-host-keys.sh` being sourced first.
|
||||
- `ssh-host-keys.sh` — `generate_host_ed25519_key`/`ssh_pubkey_to_age`,
|
||||
shared by `sync-host-keys.sh` and `prepare-host-key.sh`.
|
||||
- `sops-age.sh` — `age_pubkey_from_identity_file`/`sops_yaml_admin_pubkey`/
|
||||
@@ -316,6 +387,17 @@ Sourced by the scripts above, never run directly:
|
||||
default cores/memory, `NIX_CACHE_HOST`, `LAN_DOMAIN`) sourced by
|
||||
`create-proxmox-resource.sh` and `scripts/installer/auto-install.sh`. Add
|
||||
new cross-script config here instead of duplicating it per-script.
|
||||
- `scripts/recover-hosts.sh [<hostname> ...]` — fixes sops/SSH-key/GitHub-token
|
||||
issues on deployed NixOS hosts and triggers a `Switch-nix` rebuild on each.
|
||||
With no args discovers every known hostname; with args checks only those.
|
||||
Fixes applied automatically (prompts before rebuilding): SSH host key drift
|
||||
(restores the registered key) and stale GitHub access tokens (empties the
|
||||
rendered `nix-github-token.conf` so Nix falls back to unauthenticated requests
|
||||
until sops-nix re-renders the correct token after the next successful rebuild).
|
||||
- `scripts/gc-hosts.sh [--dry-run]` — runs `nix-collect-garbage -d` on all live
|
||||
NixOS hosts (workstation first, then pve1, then all Proxmox guests). Excludes
|
||||
`nix-cache` (gc-ing the shared binary cache evicts store paths other hosts
|
||||
depend on). Uses passwordless sudo where available; falls back to user-level gc.
|
||||
- `scripts/bump-nixpkgs-release.sh` — bumps `flake.nix`'s `nixpkgs.url`/
|
||||
`home-manager.url` in place. Exists because flake input URLs can't
|
||||
reference `variables.nix` (confirmed empirically — `nix flake metadata`
|
||||
@@ -354,12 +436,13 @@ nixosSystem {
|
||||
```
|
||||
|
||||
Platforms: `linode`, `proxmox`, `lxc`, `baremetal`. Build types: `minimal`,
|
||||
`nix-cache`, `server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`,
|
||||
`tor-relay`. Not every combination is built — e.g. `pxe-boot` has no `linode`
|
||||
`nix-cache`, `docker`, `gui`, `pxe-boot`, `tailscale-router`, `tor-relay`,
|
||||
`ha-server`. Not every combination is built — e.g. `pxe-boot` has no `linode`
|
||||
variant (PXE/DHCP/TFTP need LAN L2 adjacency a Linode VPS doesn't have),
|
||||
`tor-relay` currently only exists as `lxc-tor-relay`, and `baremetal`
|
||||
currently only exists as `baremetal-gui` (the real gui-host hardware —
|
||||
see `hosts/nixos/host.nix` and `modules/platforms/baremetal.nix`). Treat
|
||||
`tor-relay` only exists as `lxc-tor-relay`, `ha-server` only exists as
|
||||
`proxmox-ha-server-{1,2}`, and `baremetal` only exists as `baremetal-gui`
|
||||
(the real gui-host hardware — see `hosts/nixos/host.nix` and
|
||||
`modules/platforms/baremetal.nix`). Treat
|
||||
`flake.nix`'s
|
||||
`generatedTargets` as the source
|
||||
of truth for which hosts exist — `README.md`, `AGENTS.md`,
|
||||
@@ -391,7 +474,7 @@ removing a host.
|
||||
`vzdump` backup-archive metadata this doesn't have), no install step —
|
||||
see `docs/auto-installer.md`.
|
||||
- `modules/build-types/*.nix` — what a system is for:
|
||||
minimal/server/docker/gui/pxe-boot/nix-cache/tailscale-exit-node/tor-relay.
|
||||
minimal/docker/gui/pxe-boot/nix-cache/tailscale-router/tor-relay/ha-server.
|
||||
- `modules/common/configuration.nix` — base NixOS config imported by every
|
||||
host: locale, users, nix settings, git.
|
||||
- `modules/common/home.nix` / `hosts/nixos/home.nix` — Home Manager config for
|
||||
@@ -417,7 +500,7 @@ removing a host.
|
||||
`modules/platforms/baremetal.nix` also imports
|
||||
`modules/services/zfs/enable-service.nix` for this (the `zfs_unstable`
|
||||
package, autoScrub/autoSnapshot/trim) — the only other importer today is
|
||||
`server`'s NFS data pool, an unrelated non-root ZFS use.
|
||||
`ha-server`'s NFS data pool, an unrelated non-root ZFS use.
|
||||
- `modules/boot/efi.nix` — systemd-boot + EFI vars, paired with the disko module.
|
||||
- `modules/installer/` — the auto-installer environment (ISO, also served as
|
||||
PXE netboot): `common.nix` (shared config + the generated
|
||||
@@ -431,14 +514,21 @@ removing a host.
|
||||
substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the
|
||||
full design (per-host local stores, no shared `/nix/store`, and how the
|
||||
`nixremote` signing/SSH keys fit together).
|
||||
- `modules/ha/` — HA cluster NixOS modules: `cluster-config.nix` (DRBD,
|
||||
Corosync, Pacemaker, firewall rules, cluster-wide NFS/iSCSI port
|
||||
authorisation — shared by both ha-server nodes), `pacemaker-stack.nix`
|
||||
(Pacemaker + Corosync service enablement), and supporting modules. See
|
||||
`docs/ha.md` for the cluster operational guide.
|
||||
- `modules/ipa/client.nix` — FreeIPA client enrollment: sssd, Kerberos keytab,
|
||||
and IPA host registration; imported by every real host via
|
||||
`modules/common/configuration.nix`.
|
||||
- `modules/beszel/enable-agent.nix` — enables beszel-agent, sets `HUB_URL`,
|
||||
fixes the upstream `StateDirectory` bug, and wires the universal
|
||||
`beszel-token` sops secret (from `secrets/common.yaml`) into the agent's
|
||||
`environmentFile`; see `docs/beszel.md` for the full setup guide.
|
||||
- `modules/tailscale/`, `modules/docker/`, `modules/networking/`,
|
||||
`modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose,
|
||||
single-host
|
||||
feature modules (e.g. `docker/enable-service.nix`,
|
||||
single-host feature modules (e.g. `docker/enable-service.nix`,
|
||||
`services/zfs/enable-service.nix`). Grep `modules/build-types/*.nix` for
|
||||
each build type's `imports` list to see which modules apply where.
|
||||
|
||||
@@ -462,3 +552,5 @@ duplicating config.
|
||||
- `docs/flake-lock-automation.md` — how `flake.lock` updates flow through CI
|
||||
(scheduled `nix flake update` PR + host-eval-on-PR workflow) and why hosts
|
||||
should track the committed lock file rather than `nixos-rebuild --upgrade-all`.
|
||||
- `docs/ha.md` — HA file-server cluster: DRBD + XFS + LIO iSCSI + NFS managed
|
||||
by Corosync + Pacemaker; network topology; lifecycle scripts in `scripts/ha/`.
|
||||
|
||||
@@ -9,8 +9,8 @@ Targets are named `<platform>-<buildtype>`, generated from two orthogonal
|
||||
pieces composed in `flake.nix`:
|
||||
|
||||
- **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`, `baremetal`
|
||||
- **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
|
||||
`gui`, `pxe-boot`, `tailscale-router`, `tor-relay`, `ha-server`
|
||||
- **Build types** (what it's for): `minimal`, `nix-cache`, `docker`, `gui`,
|
||||
`pxe-boot`, `tailscale-router`, `tor-relay`, `ha-server`
|
||||
|
||||
Not every combination exists — `pxe-boot` has no `linode` variant, since
|
||||
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have,
|
||||
@@ -24,7 +24,6 @@ hardware). The full list:
|
||||
| `proxmox-minimal` | Minimal NixOS host profile on Proxmox — previously the flat `nix-minimal` target |
|
||||
| `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container |
|
||||
| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder — previously the flat `nix-cache` target |
|
||||
| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host — previously the flat `server` target |
|
||||
| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack — previously the flat `docker` target |
|
||||
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target |
|
||||
| `baremetal-gui` | Same Cinnamon desktop workstation, on the real gui-host hardware — ZFS RAID0 root, systemd-boot |
|
||||
|
||||
@@ -82,15 +82,6 @@ services.beszel.agent.environment = {
|
||||
};
|
||||
```
|
||||
|
||||
The `server` host uses this to expose its ZFS data pool:
|
||||
|
||||
```nix
|
||||
services.beszel.agent.environment = {
|
||||
EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
|
||||
LOG_LEVEL = "debug";
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Optional: monitoring Docker containers
|
||||
|
||||
@@ -1,261 +0,0 @@
|
||||
# Storage/Cluster Network Segmentation Audit — pve1.sweet.home
|
||||
|
||||
**Date:** 2026-07-29
|
||||
**Scope:** Read-only discovery of pve1.sweet.home host networking, HA cluster VMs (200/201), and Docker CT (105). No changes made.
|
||||
|
||||
> **Implementation status — 2026-07-29:** All recommendations from this audit have been
|
||||
> implemented in the same session. See `docs/ip-addressing.md` for the current state.
|
||||
> Key decisions that diverged from the original recommendations:
|
||||
> - VLAN IDs renumbered: cluster → VLAN 10 (192.168.10.x), storage-client → VLAN 20 (192.168.20.x)
|
||||
> - Two Pacemaker VIPs: `vip-lan` (192.168.2.229, NFS for LAN) and `vip-storage` (192.168.20.229, NFS + iSCSI for VLAN 20)
|
||||
> - NFS served on **both** VIPs (each firewalled to its own subnet); iSCSI available on VLAN 20 but NFS is preferred for docker to support future Docker Swarm multi-host access
|
||||
> - `corosync.conf` ring1 added using LAN IPs (RF-1 resolved)
|
||||
> - `vmbr2` created and NICs added to HA VMs and docker CT (RF-6/RF-7 resolved)
|
||||
> - iSCSI portal remains on `[::0]`; firewall enforces VLAN 20 restriction (RF-4 mitigated)
|
||||
> - STONITH still disabled (RF-3 deferred — accepted risk during development phase)
|
||||
> - iSCSI ACLs not configured (RF-5 deferred — iSCSI not in active use)
|
||||
|
||||
---
|
||||
|
||||
## 1. Current State Summary
|
||||
|
||||
### pve1.sweet.home Host — Physical NICs
|
||||
|
||||
| Interface | Speed/Duplex | Notes |
|
||||
|-----------|-------------|-------|
|
||||
| `nic0` | 2500 Mb/s / Full (2.5GbE) | Only active physical NIC; sole bridge port for vmbr0 |
|
||||
| `nic1` | (not connected / no data) | Present in config, not UP |
|
||||
| `wlp4s0` | DOWN | WiFi, unused |
|
||||
|
||||
No bonding configured. Every guest's traffic ultimately funnels through the single 2.5GbE `nic0`.
|
||||
|
||||
### Proxmox Bridges
|
||||
|
||||
| Bridge | Physical NIC | Host IP | Subnet | VLAN-aware | Purpose (current) |
|
||||
|--------|-------------|---------|--------|------------|-------------------|
|
||||
| `vmbr0` | `nic0` (2.5GbE) | 192.168.2.245/24 | 192.168.2.0/24 | No | General LAN, management, **iSCSI/NFS VIP** |
|
||||
| `vmbr1` | **none** (internal-only) | — | 192.168.4.224/29 | No | Corosync heartbeat + DRBD replication |
|
||||
|
||||
`vmbr1` has `bridge-ports none` in `/etc/network/interfaces.d/vmbr1.conf` — it is a purely software bridge with zero physical uplink. All traffic on it stays inside the hypervisor's memory.
|
||||
|
||||
### Guest NIC Assignments
|
||||
|
||||
| Guest | VMID | Role | NIC | Bridge | IP | Traffic type |
|
||||
|-------|------|------|-----|--------|----|--------------|
|
||||
| nix-cache | CT 102 | Build cache | eth0 | vmbr0 | DHCP/LAN | LAN |
|
||||
| pxe-boot | CT 103 | PXE/TFTP | eth0 | vmbr0 | DHCP/LAN | LAN |
|
||||
| tor-relay | CT 104 | Tor | eth0 | vmbr0 | DHCP/LAN | LAN |
|
||||
| **docker** | **CT 105** | **Docker host** | **eth0** | **vmbr0** | **192.168.2.225/24** | **LAN only** |
|
||||
| pdm | CT 106 | Proxmox mgmt | eth0 | vmbr0 | 192.168.2.220/24 | LAN |
|
||||
| server | VM 101 | General server | net0 | vmbr0 | DHCP/LAN | LAN |
|
||||
| tailscale-router | VM 107 | Tailscale exit | net0 | vmbr0 | DHCP/LAN | LAN |
|
||||
| domain-controller | VM 108 | FreeIPA | net0 | vmbr0 | 192.168.2.253/24 | LAN |
|
||||
| **ha-server-1** | **VM 200** | **HA primary** | **net0** | **vmbr0** | **192.168.2.228/24 + VIP 192.168.2.229/24** | **LAN + VIP** |
|
||||
| **ha-server-1** | **VM 200** | **HA primary** | **net1** | **vmbr1** | **192.168.4.228/29** | **Corosync + DRBD** |
|
||||
| **ha-server-2** | **VM 201** | **HA secondary** | **net0** | **vmbr0** | **192.168.2.227/24** | **LAN** |
|
||||
| **ha-server-2** | **VM 201** | **HA secondary** | **net1** | **vmbr1** | **192.168.4.227/29** | **Corosync + DRBD** |
|
||||
|
||||
### Corosync/Pacemaker State
|
||||
|
||||
- **Transport:** knet/UDP
|
||||
- **Rings:** 1 only — ring0 on `192.168.4.228` / `192.168.4.227` (vmbr1/ens19)
|
||||
- **Cluster status:** Both nodes online, DC = ha-server-1, quorum achieved
|
||||
- **STONITH:** `stonith-enabled: false`
|
||||
- **no-quorum-policy:** `ignore`
|
||||
- **Resources (all active on ha-server-1):**
|
||||
- `ms-drbd0` — promotable DRBD clone (Primary: ha-server-1, Secondary: ha-server-2)
|
||||
- `xfs-data` — XFS on `/dev/drbd0` → `/srv/ha-data`
|
||||
- `iscsi-target` — targetctl service
|
||||
- `nfs-server` — nfs-server service
|
||||
- `vip` — IPaddr2 at **192.168.2.229/24** (no `nic=` parameter specified; floats to ens18/vmbr0 automatically based on subnet match)
|
||||
- **Resource ordering:** ha-group starts only after DRBD is promoted; collocated with Promoted DRBD clone.
|
||||
|
||||
### DRBD State
|
||||
|
||||
- **Resource:** `ha-data` (DRBD 8.4.11 kernel module, config in `/etc/drbd.conf`)
|
||||
- **Protocol:** C (synchronous)
|
||||
- **Replication endpoints:**
|
||||
- ha-server-1: `192.168.4.228:7789` (ens19 / vmbr1)
|
||||
- ha-server-2: `192.168.4.227:7789` (ens19 / vmbr1)
|
||||
- **State at audit time:** Initial sync in progress — ~20% complete, ~40 MB/s, ~34 min remaining (100 GB disk)
|
||||
- **Fencing config:** `fencing resource-only` + fence-peer/unfence-peer handlers
|
||||
|
||||
### iSCSI Target
|
||||
|
||||
- **IQN:** `iqn.2026-01.home.sweet:ha-storage`
|
||||
- **Portal:** `[::0]:3260` — confirmed listening on all interfaces (`ss -tnlp` shows `*:3260 *:*`)
|
||||
- **LUN 0:** fileio backstore — `/srv/ha-data/iscsi-lun.img` (10 GiB, write-thru)
|
||||
- **ACLs:** **None** (`no-gen-acls`, `no-auth`)
|
||||
- **VIP (intended portal):** 192.168.2.229 — on vmbr0/LAN, no storage-NIC-specific binding
|
||||
|
||||
### NFS Exports
|
||||
|
||||
Served from the same `ha-group` as iSCSI (starts/stops together):
|
||||
|
||||
| Export path | Client subnet |
|
||||
|------------|---------------|
|
||||
| `/srv/ha-data/docker/{config,databases,volumes,nextcloud-data}` | 192.168.2.0/24 |
|
||||
| `/srv/ha-data/proxmox/{iso,lxc}` | 192.168.2.0/24 |
|
||||
| `/srv/ha-data/pxe-boot/images` | 192.168.2.0/24 |
|
||||
| `/srv/ha-data/raspi/volumes` | 192.168.2.0/24 |
|
||||
|
||||
All NFS exports restrict to 192.168.2.0/24 and are served via the VIP at 192.168.2.229. `rw,sync,no_subtree_check,no_root_squash`.
|
||||
|
||||
### Docker Host Current State
|
||||
|
||||
- **Single NIC:** eth0 on vmbr0, 192.168.2.225/24, gateway 192.168.2.254
|
||||
- **Route to storage network (192.168.4.x):** none — no NIC and no route
|
||||
- **iSCSI sessions:** none
|
||||
- **iSCSI nodes discovered:** none
|
||||
- **Docker networks:** several active compose-project networks (core_traefik, core_nextcloud, core_passbolt, core_gramps, core_docker-socket-proxy, plus CI isolation networks)
|
||||
|
||||
---
|
||||
|
||||
## 2. Risk Flags
|
||||
|
||||
### RF-1: Corosync has only one ring (no heartbeat path redundancy)
|
||||
|
||||
`corosync.conf` defines only `ring0_addr` for each node, using 192.168.4.x on vmbr1. No `ring1_addr` / second knet link is configured. On a single Proxmox host, vmbr1 is a software bridge (no physical NIC), so physical link failure is not the concern — but a kernel network stack hiccup, a `pveproxy` restart dropping bridge state, or vmbr1 getting disrupted during heavy DRBD sync all leave corosync with zero fallback path. Missed heartbeats on a two-node cluster with `no-quorum-policy: ignore` do not cause a clean shutdown; they cause a false failover or split-brain.
|
||||
|
||||
Adding the LAN addresses (192.168.2.228 / 192.168.2.227 via ens18/vmbr0) as a second knet link would provide a backup path with no infrastructure changes needed.
|
||||
|
||||
### RF-2: Corosync heartbeat and DRBD replication share vmbr1 — no isolation between them
|
||||
|
||||
Both corosync (knet/UDP, ~1 kB heartbeat packets every ~100 ms) and DRBD replication (protocol C, synchronous, syncing at ~40 MB/s on a 100 GB initial fill at audit time) traverse the same `vmbr1` virtual bridge and terminate on the same ens19 NIC pair inside each HA VM. Under heavy DRBD write load, the guest-kernel scheduler's NIC transmit queue processes both flows together. While corosync's heartbeat is tiny, the absence of QoS/priority marking on vmbr1 means a DRBD burst can delay a heartbeat enough to trigger a ring fault warning. This is a latent risk that grows under high-write workloads.
|
||||
|
||||
### RF-3: STONITH disabled — split-brain protection relies solely on DRBD's resource-only fencing
|
||||
|
||||
`stonith-enabled: false` in the CIB. With `no-quorum-policy: ignore`, both nodes will continue running if corosync loses communication. DRBD's `fencing resource-only` does call `fence-peer` before allowing a Primary promotion, which provides some protection, but there is no hard external power fence to guarantee the other node actually stops. In a real split-brain (both nodes believe they are Primary), data corruption on the shared XFS filesystem is possible. **This is the highest-severity risk in the current setup.**
|
||||
|
||||
Getting STONITH to work on Proxmox-hosted VMs requires either a `fence_pve` agent (Proxmox API fencing) or `fence_virtd` (QEMU guest agent fencing). Neither is configured.
|
||||
|
||||
### RF-4: iSCSI portal bound to `[::0]:3260` — listens on every interface, not just the VIP
|
||||
|
||||
The targetcli portal is `[::0]:3260` (confirmed: `*:3260 *:*` in ss). This means the target is reachable on:
|
||||
- 192.168.2.229 (VIP — correct, failover-safe)
|
||||
- 192.168.2.228 (ha-server-1 LAN IP — does **not** move during failover; an initiator session connecting here would break on failover)
|
||||
- 192.168.4.228 (storage NIC — not reachable by the Docker host today, but unintentionally exposed)
|
||||
|
||||
Binding the portal explicitly to the VIP IP instead of wildcard eliminates the non-VIP reachability risks.
|
||||
|
||||
### RF-5: iSCSI has zero ACLs and no authentication
|
||||
|
||||
`targetcli ls` shows `acls: 0`, `no-gen-acls`, `no-auth`. Any host that can reach port 3260 on any of the above IPs can log into the LUN with no credentials. The Docker host is not yet configured as an initiator — but neither is it blocked.
|
||||
|
||||
### RF-6: Docker host has no path to the storage network — iSCSI would traverse vmbr0/nic0
|
||||
|
||||
CT 105 (docker, 192.168.2.225) has one NIC, on vmbr0. To reach the VIP at 192.168.2.229, iSCSI traffic would travel:
|
||||
|
||||
```
|
||||
docker (eth0/vmbr0) → nic0 (2.5GbE) → vmbr0 → tap200i0 (VM 200 net0/ens18)
|
||||
```
|
||||
|
||||
All of the following share this same path over vmbr0 → nic0:
|
||||
- Docker container traffic (outbound and inter-container)
|
||||
- CI/CD runner traffic (Gitea Actions jobs visible in `docker network ls`)
|
||||
- NFS mounts from pxe-boot, proxmox host itself, and other LAN clients
|
||||
- iSCSI block traffic (protocol-sensitive to latency and retransmit)
|
||||
|
||||
A Nextcloud upload or a CI `nix build` job can saturate nic0 and starve the iSCSI session, causing command timeouts and filesystem errors on the Docker host.
|
||||
|
||||
### RF-7: VIP is on the LAN interface with no storage-specific binding
|
||||
|
||||
The Pacemaker `vip` resource specifies `ip=192.168.2.229, cidr_netmask=24` with no `nic=` parameter. Pacemaker's IPaddr2 agent selects the interface by longest-prefix match, landing it on ens18 (vmbr0/LAN). There is no way to keep this VIP from competing with general LAN traffic on nic0 without moving the VIP to a separate subnet on a different virtual bridge.
|
||||
|
||||
---
|
||||
|
||||
## 3. Recommended Target Layout
|
||||
|
||||
### Design constraints
|
||||
|
||||
- Single Proxmox host: all traffic ultimately shares nic0's bandwidth. The goal is QoS partitioning via separate bridges and subnets, not true physical isolation.
|
||||
- Future physical split: bridge/VLAN IDs chosen here should map cleanly to physical uplink VLAN tags when the HA nodes move to separate hardware.
|
||||
|
||||
### Proposed bridge layout
|
||||
|
||||
| Bridge | Physical port | VLAN tag (future) | Subnet | Purpose |
|
||||
|--------|-------------|-------------------|--------|---------|
|
||||
| `vmbr0` | nic0 | untagged / VLAN 1 | 192.168.2.0/24 | **LAN/management only** — no storage traffic |
|
||||
| `vmbr1` | (none / VLAN 10 on future trunk) | VLAN 10 | 192.168.4.224/29 | **Corosync heartbeat + DRBD replication** (current, keep) |
|
||||
| `vmbr2` *(new)* | (none / VLAN 20 on future trunk) | VLAN 20 | 192.168.5.0/24 | **Storage: iSCSI + NFS client access** |
|
||||
|
||||
This is the minimum-disruption path: vmbr1 stays as-is (no DRBD reconfiguration needed), and the new vmbr2 gives the Docker host a direct path to the storage VIP without crossing vmbr0.
|
||||
|
||||
If stricter isolation is later desired, DRBD can be migrated from vmbr1 to vmbr2 in a separate maintenance window (see §5), leaving vmbr1 as corosync-only.
|
||||
|
||||
### Per-guest NIC assignments in target layout
|
||||
|
||||
| Guest | VMID | NIC | Bridge | Proposed IP | Purpose |
|
||||
|-------|------|-----|--------|-------------|---------|
|
||||
| ha-server-1 | VM 200 | net0 | vmbr0 | 192.168.2.228/24 | LAN/management (keep) |
|
||||
| ha-server-1 | VM 200 | net1 | vmbr1 | 192.168.4.228/29 | Corosync + DRBD (keep) |
|
||||
| ha-server-1 | VM 200 | **net2 (new)** | **vmbr2** | **192.168.5.1/24** | iSCSI + NFS storage client |
|
||||
| ha-server-2 | VM 201 | net0 | vmbr0 | 192.168.2.227/24 | LAN/management (keep) |
|
||||
| ha-server-2 | VM 201 | net1 | vmbr1 | 192.168.4.227/29 | Corosync + DRBD (keep) |
|
||||
| ha-server-2 | VM 201 | **net2 (new)** | **vmbr2** | **192.168.5.2/24** | iSCSI + NFS storage client |
|
||||
| docker | CT 105 | net0 | vmbr0 | 192.168.2.225/24 | LAN/management (keep) |
|
||||
| docker | CT 105 | **net1 (new)** | **vmbr2** | **192.168.5.10/24** | iSCSI + NFS |
|
||||
|
||||
**VIP target:** `192.168.5.100/24` on vmbr2. The Pacemaker `vip` resource changes from `ip=192.168.2.229` to `ip=192.168.5.100, nic=<ens20>` (whichever name the new NIC gets inside the HA VMs). The existing `192.168.2.229` LAN VIP can optionally be retained as a separate static alias on ens18 for management-plane access, but should not be the iSCSI portal target.
|
||||
|
||||
**iSCSI portal:** Bind to `192.168.5.100:3260` instead of `[::0]:3260`. In targetcli: remove the wildcard portal, add `portals/ create 192.168.5.100`.
|
||||
|
||||
**NFS exports:** NFS is a file-level protocol and is fine being accessed over a routed path. After the VIP moves, non-docker LAN clients (proxmox host, pxe-boot, raspi) can reach NFS either via a static route to 192.168.5.0/24 or by keeping a secondary static alias at 192.168.2.229 on ens18 dedicated to NFS. Either approach works — NFS handles reconnect gracefully in ways iSCSI block I/O cannot.
|
||||
|
||||
**Corosync second ring (independent, low-disruption improvement):**
|
||||
|
||||
Add a second knet link using the LAN addresses as a backup heartbeat path. Edit `corosync.conf` on both nodes:
|
||||
|
||||
```
|
||||
node { ring0_addr: 192.168.4.228; ring1_addr: 192.168.2.228; name: ha-server-1; nodeid: 1; }
|
||||
node { ring0_addr: 192.168.4.227; ring1_addr: 192.168.2.227; name: ha-server-2; nodeid: 2; }
|
||||
```
|
||||
|
||||
Requires a corosync service restart (brief cluster pause, ~5 seconds), no interface or bridge changes.
|
||||
|
||||
**Future physical-host split:**
|
||||
When ha-server-1 and ha-server-2 move to separate physical machines, vmbr1 and vmbr2 become VLAN-tagged sub-interfaces on a physical trunk (e.g. VLAN 10 → cluster, VLAN 20 → storage). The bridge/subnet/IP layout above is designed so the tag numbers can be layered onto the existing addresses without renumbering.
|
||||
|
||||
---
|
||||
|
||||
## 4. Gap List
|
||||
|
||||
| Gap | Action needed |
|
||||
|-----|--------------|
|
||||
| `vmbr2` does not exist on pve1 | Create internal bridge: `/etc/network/interfaces.d/vmbr2.conf` with `bridge-ports none`, `inet manual` |
|
||||
| VM 200 and VM 201 have no net2 | `qm set 200 --net2 virtio,bridge=vmbr2` / `qm set 201 --net2 virtio,bridge=vmbr2` (hot-plug, no reboot needed) |
|
||||
| CT 105 has no net1 | `pct set 105 --net1 name=eth1,bridge=vmbr2,ip=192.168.5.10/24` |
|
||||
| HA VMs have no OS config for the new NIC | NixOS `networking.interfaces.<ens20>` with `ipv4.addresses = [{address="192.168.5.1"; prefixLength=24;}]` per host (name may differ — check `ip link` after hotplug) |
|
||||
| VIP needs to move to 192.168.5.100 on vmbr2 | `pcs resource update vip ip=192.168.5.100 cidr_netmask=24 nic=<ens20>` |
|
||||
| iSCSI portal bound to `[::0]` | `targetcli /iscsi/iqn.2026-01.home.sweet:ha-storage/tpg1/portals delete ::0 3260` then `create 192.168.5.100`; save and restart via `pcs resource restart iscsi-target` |
|
||||
| iSCSI ACLs empty | Get Docker initiator IQN via `iscsiadm -m iface` on CT 105, then add via targetcli `acls/ create <iqn>` |
|
||||
| Docker host has no iSCSI initiator config | `iscsiadm -m discoverydb -t sendtargets -p 192.168.5.100 -D` then `iscsiadm -m node -l` once ACLs are set |
|
||||
| Corosync single ring | Add `ring1_addr` entries in `corosync.conf` using LAN IPs; restart corosync cluster-wide (one node at a time) |
|
||||
| STONITH not configured | Evaluate `fence_pve` (Proxmox API agent); document accepted risk if deferred |
|
||||
| DRBD still on vmbr1 (optional, separate window) | Stop ms-drbd0 via pcs, edit `/etc/drbd.conf` on both nodes (change `192.168.4.x` → `192.168.5.x`), restart DRBD, re-enable via pcs |
|
||||
|
||||
---
|
||||
|
||||
## 5. Migration Notes
|
||||
|
||||
### Non-disruptive (no service impact)
|
||||
|
||||
- **Create vmbr2 on pve1:** Bridge definition edit only; no effect on existing bridges or guests.
|
||||
- **Hot-add net2 to VMs 200/201:** Proxmox allows adding a NIC without reboot (`qm set 200 --net2 ...`). The NIC appears inside the VM immediately via QEMU hotplug but will be unconfigured (down) inside NixOS until the NixOS config is deployed — no impact on running services.
|
||||
- **Add net1 to CT 105:** LXC NIC hotplug works similarly; CT does not need to restart.
|
||||
- **Add corosync ring1:** Requires `systemctl restart corosync` on both nodes (one at a time). Pacemaker briefly sees corosync go offline and recover; with two nodes and `wait_for_all: 0`, this typically completes in under 5 seconds and resources stay running.
|
||||
|
||||
### Disruptive — requires maintenance window
|
||||
|
||||
- **Move VIP from 192.168.2.229 to 192.168.5.100:** `pcs resource update vip ip=192.168.5.100` causes Pacemaker to immediately stop the old VIP and start the new one. Any NFS mounts referencing 192.168.2.229 will stall until remounted at the new address (or a static alias is added at 192.168.2.229 on ens18). No iSCSI sessions exist yet, so no iSCSI disruption.
|
||||
|
||||
- **Change iSCSI portal from `[::0]` to VIP-specific:** Requires `pcs resource restart iscsi-target` after the targetcli portal change — brief target unavailability. Any initiator sessions (once configured) will need to re-login.
|
||||
|
||||
- **Migrate DRBD replication from 192.168.4.x to 192.168.5.x** (optional — only needed to give corosync sole ownership of vmbr1):
|
||||
1. `pcs resource disable ms-drbd0` — demotes DRBD Primary, stops ha-group (unmounts XFS, stops iSCSI + NFS + VIP)
|
||||
2. `drbdadm down ha-data` on both nodes
|
||||
3. Edit `/etc/drbd.conf` on both nodes (change `address` lines)
|
||||
4. `drbdadm up ha-data` on both nodes
|
||||
5. `pcs resource enable ms-drbd0` — Pacemaker re-promotes, mounts, starts services
|
||||
|
||||
DRBD does **not** require a full resync when only the address changes — the disk data and metadata are unchanged; only the TCP connection endpoint changes. However, the initial sync was in progress at audit time (~20% at ~40 MB/s). Recommend waiting for that sync to complete before scheduling this migration.
|
||||
+160
@@ -0,0 +1,160 @@
|
||||
# HA File-Server Cluster
|
||||
|
||||
Two `proxmox-ha-server-{1,2}` VMs form an active/passive file-server cluster:
|
||||
DRBD replicates a block device between nodes; Corosync + Pacemaker manage
|
||||
failover; XFS, LIO iSCSI, and NFS are brought up as a collocated resource
|
||||
group on whichever node holds the DRBD Primary role.
|
||||
|
||||
NixOS modules: `modules/ha/`. Lifecycle scripts: `scripts/ha/`.
|
||||
Cluster-wide constants: `variables.nix` (`haServer*` vars).
|
||||
|
||||
---
|
||||
|
||||
## Network layout
|
||||
|
||||
Three subnets — all internal to pve1 (`vmbr0`/`vmbr1`/`vmbr2`):
|
||||
|
||||
| Subnet | VLAN | CIDR | Bridge | Purpose |
|
||||
|---|---|---|---|---|
|
||||
| LAN | 2 | `192.168.2.0/24` | `vmbr0` | Management, LAN NFS |
|
||||
| Cluster | 10 | `192.168.10.224/29` | `vmbr1` | Corosync ring0 + DRBD replication |
|
||||
| Storage-client | 20 | `192.168.20.0/24` | `vmbr2` | NFS + iSCSI for docker/swarm |
|
||||
|
||||
Each HA VM has three NICs: `ens18` (LAN/vmbr0), `ens19` (cluster/vmbr1),
|
||||
`ens20` (storage-client/vmbr2). See `docs/ip-addressing.md` for all IPs.
|
||||
|
||||
Corosync ring0 uses the cluster NIC; ring1 (backup heartbeat) uses the LAN
|
||||
NIC. DRBD replicates over the cluster NIC. No storage traffic crosses the LAN.
|
||||
|
||||
---
|
||||
|
||||
## Pacemaker resources
|
||||
|
||||
All resources run collocated on whichever node is Primary, in this order:
|
||||
|
||||
```
|
||||
ms-drbd0 (promotable DRBD clone)
|
||||
→ xfs-data (XFS mount on /dev/drbd0 → /srv/ha-data)
|
||||
→ iscsi-target (targetctl)
|
||||
→ nfs-server (nfs-server.service)
|
||||
→ vip-lan (192.168.2.229/24 on vmbr0 — NFS for LAN clients)
|
||||
→ vip-storage (192.168.20.229/24 on vmbr2 — NFS + iSCSI for VLAN 20)
|
||||
```
|
||||
|
||||
`vip-lan` serves pxe-boot and other LAN-only NFS clients.
|
||||
`vip-storage` serves docker and any future swarm nodes; iSCSI is available on
|
||||
VLAN 20 but NFS is preferred for multi-host volume sharing.
|
||||
|
||||
---
|
||||
|
||||
## DRBD fencing
|
||||
|
||||
`fencing resource-only` with `crm-fence-peer.sh`/`crm-unfence-peer.sh`
|
||||
wrappers (`modules/ha/cluster-config.nix`). The DRBD kernel module invokes
|
||||
these via the User Mode Helper with a minimal PATH; the wrappers prepend
|
||||
`/run/current-system/sw/bin` before exec-ing the real handlers so Pacemaker
|
||||
tools (`cibadmin`, `crm_mon`, etc.) are found.
|
||||
|
||||
STONITH is initially disabled (`stonith-enabled: false`,
|
||||
`no-quorum-policy: ignore`). Enable it once the `fence_pve_ssh` fence agent
|
||||
(`scripts/ha/fence-pve-ssh.py`) is deployed and authorised:
|
||||
|
||||
```bash
|
||||
scripts/ha/cluster-enable-stonith.sh # run as root on ha-server-1
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Deploying the cluster from scratch
|
||||
|
||||
Use `scripts/ha/deploy.sh` — it orchestrates all phases:
|
||||
|
||||
```bash
|
||||
# Against pve-test (safe — Claude's default target):
|
||||
scripts/ha/deploy.sh --node "$PVE_TEST_HOST" [--dry-run]
|
||||
|
||||
# Against pve1 (production — requires explicit operator go-ahead):
|
||||
scripts/ha/deploy.sh --node "$PVE1_HOST"
|
||||
```
|
||||
|
||||
Phases (each skippable with `--skip-<phase>`):
|
||||
1. `ensure-bridge` — creates `vmbr1`/`vmbr2` on the Proxmox node if absent
|
||||
2. `sync-keys` — generates SSH host keys for both nodes; registers sops recipients
|
||||
3. `create-vms` — builds disk images, creates VMs via `create-proxmox-resource.sh`
|
||||
4. `add-hardware` — attaches storage NIC and DRBD data disk to each VM
|
||||
5. `init-cluster` — runs `scripts/ha/cluster-init.sh` on ha-server-1
|
||||
|
||||
`--destroy` runs the teardown sequence.
|
||||
|
||||
---
|
||||
|
||||
## Day-to-day operations
|
||||
|
||||
```bash
|
||||
# Read-only health check (safe from workstation):
|
||||
scripts/ha/health.sh
|
||||
|
||||
# Graceful failover (prompts for confirmation):
|
||||
scripts/ha/failover.sh [--to node1|node2]
|
||||
|
||||
# Online data-disk growth (no downtime):
|
||||
scripts/ha/resize-data-disk.sh --size +20G
|
||||
|
||||
# Acceptance tests (run after any significant change):
|
||||
scripts/ha/acceptance-tests.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Adding FreeIPA host accounts
|
||||
|
||||
IPA host registration is automated:
|
||||
|
||||
```bash
|
||||
scripts/ipa/create-nixos-ipa-host-account.sh <hostname>
|
||||
```
|
||||
|
||||
This runs `ipa host-add`, fetches a keytab from the domain controller, and
|
||||
writes a sops-encrypted `secrets/<hostname>.keytab` in one step. The module
|
||||
`modules/ipa/client.nix` (imported by every host via
|
||||
`modules/common/configuration.nix`) consumes the keytab via sops-nix.
|
||||
|
||||
---
|
||||
|
||||
## Storage layout
|
||||
|
||||
```
|
||||
/srv/ha-data/
|
||||
docker/
|
||||
config/ NFS → docker:/mnt/docker/config
|
||||
databases/ NFS → docker:/mnt/docker/databases
|
||||
volumes/ NFS → docker:/mnt/docker/volumes
|
||||
nextcloud-data/ NFS → docker:/mnt/docker/nextcloud-data
|
||||
proxmox/
|
||||
iso/ NFS → pve1 ISO storage
|
||||
lxc/ NFS → pve1 CT template storage
|
||||
pxe-boot/
|
||||
images/ NFS → pxe-boot:/srv/pxe/http/images (PXE assets)
|
||||
raspi/
|
||||
volumes/ NFS → raspi NFS mounts
|
||||
iscsi-lun.img iSCSI fileio backstore (VLAN 20 only, not in active use)
|
||||
```
|
||||
|
||||
All shares are defined in `variables.nix` (`vars.nfsShares.*`). The NFS
|
||||
export list lives in `modules/ha/nfs-exports.nix`.
|
||||
|
||||
---
|
||||
|
||||
## Key variables
|
||||
|
||||
| Variable | Description |
|
||||
|---|---|
|
||||
| `vars.haServer1Ip` / `vars.haServer2Ip` | LAN management IPs |
|
||||
| `vars.haServer1StorageIp` / `vars.haServer2StorageIp` | Cluster NIC IPs (DRBD/Corosync ring0) |
|
||||
| `vars.haServerLanVip` | Pacemaker `vip-lan` — NFS for LAN (192.168.2.229) |
|
||||
| `vars.haServerVip` | Pacemaker `vip-storage` — NFS + iSCSI for VLAN 20 (192.168.20.229) |
|
||||
| `vars.haLanNfsFqdn` | FQDN of `vip-lan`: `ha-vip-lan.sweet.home` |
|
||||
| `vars.haStorageRoot` | XFS mount point: `/srv/ha-data` |
|
||||
| `vars.haServerDrbdDisk` | Block device for DRBD backing store |
|
||||
| `vars.haStorageCidr` | Cluster subnet CIDR (`192.168.10.224/29`) |
|
||||
| `vars.haClientCidr` | Storage-client subnet CIDR (`192.168.20.0/24`) |
|
||||
@@ -0,0 +1,330 @@
|
||||
# Docker Swarm Cutover Plan
|
||||
|
||||
Migration guide for moving containerised services from the existing single-host
|
||||
Docker LXC container (CT 105, `docker.sweet.home`, 192.168.2.225) to the new
|
||||
Docker Swarm cluster (`ha-docker-1` / `ha-docker-2`, 192.168.2.230–231).
|
||||
|
||||
CT 105 stays running throughout. Services migrate one stack at a time.
|
||||
Roll back any stack by restarting it on CT 105 if anything goes wrong.
|
||||
|
||||
---
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Swarm cluster deployed and healthy (`scripts/docker-swarm/deploy.sh`).
|
||||
- Both nodes show `Ready / Active / Manager` in `docker node ls`.
|
||||
- NFS mounts healthy on both swarm nodes (`/mnt/docker/config`, `/mnt/docker/databases`, `/mnt/docker/volumes`).
|
||||
- Access to FreeIPA DNS admin to update A records during cutover.
|
||||
|
||||
---
|
||||
|
||||
## 1. Traefik — switch to Docker log rotation
|
||||
|
||||
**Current state (CT 105):** Traefik writes access logs to the NFS volume at
|
||||
`/mnt/docker/volumes/traefik-data/logs/`. `modules/traefik/rotate-logs.nix`
|
||||
rotates those files via `logrotate`.
|
||||
|
||||
**Swarm approach:** Remove file-based access logging from Traefik's static
|
||||
config and rely on Docker's json-file log driver with built-in rotation.
|
||||
Traefik container logs (including access events) then live under
|
||||
`/var/lib/docker/containers/<id>/` on the node running Traefik.
|
||||
|
||||
### Steps
|
||||
|
||||
**1a.** In the Traefik stack definition, add logging config to the service:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
traefik:
|
||||
logging:
|
||||
driver: "json-file"
|
||||
options:
|
||||
max-size: "100m"
|
||||
max-file: "20"
|
||||
```
|
||||
|
||||
**1b.** In `traefik.yml` (Traefik's static config), remove the `accessLog`
|
||||
file path if present. To keep structured access logs, use Traefik's
|
||||
`accessLog.format: json` with no `filePath` — logs then go to stdout and are
|
||||
captured by the json-file driver above.
|
||||
|
||||
**1c.** Deploy Traefik to the swarm:
|
||||
|
||||
```bash
|
||||
# On either swarm manager:
|
||||
docker stack deploy -c /mnt/docker/config/traefik/docker-compose.yml traefik
|
||||
```
|
||||
|
||||
Traefik should be deployed as a **global mode** service so it runs on all
|
||||
swarm nodes and handles ingress on whichever node a request arrives at:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
traefik:
|
||||
deploy:
|
||||
mode: global
|
||||
placement:
|
||||
constraints:
|
||||
- node.role == manager
|
||||
```
|
||||
|
||||
**1d.** After confirming Traefik works on the swarm, remove
|
||||
`traefik/rotate-logs.nix` from the `docker` build type in
|
||||
`modules/build-types/docker.nix` and rebuild CT 105.
|
||||
|
||||
**DNS:** Update `docker.sweet.home` and any service FQDNs that point at
|
||||
192.168.2.225 to a swarm VIP or round-robin A records once Traefik is running
|
||||
on the swarm. See section 8 (DNS cutover).
|
||||
|
||||
---
|
||||
|
||||
## 2. Nextcloud — migrate cron job to sidecar container
|
||||
|
||||
**Current state (CT 105):** `modules/docker/nextcloud-cron-job.nix` runs a
|
||||
systemd timer every 5 minutes that calls:
|
||||
```bash
|
||||
docker exec nextcloud-webapp php ./cron.php
|
||||
```
|
||||
|
||||
**Swarm problem:** `docker exec` only works against the local daemon. If
|
||||
Nextcloud is scheduled on the other swarm node, the exec fails silently and
|
||||
cron never runs.
|
||||
|
||||
**Swarm approach:** Add a `nextcloud-cron` sidecar container to the Nextcloud
|
||||
stack definition, pinned to the same node as the main Nextcloud container via
|
||||
placement constraints.
|
||||
|
||||
### Steps
|
||||
|
||||
**2a.** Choose which swarm node will host Nextcloud (e.g. `ha-docker-1`).
|
||||
Label that node:
|
||||
|
||||
```bash
|
||||
# On either swarm manager:
|
||||
docker node update --label-add nextcloud=true ha-docker-1
|
||||
```
|
||||
|
||||
**2b.** In the Nextcloud stack compose file, add the sidecar and pin both
|
||||
services to the labelled node:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
nextcloud-webapp:
|
||||
image: nextcloud:production # pin same version as CT 105
|
||||
deploy:
|
||||
replicas: 1
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.nextcloud == true
|
||||
# ... existing volumes, env, networks ...
|
||||
|
||||
nextcloud-cron:
|
||||
image: nextcloud:production # same image, different entrypoint
|
||||
entrypoint: /cron.sh
|
||||
deploy:
|
||||
replicas: 1
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.nextcloud == true # must co-locate with webapp
|
||||
volumes:
|
||||
# Same data volume as nextcloud-webapp so cron sees the same files.
|
||||
- nextcloud-data:/var/www/html
|
||||
# No ports exposed — cron only runs PHP inside the container.
|
||||
```
|
||||
|
||||
`/cron.sh` is Nextcloud's built-in cron entrypoint. It runs
|
||||
`php -f /var/www/html/cron.php` in a loop, sleeping for 5 minutes between
|
||||
runs — identical to the current systemd timer.
|
||||
|
||||
**2c.** Migrate Nextcloud's data volume to the swarm:
|
||||
|
||||
```
|
||||
/mnt/docker/volumes/nextcloud-data/ → already on NFS, no migration needed
|
||||
/mnt/docker/databases/nextcloud/ → already on NFS, no migration needed
|
||||
```
|
||||
|
||||
The NFS paths are identical on the swarm nodes (`mount-data.nix` mounts the
|
||||
same shares from the same VIP). Stop Nextcloud on CT 105, deploy on the
|
||||
swarm, confirm it starts cleanly.
|
||||
|
||||
**2d.** Remove `nextcloud-cron-job.nix` from `modules/build-types/docker.nix`
|
||||
and rebuild CT 105 after confirming Nextcloud works on the swarm.
|
||||
|
||||
---
|
||||
|
||||
## 3. docker-health-to-gotify — update for swarm awareness
|
||||
|
||||
**Current state (CT 105):** The script at
|
||||
`/home/nixos/docker/monitoring/gotify/docker-health-to-gotify.sh` runs every
|
||||
minute, calls `docker ps --filter health=unhealthy`, and notifies Gotify.
|
||||
|
||||
**Swarm behaviour:** The same script runs on both swarm nodes independently,
|
||||
each monitoring its own local Docker daemon. This gives per-node coverage
|
||||
across the swarm.
|
||||
|
||||
**Changes needed in the script** (edit the copy on the NFS volume — it takes
|
||||
effect on both nodes simultaneously on the next timer fire):
|
||||
|
||||
### 3a. Strip the Swarm task suffix from service names
|
||||
|
||||
In swarm mode, `docker ps --format '{{.Names}}'` returns names like
|
||||
`nextcloud-webapp.1.abc123xyz`. The notification should show `nextcloud-webapp`,
|
||||
not the full task name.
|
||||
|
||||
```bash
|
||||
# Before:
|
||||
CONTAINER_NAME=$(docker ps --format '{{.Names}}' ...)
|
||||
|
||||
# After:
|
||||
CONTAINER_NAME=$(docker ps --format '{{.Names}}' ... | cut -d. -f1)
|
||||
```
|
||||
|
||||
### 3b. Include the reporting node in the Gotify message
|
||||
|
||||
Add `$(hostname)` to the notification payload so you know which swarm node
|
||||
detected the problem:
|
||||
|
||||
```bash
|
||||
MESSAGE="[$(hostname)] ${CONTAINER_NAME} is unhealthy"
|
||||
```
|
||||
|
||||
### 3c. Extend to catch swarm service replica failures
|
||||
|
||||
`docker ps` only shows what's running locally. If a service has zero healthy
|
||||
replicas (task crash-looping) it may not show up on either node's `docker ps`
|
||||
at the same moment. Add a swarm-level check:
|
||||
|
||||
```bash
|
||||
# Run only on managers (both ha-docker nodes are managers):
|
||||
if docker info --format '{{.Swarm.ControlAvailable}}' 2>/dev/null | grep -q true; then
|
||||
# Find services where running replicas < desired replicas
|
||||
docker service ls --format '{{.Name}}\t{{.Replicas}}' | \
|
||||
awk -F'\t' '$2 !~ /^[0-9]+\/[0-9]+$/ || split($2,a,"/") && a[1] < a[2] { print $1, $2 }' | \
|
||||
while read -r svc_name replicas; do
|
||||
# Send Gotify notification for degraded service
|
||||
curl -s -X POST "${GOTIFY_URL}/message" \
|
||||
-H "X-Gotify-Key: ${GOTIFY_TOKEN}" \
|
||||
-d "title=Swarm service degraded" \
|
||||
-d "message=[$(hostname)] ${svc_name}: ${replicas} replicas"
|
||||
done
|
||||
fi
|
||||
```
|
||||
|
||||
This catches the case where a service's desired replicas are not running
|
||||
(e.g. OOM kill, image pull failure) — a failure mode that doesn't produce a
|
||||
Docker health event on any node.
|
||||
|
||||
---
|
||||
|
||||
## 4. Passbolt migration
|
||||
|
||||
Passbolt has strict data integrity requirements. Migrate with care:
|
||||
|
||||
1. **Backup first** — `docker exec passbolt-webapp php /usr/share/php/passbolt/bin/cake passbolt export_keys` and a database dump.
|
||||
2. Database is on NFS (`/mnt/docker/databases/passbolt/`) — no data copy needed.
|
||||
3. Pin Passbolt to a specific node: `docker node update --label-add passbolt=true ha-docker-1`
|
||||
4. Add placement constraint `node.labels.passbolt == true` to the Passbolt stack.
|
||||
5. Stop on CT 105, deploy on swarm, verify login works.
|
||||
6. Test email delivery and 2FA.
|
||||
|
||||
---
|
||||
|
||||
## 5. Gitea migration
|
||||
|
||||
Gitea's data directory is on NFS (`/mnt/docker/volumes/gitea-data/`).
|
||||
|
||||
1. Stop Gitea on CT 105: `docker stop gitea`
|
||||
2. Deploy to swarm with placement constraint (pin to `ha-docker-1` initially).
|
||||
3. Verify web UI and SSH clone/push work.
|
||||
4. Update DNS: `gitea.lan.ddnsgeek.com` → swarm Traefik endpoint.
|
||||
5. Update the flake remote URL in `variables.nix` (`giteaDomain`) if the address changes.
|
||||
|
||||
---
|
||||
|
||||
## 6. Other services
|
||||
|
||||
Deploy remaining services (Grafana, InfluxDB, NodeRed, Prometheus, etc.)
|
||||
as swarm stacks. Most have no special migration concern — they use NFS
|
||||
volumes already on the shared storage.
|
||||
|
||||
Services with stateful databases (PostgreSQL, MariaDB) should follow the
|
||||
pattern: stop on CT 105, confirm NFS database directory is intact, deploy on
|
||||
swarm, verify.
|
||||
|
||||
---
|
||||
|
||||
## 7. Monitoring — Beszel
|
||||
|
||||
The Beszel hub runs on CT 105 (`docker.sweet.home:8090`). Both swarm nodes
|
||||
run `beszel-agent` (from `modules/beszel/enable-agent.nix`), pointing at the
|
||||
existing hub URL.
|
||||
|
||||
No migration needed for Beszel itself during the container migration. Once
|
||||
all services are on the swarm, you may wish to move the Beszel hub too (as a
|
||||
swarm service with a placement constraint) but this is optional.
|
||||
|
||||
---
|
||||
|
||||
## 8. DNS cutover
|
||||
|
||||
When a service is confirmed working on the swarm, update the FreeIPA DNS
|
||||
A record from the CT 105 IP (192.168.2.225) to a swarm node IP or, when a
|
||||
shared Traefik frontend is in place, to a round-robin record across both nodes.
|
||||
|
||||
**Recommended approach — Traefik as the single entry point:**
|
||||
|
||||
```
|
||||
service.lan.ddnsgeek.com → Traefik on swarm (global mode)
|
||||
docker.sweet.home → keep as 192.168.2.225 (CT 105) until fully decommissioned
|
||||
```
|
||||
|
||||
For LAN-only services using `*.sweet.home` names, update FreeIPA directly:
|
||||
|
||||
```bash
|
||||
# On domain-controller (or via SSH):
|
||||
ipa dnsrecord-mod sweet.home nextcloud --a-rec=192.168.2.230
|
||||
# Add 192.168.2.231 as a second A record for round-robin (optional):
|
||||
ipa dnsrecord-add sweet.home nextcloud --a-rec=192.168.2.231
|
||||
```
|
||||
|
||||
Services behind Traefik don't need their own DNS updates — only Traefik's
|
||||
own entry point IPs need to change.
|
||||
|
||||
---
|
||||
|
||||
## 9. NixOS cleanup — CT 105
|
||||
|
||||
Once all services are migrated:
|
||||
|
||||
**Remove from `modules/build-types/docker.nix`:**
|
||||
- `../docker/nextcloud-cron-job.nix` — replaced by sidecar container
|
||||
- `../traefik/rotate-logs.nix` — replaced by Docker log driver
|
||||
|
||||
**Keep in `modules/build-types/docker.nix` until CT 105 is decommissioned:**
|
||||
- `../docker/docker-health-to-gotify.nix` — still monitors CT 105's own daemon
|
||||
- Everything else
|
||||
|
||||
**When decommissioning CT 105:**
|
||||
1. Confirm all NFS volumes are in use only by swarm services (not CT 105).
|
||||
2. Stop CT 105: `pct stop 105` on pve1.
|
||||
3. Archive/remove the `lxc-docker` and `proxmox-docker` targets from `flake.nix`.
|
||||
4. Remove `hosts/docker/`, `modules/build-types/docker.nix`, and `modules/docker/`.
|
||||
5. Update `variables.nix` to remove `dockerIp`, `dockerStorageIp`, `dockerHost`
|
||||
(or reassign `dockerHost` to point at a swarm node for Beszel hub resolution).
|
||||
|
||||
---
|
||||
|
||||
## Rollback
|
||||
|
||||
Any stack can be rolled back to CT 105 independently:
|
||||
|
||||
```bash
|
||||
# On CT 105:
|
||||
docker start <service-name>
|
||||
# Update DNS A record back to 192.168.2.225
|
||||
ipa dnsrecord-mod sweet.home <service> --a-rec=192.168.2.225
|
||||
```
|
||||
|
||||
CT 105 remains running throughout the cutover. Only decommission it after
|
||||
every service is confirmed stable on the swarm and you have run one full
|
||||
backup cycle from the new hosts.
|
||||
+41
-38
@@ -7,6 +7,10 @@
|
||||
| LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
|
||||
| Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
|
||||
| Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink |
|
||||
| Swarm cluster | 30 | `192.168.30.0/24` | Docker Swarm gossip (TCP/UDP 7946) + VXLAN overlay (UDP 4789) | No — internal `vmbr3` only, no uplink |
|
||||
|
||||
When expanded to a second Proxmox node, VLAN 10 (cluster), VLAN 20 (storage-client), and VLAN 30 (swarm) all share
|
||||
the same inter-node trunk NIC via 802.1q VLAN tagging — different VLAN IDs, same physical cable.
|
||||
|
||||
The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
|
||||
bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
|
||||
@@ -27,9 +31,9 @@ is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20
|
||||
|
||||
## DNS Zones
|
||||
|
||||
FreeIPA (domain-controller.sweet.home) is authoritative for all zones. Three
|
||||
zones correspond to the three subnets — one per VLAN. All zones are internal
|
||||
only; no external delegation.
|
||||
FreeIPA (domain-controller.sweet.home) is authoritative for all zones.
|
||||
|
||||
Four zones correspond to the four subnets. All zones are internal only; no external delegation.
|
||||
|
||||
### sweet.home — VLAN 2 (192.168.2.x)
|
||||
|
||||
@@ -132,12 +136,14 @@ All VMs and LXC containers run on pve1.
|
||||
| `192.168.2.228` | ha-node1 | HA file server node 1 — management NIC | Active |
|
||||
| `192.168.2.227` | ha-node2 | HA file server node 2 — management NIC | Active |
|
||||
| `192.168.2.226` | server | Former NFS/ZFS file server — decommissioned | Removed from flake |
|
||||
| `192.168.2.225` | docker | Docker / Traefik stack | Active |
|
||||
| `192.168.2.225` | docker | Docker / Traefik stack (CT 105 — existing single-host) | Active |
|
||||
| `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
|
||||
| `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active |
|
||||
| `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active |
|
||||
| `192.168.2.221` | tor-relay | Tor relay | Active |
|
||||
| `192.168.2.220` | pdm | Proxmox Deploy Manager | Active |
|
||||
| `192.168.2.231` | ha-docker-2 | Docker Swarm node 2 — management NIC | Active |
|
||||
| `192.168.2.230` | ha-docker-1 | Docker Swarm node 1 — management NIC | Active |
|
||||
|
||||
### Client DHCP pool (.10–.59)
|
||||
|
||||
@@ -175,7 +181,9 @@ Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached.
|
||||
| `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
|
||||
| `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
|
||||
| `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — decommissioned |
|
||||
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client |
|
||||
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client (CT 105) |
|
||||
| `192.168.20.231` | ha-docker-2 | Storage-client NIC (ens19 / vmbr2) — NFS client |
|
||||
| `192.168.20.230` | ha-docker-1 | Storage-client NIC (ens19 / vmbr2) — NFS client |
|
||||
| — | no gateway | Isolated — not routed to LAN or internet |
|
||||
|
||||
NFS clients mount from `192.168.20.229` (surviving failover transparently via the VIP).
|
||||
@@ -186,40 +194,35 @@ active HA node.
|
||||
|
||||
---
|
||||
|
||||
## Migration reference
|
||||
## Swarm cluster network — VLAN 30 — 192.168.30.0/24
|
||||
|
||||
Current → target IP for every host being renumbered.
|
||||
Internal to pve1 only. Proxmox bridge `vmbr3`, no physical NIC attached.
|
||||
Carries Docker Swarm inter-node traffic only: Raft consensus (TCP 2377),
|
||||
Serf gossip (TCP/UDP 7946), and VXLAN overlay data path (UDP 4789).
|
||||
Docker Swarm is initialised with `--advertise-addr` and `--data-path-addr`
|
||||
both pointing to this subnet so all cluster traffic stays on `vmbr3` and
|
||||
never crosses the LAN.
|
||||
|
||||
| Host | Current IP | New IP | Config location |
|
||||
|---|---|---|---|
|
||||
| router | `192.168.2.254` | `192.168.2.254` | unchanged |
|
||||
| domain-controller | `192.168.2.138` | `192.168.2.253` | `/etc/sysconfig/network-scripts/ifcfg-eth0` on guest |
|
||||
| pve1 | `192.168.2.250` | `192.168.2.245` | `/etc/network/interfaces` on Proxmox host |
|
||||
| pbs | `192.168.2.108` | `192.168.2.244` | static config on PBS host |
|
||||
| nixos workstation | `192.168.2.119` | `192.168.2.243` | `networking.interfaces` / NetworkManager on guest |
|
||||
| ha-node1 | — | `192.168.2.228` (LAN), `192.168.10.228` (cluster/VLAN 10), `192.168.20.228` (storage/VLAN 20) | active |
|
||||
| ha-node2 | — | `192.168.2.227` (LAN), `192.168.10.227` (cluster/VLAN 10), `192.168.20.227` (storage/VLAN 20) | active |
|
||||
| ha-vip-lan | — | `192.168.2.229` (vmbr0 / Pacemaker `vip-lan`) — NFS endpoint for LAN clients | active |
|
||||
| ha-vip-storage | — | `192.168.20.229` (vmbr2 / Pacemaker `vip-storage`) — iSCSI endpoint for VLAN 20 clients | active |
|
||||
| server | `192.168.2.252` | `192.168.2.226` | static config on guest |
|
||||
| docker | `192.168.2.249` | `192.168.2.225` | static config on guest |
|
||||
| nix-cache | `192.168.2.120` | `192.168.2.224` | static config on guest |
|
||||
| pxe-boot | `192.168.2.247` | `192.168.2.223` | static config on guest; update `vars.pxeServerIp` in `variables.nix` ✓ |
|
||||
| tailscale-router | `192.168.2.121` | `192.168.2.222` | static config on guest |
|
||||
| tor-relay | `192.168.2.107` | `192.168.2.221` | static config on guest |
|
||||
| pdm | `192.168.2.248` | `192.168.2.220` | static config on guest |
|
||||
| IP | Hostname | Interface / role |
|
||||
|---|---|---|
|
||||
| `192.168.30.231` | ha-docker-2 | Swarm cluster NIC (ens20 / vmbr3) |
|
||||
| `192.168.30.230` | ha-docker-1 | Swarm cluster NIC (ens20 / vmbr3) |
|
||||
| — | no gateway | Isolated — not routed to LAN or internet |
|
||||
|
||||
### Cutover notes
|
||||
### DNS zone: `swarm.home` — VLAN 30 (192.168.30.x)
|
||||
|
||||
| Hostname | A record | Notes |
|
||||
|---|---|---|
|
||||
| `ha-docker-1.swarm.home` | `192.168.30.230` | Swarm NIC — debugging only |
|
||||
| `ha-docker-2.swarm.home` | `192.168.30.231` | Swarm NIC — debugging only |
|
||||
|
||||
Operators reach the Docker API on the LAN IPs (`192.168.2.230`/`.231`), not these addresses.
|
||||
The `swarm.home` records exist for diagnostic convenience (e.g. confirming `vmbr3` routing).
|
||||
|
||||
### Multi-node Proxmox expansion
|
||||
|
||||
When a second Proxmox node (pve2) is added, VLAN 10 (cluster), VLAN 20 (storage-client),
|
||||
and VLAN 30 (swarm) all extend to pve2 via 802.1q VLAN tagging on the inter-node trunk
|
||||
link. All three internal networks share the same physical NIC between hypervisors —
|
||||
VLAN tags provide the logical separation.
|
||||
|
||||
- **Do domain-controller first** — it becomes the DNS server; everything else depends on it
|
||||
having its new IP and FreeIPA DNS configured before Pi-hole is retired.
|
||||
- **pve1 last among physical hosts** — changing the Proxmox management IP drops the web UI
|
||||
briefly; all guests keep running.
|
||||
- **Update Pi-hole custom.list / FreeIPA DNS A records** to new IPs before flipping any host,
|
||||
so name resolution stays valid throughout the migration.
|
||||
- **variables.nix already updated** for `pxeServerIp` (.247→.223), `pbsIp` (.108→.244), and
|
||||
new `domainControllerIp` (.253). Rebuild affected hosts after renumbering.
|
||||
- **Router DHCP**: once domain-controller is at .253 and FreeIPA DNS is serving `sweet.home`,
|
||||
switch router DHCP on with pool .10–.59 and DNS option pointing to .253; retire Pi-hole CT.
|
||||
- **Pi-hole's iPXE dnsmasq config** (`99-ipxe-chainload.conf`) moves to the pxe-boot CT as a
|
||||
dnsmasq proxy-mode config before Pi-hole is decommissioned.
|
||||
|
||||
@@ -1,448 +0,0 @@
|
||||
# Network Cutover Plan
|
||||
|
||||
Moves the LAN from the current flat/Pi-hole-managed state to the new IP scheme
|
||||
defined in `docs/ip-addressing.md`. Works in five independent stages — each
|
||||
stage is safe to pause after and resume later. Rollback steps are given at
|
||||
every point where something can break.
|
||||
|
||||
**Before starting anything:** confirm you have
|
||||
- SSH access to `192.168.2.138` (domain-controller, current IP)
|
||||
- SSH access to `192.168.2.250` (pve1)
|
||||
- Browser access to Pi-hole admin at `http://192.168.2.253`
|
||||
- Browser access to router admin at `http://192.168.2.254`
|
||||
- The FreeIPA `admin` password to hand
|
||||
|
||||
---
|
||||
|
||||
## Stage 1 — Prepare FreeIPA DNS (zero downtime)
|
||||
|
||||
Everything here is additive. Pi-hole keeps running. Nothing breaks if you stop
|
||||
mid-stage.
|
||||
|
||||
### 1a. Add NextDNS forwarders
|
||||
|
||||
```bash
|
||||
ssh wayne@192.168.2.138
|
||||
kinit admin # enter FreeIPA admin password when prompted
|
||||
ipa dnsconfig-mod \
|
||||
--forwarder=45.90.28.142 \
|
||||
--forwarder=45.90.30.142 \
|
||||
--forward-policy=only
|
||||
```
|
||||
|
||||
**Verify external resolution works through FreeIPA before continuing:**
|
||||
```bash
|
||||
dig @127.0.0.1 google.com +short # must return an IP, not SERVFAIL
|
||||
```
|
||||
|
||||
### 1b. Add A records for every host at their CURRENT IPs
|
||||
|
||||
These represent the live state now. You'll update each record to the new IP
|
||||
when you renumber that host in Stage 5.
|
||||
|
||||
```bash
|
||||
ipa dnsrecord-add sweet.home pve1 --a-rec 192.168.2.250
|
||||
ipa dnsrecord-add sweet.home pbs --a-rec 192.168.2.108
|
||||
ipa dnsrecord-add sweet.home nixos --a-rec 192.168.2.119
|
||||
ipa dnsrecord-add sweet.home server --a-rec 192.168.2.252
|
||||
ipa dnsrecord-add sweet.home docker --a-rec 192.168.2.249
|
||||
ipa dnsrecord-add sweet.home nix-cache --a-rec 192.168.2.120
|
||||
ipa dnsrecord-add sweet.home pxe-boot --a-rec 192.168.2.247
|
||||
ipa dnsrecord-add sweet.home tailscale-router --a-rec 192.168.2.121
|
||||
ipa dnsrecord-add sweet.home tor-relay --a-rec 192.168.2.107
|
||||
ipa dnsrecord-add sweet.home pdm --a-rec 192.168.2.248
|
||||
ipa dnsrecord-add sweet.home router --a-rec 192.168.2.254
|
||||
```
|
||||
|
||||
### 1c. Clean up stale reverse-zone PTR records
|
||||
|
||||
FreeIPA already has PTR records from an earlier import but some are wrong.
|
||||
Fix them now so reverse DNS is accurate from day one.
|
||||
|
||||
```bash
|
||||
# Remove stale "win11" entry at .250 (should be pve1)
|
||||
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec win11.
|
||||
ipa dnsrecord-add 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
|
||||
|
||||
# Fix unqualified PTR records (missing .sweet.home. suffix)
|
||||
ipa dnsrecord-mod 2.168.192.in-addr.arpa 108 --ptr-rec pbs.sweet.home.
|
||||
ipa dnsrecord-mod 2.168.192.in-addr.arpa 248 --ptr-rec pdm.sweet.home.
|
||||
ipa dnsrecord-mod 2.168.192.in-addr.arpa 249 --ptr-rec docker.sweet.home.
|
||||
ipa dnsrecord-mod 2.168.192.in-addr.arpa 252 --ptr-rec server.sweet.home.
|
||||
|
||||
# Add any missing PTR records
|
||||
ipa dnsrecord-add 2.168.192.in-addr.arpa 119 --ptr-rec nixos.sweet.home.
|
||||
ipa dnsrecord-add 2.168.192.in-addr.arpa 120 --ptr-rec nix-cache.sweet.home.
|
||||
ipa dnsrecord-add 2.168.192.in-addr.arpa 121 --ptr-rec tailscale-router.sweet.home.
|
||||
ipa dnsrecord-add 2.168.192.in-addr.arpa 247 --ptr-rec pxe-boot.sweet.home.
|
||||
ipa dnsrecord-add 2.168.192.in-addr.arpa 254 --ptr-rec router.sweet.home.
|
||||
```
|
||||
|
||||
### 1d. Point domain-controller's own DNS at itself
|
||||
|
||||
```bash
|
||||
sudo nmcli connection modify "System eth0" ipv4.dns "127.0.0.1"
|
||||
sudo nmcli connection up "System eth0"
|
||||
```
|
||||
|
||||
**Verify:**
|
||||
```bash
|
||||
dig pve1.sweet.home +short # must return 192.168.2.250
|
||||
dig google.com +short # must return an IP (NextDNS forwarding)
|
||||
```
|
||||
|
||||
**Rollback 1d:** `sudo nmcli connection modify "System eth0" ipv4.dns "192.168.2.253" && sudo nmcli connection up "System eth0"`
|
||||
|
||||
---
|
||||
|
||||
## Stage 2 — Move pxe-boot DHCP options off Pi-hole (zero downtime)
|
||||
|
||||
Pi-hole's dnsmasq currently serves the iPXE boot options via
|
||||
`99-ipxe-chainload.conf`. Before Pi-hole is retired, that config must move to
|
||||
the pxe-boot CT running dnsmasq in proxy mode so PXE boot keeps working.
|
||||
|
||||
### 2a. Add dnsmasq proxy config to the pxe-boot NixOS module
|
||||
|
||||
In `modules/build-types/pxe-boot.nix`, add:
|
||||
|
||||
```nix
|
||||
services.dnsmasq = {
|
||||
enable = true;
|
||||
settings = {
|
||||
# Proxy mode: respond only to PXE DHCP requests, leave normal leases to router
|
||||
dhcp-range = [ "192.168.2.0,proxy" ];
|
||||
# iPXE client detection
|
||||
dhcp-match = [
|
||||
"set:ipxe,175"
|
||||
"set:efi64,option:client-arch,7"
|
||||
"set:efi64,option:client-arch,9"
|
||||
];
|
||||
dhcp-userclass = "set:ipxe,iPXE";
|
||||
# Boot file selection
|
||||
dhcp-boot = [
|
||||
"tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
|
||||
"tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
|
||||
"tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
|
||||
"undionly.kpxe,,${vars.pxeServerIp}"
|
||||
];
|
||||
};
|
||||
};
|
||||
```
|
||||
|
||||
### 2b. Rebuild and deploy the pxe-boot CT
|
||||
|
||||
```bash
|
||||
# On pve1 — build the new tarball
|
||||
nix build .#lxc-pxe-boot.config.system.build.tarball
|
||||
|
||||
# Verify dnsmasq starts correctly in the CT after deploy
|
||||
ssh nixos@192.168.2.247 systemctl status dnsmasq
|
||||
```
|
||||
|
||||
### 2c. Remove the iPXE config from Pi-hole
|
||||
|
||||
In the Pi-hole CT, remove `/etc/dnsmasq.d/99-ipxe-chainload.conf` and
|
||||
restart the FTL service:
|
||||
|
||||
```bash
|
||||
ssh wayne@pve1.sweet.home \
|
||||
"sudo pct exec 100 -- bash -c 'rm /etc/dnsmasq.d/99-ipxe-chainload.conf && systemctl restart pihole-FTL'"
|
||||
```
|
||||
|
||||
**Verify:** PXE boot a test machine — it should still get an iPXE response and
|
||||
reach the boot menu.
|
||||
|
||||
**Rollback 2c:** restore the file from the Pi-hole config backup at
|
||||
`/etc/pihole/config_backups/` and restart pihole-FTL.
|
||||
|
||||
---
|
||||
|
||||
## Stage 3 — DHCP migration: Pi-hole → router (brief maintenance window)
|
||||
|
||||
**Do this in the evening.** Existing DHCP leases stay valid during the
|
||||
switchover so connected devices don't drop — only new lease requests fail
|
||||
during the gap, which is under 60 seconds if you follow the steps in order.
|
||||
|
||||
The key: configure the router's DHCP DNS option to point at `.253` (Pi-hole's
|
||||
current IP). This way, all new leases issued by the router still get the same
|
||||
DNS server address — clients never need to change their DNS config. When Pi-hole
|
||||
is retired and the DC takes `.253` in Stage 4, `.253` just starts answering
|
||||
differently. No client reconfiguration.
|
||||
|
||||
### 3a. Pre-configure router DHCP (do not enable yet)
|
||||
|
||||
Log into `http://192.168.2.254`, find the DHCP settings and fill in — but
|
||||
leave DHCP **disabled** until step 3b:
|
||||
|
||||
| Setting | Value |
|
||||
|---|---|
|
||||
| Start IP | 192.168.2.10 |
|
||||
| End IP | 192.168.2.59 |
|
||||
| Subnet mask | 255.255.255.0 |
|
||||
| Gateway | 192.168.2.254 |
|
||||
| Primary DNS | 192.168.2.253 |
|
||||
| Secondary DNS | *(leave blank)* |
|
||||
| Lease time | 24h |
|
||||
|
||||
Save without enabling.
|
||||
|
||||
### 3b. Switchover (do steps in quick succession)
|
||||
|
||||
1. **Disable Pi-hole DHCP:** Pi-hole admin UI → Settings → DHCP → uncheck
|
||||
"DHCP server enabled" → Save
|
||||
2. **Enable router DHCP** immediately after step 1
|
||||
|
||||
### 3c. Verify router DHCP is working
|
||||
|
||||
On a phone or laptop, disconnect from WiFi and reconnect (or run
|
||||
`sudo dhclient -r && sudo dhclient` on a Linux host):
|
||||
|
||||
```bash
|
||||
ip addr show # IP should be in 192.168.2.10–59 range
|
||||
dig google.com # should resolve (Pi-hole DNS still running at .253)
|
||||
dig pve1.sweet.home # should resolve via FreeIPA at .138 (relayed via Pi-hole)
|
||||
```
|
||||
|
||||
Wait 10–15 minutes for the most active devices to renew their leases. There's
|
||||
no need to wait for all leases to expire before proceeding.
|
||||
|
||||
**Rollback 3b:** Re-enable Pi-hole DHCP. Disable router DHCP. Done — existing
|
||||
leases remain valid so most devices are unaffected.
|
||||
|
||||
---
|
||||
|
||||
## Stage 4 — Move domain-controller from .138 to .253
|
||||
|
||||
Pi-hole lives at `.253`. The DC must take `.253` the moment Pi-hole stops so
|
||||
clients that still have `.253` as their DNS server don't notice the change.
|
||||
Script these commands in advance and run them in rapid succession.
|
||||
|
||||
**Pre-stage: have this SSH command ready before running step 4a:**
|
||||
```bash
|
||||
ssh wayne@192.168.2.138 "
|
||||
sudo nmcli connection modify 'System eth0' \
|
||||
ipv4.addresses '192.168.2.253/24' \
|
||||
ipv4.gateway '192.168.2.254' \
|
||||
ipv4.dns '127.0.0.1' \
|
||||
ipv4.method manual && \
|
||||
sudo nmcli connection up 'System eth0'
|
||||
"
|
||||
```
|
||||
|
||||
**Also update the Proxmox VM config to match (run from pve1):**
|
||||
```bash
|
||||
sudo qm set 108 \
|
||||
--ipconfig0 ip=192.168.2.253/24,gw=192.168.2.254 \
|
||||
--nameserver 192.168.2.253
|
||||
```
|
||||
|
||||
### 4a. Stop Pi-hole
|
||||
|
||||
```bash
|
||||
ssh wayne@pve1.sweet.home "sudo pct stop 100"
|
||||
```
|
||||
|
||||
### 4b. Immediately: change DC's IP to .253
|
||||
|
||||
Run the pre-staged SSH command from above. You have ~30 seconds before any
|
||||
client notices Pi-hole is gone. If SSH to `.138` refuses (the IP is already
|
||||
changing), open a Proxmox console to VM 108 and run the `nmcli` commands
|
||||
there.
|
||||
|
||||
### 4c. Update Proxmox VM config
|
||||
|
||||
Run the pre-staged `qm set 108` command from above.
|
||||
|
||||
### 4d. Verify
|
||||
|
||||
```bash
|
||||
ssh wayne@192.168.2.253 # must connect (new DC IP)
|
||||
dig @192.168.2.253 pve1.sweet.home +short # must return 192.168.2.250
|
||||
dig @192.168.2.253 google.com +short # must return an IP
|
||||
```
|
||||
|
||||
From a client device that renewed its DHCP lease in Stage 3:
|
||||
```bash
|
||||
cat /etc/resolv.conf # should show 192.168.2.253
|
||||
dig pve1.sweet.home # should resolve
|
||||
```
|
||||
|
||||
**Rollback 4:** `ssh wayne@pve1.sweet.home "sudo pct start 100"`. Change DC IP
|
||||
back to .138 via Proxmox console. This restores full Pi-hole DNS/DHCP service.
|
||||
Leave Pi-hole CT stopped-but-intact for 48 hours before deleting it.
|
||||
|
||||
---
|
||||
|
||||
## Stage 5 — Host renumbering (one at a time, any order)
|
||||
|
||||
For each host:
|
||||
1. Update FreeIPA DNS A record and PTR record to the new IP
|
||||
2. Change the static IP on the host itself
|
||||
3. Verify SSH to new IP
|
||||
4. Update `variables.nix` if that host has an IP variable (pxe-boot, pbs — already done in this PR)
|
||||
|
||||
**FreeIPA record update template** (run as admin on domain-controller):
|
||||
```bash
|
||||
ipa dnsrecord-mod sweet.home <hostname> --a-rec <new-ip>
|
||||
ipa dnsrecord-del 2.168.192.in-addr.arpa <old-last-octet> --ptr-rec <hostname>.sweet.home.
|
||||
ipa dnsrecord-add 2.168.192.in-addr.arpa <new-last-octet> --ptr-rec <hostname>.sweet.home.
|
||||
```
|
||||
|
||||
### Renumbering order
|
||||
|
||||
| # | Host | Old IP | New IP | How to change IP |
|
||||
|---|---|---|---|---|
|
||||
| 1 | nixos workstation | .119 | .243 | NetworkManager on guest; or `nmcli connection modify` |
|
||||
| 2 | nix-cache | .120 | .224 | `pct set 102 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.224/24,gw=192.168.2.254` then `pct reboot 102` |
|
||||
| 3 | tailscale-router | .121 | .222 | Static config on guest; check Tailscale ACLs if IP is referenced there |
|
||||
| 4 | tor-relay | .107 | .221 | `pct set 104 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.221/24,gw=192.168.2.254` then `pct reboot 104` |
|
||||
| 5 | pdm | .248 | .220 | `pct set 106 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.220/24,gw=192.168.2.254` then `pct reboot 106` |
|
||||
| 6 | pxe-boot | .247 | .223 | `pct set 103 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.223/24,gw=192.168.2.254` then rebuild NixOS (already updated in variables.nix) |
|
||||
| 7 | server | .252 | .226 | Static config on guest; NFS clients (docker) lose mounts briefly — they remount automatically |
|
||||
| 8 | docker | .249 | .225 | Static config on guest; do this after server is at .226 |
|
||||
| 9 | pbs | .108 | .244 | Static config on PBS host itself; update in `pbsIp` already done in variables.nix |
|
||||
| 10 | pve1 | .250 | .245 | Edit `/etc/network/interfaces` on the Proxmox host — see below |
|
||||
|
||||
### pve1 renumber (step 10 — do last)
|
||||
|
||||
All guests keep running; only the Proxmox web UI is briefly unreachable.
|
||||
|
||||
```bash
|
||||
ssh wayne@pve1.sweet.home
|
||||
|
||||
# Edit /etc/network/interfaces: change address from .250 to .245
|
||||
sudo nano /etc/network/interfaces
|
||||
# Change: address 192.168.2.250/24
|
||||
# To: address 192.168.2.245/24
|
||||
|
||||
sudo systemctl restart networking
|
||||
# SSH will drop here — reconnect to new IP
|
||||
```
|
||||
|
||||
```bash
|
||||
ssh wayne@192.168.2.245 # verify
|
||||
```
|
||||
|
||||
Update FreeIPA DNS:
|
||||
```bash
|
||||
ipa dnsrecord-mod sweet.home pve1 --a-rec 192.168.2.245
|
||||
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
|
||||
ipa dnsrecord-add 2.168.192.in-addr.arpa 245 --ptr-rec pve1.sweet.home.
|
||||
```
|
||||
|
||||
**Rollback any step 5 host:** change the IP back on the guest and update the
|
||||
FreeIPA record back to the old IP. The old IP is unoccupied so you can
|
||||
temporarily use either.
|
||||
|
||||
---
|
||||
|
||||
## Stage 6 — HA storage cutover (docker NFS remount)
|
||||
|
||||
> **Prerequisites:**
|
||||
> - HA cluster fully deployed and `vip-storage` (`nfs.storage.home` → 192.168.20.229) serving NFS ✓
|
||||
> - DNS configured: `storage.home` zone populated, `nfs.storage.home` resolves to 192.168.20.229 ✓
|
||||
> - docker CT has eth1 on vmbr2 (`docker.storage.home` → 192.168.20.225) ✓
|
||||
> - Final rsync from server.sweet.home to `/srv/ha-data` complete before step 6b
|
||||
|
||||
docker.sweet.home currently NFS-mounts its persistent volumes from `server.sweet.home`
|
||||
(`192.168.2.226:/tank/docker/...`). This stage moves those mounts to the HA cluster's
|
||||
storage VIP so server can be decommissioned.
|
||||
|
||||
### 6a. Final rsync from server to HA cluster
|
||||
|
||||
Run from server.sweet.home (or over SSH from the workstation) to sync any data written
|
||||
since the initial rsync:
|
||||
|
||||
```bash
|
||||
# Confirm active HA node and mount point
|
||||
ssh wayne@192.168.2.228 'sudo findmnt /srv/ha-data' # check which node is active
|
||||
|
||||
# rsync each dataset (adjust source paths to match /tank layout on server)
|
||||
sudo rsync -av --delete /tank/docker/config/ wayne@<active-node-ip>:/srv/ha-data/docker/config/
|
||||
sudo rsync -av --delete /tank/docker/databases/ wayne@<active-node-ip>:/srv/ha-data/docker/databases/
|
||||
sudo rsync -av --delete /tank/docker/volumes/ wayne@<active-node-ip>:/srv/ha-data/docker/volumes/
|
||||
sudo rsync -av --delete /tank/docker/nextcloud-data/ wayne@<active-node-ip>:/srv/ha-data/docker/nextcloud-data/
|
||||
```
|
||||
|
||||
### 6b. Update docker NixOS config to mount from vip-storage
|
||||
|
||||
In `hosts/docker/host.nix` (or wherever the NFS mount fileSystems are declared), change
|
||||
the NFS server from `server.sweet.home` / `192.168.2.226` to `nfs.storage.home`:
|
||||
|
||||
```nix
|
||||
# Before:
|
||||
fileSystems."/mnt/docker/config" = {
|
||||
device = "server:/tank/docker/config"; # or 192.168.2.226:...
|
||||
...
|
||||
};
|
||||
|
||||
# After:
|
||||
fileSystems."/mnt/docker/config" = {
|
||||
device = "nfs.storage.home:/srv/ha-data/docker/config";
|
||||
...
|
||||
};
|
||||
```
|
||||
|
||||
Using the DNS name (`nfs.storage.home`) rather than the VIP IP means the mount
|
||||
config survives a future VIP renumber without touching the NixOS config.
|
||||
Repeat for all four docker shares (`config`, `databases`, `volumes`, `nextcloud-data`).
|
||||
Then rebuild docker:
|
||||
|
||||
```bash
|
||||
# On the workstation — or via Switch-nix on docker itself
|
||||
sudo nixos-rebuild switch --no-write-lock-file --refresh \
|
||||
--flake "git+https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos.git#lxc-docker"
|
||||
```
|
||||
|
||||
### 6c. Verify mounts and container health
|
||||
|
||||
```bash
|
||||
ssh wayne@192.168.2.225 'findmnt | grep 192.168.20' # mounts should show vip-storage
|
||||
ssh wayne@192.168.2.225 'docker ps' # all containers running
|
||||
```
|
||||
|
||||
Spot-check Nextcloud, Traefik, and any database containers for connectivity.
|
||||
|
||||
### 6d. Decommission server.sweet.home
|
||||
|
||||
Once docker is confirmed healthy on the HA NFS mounts:
|
||||
|
||||
```bash
|
||||
# Stop server VM on pve1
|
||||
ssh wayne@192.168.2.245 'sudo qm stop 101'
|
||||
|
||||
# (Optional) Archive the ZFS pool snapshot before destroying
|
||||
# Then after a settling period:
|
||||
ssh wayne@192.168.2.245 'sudo qm destroy 101 --destroy-unreferenced-disks 1'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Stage 7 — Final cleanup
|
||||
|
||||
Once all hosts are at their new IPs and verified:
|
||||
|
||||
```bash
|
||||
# Delete the Pi-hole CT (already stopped since Stage 4)
|
||||
ssh wayne@pve1.sweet.home "sudo pct destroy 100"
|
||||
|
||||
# Remove stale FreeIPA records for retired addresses
|
||||
ipa dnsrecord-del sweet.home pihole --del-all
|
||||
ipa dnsrecord-del 2.168.192.in-addr.arpa 253 --ptr-rec pihole.sweet.home.
|
||||
|
||||
# Rebuild any NixOS hosts that reference pbsIp or pxeServerIp to pick up
|
||||
# the updated variables.nix values (pxe-boot mandatory; others as convenient)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Rollback summary
|
||||
|
||||
| What broke | How to roll back |
|
||||
|---|---|
|
||||
| FreeIPA DNS not resolving | Check `systemctl status named` on DC; restart if failed |
|
||||
| FreeIPA DNS unreachable | `pct start 100` on pve1 (restores Pi-hole) |
|
||||
| Router DHCP not handing out leases | Re-enable Pi-hole DHCP; disable router DHCP |
|
||||
| DC unreachable after IP change | Proxmox console on VM 108 → `nmcli connection up "System eth0"` with old IP |
|
||||
| Host unreachable after renumber | Proxmox console → revert IP; or `pct set <id> --net0 ...` old IP and reboot CT |
|
||||
| pve1 web UI gone after renumber | SSH to .245 and check `/etc/network/interfaces`; if wrong, fix and restart networking |
|
||||
@@ -130,6 +130,9 @@
|
||||
|
||||
proxmox-ha-server-1 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-1/host.nix; nameSuffix = "-1"; };
|
||||
proxmox-ha-server-2 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-2/host.nix; nameSuffix = "-2"; };
|
||||
|
||||
proxmox-ha-docker-1 = mkTarget { platform = "proxmox"; buildType = "ha-docker"; hostPath = ./hosts/ha-docker-1/host.nix; nameSuffix = "-1"; };
|
||||
proxmox-ha-docker-2 = mkTarget { platform = "proxmox"; buildType = "ha-docker"; hostPath = ./hosts/ha-docker-2/host.nix; nameSuffix = "-2"; };
|
||||
};
|
||||
|
||||
# Auto-install environments (migrated from the former nix-auto-installer
|
||||
@@ -200,7 +203,11 @@
|
||||
(modulesPath + "/installer/netboot/netboot-minimal.nix")
|
||||
];
|
||||
})
|
||||
{ networking.hostName = "nixos-minimal"; }
|
||||
{
|
||||
networking.hostName = "nixos-minimal";
|
||||
system.stateVersion = "26.05";
|
||||
boot.zfs.forceImportRoot = false;
|
||||
}
|
||||
];
|
||||
};
|
||||
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
{ vars, ... }:
|
||||
{
|
||||
networking = {
|
||||
hostName = vars.haDocker1Host;
|
||||
hostId = "a1d0c4e1";
|
||||
useDHCP = false;
|
||||
interfaces = {
|
||||
# ens18 — LAN management NIC (vmbr0, 192.168.2.0/24)
|
||||
${vars.vmLanInterface}.ipv4.addresses = [{
|
||||
address = vars.haDocker1Ip;
|
||||
prefixLength = vars.lanPrefixLength;
|
||||
}];
|
||||
# ens19 — storage-client NIC (vmbr2, 192.168.20.0/24) — NFS from HA cluster
|
||||
${vars.haDockerStorageInterface}.ipv4.addresses = [{
|
||||
address = vars.haDocker1StorageIp;
|
||||
prefixLength = vars.haClientPrefixLength;
|
||||
}];
|
||||
# ens20 — swarm cluster NIC (vmbr3, 192.168.30.0/24) — Docker gossip + VXLAN
|
||||
${vars.haDockerSwarmInterface}.ipv4.addresses = [{
|
||||
address = vars.haDocker1SwarmIp;
|
||||
prefixLength = vars.haDockerSwarmPrefixLength;
|
||||
}];
|
||||
};
|
||||
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
|
||||
nameservers = [ vars.domainControllerIp ];
|
||||
};
|
||||
|
||||
# Only register the LAN IP with IPA DNS. Without this, sssd dyndns
|
||||
# would also register Docker bridge IPs (172.x.x.x) and the storage/swarm
|
||||
# NIC IPs as A records for ha-docker-1.sweet.home.
|
||||
security.ipa.dyndns.interface = vars.vmLanInterface;
|
||||
|
||||
system.stateVersion = "26.05";
|
||||
}
|
||||
@@ -0,0 +1,32 @@
|
||||
{ vars, ... }:
|
||||
{
|
||||
networking = {
|
||||
hostName = vars.haDocker2Host;
|
||||
hostId = "a2d0c4e2";
|
||||
useDHCP = false;
|
||||
interfaces = {
|
||||
# ens18 — LAN management NIC (vmbr0, 192.168.2.0/24)
|
||||
${vars.vmLanInterface}.ipv4.addresses = [{
|
||||
address = vars.haDocker2Ip;
|
||||
prefixLength = vars.lanPrefixLength;
|
||||
}];
|
||||
# ens19 — storage-client NIC (vmbr2, 192.168.20.0/24) — NFS from HA cluster
|
||||
${vars.haDockerStorageInterface}.ipv4.addresses = [{
|
||||
address = vars.haDocker2StorageIp;
|
||||
prefixLength = vars.haClientPrefixLength;
|
||||
}];
|
||||
# ens20 — swarm cluster NIC (vmbr3, 192.168.30.0/24) — Docker gossip + VXLAN
|
||||
${vars.haDockerSwarmInterface}.ipv4.addresses = [{
|
||||
address = vars.haDocker2SwarmIp;
|
||||
prefixLength = vars.haDockerSwarmPrefixLength;
|
||||
}];
|
||||
};
|
||||
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
|
||||
nameservers = [ vars.domainControllerIp ];
|
||||
};
|
||||
|
||||
# Only register the LAN IP with IPA DNS — same reasoning as ha-docker-1.
|
||||
security.ipa.dyndns.interface = vars.vmLanInterface;
|
||||
|
||||
system.stateVersion = "26.05";
|
||||
}
|
||||
@@ -11,9 +11,7 @@
|
||||
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
|
||||
nameservers = [ vars.domainControllerIp ];
|
||||
};
|
||||
services.beszel.agent.environment = {
|
||||
# KEY = "";
|
||||
};
|
||||
services.beszel.agent.environment = { };
|
||||
# Preserved from the pre-refactor `pxe-boot` target — stateVersion must
|
||||
# never be bumped on an already-installed machine.
|
||||
system.stateVersion = "25.05";
|
||||
|
||||
@@ -0,0 +1,84 @@
|
||||
# Docker Swarm node build type.
|
||||
#
|
||||
# Produces NixOS hosts that form a Docker Swarm manager cluster. Two nodes
|
||||
# (ha-docker-1, ha-docker-2) are both managers so either can accept Docker
|
||||
# API and `docker stack` commands.
|
||||
#
|
||||
# Key differences from the existing `docker` build type (used by CT 105):
|
||||
# - nextcloud-cron-job.nix is EXCLUDED — `docker exec` breaks in swarm
|
||||
# because the target container may be on the other node. The cron job
|
||||
# is replaced by a nextcloud-cron sidecar in the Nextcloud stack.
|
||||
# See docs/internal/docker-swarm-cutover.md.
|
||||
# - traefik/rotate-logs.nix is EXCLUDED — log rotation moves to Docker's
|
||||
# json-file log driver (max-size/max-file on the Traefik service
|
||||
# definition). See docs/internal/docker-swarm-cutover.md.
|
||||
# - raspi/mount-data.nix is EXCLUDED — specific to CT 105's backup role.
|
||||
# - Swarm firewall ports (2377/tcp, 7946/tcp+udp, 4789/udp) are opened
|
||||
# on the swarm NIC (ens20/vmbr3) only.
|
||||
# - checkReversePath = "loose" is required for the Swarm ingress routing
|
||||
# mesh: VXLAN return traffic is asymmetric (arrives ens20, exits ens18).
|
||||
# - beszel-agent is enabled for host-level monitoring.
|
||||
{ pkgs, vars, ... }:
|
||||
|
||||
{
|
||||
# Pin Docker Engine to version 29, matching CT 105, so image layers cached
|
||||
# on NFS volumes remain compatible across old and new hosts.
|
||||
nixpkgs.overlays = [
|
||||
(final: prev: {
|
||||
docker = prev.docker_29;
|
||||
docker_cli = prev.docker_29;
|
||||
})
|
||||
];
|
||||
|
||||
imports = [
|
||||
../docker/enable-service.nix
|
||||
../docker/mount-data.nix
|
||||
../docker/docker-health-to-gotify.nix
|
||||
../beszel/enable-agent.nix
|
||||
../services/enable-rpcbind.nix
|
||||
];
|
||||
|
||||
environment.systemPackages = with pkgs; [
|
||||
nfs-utils
|
||||
];
|
||||
|
||||
boot.supportedFilesystems = [ "nfs" ];
|
||||
|
||||
systemd.tmpfiles.rules = [
|
||||
# Symlink ~/docker → NFS config mount so the docker-health-to-gotify
|
||||
# script (and operator convenience) resolves ~/docker/... correctly.
|
||||
"L+ /home/${vars.primaryUser}/docker - - - - ${vars.nfsShares.dockerConfig.mountpoint}"
|
||||
"d /mnt/docker 0755 ${vars.primaryUser} users -"
|
||||
];
|
||||
|
||||
users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
|
||||
|
||||
networking.firewall = {
|
||||
# LAN-facing service ports — same as the existing docker build type.
|
||||
allowedTCPPorts = [
|
||||
vars.ports.dockerHttp
|
||||
vars.ports.dockerHttps
|
||||
vars.ports.dockerExtra
|
||||
vars.ports.beszelHub
|
||||
];
|
||||
|
||||
# Swarm inter-node ports restricted to the swarm NIC (ens20/vmbr3).
|
||||
# vmbr3 is an isolated internal bridge — no LAN reachability.
|
||||
interfaces.${vars.haDockerSwarmInterface} = {
|
||||
allowedTCPPorts = [
|
||||
vars.ports.dockerSwarmMgmt # 2377 — Raft + cluster management
|
||||
vars.ports.dockerSwarmDisc # 7946 — Serf gossip (TCP half)
|
||||
];
|
||||
allowedUDPPorts = [
|
||||
vars.ports.dockerSwarmDisc # 7946 — Serf gossip (UDP half)
|
||||
vars.ports.dockerSwarmVxlan # 4789 — VXLAN overlay data path
|
||||
];
|
||||
};
|
||||
|
||||
# Docker Swarm ingress routing mesh creates asymmetric routes: a request
|
||||
# arrives on ens18 (LAN) for a container that lives on ens20's VXLAN
|
||||
# overlay; the return path differs from the incoming interface. Strict
|
||||
# rp_filter drops these packets. "loose" allows them.
|
||||
checkReversePath = "loose";
|
||||
};
|
||||
}
|
||||
@@ -6,6 +6,10 @@ let
|
||||
tftpRoot = "${pxeRoot}/tftp";
|
||||
pxeBaseUrl = "http://${vars.pxeServerIp}";
|
||||
|
||||
# Base network address extracted from lanCidr (e.g. "192.168.2.0" from
|
||||
# "192.168.2.0/24") — used by dnsmasq's proxy DHCP range directive.
|
||||
lanBaseAddr = lib.head (lib.splitString "/" vars.lanCidr);
|
||||
|
||||
bootIpxe = pkgs.writeText "boot.ipxe" ''
|
||||
#!ipxe
|
||||
|
||||
@@ -73,16 +77,16 @@ let
|
||||
boot
|
||||
'';
|
||||
|
||||
# Kickstart file for domain-controller.sweet.home.
|
||||
# Installs Rocky Linux 9, sets a static IP, creates wayne with the
|
||||
# admin SSH key, then on first reboot runs ipa-server-install via a
|
||||
# systemd oneshot service. Passwords are generated at %post time,
|
||||
# written to /root/ipa-credentials.txt (chmod 600), and read back by
|
||||
# the first-boot script — never hardcoded here or in the repo.
|
||||
# Kickstart file for ${vars.ipaServer}.
|
||||
# Installs Rocky Linux 9, sets a static IP, creates ${vars.ipaUser} with
|
||||
# the admin SSH key, then on first reboot runs ipa-server-install via a
|
||||
# systemd oneshot service. Passwords are generated at %post time, written
|
||||
# to /root/ipa-credentials.txt (chmod 600), and read back by the
|
||||
# first-boot script — never hardcoded here or in the repo.
|
||||
rockyFreeIpaKs = pkgs.writeText "rocky-freeipa.ks" ''
|
||||
#version=RHEL9
|
||||
# Unattended Rocky Linux 9 + FreeIPA install
|
||||
# Target: domain-controller.${vars.homeDomain} ${vars.domainControllerIp}
|
||||
# Target: ${vars.ipaServer} ${vars.domainControllerIp}
|
||||
|
||||
url --url=${rockyMirror}/BaseOS/${rockyArch}/os/
|
||||
repo --name=appstream --baseurl=${rockyMirror}/AppStream/${rockyArch}/os/
|
||||
@@ -93,14 +97,14 @@ let
|
||||
|
||||
# DHCP during install; static IP configured in %post via NM config file
|
||||
network --bootproto=dhcp --device=link --activate
|
||||
network --hostname=domain-controller.sweet.home
|
||||
network --hostname=${vars.ipaServer}
|
||||
|
||||
selinux --enforcing
|
||||
firewall --enabled --service=ssh
|
||||
|
||||
rootpw --lock
|
||||
user --name=wayne --groups=wheel --shell=/bin/bash
|
||||
sshkey --username=wayne "${vars.adminSshKey}"
|
||||
user --name=${vars.ipaUser} --groups=wheel --shell=/bin/bash
|
||||
sshkey --username=${vars.ipaUser} "${vars.adminSshKey}"
|
||||
|
||||
zerombr
|
||||
clearpart --all --initlabel --drives=sda
|
||||
@@ -148,7 +152,7 @@ let
|
||||
|
||||
# -- /etc/hosts: FQDN must resolve to the real IP (not loopback) for IPA --
|
||||
sed -i '/domain-controller/d' /etc/hosts
|
||||
echo '${vars.domainControllerIp} domain-controller.${vars.homeDomain} domain-controller' >> /etc/hosts
|
||||
echo '${vars.domainControllerIp} ${vars.ipaServer} domain-controller' >> /etc/hosts
|
||||
|
||||
# -- Generate IPA passwords and store securely --
|
||||
DM_PASS=$(openssl rand -base64 24 | tr -dc 'A-Za-z0-9' | head -c 24)
|
||||
@@ -168,13 +172,13 @@ let
|
||||
ADMIN_PASS=$(grep '^IPA Admin:' /root/ipa-credentials.txt | awk '{print $NF}')
|
||||
|
||||
ipa-server-install \
|
||||
--realm=SWEET.HOME \
|
||||
--domain=sweet.home \
|
||||
--hostname=domain-controller.sweet.home \
|
||||
--realm=${lib.strings.toUpper vars.homeDomain} \
|
||||
--domain=${vars.homeDomain} \
|
||||
--hostname=${vars.ipaServer} \
|
||||
--ds-password="$DM_PASS" \
|
||||
--admin-password="$ADMIN_PASS" \
|
||||
--setup-dns \
|
||||
--forwarder=192.168.2.253 \
|
||||
--forwarder=${vars.domainControllerIp} \
|
||||
--no-dnssec-validation \
|
||||
--no-ntp \
|
||||
--unattended
|
||||
@@ -341,9 +345,7 @@ in
|
||||
atftpd = {
|
||||
enable = true;
|
||||
root = tftpRoot;
|
||||
extraOptions = [
|
||||
"--verbose=5"
|
||||
];
|
||||
extraOptions = [ "--verbose=5" ];
|
||||
};
|
||||
|
||||
openssh.settings.PermitRootLogin = "yes";
|
||||
@@ -428,7 +430,7 @@ in
|
||||
# Without this dnsmasq tries to bind port 53 which systemd-resolved
|
||||
# already owns, causing startup failure.
|
||||
port = 0;
|
||||
dhcp-range = [ "192.168.2.0,proxy" ];
|
||||
dhcp-range = [ "${lanBaseAddr},proxy" ];
|
||||
dhcp-match = [
|
||||
"set:ipxe,175"
|
||||
"set:efi64,option:client-arch,7"
|
||||
@@ -445,5 +447,5 @@ in
|
||||
};
|
||||
|
||||
networking.firewall.allowedTCPPorts = [ vars.ports.pxeBootHttp ];
|
||||
networking.firewall.allowedUDPPorts = [ vars.ports.pxeBootTftp 67 ];
|
||||
networking.firewall.allowedUDPPorts = [ vars.ports.pxeBootTftp vars.ports.dhcp ];
|
||||
}
|
||||
|
||||
@@ -5,13 +5,13 @@ let
|
||||
sudo nixos-rebuild switch \
|
||||
--no-write-lock-file \
|
||||
--refresh \
|
||||
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
|
||||
--flake git+https://${vars.giteaDomain}/${vars.giteaRepoPath}.git?dir=${vars.giteaRepoFlakePath}#$(cat /etc/flake-target)
|
||||
'';
|
||||
testCmd = ''
|
||||
sudo nixos-rebuild test \
|
||||
--no-write-lock-file \
|
||||
--refresh \
|
||||
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
|
||||
--flake git+https://${vars.giteaDomain}/${vars.giteaRepoPath}.git?dir=${vars.giteaRepoFlakePath}#$(cat /etc/flake-target)
|
||||
'';
|
||||
buildImageFn = ''
|
||||
buildImage() {
|
||||
@@ -39,21 +39,18 @@ in
|
||||
};
|
||||
interactiveShellInit = buildImageFn;
|
||||
};
|
||||
|
||||
networking.networkmanager.enable = true;
|
||||
|
||||
# Recommended over the true default (bypasses ZFS's own import safeguards)
|
||||
# per the option's own docs; matches hosts/docker/host.nix and
|
||||
# modules/services/zfs/enable-service.nix, which already set this
|
||||
# explicitly. Harmless no-op on hosts that don't use ZFS at all.
|
||||
# modules/services/zfs/enable-service.nix. Harmless no-op on hosts without ZFS.
|
||||
boot.zfs.forceImportRoot = false;
|
||||
|
||||
# Set your time zone.
|
||||
time.timeZone = vars.timeZone;
|
||||
|
||||
# Enable QEMU agent
|
||||
services.qemuGuest.enable = true;
|
||||
|
||||
# Enable docker-compose
|
||||
environment.systemPackages = with pkgs; [
|
||||
vim
|
||||
btop
|
||||
@@ -63,11 +60,10 @@ in
|
||||
];
|
||||
|
||||
# Secrets shared by every host, decrypted at activation via each host's
|
||||
# existing SSH host key (sops-nix derives the age key from
|
||||
# /etc/ssh/ssh_host_ed25519_key automatically — see modules/common/README
|
||||
# or docs/ for the sops workflow). hashedPassword/hashedPasswordFile need
|
||||
# SSH host key (sops-nix derives the age key from
|
||||
# /etc/ssh/ssh_host_ed25519_key automatically). hashedPassword secrets need
|
||||
# neededForUsers so they're available before the normal secret-activation
|
||||
# step, since user creation happens very early in boot.
|
||||
# step — user creation happens very early in boot.
|
||||
sops = {
|
||||
defaultSopsFile = ../../secrets/common.yaml;
|
||||
|
||||
@@ -75,27 +71,27 @@ in
|
||||
"root-hashedPassword".neededForUsers = true;
|
||||
"nixos-hashedPassword".neededForUsers = true;
|
||||
"nix-github-token" = { };
|
||||
"nix-gitea-token" = { };
|
||||
};
|
||||
|
||||
# nix.conf doesn't support a *File-style option for access-tokens, so the
|
||||
# token is rendered into a runtime-only file (never touches the Nix store)
|
||||
# and pulled in via nix.conf's native !include directive.
|
||||
templates."nix-github-token.conf".content = ''
|
||||
access-tokens = github.com=${config.sops.placeholder."nix-github-token"}
|
||||
# nix.conf has no *File-style option for access-tokens, so tokens are
|
||||
# rendered into a runtime-only file (never touches the Nix store) and
|
||||
# pulled in via nix.conf's native !include directive.
|
||||
templates."nix-access-tokens.conf".content = ''
|
||||
access-tokens = github.com=${config.sops.placeholder."nix-github-token"} ${vars.giteaDomain}=${config.sops.placeholder."nix-gitea-token"}
|
||||
'';
|
||||
};
|
||||
|
||||
nix.extraOptions = ''
|
||||
!include ${config.sops.templates."nix-github-token.conf".path}
|
||||
!include ${config.sops.templates."nix-access-tokens.conf".path}
|
||||
'';
|
||||
|
||||
users = {
|
||||
# With mutableUsers = false, update-users-groups.pl enforces hashedPasswordFile
|
||||
# on every activation regardless of whether the account already exists in
|
||||
# /etc/shadow. The default (true) only applies hashedPasswordFile to newly-
|
||||
# created accounts — which means a freshly-built proxmox disk image (where
|
||||
# activation runs without a usable sops key, so both accounts land in shadow
|
||||
# with ‘!’) will never have its passwords fixed by subsequent boots.
|
||||
# mutableUsers = false makes update-users-groups.pl enforce hashedPasswordFile
|
||||
# on every activation, not just on newly-created accounts. Without this, a
|
||||
# freshly-built proxmox disk image (activation runs without a usable sops key,
|
||||
# so both accounts land in shadow with '!') will never have its passwords fixed
|
||||
# by subsequent boots.
|
||||
mutableUsers = false;
|
||||
|
||||
users.root = {
|
||||
@@ -104,39 +100,23 @@ in
|
||||
|
||||
users.${vars.primaryUser} = {
|
||||
isNormalUser = true;
|
||||
extraGroups = [ "wheel" ]; # Enable ‘sudo’ for the user.
|
||||
packages = with pkgs; [
|
||||
tree
|
||||
];
|
||||
extraGroups = [ "wheel" ];
|
||||
packages = with pkgs; [ tree ];
|
||||
hashedPasswordFile = config.sops.secrets."nixos-hashedPassword".path;
|
||||
openssh.authorizedKeys.keys = [
|
||||
vars.adminSshKey
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGygkCljN6uKpdJbHTOQtn8ZnH+wKXDLAwrDFbLrE/65 nixos@nixos"
|
||||
];
|
||||
openssh.authorizedKeys.keys = [ vars.adminSshKey ] ++ vars.extraAdminSshKeys;
|
||||
};
|
||||
};
|
||||
|
||||
|
||||
# Enable the OpenSSH daemon.
|
||||
services.openssh.enable = true;
|
||||
|
||||
#Enable flakes
|
||||
|
||||
nix.settings = {
|
||||
experimental-features = [ "nix-command" "flakes" ];
|
||||
auto-optimise-store = true;
|
||||
};
|
||||
|
||||
|
||||
programs.git = {
|
||||
enable = true;
|
||||
package = pkgs.git;
|
||||
config = {
|
||||
credential.helper = "store";
|
||||
};
|
||||
config.credential.helper = "store";
|
||||
};
|
||||
|
||||
|
||||
|
||||
}
|
||||
|
||||
@@ -42,12 +42,8 @@ let
|
||||
'';
|
||||
in
|
||||
{
|
||||
# Root SSH access — same key set as nixos user so all admin keys can reach root.
|
||||
users.users.root.openssh.authorizedKeys.keys = [
|
||||
vars.adminSshKey
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGygkCljN6uKpdJbHTOQtn8ZnH+wKXDLAwrDFbLrE/65 nixos@nixos"
|
||||
];
|
||||
# Root SSH access — same key set as the nixos user so all admin keys can reach root.
|
||||
users.users.root.openssh.authorizedKeys.keys = [ vars.adminSshKey ] ++ vars.extraAdminSshKeys;
|
||||
|
||||
# Passwordless sudo for wheel — operator SSHes as nixos and uses sudo for
|
||||
# cluster management commands (drbdadm, crm*, pcs, etc.)
|
||||
|
||||
@@ -1,10 +1,10 @@
|
||||
{ config, lib, vars, ... }:
|
||||
|
||||
let
|
||||
# FQDN of the LAN NFS VIP (Pacemaker vip-lan, 192.168.2.229). Using the
|
||||
# FQDN rather than a raw IP or bare hostname avoids systemd-resolved LLMNR
|
||||
# quirks and survives a future VIP renumber via a DNS-only update.
|
||||
nfsServer = "ha-vip-lan.${vars.homeDomain}";
|
||||
# FQDN of the LAN NFS VIP (Pacemaker vip-lan, 192.168.2.229). Defined in
|
||||
# variables.nix as haLanNfsFqdn; using the FQDN avoids systemd-resolved
|
||||
# LLMNR quirks and survives a future VIP renumber via a DNS-only update.
|
||||
nfsServer = vars.haLanNfsFqdn;
|
||||
in
|
||||
{
|
||||
fileSystems.${vars.nfsShares.pxebootImages.mountpoint} = {
|
||||
|
||||
@@ -3,12 +3,11 @@
|
||||
{
|
||||
# Run dnsmasq on the LAN interface as a forwarding-only resolver for
|
||||
# *.ts.net (Tailscale MagicDNS names). FreeIPA's bind-dyndb-ldap
|
||||
# cannot reach 100.100.100.100 (Tailscale's internal resolver) directly
|
||||
# because the DC is not a Tailscale node. This host IS a Tailscale node
|
||||
# and can reach 100.100.100.100 via its tailscale0 interface, so it
|
||||
# acts as an intermediary: FreeIPA has a conditional forward zone for
|
||||
# ts.net pointing here (vars.tailscaleRouterIp), and this dnsmasq
|
||||
# instance forwards those queries onward to Tailscale's resolver.
|
||||
# cannot reach vars.tailscaleResolverIp directly because the DC is not a
|
||||
# Tailscale node. This host IS a Tailscale node and can reach it via
|
||||
# tailscale0, so it acts as an intermediary: FreeIPA has a conditional
|
||||
# forward zone for ts.net pointing here (vars.tailscaleRouterIp), and this
|
||||
# dnsmasq instance forwards those queries onward to Tailscale's resolver.
|
||||
#
|
||||
# Configure FreeIPA once after deploying this host:
|
||||
# kinit admin
|
||||
@@ -29,32 +28,26 @@
|
||||
resolveLocalQueries = false;
|
||||
settings = {
|
||||
# Listen only on the LAN interface — not tailscale0 or loopback.
|
||||
# bind-interfaces prevents dnsmasq from binding to 0.0.0.0 and
|
||||
# then filtering by interface later; combined with `interface` this
|
||||
# ensures it genuinely listens only on eth0.
|
||||
# bind-interfaces prevents dnsmasq from binding to 0.0.0.0 and then
|
||||
# filtering by interface later; combined with `interface` this ensures
|
||||
# it genuinely listens only on eth0.
|
||||
bind-interfaces = true;
|
||||
interface = [ vars.lxcLanInterface ];
|
||||
|
||||
# Forward-only: no local /etc/hosts or /etc/resolv.conf reading,
|
||||
# no negative caching of NXDOMAIN for names this instance doesn't
|
||||
# serve. All ts.net queries come from FreeIPA's conditional forwarder
|
||||
# and must be answered by Tailscale's resolver.
|
||||
# Forward-only: no local /etc/hosts or /etc/resolv.conf reading, no
|
||||
# negative caching of NXDOMAIN for names this instance doesn't serve.
|
||||
# All ts.net queries come from FreeIPA's conditional forwarder and must
|
||||
# be answered by Tailscale's resolver.
|
||||
no-hosts = true;
|
||||
no-resolv = true;
|
||||
|
||||
# Tailscale's internal "Quad100" resolver — reachable from any
|
||||
# Tailscale node via the tailscale0 interface. Scoped to the
|
||||
# specific tailnet subdomain (vars.tailnetDomain) rather than
|
||||
# all of ts.net: FreeIPA refuses to shadow ts.net (a real public
|
||||
# TLD with DNSimple nameservers) so the conditional forward zone
|
||||
# in FreeIPA must use the tailnet-specific subdomain instead:
|
||||
# ipa dnsforwardzone-add ${vars.tailnetDomain} \
|
||||
# --forwarder=${vars.tailscaleRouterIp} \
|
||||
# --forward-policy=only
|
||||
server = [ "/${vars.tailnetDomain}/100.100.100.100" ];
|
||||
# Forward *.tailnetDomain to Tailscale's internal resolver, scoped to
|
||||
# the tailnet-specific subdomain rather than all of ts.net (FreeIPA
|
||||
# refuses to shadow ts.net, a real public TLD).
|
||||
server = [ "/${vars.tailnetDomain}/${vars.tailscaleResolverIp}" ];
|
||||
};
|
||||
};
|
||||
|
||||
networking.firewall.allowedUDPPorts = [ 53 ];
|
||||
networking.firewall.allowedTCPPorts = [ 53 ];
|
||||
networking.firewall.allowedUDPPorts = [ vars.ports.dns ];
|
||||
networking.firewall.allowedTCPPorts = [ vars.ports.dns ];
|
||||
}
|
||||
|
||||
@@ -37,6 +37,17 @@ source "${script_dir}/lib/nix-parallel.sh"
|
||||
repo_root="$(cd "${script_dir}/.." && pwd)"
|
||||
cd "$repo_root"
|
||||
|
||||
# When this repo is a subdirectory of a larger git repo (e.g. a mono-repo
|
||||
# subtree), `git diff --name-only` outputs paths relative to the outer git
|
||||
# root, not this directory. Compute a prefix to strip so pattern matching
|
||||
# below works correctly regardless of nesting depth.
|
||||
_git_root="$(git rev-parse --show-toplevel 2>/dev/null || echo "$repo_root")"
|
||||
if [[ "$repo_root" != "$_git_root" ]]; then
|
||||
_subtree_prefix="${repo_root#"$_git_root"/}/"
|
||||
else
|
||||
_subtree_prefix=""
|
||||
fi
|
||||
|
||||
full_check=false
|
||||
dry_run=false
|
||||
|
||||
@@ -120,7 +131,7 @@ if ! $full_check; then
|
||||
base_ref="$(resolve_base_ref)"
|
||||
echo
|
||||
echo "Changed-files scope: diffing against ${base_ref}"
|
||||
mapfile -t changed_files < <(git diff --name-only --diff-filter=ACMR "$base_ref" -- . | sort -u)
|
||||
mapfile -t changed_files < <(git diff --name-only --diff-filter=ACMR "$base_ref" -- . | sort -u | sed "s|^${_subtree_prefix}||")
|
||||
|
||||
if [[ ${#changed_files[@]} -eq 0 ]]; then
|
||||
echo "No changed files detected."
|
||||
|
||||
Executable
+595
@@ -0,0 +1,595 @@
|
||||
#!/usr/bin/env bash
|
||||
# deploy.sh — Full lifecycle management for the Docker Swarm HA cluster.
|
||||
#
|
||||
# Provisions two NixOS Proxmox VMs (ha-docker-1, ha-docker-2) as dual-manager
|
||||
# Docker Swarm nodes sharing NFS storage from the existing HA file-server
|
||||
# cluster. Both nodes are managers so either can accept Docker API and
|
||||
# `docker stack` commands.
|
||||
#
|
||||
# Usage:
|
||||
# scripts/docker-swarm/deploy.sh [options]
|
||||
# scripts/docker-swarm/deploy.sh --destroy [options]
|
||||
#
|
||||
# Phases (all run by default; skip any with --skip-<phase>):
|
||||
# 1. ensure-bridge Create vmbr3 (swarm cluster bridge) on the Proxmox node.
|
||||
# 2. sync-keys Generate SSH host keys for both nodes (clan vars).
|
||||
# 3. ipa-hosts Create IPA host objects + sops-encrypted keytabs.
|
||||
# 4. create-vms Build NixOS disk images and create VMs via create-proxmox-resource.sh.
|
||||
# 5. add-hardware Attach vmbr2 (storage) and vmbr3 (swarm) NICs; start VMs.
|
||||
# 6. boot-wait Wait for SSH on both LAN IPs.
|
||||
# 7. refresh-sops-keys Detect disko key drift; re-encrypt secrets; commit.
|
||||
# 8. init-swarm docker swarm init on node1; manager join on node2; label nodes.
|
||||
# 9. dns Register storage.home and swarm.home A records in FreeIPA.
|
||||
# 10. verify docker node ls; NFS mount check; swarm health.
|
||||
#
|
||||
# Options:
|
||||
# --node <host> Proxmox host (default: pve1.sweet.home)
|
||||
# --vmid1 <n> VMID for ha-docker-1 (default: 202)
|
||||
# --vmid2 <n> VMID for ha-docker-2 (default: 203)
|
||||
# --storage <pool> Proxmox storage pool (default: local-zfs)
|
||||
# --swarm-bridge <br> Bridge for Docker Swarm cluster network (default: vmbr3)
|
||||
# --storage-bridge <br> Bridge for NFS storage network (default: vmbr2)
|
||||
# --memory <MB> RAM per node (default: 4096)
|
||||
# --cores <n> vCPUs per node (default: 4)
|
||||
# --skip-ensure-bridge Skip vmbr3 creation/check
|
||||
# --skip-sync-keys Skip sync-host-keys.sh (clan vars already exist)
|
||||
# --skip-ipa-hosts Skip IPA host account creation (keytabs already exist)
|
||||
# --skip-create-vms Skip VM creation (VMs already exist)
|
||||
# --skip-add-hardware Skip NIC attachment (already attached)
|
||||
# --skip-boot-wait Skip boot/SSH wait (VMs already running)
|
||||
# --skip-refresh-sops-keys Skip sops host-key drift fix
|
||||
# --skip-init-swarm Skip swarm initialisation (already initialised)
|
||||
# --skip-dns Skip FreeIPA DNS record creation
|
||||
# --skip-verify Skip post-deploy health checks
|
||||
# --force-rebuild Pass --force-rebuild to create-proxmox-resource.sh
|
||||
# --destroy Stop and delete both VMs (skip all other phases)
|
||||
# --dry-run Print what would run without executing
|
||||
# -h|--help Show this message
|
||||
#
|
||||
# Prerequisites:
|
||||
# - SSH access to the Proxmox node as $PROXMOX_SSH_USER (wayne).
|
||||
# - sops age key in the standard location (used by sync-host-keys.sh).
|
||||
# - SSH access to domain-controller.sweet.home as $PROXMOX_SSH_USER for DNS phase.
|
||||
# - For --skip-sync-keys: clan vars already in vars/per-machine/proxmox-ha-docker-{1,2}/.
|
||||
# - For --skip-ipa-hosts: secrets/ha-docker-{1,2}.keytab already exist and are committed.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||
|
||||
# shellcheck source=../env.sh
|
||||
source "${REPO_ROOT}/scripts/env.sh"
|
||||
|
||||
# ── Defaults ──────────────────────────────────────────────────────────────────
|
||||
|
||||
NODE="${PVE1_HOST}" # deploy.sh targets pve1 by default (authorised for this cluster)
|
||||
VMID1=202
|
||||
VMID2=203
|
||||
STORAGE="${PROXMOX_STORAGE:-local-zfs}"
|
||||
SWARM_BRIDGE="vmbr3"
|
||||
STORAGE_BRIDGE="vmbr2"
|
||||
MEMORY_MB=4096
|
||||
CORES=4
|
||||
|
||||
SKIP_ENSURE_BRIDGE=false
|
||||
SKIP_SYNC_KEYS=false
|
||||
SKIP_IPA_HOSTS=false
|
||||
SKIP_CREATE_VMS=false
|
||||
SKIP_ADD_HARDWARE=false
|
||||
SKIP_BOOT_WAIT=false
|
||||
SKIP_REFRESH_SOPS_KEYS=false
|
||||
SKIP_INIT_SWARM=false
|
||||
SKIP_DNS=false
|
||||
SKIP_VERIFY=false
|
||||
FORCE_REBUILD=false
|
||||
DESTROY=false
|
||||
DRY_RUN=false
|
||||
|
||||
# ── Variables from repo (mirrors variables.nix) ───────────────────────────────
|
||||
|
||||
NODE1_HOST="ha-docker-1"
|
||||
NODE2_HOST="ha-docker-2"
|
||||
NODE1_LAN_IP="192.168.2.230"
|
||||
NODE2_LAN_IP="192.168.2.231"
|
||||
NODE1_SWARM_IP="192.168.30.230"
|
||||
NODE2_SWARM_IP="192.168.30.231"
|
||||
NODE1_STORAGE_IP="192.168.20.230"
|
||||
NODE2_STORAGE_IP="192.168.20.231"
|
||||
SWARM_CIDR="192.168.30.0/24"
|
||||
STORAGE_CIDR="192.168.20.0/24"
|
||||
STORAGE_ZONE="storage.home"
|
||||
SWARM_ZONE="swarm.home"
|
||||
SSH_USER="${PROXMOX_SSH_USER:-wayne}"
|
||||
DC_HOST="${IPA_SERVER:-domain-controller.sweet.home}"
|
||||
|
||||
# ── Argument parsing ──────────────────────────────────────────────────────────
|
||||
|
||||
usage() {
|
||||
sed -n '/^# Usage:/,/^[^#]/{ /^#/{ s/^# \?//; p } }' "$0"
|
||||
exit "${1:-0}"
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--node) NODE="$2"; shift 2 ;;
|
||||
--vmid1) VMID1="$2"; shift 2 ;;
|
||||
--vmid2) VMID2="$2"; shift 2 ;;
|
||||
--storage) STORAGE="$2"; shift 2 ;;
|
||||
--swarm-bridge) SWARM_BRIDGE="$2"; shift 2 ;;
|
||||
--storage-bridge) STORAGE_BRIDGE="$2"; shift 2 ;;
|
||||
--memory) MEMORY_MB="$2"; shift 2 ;;
|
||||
--cores) CORES="$2"; shift 2 ;;
|
||||
--skip-ensure-bridge) SKIP_ENSURE_BRIDGE=true; shift ;;
|
||||
--skip-sync-keys) SKIP_SYNC_KEYS=true; shift ;;
|
||||
--skip-ipa-hosts) SKIP_IPA_HOSTS=true; shift ;;
|
||||
--skip-create-vms) SKIP_CREATE_VMS=true; shift ;;
|
||||
--skip-add-hardware) SKIP_ADD_HARDWARE=true; shift ;;
|
||||
--skip-boot-wait) SKIP_BOOT_WAIT=true; shift ;;
|
||||
--skip-refresh-sops-keys) SKIP_REFRESH_SOPS_KEYS=true; shift ;;
|
||||
--skip-init-swarm) SKIP_INIT_SWARM=true; shift ;;
|
||||
--skip-dns) SKIP_DNS=true; shift ;;
|
||||
--skip-verify) SKIP_VERIFY=true; shift ;;
|
||||
--force-rebuild) FORCE_REBUILD=true; shift ;;
|
||||
--destroy) DESTROY=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage 0 ;;
|
||||
*) echo "Unknown option: $1" >&2; usage 1 ;;
|
||||
esac
|
||||
done
|
||||
|
||||
# ── Helpers ───────────────────────────────────────────────────────────────────
|
||||
|
||||
log() { echo "==> $*"; }
|
||||
logn() { echo " $*"; }
|
||||
err() { echo "ERROR: $*" >&2; exit 1; }
|
||||
|
||||
run() {
|
||||
if $DRY_RUN; then
|
||||
echo "[dry-run] $*"
|
||||
else
|
||||
"$@"
|
||||
fi
|
||||
}
|
||||
|
||||
pve() {
|
||||
if $DRY_RUN; then
|
||||
echo "[dry-run] ssh ${SSH_USER}@${NODE} sudo $*"
|
||||
else
|
||||
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
|
||||
fi
|
||||
}
|
||||
|
||||
pve_check() {
|
||||
# Read-only probe — always executes even in dry-run.
|
||||
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
|
||||
}
|
||||
|
||||
SWARM_USER="nixos"
|
||||
|
||||
n1() {
|
||||
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
|
||||
"${SWARM_USER}@${NODE1_LAN_IP}" "$@" 2>/dev/null
|
||||
}
|
||||
|
||||
n2() {
|
||||
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
|
||||
"${SWARM_USER}@${NODE2_LAN_IP}" "$@" 2>/dev/null
|
||||
}
|
||||
|
||||
dc() {
|
||||
# Run ipa commands on domain-controller as $SSH_USER.
|
||||
if $DRY_RUN; then
|
||||
echo "[dry-run] ssh ${SSH_USER}@${DC_HOST} $*"
|
||||
return 0
|
||||
fi
|
||||
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${DC_HOST}" "$@"
|
||||
}
|
||||
|
||||
wait_for_ssh() {
|
||||
local ip="$1" label="$2"
|
||||
if $DRY_RUN; then
|
||||
logn "[dry-run] Skipping SSH wait for ${label} (${ip})"
|
||||
return 0
|
||||
fi
|
||||
local deadline=$(( $(date +%s) + 300 ))
|
||||
log "Waiting for SSH on ${label} (${ip}) — up to 5 min..."
|
||||
while [[ $(date +%s) -lt $deadline ]]; do
|
||||
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=3 \
|
||||
-o BatchMode=yes "${SWARM_USER}@${ip}" true 2>/dev/null; then
|
||||
logn "${label} is up."
|
||||
return 0
|
||||
fi
|
||||
sleep 5
|
||||
done
|
||||
err "Timed out waiting for SSH on ${label} (${ip})"
|
||||
}
|
||||
|
||||
# ── Destroy mode ──────────────────────────────────────────────────────────────
|
||||
|
||||
if $DESTROY; then
|
||||
log "Destroying Docker Swarm VMs (${VMID1}=${NODE1_HOST}, ${VMID2}=${NODE2_HOST}) on ${NODE}"
|
||||
for vmid in "$VMID1" "$VMID2"; do
|
||||
STATUS=$(pve "qm status ${vmid} 2>/dev/null" 2>/dev/null || true)
|
||||
if echo "$STATUS" | grep -q "running"; then
|
||||
log "Stopping VMID ${vmid}..."
|
||||
pve "qm stop ${vmid} --skiplock 1"
|
||||
sleep 5
|
||||
fi
|
||||
if $DRY_RUN || pve "qm config ${vmid} >/dev/null 2>&1"; then
|
||||
log "Deleting VMID ${vmid}..."
|
||||
run pve "qm destroy ${vmid} --purge 1"
|
||||
else
|
||||
logn "VMID ${vmid} not found — already gone."
|
||||
fi
|
||||
done
|
||||
log "Done — swarm VMs destroyed."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Phase 1: Ensure swarm bridge ──────────────────────────────────────────────
|
||||
|
||||
if ! $SKIP_ENSURE_BRIDGE; then
|
||||
log "Phase 1: Ensuring swarm bridge ${SWARM_BRIDGE} on ${NODE}"
|
||||
if pve_check "test -d /sys/class/net/${SWARM_BRIDGE}" &>/dev/null; then
|
||||
logn "${SWARM_BRIDGE} already exists — skipping."
|
||||
else
|
||||
logn "Creating isolated internal bridge ${SWARM_BRIDGE} (no upstream port, ${SWARM_CIDR})"
|
||||
BRIDGE_CONF="auto ${SWARM_BRIDGE}
|
||||
iface ${SWARM_BRIDGE} inet manual
|
||||
bridge-ports none
|
||||
bridge-stp off
|
||||
bridge-fd 0"
|
||||
if $DRY_RUN; then
|
||||
echo "[dry-run] Would write /etc/network/interfaces.d/${SWARM_BRIDGE}.conf and ifup it"
|
||||
else
|
||||
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
|
||||
"echo '${BRIDGE_CONF}' | sudo tee /etc/network/interfaces.d/${SWARM_BRIDGE}.conf > /dev/null && sudo ifup ${SWARM_BRIDGE}"
|
||||
logn "${SWARM_BRIDGE} created and brought up."
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Phase 2: Sync host keys ───────────────────────────────────────────────────
|
||||
|
||||
if ! $SKIP_SYNC_KEYS; then
|
||||
log "Phase 2: Syncing SSH host keys for both swarm targets"
|
||||
for target in proxmox-ha-docker-1 proxmox-ha-docker-2; do
|
||||
CLAN_DIR="${REPO_ROOT}/vars/per-machine/${target}/openssh"
|
||||
if [[ -d "$CLAN_DIR" ]]; then
|
||||
logn "Clan vars for ${target} already exist — skipping."
|
||||
else
|
||||
logn "Generating host keys for ${target}..."
|
||||
run bash "${REPO_ROOT}/scripts/secrets/sync-host-keys.sh" "$target"
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Phase 3: IPA host accounts ────────────────────────────────────────────────
|
||||
|
||||
if ! $SKIP_IPA_HOSTS; then
|
||||
log "Phase 3: Creating IPA host accounts and keytabs"
|
||||
IPA_SCRIPT="${REPO_ROOT}/scripts/ipa/create-nixos-ipa-host-account.sh"
|
||||
for host in "${NODE1_HOST}" "${NODE2_HOST}"; do
|
||||
KEYTAB="${REPO_ROOT}/secrets/${host}.keytab"
|
||||
if [[ -f "$KEYTAB" ]]; then
|
||||
logn "Keytab for ${host} already exists — skipping."
|
||||
else
|
||||
logn "Creating IPA host account and keytab for ${host}..."
|
||||
run bash "$IPA_SCRIPT" "$host"
|
||||
fi
|
||||
done
|
||||
|
||||
if ! $DRY_RUN; then
|
||||
# Keytabs must be committed and pushed before VMs rebuild from Gitea.
|
||||
CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"
|
||||
logn "Committing keytabs and pushing to Gitea (branch: ${CURRENT_BRANCH})..."
|
||||
(cd "${REPO_ROOT}" && \
|
||||
git add secrets/ha-docker-1.keytab secrets/ha-docker-2.keytab .sops.yaml && \
|
||||
git commit -m "secrets(ha-docker): add IPA keytabs for ha-docker-1 and ha-docker-2" || true && \
|
||||
git push origin "${CURRENT_BRANCH}")
|
||||
logn "Pushed."
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Phase 3.5: Prepare Proxmox node for building ─────────────────────────────
|
||||
|
||||
if ! $SKIP_CREATE_VMS && ! $DRY_RUN; then
|
||||
CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"
|
||||
|
||||
local_ssh() { ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "$*"; }
|
||||
if local_ssh "test -d /nix" &>/dev/null && ! local_ssh "test -w /nix" &>/dev/null; then
|
||||
logn "/nix exists but not writable by ${SSH_USER} — fixing ownership with sudo..."
|
||||
local_ssh "sudo chown -R ${SSH_USER} /nix"
|
||||
logn "Done."
|
||||
fi
|
||||
unset -f local_ssh
|
||||
|
||||
# Use PROXMOX_REMOTE_REPO_DIR (from env.sh) so the build path is consistent
|
||||
# with what create-proxmox-resource.sh will use. The default is
|
||||
# /home/<user>/nixos (the standalone nixos repo clone on pve1), but can be
|
||||
# overridden to e.g. /home/<user>/infrastructure/nixos when the infrastructure
|
||||
# mono-repo is checked out on pve1 instead.
|
||||
REMOTE_REPO="${PROXMOX_REMOTE_REPO_DIR:-/home/${SSH_USER}/nixos}"
|
||||
if pve_check "test -d ${REMOTE_REPO}/.git" &>/dev/null; then
|
||||
REMOTE_BRANCH=$(ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
|
||||
"cd ${REMOTE_REPO} && git rev-parse --abbrev-ref HEAD 2>/dev/null")
|
||||
if [[ "$REMOTE_BRANCH" != "$CURRENT_BRANCH" ]]; then
|
||||
logn "Remote repo (${REMOTE_REPO}) is on '${REMOTE_BRANCH}', switching to '${CURRENT_BRANCH}'..."
|
||||
if ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
|
||||
"cd ${REMOTE_REPO} && git fetch origin && git checkout '${CURRENT_BRANCH}' && git pull --ff-only" 2>&1; then
|
||||
logn "Done."
|
||||
else
|
||||
logn "WARNING: branch switch failed — proceeding anyway (create-proxmox-resource.sh will retry)"
|
||||
fi
|
||||
fi
|
||||
else
|
||||
logn "Remote repo ${REMOTE_REPO} not found on ${NODE} — create-proxmox-resource.sh will clone it."
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Phase 4: Create VMs ───────────────────────────────────────────────────────
|
||||
|
||||
if ! $SKIP_CREATE_VMS; then
|
||||
log "Phase 4: Building and creating swarm VMs on ${NODE}"
|
||||
CREATE="${REPO_ROOT}/scripts/proxmox/create-proxmox-resource.sh"
|
||||
REBUILD_FLAG=""
|
||||
$FORCE_REBUILD && REBUILD_FLAG="--force-rebuild"
|
||||
|
||||
for spec in "${VMID1}:${NODE1_HOST}:proxmox-ha-docker-1" "${VMID2}:${NODE2_HOST}:proxmox-ha-docker-2"; do
|
||||
IFS=: read -r vmid host_name flake_target <<< "$spec"
|
||||
log "Creating ${flake_target} (VMID ${vmid}) on ${NODE}..."
|
||||
# --force-rebuild is always passed: create-proxmox-resource.sh only calls
|
||||
# sync_remote_host_keys (which bakes the clan-var SSH key into the disk
|
||||
# image) when it actually builds. Reusing a cached image skips that step
|
||||
# and leaves the VM unable to decrypt sops secrets on first boot.
|
||||
run bash "$CREATE" \
|
||||
--type vm \
|
||||
--host "$host_name" \
|
||||
--vmid "$vmid" \
|
||||
--node "$NODE" \
|
||||
--storage "$STORAGE" \
|
||||
--memory "$MEMORY_MB" \
|
||||
--cores "$CORES" \
|
||||
--force-rebuild
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Phase 5: Add NICs and start VMs ──────────────────────────────────────────
|
||||
|
||||
if ! $SKIP_ADD_HARDWARE; then
|
||||
log "Phase 5: Attaching storage (${STORAGE_BRIDGE}) and swarm (${SWARM_BRIDGE}) NICs"
|
||||
for vmid in "$VMID1" "$VMID2"; do
|
||||
logn "VMID ${vmid}: stopping to add NICs..."
|
||||
pve "qm stop ${vmid} --skiplock 1 2>/dev/null; sleep 3" || true
|
||||
|
||||
logn "Adding net1 (${STORAGE_BRIDGE} — NFS storage)..."
|
||||
pve "qm set ${vmid} --net1 virtio,bridge=${STORAGE_BRIDGE},firewall=0"
|
||||
|
||||
logn "Adding net2 (${SWARM_BRIDGE} — Docker Swarm)..."
|
||||
pve "qm set ${vmid} --net2 virtio,bridge=${SWARM_BRIDGE},firewall=0"
|
||||
|
||||
logn "Starting VMID ${vmid}..."
|
||||
pve "qm start ${vmid}"
|
||||
done
|
||||
fi
|
||||
|
||||
# ── Phase 6: Wait for SSH ─────────────────────────────────────────────────────
|
||||
|
||||
if ! $SKIP_BOOT_WAIT; then
|
||||
log "Phase 6: Waiting for both nodes to come up on LAN IPs"
|
||||
wait_for_ssh "$NODE1_LAN_IP" "$NODE1_HOST"
|
||||
wait_for_ssh "$NODE2_LAN_IP" "$NODE2_HOST"
|
||||
logn "Both nodes are SSHable."
|
||||
sleep 10 # let systemd finish activation
|
||||
fi
|
||||
|
||||
# ── Phase 7: Refresh sops host-key registrations ─────────────────────────────
|
||||
#
|
||||
# Disko builds raw disk images: each new VM boots with a freshly-generated SSH
|
||||
# host key, not the one pre-seeded in clan vars. Scan the running VMs; if
|
||||
# their ed25519 keys differ from the clan var, update the clan var, rewrite
|
||||
# the .sops.yaml anchor, and re-encrypt all affected sops files.
|
||||
|
||||
if ! $SKIP_REFRESH_SOPS_KEYS; then
|
||||
if $DRY_RUN; then
|
||||
logn "[dry-run] Would scan VM host keys and refresh .sops.yaml / secrets if needed"
|
||||
else
|
||||
log "Phase 7: Refreshing sops host-key registrations (disko key drift fix)"
|
||||
SOPS_UPDATED=false
|
||||
|
||||
for spec in \
|
||||
"${NODE1_LAN_IP}:proxmox-ha-docker-1:${NODE1_HOST}" \
|
||||
"${NODE2_LAN_IP}:proxmox-ha-docker-2:${NODE2_HOST}"; do
|
||||
IFS=: read -r node_ip flake_target host_name <<< "$spec"
|
||||
CLAN_PUB="${REPO_ROOT}/vars/per-machine/${flake_target}/openssh/ssh_host_ed25519_key.pub/value"
|
||||
|
||||
logn "Scanning ed25519 host key from ${host_name} (${node_ip})..."
|
||||
RAW=$(ssh-keyscan -t ed25519 "${node_ip}" 2>/dev/null | grep -v "^#") || true
|
||||
if [[ -z "$RAW" ]]; then
|
||||
logn "WARNING: no ed25519 key returned for ${node_ip} — skipping"
|
||||
continue
|
||||
fi
|
||||
SCANNED_TYPE=$(awk '{print $2}' <<< "$RAW")
|
||||
SCANNED_KEY=$(awk '{print $3}' <<< "$RAW")
|
||||
SCANNED_PUBKEY="${SCANNED_TYPE} ${SCANNED_KEY} ${host_name}"
|
||||
|
||||
CURRENT=$(tr -d '\n' < "$CLAN_PUB" 2>/dev/null || true)
|
||||
if [[ "$SCANNED_PUBKEY" == "$CURRENT" ]]; then
|
||||
logn "${host_name}: clan var matches running key — no update needed"
|
||||
continue
|
||||
fi
|
||||
|
||||
logn "${host_name}: key drift detected — updating clan var"
|
||||
logn " old: ${CURRENT}"
|
||||
logn " new: ${SCANNED_PUBKEY}"
|
||||
echo "$SCANNED_PUBKEY" > "$CLAN_PUB"
|
||||
SOPS_UPDATED=true
|
||||
|
||||
ANCHOR="${flake_target}"
|
||||
NEW_AGE=$(echo "$SCANNED_PUBKEY" | \
|
||||
nix run --quiet --no-warn-dirty nixpkgs#ssh-to-age 2>/dev/null)
|
||||
[[ -z "$NEW_AGE" ]] && err "ssh-to-age produced no output for ${host_name}"
|
||||
logn " new age key: ${NEW_AGE}"
|
||||
sed -i "/&${ANCHOR} /s| age[a-z0-9]*$| ${NEW_AGE}|" "${REPO_ROOT}/.sops.yaml"
|
||||
done
|
||||
|
||||
if $SOPS_UPDATED; then
|
||||
logn "Running sops updatekeys on affected secrets..."
|
||||
SOPS="nix run --quiet --no-warn-dirty nixpkgs#sops --"
|
||||
(cd "${REPO_ROOT}" && \
|
||||
$SOPS updatekeys -y secrets/common.yaml && \
|
||||
$SOPS updatekeys -y secrets/ha-docker-1.keytab && \
|
||||
$SOPS updatekeys -y secrets/ha-docker-2.keytab)
|
||||
|
||||
logn "Committing refreshed host keys and re-encrypted secrets..."
|
||||
(cd "${REPO_ROOT}" && \
|
||||
git add \
|
||||
vars/per-machine/proxmox-ha-docker-1/openssh/ssh_host_ed25519_key.pub/value \
|
||||
vars/per-machine/proxmox-ha-docker-2/openssh/ssh_host_ed25519_key.pub/value \
|
||||
.sops.yaml \
|
||||
secrets/common.yaml \
|
||||
secrets/ha-docker-1.keytab \
|
||||
secrets/ha-docker-2.keytab && \
|
||||
git commit -m "secrets(ha-docker): refresh sops host-key registrations for new VM instances" || true)
|
||||
logn "Sops keys refreshed and committed."
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Phase 8: Initialise Docker Swarm ─────────────────────────────────────────
|
||||
|
||||
if ! $SKIP_INIT_SWARM; then
|
||||
log "Phase 8: Initialising Docker Swarm"
|
||||
|
||||
if $DRY_RUN; then
|
||||
logn "[dry-run] Would run: docker swarm init --advertise-addr ${NODE1_SWARM_IP} --data-path-addr ${NODE1_SWARM_IP} on ${NODE1_HOST}"
|
||||
logn "[dry-run] Would join ${NODE2_HOST} as manager"
|
||||
logn "[dry-run] Would label both nodes"
|
||||
else
|
||||
# Check if node1 is already a swarm manager.
|
||||
if n1 "docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null | grep -qx "active"; then
|
||||
logn "${NODE1_HOST} is already in a swarm — skipping init."
|
||||
else
|
||||
logn "Initialising swarm on ${NODE1_HOST} (advertise: ${NODE1_SWARM_IP})..."
|
||||
n1 "docker swarm init \
|
||||
--advertise-addr ${NODE1_SWARM_IP} \
|
||||
--data-path-addr ${NODE1_SWARM_IP}"
|
||||
logn "Swarm initialised on ${NODE1_HOST}."
|
||||
fi
|
||||
|
||||
# Check if node2 is already joined.
|
||||
if n2 "docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null | grep -qx "active"; then
|
||||
logn "${NODE2_HOST} is already in the swarm — skipping join."
|
||||
else
|
||||
logn "Fetching manager join token from ${NODE1_HOST}..."
|
||||
JOIN_TOKEN=$(n1 "docker swarm join-token manager -q")
|
||||
[[ -z "$JOIN_TOKEN" ]] && err "Failed to get swarm manager join token from ${NODE1_HOST}"
|
||||
|
||||
logn "Joining ${NODE2_HOST} as manager (advertise: ${NODE2_SWARM_IP})..."
|
||||
n2 "docker swarm join \
|
||||
--token ${JOIN_TOKEN} \
|
||||
--advertise-addr ${NODE2_SWARM_IP} \
|
||||
--data-path-addr ${NODE2_SWARM_IP} \
|
||||
${NODE1_SWARM_IP}:2377"
|
||||
logn "${NODE2_HOST} joined as manager."
|
||||
fi
|
||||
|
||||
# Label nodes for service placement constraints.
|
||||
logn "Labelling swarm nodes..."
|
||||
n1 "docker node update --label-add node=${NODE1_HOST} ${NODE1_HOST}" || true
|
||||
n1 "docker node update --label-add node=${NODE2_HOST} ${NODE2_HOST}" || true
|
||||
logn "Labels applied."
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Phase 9: DNS registration ─────────────────────────────────────────────────
|
||||
|
||||
if ! $SKIP_DNS; then
|
||||
log "Phase 9: Registering DNS records in FreeIPA"
|
||||
|
||||
if $DRY_RUN; then
|
||||
logn "[dry-run] Would create/verify ${SWARM_ZONE} zone and add A records"
|
||||
else
|
||||
# Check for and create the swarm.home zone if absent.
|
||||
if ! dc "ipa dnszone-show ${SWARM_ZONE}" >/dev/null 2>&1; then
|
||||
logn "Creating ${SWARM_ZONE} DNS zone..."
|
||||
dc "ipa dnszone-add ${SWARM_ZONE} \
|
||||
--name-server=${DC_HOST}. \
|
||||
--admin-email=hostmaster@${SWARM_ZONE}"
|
||||
# Reverse zone for 192.168.30.x
|
||||
dc "ipa dnszone-add 30.168.192.in-addr.arpa \
|
||||
--name-server=${DC_HOST}. \
|
||||
--admin-email=hostmaster@${SWARM_ZONE}" 2>/dev/null || \
|
||||
logn " (reverse zone 30.168.192.in-addr.arpa already exists or skipped)"
|
||||
else
|
||||
logn "${SWARM_ZONE} zone already exists."
|
||||
fi
|
||||
|
||||
# storage.home A records (zone already exists from HA cluster setup).
|
||||
for spec in "${NODE1_HOST}:${NODE1_STORAGE_IP}" "${NODE2_HOST}:${NODE2_STORAGE_IP}"; do
|
||||
IFS=: read -r hostname ip <<< "$spec"
|
||||
logn "Adding ${hostname}.${STORAGE_ZONE} → ${ip}"
|
||||
dc "ipa dnsrecord-add ${STORAGE_ZONE} ${hostname} --a-rec=${ip} --a-create-reverse" 2>/dev/null || \
|
||||
logn " (record already exists or reverse zone missing — continuing)"
|
||||
done
|
||||
|
||||
# swarm.home A records.
|
||||
for spec in "${NODE1_HOST}:${NODE1_SWARM_IP}" "${NODE2_HOST}:${NODE2_SWARM_IP}"; do
|
||||
IFS=: read -r hostname ip <<< "$spec"
|
||||
logn "Adding ${hostname}.${SWARM_ZONE} → ${ip}"
|
||||
dc "ipa dnsrecord-add ${SWARM_ZONE} ${hostname} --a-rec=${ip} --a-create-reverse" 2>/dev/null || \
|
||||
logn " (record already exists — continuing)"
|
||||
done
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Phase 10: Verify ──────────────────────────────────────────────────────────
|
||||
|
||||
if ! $SKIP_VERIFY; then
|
||||
log "Phase 10: Verifying swarm health"
|
||||
|
||||
if $DRY_RUN; then
|
||||
logn "[dry-run] Would verify swarm node list and NFS mounts"
|
||||
else
|
||||
logn "Swarm node list:"
|
||||
n1 "docker node ls" || err "docker node ls failed on ${NODE1_HOST}"
|
||||
|
||||
logn "Checking swarm state on both nodes..."
|
||||
for spec in "${NODE1_LAN_IP}:${NODE1_HOST}" "${NODE2_LAN_IP}:${NODE2_HOST}"; do
|
||||
IFS=: read -r ip hostname <<< "$spec"
|
||||
STATE=$(ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
|
||||
"${SWARM_USER}@${ip}" "docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null)
|
||||
if [[ "$STATE" != "active" ]]; then
|
||||
err "${hostname} swarm state is '${STATE}', expected 'active'"
|
||||
fi
|
||||
logn " ${hostname}: swarm=${STATE} ✓"
|
||||
done
|
||||
|
||||
logn "Checking NFS mounts on both nodes..."
|
||||
for spec in "${NODE1_LAN_IP}:${NODE1_HOST}" "${NODE2_LAN_IP}:${NODE2_HOST}"; do
|
||||
IFS=: read -r ip hostname <<< "$spec"
|
||||
NFS_OK=$(ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
|
||||
"${SWARM_USER}@${ip}" "df -h /mnt/docker/config 2>/dev/null | grep -c nfs || echo 0" 2>/dev/null)
|
||||
if [[ "$NFS_OK" -ge 1 ]]; then
|
||||
logn " ${hostname}: /mnt/docker/config NFS mount ✓"
|
||||
else
|
||||
logn " WARNING: ${hostname}: /mnt/docker/config does not appear to be NFS-mounted"
|
||||
logn " (automount may still be pending — try: ssh nixos@${ip} 'ls /mnt/docker/config')"
|
||||
fi
|
||||
done
|
||||
|
||||
logn "Checking overlay network..."
|
||||
NETWORKS=$(n1 "docker network ls --filter driver=overlay --format '{{.Name}}'")
|
||||
if echo "$NETWORKS" | grep -q "ingress"; then
|
||||
logn " ingress overlay network present ✓"
|
||||
else
|
||||
logn " WARNING: ingress overlay network not found — swarm may not be fully initialised"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
log "Deploy complete. Both nodes are ready for 'docker stack deploy'."
|
||||
log "Connect to either manager:"
|
||||
log " ssh nixos@${NODE1_LAN_IP} (${NODE1_HOST})"
|
||||
log " ssh nixos@${NODE2_LAN_IP} (${NODE2_HOST})"
|
||||
+131
-139
@@ -2,234 +2,226 @@ root-hashedPassword: ENC[AES256_GCM,data:Kp0nOZI7vDoLhJHiOJBwJn0rQZ5yhnwapGnAcA+
|
||||
nixos-hashedPassword: ENC[AES256_GCM,data:pT7tVRN6X4a+DNUgB7fIUUE3CbnetkjxmoSL1PxSU+ktsFU+fB0mEvJjA1uujsGH5Rcztg7YM815+M0Z67ILmHaXbza5DtFacrqhi4/b277xly0SHRX4yOvBwQh6mJG1jn/0O/wvUUIYdw==,iv:bp2nfhC8nFbk6o5iWDAugvbzu7J/a1xayFnBEtkhNpE=,tag:HqWgkIpSrSM/K9OK2WO+VQ==,type:str]
|
||||
nix-github-token: ENC[AES256_GCM,data:k1vYz7SqVhzpWa6jTL6NUD8lKOCpHCgTm+HT4IcnbzbSTUZP/bJUYw==,iv:UqAULZnr/4+VcioUDfTwvOSuwM8K9JgGhiApvYQPyoc=,tag:1LKHXhWAO/AHPDIZFBb04A==,type:str]
|
||||
beszel-token: ENC[AES256_GCM,data:OWmSRkZjb11y0Y8GdobqiE9GFwzdHOvvxCbYx69qUghGYARN,iv:i/JhGH0O7ThxPkL0SLAjfN0Fq8prm7tybI5kF2NRNpw=,tag:dBcqxOSHTnD4xngpOog55Q==,type:str]
|
||||
nix-gitea-token: ENC[AES256_GCM,data:HRQ8ymx/D8pLcL/pYIhcSTv3tlBCKOU5nhXPp/W6rNI+t+yxfOoRPQ==,iv:0Av3lrxQew2bDFf67nX/UM+cD0/8CtbQ2fZSaKWHAzM=,tag:0K+atsB/YmYd/qCN3hMdBw==,type:str]
|
||||
sops:
|
||||
age:
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiU2IxZ056SUs3SVZiUTlQ
|
||||
NXZEYmg3VzJzeWJPRnZXcUtOMGQ1MElBU0dVCm5xSURZZ0sxTjVxQnhpVVNEU092
|
||||
bDBHbFAzZ3hzejc5NmM2U0FQTkRNSTQKLS0tIC9OOURPRlcvQXIxMVZXR1JjUWRX
|
||||
bE5ja3FoWXFIZ2ZLTklqWHhMRHBTRWcK2Z4sEcdyvx1rdBIGrbNw7bYvEPXdobvF
|
||||
h0aNihn817DvtVr8eeucj22iYCS40etElQNcO5i68t5wab4/d3rUbQ==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyR3lnUnZwUnlJTkFaeVdz
|
||||
T3Q2eW52cWI2eTlhSFpHNDRHTEVNVGZJYm1NCldaMmlHaCt1K3lWRHhEZExHK3lo
|
||||
azZFck1URVg5ejBaRVdCWjdFMTc4dGsKLS0tIG9mNkxsZXI1N1hCRWk1NTZKcUUr
|
||||
SlJoWGdWbXhqZEJHM3IzZmZSQ25QbzAKPzBIA/IJiZr5NpOhB6IPkUSDGQzPwpTU
|
||||
vgFLMze8OSEviaGXKLt/ZwTXHsr5As7V9yGvJKJHhS/hzuSLRjs8rQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrN0lNNGtYbE8rRTB0U2hL
|
||||
eUNOZUNFRGk1RzE3R1VjV0Q3ckszcDNHWGxJCkhFN25JbDVSaWJIZ09uNzczbkM1
|
||||
ZWZ1Y2wrTHZ6VzlnVWRoT2JOdDZHM0UKLS0tIHQ2ZE9BaFRCeDNqbldRQkpLclI4
|
||||
dXZqOXNaR2Q2QWJtRlZUa0hLaTh2YlkKeCh3+yIt0sjh2Ig8m/YQifhMDHnTpuZE
|
||||
zHBE74SUyWFmqLgtd1b8Rx5gn9vBcS/3SMv3yKWispg4DZ3bCOeNFA==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpYmZHNWVCL0ErNEF3OTlG
|
||||
TTJDT1hZMXJNNUdLZ3ByUDNndVpLWVVGUm1RCkZSRmJmd3NyeHc0V0tpZEdYakF1
|
||||
WXhyWUJ0V1hTYzhrSUo1ZUNzR2J3OVUKLS0tIEthYXVvNVRibDVSQ2N0NWt3TmNP
|
||||
cTlteEtRYmxnUmdIUlBTREZzQXBBSVEKxQq10KDseuoVPe0cLBbk1+weuq0y6di+
|
||||
wJjaMShqVQBUI+MFWiQokBhPt8gZS7cs33LkWf3BegNALLDd6HXoOg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age19m0m7vdfg86yqy8l5mmle5jdd0unrn3f55t232w8h5ey42cqw34sfpt32n
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5c2luYmpYcVVxRHROQlpk
|
||||
cTM4cm9uNEtVS21pK2pDRVRySmxYS3NwdGg0CnQvS3g4Q2tSdlpVeWZxbzArcTM1
|
||||
YWRMcDAxcWgySFRqRC9XSElsTVorVEEKLS0tIFd6YjdrY2ZXQVBaZ01wMTIvV2sv
|
||||
ZEk0TG0ybENlTlhpZnovMXh0OGd5V1kKEbjrmKP6Su16KRDQ4PMqi0tHRS6+PfuT
|
||||
0v5DEyi8EDtIYSMzzdF7Jk2C0h5XuRux45Z6E6JhKW5c788dGQHz/A==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArOE9oU210Vkk1Z1VKMS8x
|
||||
YTJGbFlSYmx6eWgvMXExZ243SndXWkZ0ekY4ClJVUTFSVkRERjY1UVZMZjlFeE1P
|
||||
Vm40WHY4UjMwZWxVa05lblBMUW9iNEEKLS0tIGVlbDVxakR4MlBEaFlrM0tNOWRX
|
||||
elIvMnp5NmhnYVBYOFA0aUdtOE5DbEUKy+soKNLlRe0SC8kcnwrpKqvSrTGE114/
|
||||
FaX2829gQWm0bYI0M4ixeTc5ME2O2Ct2tvYlzfZQPnAuub+jVx2k8g==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1eW5kd1FrSGJaSDkwTzdY
|
||||
SnJhR2hqRkNTTGtSTWsvYmZpSlVWV0xvYkRZCmtqR1FQOTRQR3dGYU8zNFpMdkUr
|
||||
SWN2KzhXaE1rYWZYNHJsOVVvRDY3R00KLS0tIG1aUldSWHlHeHkrcjF2TENvazR3
|
||||
YXJLUXV3VVNSWEdseFlZQUJPQUF6OFkKOPKH52muUha+MV2leWDiqpuTaIj0ZKzV
|
||||
keEdyqkIck6KKfIjphNKTW5ujiuKjGQbRliquBknfCRGMnRlMNfmkA==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAzR0R1VEc2eTRIOWNxaVpa
|
||||
RXVaNkxjaDVDTmxMVWJEU0dWNkxDeE56c2swCjdkZGVQM3V0SnhGMXFxSEFEeEpP
|
||||
OFNwTWQwYTY0S0trMy9FMjNjbWFxajAKLS0tIHhjUW1lZ1EycUh2Sm1yanhzS3dP
|
||||
WFlMcndzOThUVVVDZlJJeGFBT2JLTEUKHsJ6cwSPcO0IB1CQe2RqKeid8Q92BTNF
|
||||
RfURqE7Curj1yaFB45mzv2ThBgTKN5FE6y5BWgBnF6+szdMMXRw4uQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmTDExNFpCZHZmWnpwYis4
|
||||
bFJGZXV5UksyS01MT0d3cXE2Wno2Zzh6OFFnCm9hT3MyZ0hTbnkzdXk3NmN1TUVF
|
||||
OGtCZnlJRmZJaWJWaE1pMGcrTzJzZm8KLS0tIFE1L3ZxeUl6WVA2UXN2b0lIcTBl
|
||||
YmtXZnZaMml6ejY1Y3lRUFFtTEVMS0UKZbcuunA9GRqctCfC0Fh2foKt7n3l8KkE
|
||||
FNjjxUzL7OQjNG9M1aAtRMMO5KXL7YKXRUJWOPcpiLeLUNvDLeLXvA==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlQkVPTHBYWkdMM3Jyd3NW
|
||||
bkNGa3VtYXNBanZPc1puSGdwUEN5ejIwRkFJCjlNNXFlTEV1eEZNVTFxWnNJZG5O
|
||||
UG5hci9XVmMyeXF1ZHF2Rm5DUVdPcFEKLS0tIDBEZUtZNXR2R093OVNNYzRnbXlj
|
||||
S3R2UzFoTmhZT3Y0K1d4STFjU2RRamsKU9LcaOLLjmcarmdir9Hnt/qaNvlxvSsE
|
||||
RdXIdOKuaQqJyJ1VEpuDCfuZgtIdkr7OG1360giXFUIEUDliM9OPiA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQYlp5U0FvVkdIKzBGZVRl
|
||||
UEtOOG5kVHVLQ0ptVTVpdG9Lb1Y3bzlOODNzCmNYN0Y5ZGNzTGFGYmxJcWFxaExn
|
||||
NkkrT2RLSlZTdmFiNks3RVZXQlBEYnMKLS0tIDdCVlBxUG8vYU05d3c2MXhjQjhM
|
||||
TTVJUGxlWUxWalBRYzFKQUQzd1R6dHcKPz1ciNICph7qnqSH5WYmOjUCNHsIAmTX
|
||||
lRnbFCZdD/ZIFRK8OYAyCGfQ42UYSTf16kH2jDdTFUy7Fk2+O7lNIA==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWUFJETUk2TGZTdVJqUTcz
|
||||
MUZvUWhmWGR4SGVxOW8xcy9Pd1A2ZnhPcXpzCnhvL1lPTmZJclJ3OHVqbGtGZFMw
|
||||
VysxdnY5ZzgwUEFiVUxpYStUb3NsTEUKLS0tIHRpS0dpb1ZNMXBiRUhFVVZiSHQ2
|
||||
UTRoTXFjNVFOeE84WXdsT25UQUprbE0K5U8S5xojEgUn8pAgY6X+Njllv/jqm/Qp
|
||||
tqeTbMw+w/afAhRxY80x/mTJdCAxUh4guLTKO6eojHYcFvwT6eZA4w==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiemVqWW9LdlV0RjRuRHpF
|
||||
U2MwN3RDeHU5SWhuQ2l4YStRVVE2eU9XSUc4ClNGMkNXQVdkWFkwcys0aFVhZnZv
|
||||
Y3J6RmxwakdaZVo5VWZhb0lVTUdTYzQKLS0tIEQ2NW0yb2lGQ0NLZm1PSWNFVkQ1
|
||||
a0Y1NlYwZ0dnbHhsQVB5eS9kZVdqbmsKgFLUpZMqGywO9Ext5WRwco2bpenVNrzL
|
||||
XrGCqhn/7TW9DrJlPnFe8r4fl4DFP6GWByn7axDl58Qy1OzeBZwnNg==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpY1FUZ3RKU080RVNPZFZ6
|
||||
cDExYmhLdlU4K2xWU213MkZtUWY4elVxdlF3Ci9LK0cxV2hKbXZEMEd5dzVCWGVU
|
||||
SjErTmpYbDhmcDlSSENhNnRrN1QrcTgKLS0tIFRlbVlKOW51R3ZiZ2pLd0JXdks0
|
||||
b0FsZ3VETmREVEhqTGI4Vm5KQnZaWjAKXq+8u2Qk84Vt+eDUxzE6sDk4DDm78P7H
|
||||
KVnrZfhAmwP3X7dSuBhW+dK8in8D3jaqRK50d/eHUUWX0NIqniUMKg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLRzlzWEcwTUx6M1JNYWgr
|
||||
ZWxnVXhFWkhQVjBrSndGTU9oY2MrOTQxTGhFCjhUdERPc3hzQlkzMjBEa3AvaWVX
|
||||
T0RJWXFFZHdmUVBObGhpQktCQ2NPbWcKLS0tIDN2YkE4bEVtbjJlcndWVmlrQ3ZT
|
||||
dlVvRmt4RklOdUJyZ25ia0ZoYjU4ZHcK5fL1yvRcty6+EMzfSkwwJM3qvrr6h46d
|
||||
d7Zs+EElXrsGDQvL0fevB/zDcICAtudYnVv4nP3Jf8GQnL19zWlwbg==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZU2R4VVVkWTdqS1dxbDhv
|
||||
K21KeWs2NVo4UTBvNEF6SDRYdE9WUldYUmpRClloR1lBRU9UYkpRSzYxY0NsR2c0
|
||||
WHlXT0MycWJPdHhHaThWUHltdGZYQXcKLS0tIGJuaTEwSnpmMnFGVml4UUs1ZCtT
|
||||
U1V3U2NNN0l4L2JPRHFDOWpWWE9HaVkK/i5m6YFiAR6xtms/pbcDNhKaZreqIpjT
|
||||
8tvnqHz2HDSuCMAjAnZfluvuP1USHvjJQGZpBfreZ/XGhW0oqa7D7w==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhZElRNXNLK0tVUjQ0ektQ
|
||||
Tzc5NDJReldiYVFFZnRLS3l5dWw3UVQ0WWhJCm1DYVloZGpYeVJtNUNRcEJvVUZE
|
||||
Q0xMcmR0WWdVVTBaWko0dGI5V1hqaUkKLS0tIFBBeXdNY1pFai9QT2t3NWFLV2sw
|
||||
cWpRU2RISSs1UXh3Wkl1dWI3VEk1bXMKP6v3Lm0XZkp3Gn3iJkB0K0Y0E5BeilqR
|
||||
plmPh1rKbT/wkefFwpa8e2Wwmhx80YK+bjn633MH7mLQDf6AiK83Xw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMVk8yT1FtMVoxYjF2YVBs
|
||||
UTFVbmhzblBCRno2V3YvaERRN0xWcHd2NVE4CmVyK3p1SWtodk5iWjdkQjJRc0dM
|
||||
Vm8zZHczYlFFSG91QThmUGNaWHBoN0UKLS0tIDZMMXoxbUM0TUVaNFUzRExLc2VM
|
||||
OVFrNGZNL256cjg0Q3ZBQW9xUFcxSUEKv4HqaTcTzNYRJCmNEeUQLFfEoFXyOBEC
|
||||
yZrg8D+ROBn98484bxNeoyOjQdcVn7OYiCv1y6utnyeA9sdMUB0ong==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKNklENW8yZ3VlMytlVHJs
|
||||
dUhDcDI0amdZWlhmN3UzbFJyR2x4RGVOdFNVCmovVzFlYnh6YWxYTnYxVzBNejVT
|
||||
MGFhdjZzU2hnMW42d1RucENGeFdXa28KLS0tIGhIZy8rZ0pFM3o0cDBKTXQzaFNO
|
||||
bG9XNzFGdVNyczRhWjRqQXlxNHFvL1UK8NMj76782tmIdJJ4qIzLicFytNhj6ZMk
|
||||
HIIOJGCrBnqgCtcKTiCrTGRhGbqGzhkId1oJkZkhMFRoU7kdSvwD8Q==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1Rkl0UXduS3k3WVU1RzdG
|
||||
N2IybE1TZ1kwVmM4Y2lvUS9DVG0zSkdpZGhRCnFkZnJVS2FBZm9Xa0V4SjRwaGdR
|
||||
TWpaaTh4VEk4cUFrb2pURERzbWtSR0UKLS0tIEsybHdvUWg3UlUvL0lhdHIrUE0y
|
||||
R0pMdnI5QlpIcWVWdGRYQ3JCaElLWTgK5cTfvehj4BPU22tYiR5piZdouUdeh5Gr
|
||||
x/xSbFWqYVauDB6rpzZvhGMI4KCJh2RmDdt7FrlCb/YCB3QoSgvl8w==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAzbFU5YWREdFpsK3NScXNU
|
||||
NUFMMml2VEVOQU5pWkIrUjkzM2tZOG9vcTNRCmR6Vzc5OWRQaGdvU1Naam51dXhD
|
||||
ME1yNFJObktNNTA2eXFsNExTL3JNeHMKLS0tIE13L0xYMCt5TE5ac3FEbkxVWmVv
|
||||
V3NCSk9LUCtPZFZCbVNmSjR5QkFKTmMK5qFJXtZCKLjOCg1r+sVQpMKl75GNcrbI
|
||||
Dum/K/3HU03wv5reG51UDsQ1tMrsFDsaFh2fjR+LxLKSGlG7b3Rz4w==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwL0NNbGlGS3lTZjEydGJm
|
||||
MWZRK0Z5YktuNVNudk84MndLUUdXR0RlRFF3ClB4MWRUTFBWYUdwbityeUU2SlR4
|
||||
azZMb012YXQzWHUxZnZzeVp3MmdzcU0KLS0tIFJ2SUNyMTBEY1NWejdVQ3hyUytQ
|
||||
VnlQWHRDYWZFaU1ON2RQTXBwNFNGYUEKwnRUWCdsiStia1jLHJ6UNP/kgEt4Aw8U
|
||||
pA8PKKu6TZVPI8pw7svtI/uvJuW+8ARuNcyb1FdYmX+sOHnOEqHyjg==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEWFdCNGpPYU5pSXkzd3lG
|
||||
TGU1WkhuUEhmaDdybzFUbVVaVmVqU0FyQXhBCnRmRngrbXlsTVVMSDV6dkhBS1lD
|
||||
MmNUd0NjTE5ybFEweFhkWXdGbDNwTmcKLS0tIGU5Sk5hM3Y2bUVsS3pTaDZiaTBi
|
||||
S21vU1ZVbjZ1SHE0WStab1VHQUNLRUUKBDW9hwI90Yn+B2mB7LUNTVxFGbwEFSw3
|
||||
LGGqgZdvnN5p6NMigBbJz+kSwOk1gQ/yo86HtRj8Ejllp7P7jRsfmw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxM0FyaFlTR2FzMkhGTzZx
|
||||
d1d3dWNGOG1ESllETEVxRGJlYWkrTUNIdkYwCmI4andLMUUvNVU3R0N2WCsvWWhV
|
||||
UmlkcHdMSFA3bFp0ZG5OckE3YmJrejAKLS0tIDhEVEJYOFc5VWVHcGcvMm5yanBl
|
||||
eGxaNEFxdEN4NWh3cnlGeElBUTJmUzAKvCSJF+LGf1JqEdkzT/Wk6hEzferseyWO
|
||||
vAYR8Z+u6GDwQi7JdFz9iZKD21LAs0IMzd7cZqXKIXVedyXuaIhPtg==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaMWdPVkV4WnNHNmU1dTZZ
|
||||
ekVoVTJWY3laQ1ROM1NLUnlZY2FKTnJwNkFFCm8wVWtoNFpxMS91RkxzQWRhOWRs
|
||||
V3k2STdvN1JRdjIrSHVFaFpSYmlmT00KLS0tIExnSmw2anNtdzVGMjRYdzdpcFRq
|
||||
alFrUTJVckpTVEVIcUxTK3pPbWNpUzAKSe3Z9V5u8+om2s+HqUcx6qXIaTmQUgOT
|
||||
zIrU8T0tLOHzdS1SAdh1Yb50yevN+P2+5TORdLjrtigpyzeYc2U/zQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGZ2M0cG5nMU5nYVliRzlx
|
||||
TVpDcHpwclhXNkxuc2VKanBvcnRxRW04UXh3CmNJeEl3K0FrWWVHQWRwK3Q4Tzk2
|
||||
MEZQbFV1QURLZEhCdEhaMjBRUzllaVUKLS0tIFhSUTJ4SXJwRHhVbkZFc2Rwb3Fh
|
||||
QzAxTnpWSHZaWlF3NUlFMlBHUkRUa2sK3Kim0mp2adSvruI6Necd3UpZHow0EFmT
|
||||
JFOIpIK+3Pc06gsiqrDtDAx2Qtc10Ja4ILTutPlp4va6zzmzrPABiw==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtWDI1Z2hVRG9ieHkyZnlP
|
||||
SVQ3VHNsSlZxMjdIMUsxQWdIclNQQk5OUWhzCnZSMzFZUGkvYlNSRTJiT1ZMU0h3
|
||||
L0RIWkt4R1F5bUlWNVlQQklKc0tGa3cKLS0tIEJpTXNzZkFUOWp1ZnFocjJEVExa
|
||||
dGlqRDRLd1BDYzYwSWZXaG1ueXhrSzAKdJ8yA+igZKyetNuQdoN2Woi2bl2I3Xj7
|
||||
dDBVAn8bvxx5ZG1eyDXXcHSh76xUp5ZPRlLOFCvRK9oOLWd8FntBmw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwTUZ1RzJKcVF5bnJGamlU
|
||||
bWJvcU1sTTJuME94YjhMZWRWRWR0aGJ3UUNvCjh6U1BkaEg0TDlCeTNyUlNTRWV6
|
||||
Nnd0NjE3NmplSWFjVWpSRy9qWjBUbnMKLS0tIGdMZUFzbGdMejZoMlFBVUJXSE13
|
||||
TW4rUHFjZk5oVkw0UGtpaG5mOHRRbHcKFX4ZyNQ0TUeqwXEaz1qmAXQRIOctDkvz
|
||||
TIwxJseAjS3qVxCslnxzgaqmCh8DzCfjrRxLUS/HVTnwOaE1YAh0vg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTWStVRDhlblZDMDJ3ejIr
|
||||
MkwyeHRwcnpWQVBmOXY0a3NCU1JnTjkvVGtVClVvVUpLTVhUNGNNLzY4U0o2SDRD
|
||||
bGZteUU1Uzlzc0NjTW56OVNCcnVpUUEKLS0tIC8wN1V2Nkc0cVU3OHNYa1NBVlhq
|
||||
YTV5QUpsQWppNHRpT1dMb2JDQnpoR3MKCoBngnNg5qpktg5j4CcuTUKiooOVI8IN
|
||||
04plE+1lra8q5ZK8q4r1/K8hBM23poyfq+XushO1GZ17dPtWs9wITg==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkb3RrRDNqTGZGYlR1eTls
|
||||
Y2FYQWxncmJBQ1ZiNWFpMTBMUmtlSnJzemhVCk5NTWZRcTJMWkIrTnVsbVAzcndB
|
||||
MVp2NHJSazJjMVN5UXJHOWpnc0dLK1EKLS0tIGJlelUvbEZZZkFKd3BpdWx4VVNH
|
||||
cHJSVmJ4ZWlVTlA3VXpWeE5DNkphcWsKTGpdWU/cE8vC/43lwmnwJDh4IPqHQoVV
|
||||
yjnUdZnPDGyQHwwrydVgun2fdaAKH1zxAut8TZlH9pxD4ir0B69G+g==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUU2ZSazhZNWFZVXk0UXdx
|
||||
Mjh3R1NLMThlaHUvWVFMZEQ4Z0NYOW83dWgwCkVnd2dhdzQrZkxaL1NRbDJodERQ
|
||||
SkM5S0kxcVBtY3laNWp5OXpXZHhqZ0UKLS0tIHVOT2VUZU5DZjJjVW1kcU5qS3FD
|
||||
YXNhRkpaREMzRDB5aWgrdVNvUk1qcWsKQnPqE+5xSBGeDMmLfKB0tEBt2eylBOxT
|
||||
oi26mC+EoO5T2pCIpbD+DODAAlWKLattqCYHneUO9wRk9DnhuEXRBg==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaZ0tVT09Sd1hXVHZPd0tD
|
||||
YmZKaStnUXg3cERMTjRmOVh2RmxFbG1FZ0hJCjNMWWJmaUFqTURoL2NvRnJIdzNW
|
||||
OWFmSFAxWVRrdit6S2lpaFdERGNrOWcKLS0tIEJlVVFBSE5PK3k0dTRTRlZYMGZt
|
||||
elcxVlltUnJVd256NHQ4TithUmFnSkkKkbFSRRktuO+tNxlfIEuJ27C7LWalHJMb
|
||||
aCnjE9301XMceGB8SstPtGi5SJRSHD4WrJleeJbKXYMronfvWnYDSg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZUlFlenlOVDk1NmxQVGUx
|
||||
UHhVK3ZtMHdzaXdtWmE1TXhOekpsTmUwSlJ3CmdncjMxOUw0alBQZkhZZjA0YzJZ
|
||||
bFNVOHUySXRuYW8wVjdHMDN4ZThuQmsKLS0tIGdvUUdHcXlpWlZkL0xBcVpMN1E1
|
||||
NGZCQURYQVY5ZzdDODNNY0o3SG04eDAKiYVuKsJLaSC7/bQRB51tX282/yqpAjWT
|
||||
XNnqPQBQ7H3xuK0tDVdCAUMruOSlqAe9YRZwmiou1irpg9vZYwgTPA==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEa3E2ek9BcHZRRUR1cUpK
|
||||
UEFERzc1ZEtrYjc4bEdEeDRnSDNkWHV6elhFCjljdnJLa2syR0UzYTRveGhPYVk1
|
||||
cGNnWkNMNUVwbXFrOUZ1aE5UelFyMGsKLS0tIFNyQ29ieUpReW96OGhJaE4yU2Ni
|
||||
Y3VWNUFYTU5LdHlmd0krK0NyQk9mMVEKfh7I/9+V9+0DLkzTf6n4sBKs+oZlMlTx
|
||||
ar203b95cR/jjsUekF0NoZjj6MW1IZSV7BkaDoizVwrAxh7WS0vwqQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnUUlPS3ZPS1lSVmxwV0lq
|
||||
WWE2QXltb1BzU3p3cFRuMzVieGJta05nWno4Cnk1RG1CUXZjM3dNQzlJK1ZGaHpZ
|
||||
b3drMTB2L3hidkpoTXdMQ05vOUtTb28KLS0tIHhyUllpQnlzL093MCs4YWx4VlVn
|
||||
S3Q3QnpVNEx6ek54RGx5bmlqUk5hbVEKHVUvfk/xKtlOk2o73LodweIivSB4Oii/
|
||||
6dwnE8wa+Qx/OU0vDJVelKOUE6JiLwckX/0u2YIUOybkx49n+tYs/Q==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtSUcxeE5pM3NZVHlhcGNt
|
||||
d1EwODM3Wm5BOGNybkNMdnM0SnJmTG5ScEhNCk5yZHhieVN4QnNPMkxzbUZ3SS9G
|
||||
aG5WVENIbVBYRGI5ZzQ3QndUTG12NjgKLS0tIFF6ZFp5YWJ6UzlqMWRsb3pTS3l0
|
||||
ODRZZG9pRkNxL1Mxa1dwUG9vdkJZZTgKYrjkSkjNQCU2wMIuZMvJnd/regVplzMi
|
||||
3kPGRBMWzO1t4CSFtf2PMqv5AHw754I+vVNs5CpTXBuxVUcQDcGJCA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3MHlrTTdxYit0MTkxa293
|
||||
eFFIcmRLMUpkZjd3eWdsOXYySEljNDB2NUdNCnAyanVkS0dmenZSa2dpSWZTSkZw
|
||||
Tm5LMkRoWEk3L2ExVktnZDZMM09hZ0EKLS0tIGVGcHFuMjh4OTUxdHNJQSsrR1pq
|
||||
WU1jVG5UN2hMbW9iQ3JmbHlwc2VveFUK4JC32glesv/9nMWBR0lFeX7MPgIt9sIo
|
||||
9HsAkOMTxPTI0NZW8q79+G5hq8qG+QvQhwO6wQtwz/KEHPXntKDX+Q==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGNHY0Wm9oNk1BWFBiYTRm
|
||||
OXBxNU9ETDNyTEhyMnBxZFRnZmFjVGVHN0RjCkdPNWpvRkVMTCtWdDhlR3Q0U3Nw
|
||||
QUh6KzBVMU1ONXh2TTYrd0FaWlhCU28KLS0tIEhsaGxvcGFtaVF5T1NIc1NUOFBt
|
||||
d2NGeW9YNVVxdUpYbFRoVnhUM2VTUGsK4adI9pgC2PipcAY4zXMRf9hPv5kilTvc
|
||||
BsUNz0Qr4YdRfVfrPlPBzMCPOTDofTp6qBv+Gc8FE4hvwxtbjIcJ3A==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRbEZGNmtlUjJVZ2dlVGZC
|
||||
MGVEQXV1SFRpMjFqWks1Y2F6c29TY25oL1MwCjVFRk90Q2dJRXhUMmIrRVdHVFRD
|
||||
TWgvQS91VWdFWERJWEYrL3VnRXFraUkKLS0tIExUU01VRWlENlpBSGt3V1Izc1dS
|
||||
RFBhSTVKTGV2KzFidDQ3b3lmUHZ3d0UKHWvWO/Om6ahwBkONreGeENqdFG9UI6JB
|
||||
fRA3gRHy98EorEiia1him6lGY+ewYkTtVNo6iEt0abncW5x5xMQywQ==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBT25YN3BtMVZrVnhDazhO
|
||||
OXpzOEd0Wk5KUnZ4a2JjRGI3U0RsMExXN1FZClM3UFoyeUIzY3FVNnlkTU1NSCtn
|
||||
dXhNb3c4ZFlMTU5VTFc3blBUV0c5MkkKLS0tIHUzUzliL1NEUSt3cWx2d0tjRHcv
|
||||
a1Nmbm1MZEptQjJuZ2ROWGUzN01zbDQKy3dPKE5oYwsTwE2vnhUi3auqJ/KBOBPX
|
||||
vSKOROUuS/uEfsdo8NoVxuZ2RaJHetrbHAYPHFHOMgpe/X72pCcQNQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dFFpeDU2Y2dtQU12cytZ
|
||||
WUJjKzZ5S2RiZmtvYVJScStYVHI5d2FvcmpzCm9COGZmRllyaElqb2NsczhKRjhO
|
||||
ZGhwZCtoaFcrWmNZbGVHMUNLMFQ3Ym8KLS0tIHJQR1JOblZlY0cydnhNUnBCT1Fh
|
||||
aHJiZVpMU3Y3NXd0ajc0MGMxSGp4MmcKky1oaS8yFydszcvZE2pAL90ZUKoN9MnJ
|
||||
1TdqatDSMtmo3PWkdFeHbWxPjTizQGIcKEcsv6GmO/5KCAJDCHDNMg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrd3QwWE4yaVlIakdiYkNX
|
||||
bFVmR01yNjdWa0d6V29tcWh4M0hWRmQwR3pBCkFRV2gzSVdPQmQ5dXZYc0F1dW1r
|
||||
OFZNTkRwSzRFR0tlUXE2UzFxeU1Ld0kKLS0tIEZ6RlZkMFFiZytPZ05DSEI1Ni9X
|
||||
WEtpUlMycVhkMERsWDRGQ2lrNHdQcDAKRqVAOzQY4ZM1uH7LyZSqX1T6+1PwNhBY
|
||||
5M0RxSuI43E8YKC9axcppvKFJMQI1GAGvgfQxg2J0CEpnmPSVsO5DA==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2Q24xSkJaVDVUZlBWcW11
|
||||
cWFOdjhXZEtuNkJtaUhQTEN3QkVUQnRRRjBZCnY4V1lCOTBNM3Mrc09naWNTQndn
|
||||
VXU2ZWt5Zi9CbjBFa3YrejFHM041VFEKLS0tIHd5L0hRN00wdmdsSUhkdFpTcUNS
|
||||
YUNiUGhVR2M2clFrOUkxNW5idkR2cnMKXO02RJh4ew22tf5GWH4lNdLlhf4bWyef
|
||||
+hW9R4TGCJOnIO1xSGpBX0wJsM3oiW4qy/1tgfQw2ejVih5qUr7JRg==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvb0xNMloyRW5tTUUvZGph
|
||||
YWxIc0ZhSWI5RHNmaEIvdjZJNHFDM2xBMkNzCldlM2RUeFZCNE9tMVp0MFlWM2sr
|
||||
Sm44MjkrWlNkY0U0eUlyNlE2ODl5SFkKLS0tIEdISElQa2VBZlduajhsV0ZaU3dZ
|
||||
ci9COWE4bldUMS9lSVB1eEk4TG1sVjQKYu9zby36DP+41dTV8sRdtDil3Az40pf0
|
||||
vIhqhCTxQymnJHEwEukiEzM/w3Z85R/W/MbOKyzEZmxzG0a5O5svhg==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1SnhhMEpwOS9vUWVuT0Iz
|
||||
bWFCUzE4djFyUzN0alRvOG94bGJldUhlb1ZnCkxHd1pUU3B4b0ZhRklnczUyRzBR
|
||||
djF1bS9scjBlQXJEbmtiUFgzTndKQVEKLS0tIDRpUnhWYTM2U3lLcWZyVFkwcVNL
|
||||
NFpldXQraURraWZNdUNqZFNwVFY0WlEKhV8IZYbBXKGb0x+2E5pJkoMavH2ox4qp
|
||||
dK/XRlBpEG0SCUkKZ5sDegzz/HyqbxRF26jC3IOL1uR8A5nsnwC2Cw==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZlFtOHI0NnlMeWdjZEsr
|
||||
UnQ0UFJXY1pHTm4yLyszY0dBaldtUkIxdzJvCkNQbk9mN0wzK3BtN2pEa3JSWWFk
|
||||
RXNoemFFbDZ1SlUxOEZMUm9mY1RaZW8KLS0tIEk3bCtyaUVGM3MwNUwxS3QyTjA4
|
||||
YVpTTFlOdzljbmI2cnZUNG1nd1NrVFkKxARm3bpBL685Wky4Y06eA18VvUZRkAl3
|
||||
BBYPJMn3lfDgiQPvfXINfhJI6O+bWyjt0WnfodCGFM6EJLARXiTaqA==
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEWmIvVG1GWEZmbnpFU252
|
||||
bFhOajA2dnJXNlFGMmF0TDRuR1YrYmZqc0hnCk85WGFUMGZXdElWU2tvdFV3eE1v
|
||||
czEyTHBkZXFnSi9MVWFQVHJabEpUb1UKLS0tIC9TNGZiQ0xYR1dQWHl6NTAxbzlD
|
||||
YmNhQkFVZTZidWFtbURqYnY2eGMxZ2sKaUPhe5mQ4QSyxuQMqLNI1jqfkCUBnBWc
|
||||
HR5ck3H80/H0pMwzP5uaBnN/hD5L4CT2FcZBgUygbyNz4G3VgbjaMA==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
|
||||
lastmodified: "2026-07-29T11:33:40Z"
|
||||
mac: ENC[AES256_GCM,data:hXns2uTM8+bX9J1EWxKcA7v1oYayBMfyB57pMgayEQHV8PvdIWi7rYez+4OCuMBZzBNVAZWn2AfzYxC2VJJnKrajrvwGZ/neJ1qvd2sfY9e9TskZv+c6nqnRuXl0OGi+kP+6EVSmZPiSFUODZd+xtAku8dIMGO8LZeZoGYjCBfs=,iv:EFZZpw/QKiuEtShBhB2bP6qq69T6jlrJk81FpqSt5m8=,tag:VCqctIBaFK2DQqoj1sxv7g==,type:str]
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlU3BybGcyckNaT2dQR0ZO
|
||||
dDVXa2NtNGFTS1RqZ0ViLzVJSk5PbWdYU3lVCkRZNTJ4MytYYVpON0JBQTh4SWRY
|
||||
ay9xdkViektHNGRuSFNSeVoyMFdrNE0KLS0tIGJ1QW5RUjhWR2NlclR2dnh4SUQ5
|
||||
SlJEWHhBTFI5Rm1rZVFjTGZlUWhZMGcKReNE3l+U35iutlQ07AZ+3fOFF4YdVbdx
|
||||
bR/Sz3NqpqmZqBEmgjUjjjQI4h4Xturv4tT8/JUzEmVnEyDsqCbhHQ==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age16y0vfts5v2k20e2rj23qa5lc5gm96gm64uwsqsr0y688xl0ne46qcfr0sm
|
||||
- enc: |
|
||||
-----BEGIN AGE ENCRYPTED FILE-----
|
||||
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTMjhleFpqRGRBMGIrVmU0
|
||||
aXcrclh1TmtIV1JZSlNWRXRRc2RCYThyUVMwCmZOdXptWnhvajYxeERlOEJFVTU4
|
||||
WTY0R2NZSXplMDNNSzZ6eCtEeTN1NjgKLS0tIFIvYmY3VHM4T1RTZzljR2RnV0ND
|
||||
Z0ZhY2JpL0psb0RJcEJNN3FHUHJVQlEKJdoXKdvm6fGA+C21lsxY4Zq+VVPt/6k4
|
||||
vH0IirsPd9Cg33+tBzrySwkF+GVNtYLtekup2L61pFQ6/Y/necfQ2A==
|
||||
-----END AGE ENCRYPTED FILE-----
|
||||
recipient: age1tm3zj5lp3elw2j832az4nj9xxmhqd66ag3cqcv3vskvmd7qdmqmsdpdcn0
|
||||
lastmodified: "2026-07-30T07:49:07Z"
|
||||
mac: ENC[AES256_GCM,data:D5Y+zyLvs3gz8BWS3+lrra/tc1TsDIk7FykxSPiz0Er1jUB5LOe4IKircytMpwGqdLcVhUKgXYdRA+C0KAXaI9KpQp5qKfRDvX2jX24jveVNFPgbvxA2LidYasVMzA7ayTU0I+oirPcEtj+5VH/ahSjDz9YtarTDeHMpXvBZMLU=,iv:KcIGEs+61yS8Ul734fqsrC95iA82wIinf72DE2bLk/A=,tag:RzL48FqVcMro2W6VAgIxzQ==,type:str]
|
||||
unencrypted_suffix: _unencrypted
|
||||
version: 3.13.3
|
||||
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"data": "ENC[AES256_GCM,data:81JCCVaOeEYNyqTT3vXkFDDV1oSAlOrElGmvN+1Jy+U+dF6EaCSeFYT0U2i2BvUu9VAYEiY6NRAXwWUgZXeQXsI65eziB6d/8NKTr7UWb0eQxvhHbthgzxoCfdCvofZH49DdGuCEl8kU6hppSeM+wnDIIYvEgWljO4JQJr4/qnQRegMqJ33564+ZvhiYqrbfojcBYpTjVBzh+4aitqZEAYoFx4xhFAtdBfmAA1W6FwDvZLrX09bbdFT5jOqnSSkPunvTHiyWtyAfLgaNzG656F1U+3eRDv4eZQYOaLXXo9H4BaBMC5jyJFV1rDgl5s3diCuQggSyDbWUnwSS3Prcixj3dkrF8h8heNk/LDV5lGBtGEV3BQsEAlCPl01PIe3O5McfsRaR8IqwD7wQQ8ULyA56fHHAYJR1xe9bNGgdLryhqLDBFWi06Ztidi1MQXFLnNFJxoIhvANIAmt//TBWimkHnXidkg==,iv:WNChOcSG+QiZOMUi8jpb2gmOCLonM3zK7vcZCrXt4Yg=,tag:zq6Zvf3xu4kofaCechUpcg==,type:str]",
|
||||
"sops": {
|
||||
"age": [
|
||||
{
|
||||
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxMzVHK0tOazNoazU0RWZF\nT1h2N01NNktHRjVMQ0pEOWR4RDIxNVlQTWlZCkV2a2pQS0o0VFZWN1lyaklLWllp\nZXJvZEt6Zng1Q09xQTFkNnVtTDNNVDQKLS0tIFhnNW1sY09ZcWpVcGdmM1hrK0o3\nZ3VscjBXMVFEa0xtbVlZM0lMN1dHS2sKcYtCKhw6D1ax3Isf5Vk93cDteUEjx79j\n1fheqgOjytY3W9t8U2NOEpUP1hT7zWYswnaq+agm8nfEjf1fbwR9MQ==\n-----END AGE ENCRYPTED FILE-----\n",
|
||||
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
|
||||
},
|
||||
{
|
||||
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA4K3VSL1NrSm9rMm5WNTk1\nVEpudHhJUkJUeWxieEtKSUdCL2c1YkNHWVdNCk9EeGNoWHdLOVNUR2dRY2YrMVlJ\nblliN08yQW1xdXN5VTRjbm9vZUxZaDgKLS0tICtLaldjemJHSkZzY2R1MHlySEsy\ncXJ6UGdZMi9hTnNjRHRTVW5UMlNMQkkKWhLTte9gTpppGCmA0lf/FEYu5b/ZaGYT\njn9j9qj3Lets0gNj9qBVaHUkCoJ2TAGORdkhnLEdY/Tm/J1w3XfjNQ==\n-----END AGE ENCRYPTED FILE-----\n",
|
||||
"recipient": "age16y0vfts5v2k20e2rj23qa5lc5gm96gm64uwsqsr0y688xl0ne46qcfr0sm"
|
||||
}
|
||||
],
|
||||
"lastmodified": "2026-07-30T09:34:38Z",
|
||||
"mac": "ENC[AES256_GCM,data:oC5OPZyZ4kEh0A8Mmwoi5oZ3+fBNvEDN3A+o3n099RdPREn6mcH5QC+XD9BXvWVxp9BzQEwRBmExlyZBEoRuwrB6DQy5Fn8TneEjQoCvZrudpbXIB66/EvORk1utpUsbAov3wn9A8tXUDAUJaRsY93/Pe1mTc8KzIOGv/QTMR9w=,iv:+p87aL7dKsbyAfCWxc0pecVB36yUUa+ltfzQZc4oovk=,tag:tNRRC056bqMToaHz1UrqQQ==,type:str]",
|
||||
"version": "3.13.3"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"data": "ENC[AES256_GCM,data:6Is5g/gqFdQ9aTE6dpe80g7kSgDbhRr3yIwffv6tT5CLqsQLA6i+V5XPAqqhH0zbpXnT7k5Rc2tA2880MDa0eVoI+x7qDnhadfLKVgErWZNwQ4Nz9ukovf+lDfoKXn9AWNFxbiBckpCoxNOms3zMv6kqf1gbEwV7eRIoiq/i2xypke4nOJSmRH22dqFUldfQOOUL1D0H6RFQpNK+5O/Okb7URxIzdUgmq6/9zIsZTOlX5j1V7kzpx8U4P+YPyoPoRSLTOxpK/5gh+Gx1BQbXVTf8z5lm42tJxuQAt5kXqu0Tu8aXZzUUcMKf1oGVbo4TwRaQN+9tFgkhegVMSS/l2don3X3vWPimBcYHTfXfvvCh3krH3hoSmej8es2jCG8AesObDEmCyMNu9hdqhGGdCWspakrXVEuefCJ2Tq8fJfOn5XQhh4AX1wAf04Us0FJO4NjByNrgW8Bo1344wocAStxBxuD3rw==,iv:kkk5muigMC6iTIiTHKwXFreCuJGl7CWmwp1W7ILmq7U=,tag:W49PPiOISdFjk54wroD8ZQ==,type:str]",
|
||||
"sops": {
|
||||
"age": [
|
||||
{
|
||||
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQN0NWRllWaEtPdTdlOHFz\nUXJBbG1IRXlPbnNzMEJhT0FuazJLYVZHWUdFCnJXdUtvV2RvUU05eDBaaVdxSlBw\nUVh1azM0T2dFZlpMMFhsU2tWbm4vSVUKLS0tIGNlUXMrbUlDYWpiRTQzV2svcXJH\nd1RIcmpGbkYvSlZwYkwxanBZajVTMUEKt4HCEEPjsSDKvp6XuSSrQjVXFQLQHwk0\n1Fi/HrkhhdkIO1f4DyTOOznvK5bc+Z4JKT5lrOkKQln92uAho2ECiw==\n-----END AGE ENCRYPTED FILE-----\n",
|
||||
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
|
||||
},
|
||||
{
|
||||
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBock0vOGVLMC9HZ0NYckxr\nZWRwK0NhbkVaV1FVVndDNmRoNWhGVXlUOEdjCk50NFVtdmdxbEFuZE92YU1IUDlN\nbVJ6QnN1TEk5c1VmWDQwNXJReFdCSXMKLS0tIEl6TW1iV1NvTk5OeUkyOTRtT0ZZ\nR3NsV1lwa00rdHdTRUkwbzdrc1VaVkkK2ioVnzacNrQD6cpNOomKz9WfRRq+B3oK\nnabokq5rEOoNsSFNip5TBeDjo/34kOTXZCKXpFYwmBSoHCrkdFHvYg==\n-----END AGE ENCRYPTED FILE-----\n",
|
||||
"recipient": "age1tm3zj5lp3elw2j832az4nj9xxmhqd66ag3cqcv3vskvmd7qdmqmsdpdcn0"
|
||||
}
|
||||
],
|
||||
"lastmodified": "2026-07-30T09:34:40Z",
|
||||
"mac": "ENC[AES256_GCM,data:QOvDbiPiQBPYBGg3BzJNZzLBv//UccDfOnoOVqnpdskOpPw88uQOcpK0mcF/unW8o+B0AJe4wzr8/XhlhtlLSRi6buES2uZr8pjgCqrgFMCUX8WP+4rxiSq9DkZtmLO1XBtBtjJy37PgFr8bwUw1p8fNdhnOEJBYR75MDC8JG38=,iv:YCCfUyncprAkIbvxWwMGGZZywNHLTrr5bYjBBdQp2OI=,tag:mHyHw11ok4jm79iYn6L74Q==,type:str]",
|
||||
"version": "3.13.3"
|
||||
}
|
||||
}
|
||||
+223
-147
@@ -1,43 +1,75 @@
|
||||
{
|
||||
# Network / domains
|
||||
lanDomain = "gitea.lan.ddnsgeek.com"; # Gitea/DDNS domain
|
||||
homeDomain = "sweet.home"; # base LAN domain for service subdomains (pve., docker.)
|
||||
tailnetDomain = "tail13f623.ts.net"; # Tailscale MagicDNS suffix
|
||||
lanCidr = "192.168.2.0/24"; # LAN subnet
|
||||
lanGateway = "192.168.2.254"; # LAN default gateway (router)
|
||||
lanPrefixLength = 24; # LAN subnet prefix length (/24 = 255.255.255.0)
|
||||
lxcLanInterface = "eth0"; # LAN NIC name in LXC containers (set by Proxmox --net0 name=eth0)
|
||||
lxcStorageInterface = "eth1"; # storage-client NIC name in LXC containers (vmbr2, --net1)
|
||||
vmLanInterface = "ens18"; # LAN NIC name in Proxmox VMs (virtio, first NIC)
|
||||
vmStorageInterface = "ens19"; # cluster-internal NIC in HA VMs (vmbr1 — DRBD + Corosync only)
|
||||
rec {
|
||||
# ── Gitea / flake remote ──────────────────────────────────────────────────
|
||||
|
||||
# External Gitea/DDNS domain — used only for the remote flake URL in
|
||||
# Switch-nix / Test-nix aliases (modules/common/configuration.nix).
|
||||
giteaDomain = "gitea.lan.ddnsgeek.com";
|
||||
|
||||
# Org/repo path within Gitea, combined with giteaDomain to form the
|
||||
# git+https:// URL used by Switch-nix / Test-nix.
|
||||
giteaRepoPath = "beatzaplenty/infrastructure";
|
||||
|
||||
# internal repo path to flake
|
||||
giteaRepoFlakePath = "nixos";
|
||||
|
||||
# ── Network ───────────────────────────────────────────────────────────────
|
||||
|
||||
# Base LAN domain for service subdomains (pve., docker., nix-cache., …)
|
||||
homeDomain = "sweet.home";
|
||||
|
||||
# Tailscale MagicDNS suffix for this tailnet
|
||||
tailnetDomain = "tail13f623.ts.net";
|
||||
|
||||
lanCidr = "192.168.2.0/24";
|
||||
lanGateway = "192.168.2.254";
|
||||
lanPrefixLength = 24;
|
||||
|
||||
# NIC names inside guests — determined by the hypervisor/platform, not the OS.
|
||||
lxcLanInterface = "eth0"; # LAN NIC in LXC containers (Proxmox --net0 name=eth0)
|
||||
lxcStorageInterface = "eth1"; # storage-client NIC in LXC containers (vmbr2, --net1)
|
||||
vmLanInterface = "ens18"; # LAN NIC in Proxmox VMs (virtio, first NIC)
|
||||
vmStorageInterface = "ens19"; # cluster-internal NIC in HA VMs (vmbr1 — DRBD + Corosync)
|
||||
vmStorageClientInterface = "ens20"; # storage-client NIC in HA VMs (vmbr2 — iSCSI/NFS VIP)
|
||||
|
||||
# ── Host IPs ──────────────────────────────────────────────────────────────
|
||||
|
||||
pxeServerIp = "192.168.2.223"; # pxe-boot LXC container LAN IP
|
||||
nixCacheIp = "192.168.2.224"; # nix-cache LXC container LAN IP
|
||||
tailscaleRouterIp = "192.168.2.222"; # tailscale-router LXC container LAN IP
|
||||
torRelayIp = "192.168.2.221"; # tor-relay LXC container LAN IP
|
||||
dockerIp = "192.168.2.225"; # docker Proxmox VM LAN IP
|
||||
pbsIp = "192.168.2.244"; # Proxmox Backup Server LAN IP (not NixOS-managed)
|
||||
domainControllerIp = "192.168.2.253"; # FreeIPA domain controller — authoritative DNS for sweet.home (not NixOS-managed)
|
||||
ipaServer = "domain-controller.sweet.home"; # FreeIPA server hostname (used by security.ipa and Kerberos; must be a resolvable FQDN, not an IP)
|
||||
domainControllerIp = "192.168.2.253"; # FreeIPA — authoritative DNS for sweet.home (not NixOS-managed)
|
||||
|
||||
# FreeIPA server FQDN used by security.ipa and Kerberos. Must be a
|
||||
# resolvable name (not an IP); resolves to domainControllerIp.
|
||||
ipaServer = "domain-controller.${homeDomain}";
|
||||
|
||||
# ── Cross-host references ─────────────────────────────────────────────────
|
||||
|
||||
# Cross-host references (LAN hostnames/users other hosts reach over the network)
|
||||
nixCacheHost = "nix-cache"; # substituter/remote-builder hostname
|
||||
dockerHost = "docker"; # docker-compose stack host
|
||||
|
||||
# Raspberry Pi's own Tailscale hostname (not fronted by `server` — it
|
||||
# Raspberry Pi's own Tailscale hostname (not fronted by any server — it
|
||||
# exports its own NFS share directly). Resolved as
|
||||
# "${raspberryPiHost}.${tailnetDomain}" in modules/raspi/mount-data.nix.
|
||||
raspberryPiHost = "raspberrypi";
|
||||
|
||||
remoteBuilderUser = "nixremote"; # remote builder SSH user
|
||||
remoteBuilderUser = "nixremote";
|
||||
|
||||
# nix-cache's own SSH host public key (not a secret — the private half
|
||||
# never leaves the host). Wired into every client's
|
||||
# programs.ssh.knownHosts by modules/nix-cache/remote-builder-client.nix
|
||||
# so distributed builds don't hit "Host key verification failed" on a
|
||||
# fresh client that has never manually ssh'd to nix-cache before. Update
|
||||
# this if nix-cache's host key is ever rotated or the host is rebuilt
|
||||
# from scratch.
|
||||
# Tailscale's internal "Quad100" DNS resolver, reachable from any Tailscale
|
||||
# node via tailscale0. Used by modules/tailscale/ts-dns-forwarder.nix to
|
||||
# forward *.tailnetDomain queries on behalf of FreeIPA's conditional
|
||||
# forwarder zone.
|
||||
tailscaleResolverIp = "100.100.100.100";
|
||||
|
||||
# ── SSH keys ──────────────────────────────────────────────────────────────
|
||||
|
||||
# nix-cache's SSH host public key (not a secret — private half never leaves
|
||||
# the host). Wired into every client's programs.ssh.knownHosts by
|
||||
# modules/nix-cache/remote-builder-client.nix so distributed builds don't
|
||||
# hit "Host key verification failed" on a fresh client. Update if nix-cache
|
||||
# is ever rebuilt with a new host key.
|
||||
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICuHUxGNH6ei3BZD+EfZs3l4X8uJNcjQiOsM/G4yo4O/ lxc-nix-cache";
|
||||
|
||||
# Beszel hub's SSH public key — used by every agent to authenticate the
|
||||
@@ -46,8 +78,8 @@
|
||||
beszelHubKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
|
||||
|
||||
# Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
|
||||
# host (modules/nix-cache/server.nix) — one per client host that's allowed
|
||||
# to use it as a distributed builder.
|
||||
# host (modules/nix-cache/server.nix) — one per client host allowed to use
|
||||
# it as a distributed builder.
|
||||
remoteBuilderAuthorizedKeys = [
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+ioWPhHixlgCB9KIQ0QTHTz6A+Oo2F3uKiINLip5rO root@docker"
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
|
||||
@@ -61,156 +93,202 @@
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMseQwPpmaa6cgV5U8KhUsiVSYARG85zGa9rho0LJWks wayne@pve1"
|
||||
];
|
||||
|
||||
# Admin SSH public key, authorized on the primary user of every host and
|
||||
# the installer image's nixos/root users.
|
||||
# Primary admin SSH public key, authorized on the primary user of every
|
||||
# host and the installer image's nixos/root users.
|
||||
adminSshKey = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCq/Q5LvIXlZwO2kdeAN5nLGZ59nZB7JHYMEszHxmNtGMzv1lM31jiPNsr0z2EKVZhE7OOfa2IF9rhWYD7JUA9G0yzdZ4WTXFNGVVOJoOVH6vAF3XCxoVilOEwTc7h2Wiy+rzd0B28/3spffzQQWJhY6GRQVa8j+6xAGF60Fcvl1vLosYT9Bn2ZbK4TCWOwAn2jqXIieGpZdn/UNZbGOeKRiCvhktDfMAzuQzN/9jMu/oF4pkPn2X1UrsQdNlvp0Ci8md612MozIpncQJyAF1ADhunr3sMx0isUXiqD29R5DS4TftpekqLNLak+zcxFa8N7DcRNp3DcKfJvyTkwQrR4r+b7lFLYOLHLagSso9CzeW/paAS2q9I5SBm/2DtE1diLLg2jZikYcstsu/G5RgvbzbKqjiaMwTdXC3AMvDxQrs7U5pDRZFzoofG3cpODbTm+uy3m0kP70z0M1K45UbDG0p+itnTu9x40JbQEgefbx38AItNvAIx1A8HO4I1VX28= wayne@stream";
|
||||
|
||||
# Additional SSH keys granted access alongside adminSshKey on every host
|
||||
# (modules/common/configuration.nix) and on HA cluster root
|
||||
# (modules/ha/cluster-config.nix). Single definition here prevents the
|
||||
# two modules from drifting out of sync.
|
||||
extraAdminSshKeys = [
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
|
||||
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGygkCljN6uKpdJbHTOQtn8ZnH+wKXDLAwrDFbLrE/65 nixos@nixos"
|
||||
];
|
||||
|
||||
# ── Wifi ──────────────────────────────────────────────────────────────────
|
||||
|
||||
# Prestaged wifi SSID for the gui host's NetworkManager profile
|
||||
# (modules/networking/wifi.nix). The password is not here -- it's
|
||||
# sops-encrypted in secrets/gui.yaml (wifi-password) instead, since this
|
||||
# file isn't a secret store.
|
||||
# (modules/networking/wifi.nix). Password is sops-encrypted in
|
||||
# secrets/gui.yaml (wifi-password) — not stored here.
|
||||
wifiSsid = "nbn-fttp-net-5G";
|
||||
|
||||
# Bare-metal gui host's two disks for a ZFS RAID0 (striped) root pool
|
||||
# (modules/disko/baremetal.nix). Only used transiently at disko-format
|
||||
# time (partitioning); the resulting fileSystems/zpool import reference
|
||||
# by-partlabel/by-id paths afterward regardless, same as
|
||||
# modules/disko/proxmox.nix's own plain "/dev/sda".
|
||||
# ── Bare-metal GUI host ───────────────────────────────────────────────────
|
||||
|
||||
# Two disks for the ZFS RAID0 (striped) root pool on baremetal-gui
|
||||
# (modules/disko/baremetal.nix). Only referenced at disko-format time;
|
||||
# afterward the pool imports by-partlabel/by-id paths regardless.
|
||||
guiRootDisk1 = "/dev/sda";
|
||||
guiRootDisk2 = "/dev/sdb";
|
||||
|
||||
# System
|
||||
# ── System / users ────────────────────────────────────────────────────────
|
||||
|
||||
timeZone = "Australia/Brisbane";
|
||||
|
||||
# Main interactive user on every host. Every module that grants this user
|
||||
# a group, a home directory, or tmpfiles ownership should reference
|
||||
# vars.primaryUser rather than the literal "nixos", so renaming it is a
|
||||
# one-line change.
|
||||
# Main interactive user on every host. Modules that grant this user a
|
||||
# group, home directory, or tmpfiles ownership reference this so a rename
|
||||
# is a one-line change here.
|
||||
primaryUser = "nixos";
|
||||
|
||||
# Primary IPA/domain user. Home Manager is configured for this user on every
|
||||
# IPA-enrolled host (see modules/ipa/client.nix) to manage the environment
|
||||
# that IPA itself doesn't cover: dotfiles, user packages, session variables.
|
||||
# Primary IPA/domain user. Home Manager is configured for this user on
|
||||
# every IPA-enrolled host (modules/ipa/client.nix).
|
||||
ipaUser = "wayne";
|
||||
|
||||
# GID of the IPA "docker-access" group (GID 50010 on the IPA server).
|
||||
# The local "docker" group is pinned to this GID on every host that runs
|
||||
# Docker so that IPA group membership alone grants docker socket access -
|
||||
# no per-host users.groups.docker.members entry for the IPA user needed.
|
||||
# GID of the IPA "docker-access" group (GID 50010 on the IPA server). The
|
||||
# local "docker" group is pinned to this GID on every Docker host so IPA
|
||||
# group membership alone grants socket access — no per-host
|
||||
# users.groups.docker.members entries needed.
|
||||
dockerAccessGid = 50010;
|
||||
|
||||
# HA file server cluster
|
||||
# LAN IPs (vmbr0 / ens18) — management only after storage migration.
|
||||
# Cluster IPs (vmbr1 / ens19) — VLAN 10 (192.168.10.x), isolated internal bridge,
|
||||
# DRBD replication and Corosync heartbeat only; never leaves pve1.
|
||||
# Storage-client IPs (vmbr2 / ens20) — VLAN 20 (192.168.20.x), isolated internal
|
||||
# bridge for iSCSI; docker and server VMs connect here instead of crossing vmbr0.
|
||||
# haServerVip: floating virtual IP on vmbr2, managed by Pacemaker IPaddr2;
|
||||
# iSCSI clients connect here regardless of which node is Active.
|
||||
# Protocol separation: iSCSI on storage-client subnet (VLAN 20) only;
|
||||
# NFS on LAN subnet (VLAN 2) only. Enforced by firewall on the HA nodes.
|
||||
# ── HA file-server cluster ────────────────────────────────────────────────
|
||||
#
|
||||
# Three network segments, all internal to pve1:
|
||||
# LAN VLAN 2 / vmbr0 / 192.168.2.x — management only
|
||||
# Cluster VLAN 10 / vmbr1 / 192.168.10.x — DRBD replication + Corosync ring0
|
||||
# Storage-client VLAN 20 / vmbr2 / 192.168.20.x — iSCSI + NFS client access
|
||||
#
|
||||
# The host octet is consistent across subnets: node1 = .228, node2 = .227,
|
||||
# VIP = .229 everywhere.
|
||||
#
|
||||
# Protocol separation (firewall-enforced on HA nodes):
|
||||
# NFS — both subnets; LAN VIP for pxe-boot/LAN clients, storage VIP for docker
|
||||
# iSCSI — storage-client subnet only
|
||||
|
||||
haServer1Host = "ha-server-1";
|
||||
haServer2Host = "ha-server-2";
|
||||
|
||||
haServer1Ip = "192.168.2.228"; # LAN IP, node 1 (vmbr0 / ens18)
|
||||
haServer2Ip = "192.168.2.227"; # LAN IP, node 2 (vmbr0 / ens18)
|
||||
|
||||
haServer1StorageIp = "192.168.10.228"; # cluster-net IP, node 1 (vmbr1 / ens19, VLAN 10)
|
||||
haServer2StorageIp = "192.168.10.227"; # cluster-net IP, node 2 (vmbr1 / ens19, VLAN 10)
|
||||
haStorageCidr = "192.168.10.224/29"; # cluster subnet — VLAN 10, internal to pve1 only
|
||||
haStoragePrefixLength = 29; # cluster subnet prefix length (/29)
|
||||
haStorageCidr = "192.168.10.224/29"; # cluster subnet — VLAN 10, internal to pve1
|
||||
haStoragePrefixLength = 29;
|
||||
|
||||
haServer1ClientIp = "192.168.20.228"; # storage-client IP, node 1 (vmbr2 / ens20, VLAN 20)
|
||||
haServer2ClientIp = "192.168.20.227"; # storage-client IP, node 2 (vmbr2 / ens20, VLAN 20)
|
||||
haServerVip = "192.168.20.229"; # storage-client floating VIP on vmbr2 (Pacemaker IPaddr2 vip-storage, VLAN 20)
|
||||
haServerLanVip = "192.168.2.229"; # LAN floating VIP on vmbr0 (Pacemaker IPaddr2 vip-lan) — NFS access
|
||||
haServerVip = "192.168.20.229"; # storage-client floating VIP (Pacemaker vip-storage, VLAN 20)
|
||||
haServerLanVip = "192.168.2.229"; # LAN floating VIP (Pacemaker vip-lan) — NFS for LAN clients
|
||||
dockerStorageIp = "192.168.20.225"; # docker CT storage-client IP (vmbr2 / eth1, VLAN 20)
|
||||
haClientCidr = "192.168.20.0/24"; # storage-client subnet — VLAN 20, internal to pve1 only
|
||||
haClientPrefixLength = 24; # storage-client subnet prefix length (/24)
|
||||
haClientCidr = "192.168.20.0/24"; # storage-client subnet — VLAN 20, internal to pve1
|
||||
haClientPrefixLength = 24;
|
||||
|
||||
haStorageRoot = "/srv/ha-data"; # XFS-over-DRBD mount point on the Active node
|
||||
haStorageNfsFqdn = "nfs.storage.home"; # NFS VIP FQDN (storage.home zone) — resolves to haServerVip; use this in fileSystems device strings
|
||||
|
||||
# NFS VIP FQDNs — use these in fileSystems device strings so mounts
|
||||
# survive a future VIP renumber via a DNS-only update, not a NixOS rebuild.
|
||||
haStorageNfsFqdn = "nfs.storage.home"; # storage-client VIP (VLAN 20) — docker + future swarm
|
||||
haLanNfsFqdn = "ha-vip-lan.${homeDomain}"; # LAN VIP (VLAN 2) — pxe-boot + other LAN clients
|
||||
|
||||
haIscsiIqn = "iqn.2026-01.home.sweet:ha-storage";
|
||||
# DRBD backing disk — identified by SCSI controller path so it resolves to the
|
||||
# correct block device regardless of OS-level naming (sda vs sdb can differ
|
||||
# between Proxmox VMs depending on disk-add order). drive-scsi1 is always the
|
||||
# dedicated data disk on all HA nodes; drive-scsi0 is the OS disk.
|
||||
|
||||
# DRBD backing disk — identified by SCSI controller path so it resolves to
|
||||
# the correct block device regardless of OS-level naming (sda vs sdb can
|
||||
# differ between VMs depending on disk-add order). drive-scsi1 is always
|
||||
# the data disk; drive-scsi0 is the OS disk.
|
||||
haServerDrbdDisk = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1";
|
||||
|
||||
# Storage
|
||||
# ── Docker Swarm cluster ──────────────────────────────────────────────────
|
||||
#
|
||||
# Three network segments, all internal to pve1:
|
||||
# LAN vmbr0 192.168.2.0/24 — management; SSH + external service traffic
|
||||
# Storage-client vmbr2 192.168.20.0/24 — NFS from HA cluster VIP (shared with HA nodes)
|
||||
# Swarm cluster vmbr3 192.168.30.0/24 — Docker Swarm gossip + VXLAN overlay
|
||||
#
|
||||
# Host octet consistent across subnets: node1 = .230, node2 = .231.
|
||||
# IPs from the .230–.239 expansion buffer documented in docs/ip-addressing.md.
|
||||
#
|
||||
# Docker Swarm uses --advertise-addr and --data-path-addr on the swarm NIC
|
||||
# (ens20/vmbr3) so all inter-node cluster traffic stays on the isolated
|
||||
# internal bridge and never crosses the LAN.
|
||||
#
|
||||
# When expanding to a second Proxmox node, vmbr3 (VLAN 30) and vmbr1
|
||||
# (VLAN 10) share the same inter-node trunk NIC via VLAN tagging — same
|
||||
# physical wire, different VLAN IDs.
|
||||
|
||||
haDocker1Host = "ha-docker-1";
|
||||
haDocker2Host = "ha-docker-2";
|
||||
|
||||
haDocker1Ip = "192.168.2.230"; # LAN management NIC (ens18, vmbr0)
|
||||
haDocker2Ip = "192.168.2.231";
|
||||
|
||||
haDocker1StorageIp = "192.168.20.230"; # storage-client NIC (ens19, vmbr2)
|
||||
haDocker2StorageIp = "192.168.20.231";
|
||||
|
||||
haDocker1SwarmIp = "192.168.30.230"; # swarm cluster NIC (ens20, vmbr3)
|
||||
haDocker2SwarmIp = "192.168.30.231";
|
||||
|
||||
haDockerSwarmCidr = "192.168.30.0/24";
|
||||
haDockerSwarmPrefixLength = 24;
|
||||
|
||||
# NIC names for ha-docker VMs. ens19/ens20 occupy the same guest bus
|
||||
# positions as vmStorageInterface/vmStorageClientInterface on ha-server VMs
|
||||
# but are attached to different bridges — storage (vmbr2) and swarm (vmbr3)
|
||||
# respectively. Kept as named variables to avoid bare literals in modules.
|
||||
haDockerStorageInterface = "ens19"; # vmbr2 — NFS client
|
||||
haDockerSwarmInterface = "ens20"; # vmbr3 — Docker Swarm gossip + VXLAN
|
||||
|
||||
# ── Storage / NFS ─────────────────────────────────────────────────────────
|
||||
|
||||
# NFS share definitions — used by ha-server.nix (exports), docker/mount-data.nix,
|
||||
# and pxe-boot/mount-pxe-images.nix (mounts). `subpath` is relative to
|
||||
# haStorageRoot; `mountpoint` is the absolute local path on each client.
|
||||
# Renaming a share only needs changing it here — exports and all client
|
||||
# Renaming a share only requires changing it here — exports and all client
|
||||
# mounts follow automatically.
|
||||
nfsShares = {
|
||||
options = "(rw,sync,no_subtree_check,no_root_squash)";
|
||||
dockerConfig = {
|
||||
subpath = "docker/config";
|
||||
mountpoint = "/mnt/docker/config";
|
||||
};
|
||||
dockerDatabases = {
|
||||
subpath = "docker/databases";
|
||||
mountpoint = "/mnt/docker/databases";
|
||||
};
|
||||
dockerVolumes = {
|
||||
subpath = "docker/volumes";
|
||||
mountpoint = "/mnt/docker/volumes";
|
||||
};
|
||||
nextcloudData = {
|
||||
subpath = "docker/nextcloud-data";
|
||||
mountpoint = "/mnt/nextcloud-data";
|
||||
};
|
||||
raspiVolumes = {
|
||||
subpath = "raspi/volumes";
|
||||
mountpoint = "/mnt/raspi-backup";
|
||||
};
|
||||
proxmoxIsos = {
|
||||
subpath = "proxmox/iso";
|
||||
mountpoint = "/mnt/iso";
|
||||
};
|
||||
proxmoxLxcImages = {
|
||||
subpath = "proxmox/lxc";
|
||||
mountpoint = "/mnt/lxc";
|
||||
};
|
||||
pxebootImages = {
|
||||
subpath = "pxe-boot/images";
|
||||
mountpoint = "/mnt/pxe-images";
|
||||
};
|
||||
dockerConfig = { subpath = "docker/config"; mountpoint = "/mnt/docker/config"; };
|
||||
dockerDatabases = { subpath = "docker/databases"; mountpoint = "/mnt/docker/databases"; };
|
||||
dockerVolumes = { subpath = "docker/volumes"; mountpoint = "/mnt/docker/volumes"; };
|
||||
nextcloudData = { subpath = "docker/nextcloud-data"; mountpoint = "/mnt/nextcloud-data"; };
|
||||
raspiVolumes = { subpath = "raspi/volumes"; mountpoint = "/mnt/raspi-backup"; };
|
||||
proxmoxIsos = { subpath = "proxmox/iso"; mountpoint = "/mnt/iso"; };
|
||||
proxmoxLxcImages = { subpath = "proxmox/lxc"; mountpoint = "/mnt/lxc"; };
|
||||
pxebootImages = { subpath = "pxe-boot/images"; mountpoint = "/mnt/pxe-images"; };
|
||||
};
|
||||
|
||||
# The Raspberry Pi's own NFS export — not under storageRoot/nfsServerHost,
|
||||
# served directly by the Pi itself over Tailscale (see raspberryPiHost
|
||||
# above) and mounted at raspiMountpoint by modules/raspi/mount-data.nix.
|
||||
# The Raspberry Pi's own NFS export — not under haStorageRoot, served
|
||||
# directly by the Pi over Tailscale (see raspberryPiHost) and mounted by
|
||||
# modules/raspi/mount-data.nix.
|
||||
raspiNfsPath = "/home/raspi/raspi";
|
||||
raspiMountpoint = "/mnt/raspi";
|
||||
|
||||
# ── Ports ─────────────────────────────────────────────────────────────────
|
||||
#
|
||||
# Every literal port referenced from modules/ or hosts/, grouped by the
|
||||
# service/host that opens or connects to it — kept as separate entries
|
||||
# even where two happen to share a number today (e.g. nixCacheHttp and
|
||||
# pxeBootHttp are both 80) so changing one service's port can never
|
||||
# silently change an unrelated one.
|
||||
# service that opens or connects to it. Kept as separate entries even where
|
||||
# two share a number today (e.g. nixCacheHttp and pxeBootHttp are both 80)
|
||||
# so changing one service's port never silently changes another.
|
||||
|
||||
ports = {
|
||||
# nix-cache's nginx reverse proxy in front of nix-serve
|
||||
# (modules/nix-cache/server.nix).
|
||||
# (modules/nix-cache/server.nix)
|
||||
nixCacheHttp = 80;
|
||||
|
||||
# pxe-boot's nginx asset server, also used to build pxeBaseUrl
|
||||
# (modules/build-types/pxe-boot.nix).
|
||||
# pxe-boot's nginx asset server; also used to build pxeBaseUrl
|
||||
# (modules/build-types/pxe-boot.nix)
|
||||
pxeBootHttp = 80;
|
||||
|
||||
# pxe-boot's atftpd TFTP server — UDP, not TCP
|
||||
# (modules/build-types/pxe-boot.nix).
|
||||
# pxe-boot's atftpd TFTP server — UDP (modules/build-types/pxe-boot.nix)
|
||||
pxeBootTftp = 69;
|
||||
|
||||
# `server`'s NFS exports: portmapper (rpcbind), NFS data, and the
|
||||
# mountd RPC service (used by showmount/NFSv3 mount protocol).
|
||||
# Mountd listens on a fixed port so the firewall can whitelist it
|
||||
# explicitly rather than opening all of rpcbind's dynamic range.
|
||||
# All three need both TCP and UDP (modules/build-types/server.nix and
|
||||
# modules/build-types/ha-server.nix).
|
||||
# DHCP proxy port opened by dnsmasq on the pxe-boot host
|
||||
# (modules/build-types/pxe-boot.nix)
|
||||
dhcp = 67;
|
||||
|
||||
# DNS port opened on tailscale-router for FreeIPA's conditional forwarder
|
||||
# (modules/tailscale/ts-dns-forwarder.nix)
|
||||
dns = 53;
|
||||
|
||||
# NFS stack: portmapper (rpcbind), NFS data, and mountd RPC service.
|
||||
# Mountd is pinned to a fixed port so the firewall can whitelist it
|
||||
# without opening rpcbind's full dynamic range. All three need TCP + UDP
|
||||
# (modules/build-types/ha-server.nix).
|
||||
nfsRpcbind = 111;
|
||||
nfsd = 2049;
|
||||
nfsMountd = 20048;
|
||||
|
||||
# HA cluster ports opened on ha-server-1 and ha-server-2
|
||||
# (modules/build-types/ha-server.nix / modules/ha/cluster-config.nix).
|
||||
# HA cluster ports (modules/ha/cluster-config.nix)
|
||||
haServerDrbd = 7789; # DRBD replication (TCP)
|
||||
haServerIscsi = 3260; # iSCSI target (TCP)
|
||||
haServerCorosync1 = 5404; # Corosync totem ring (UDP)
|
||||
@@ -219,49 +297,47 @@
|
||||
haServerPacemakerRemoted = 3121; # pacemaker-remoted (TCP)
|
||||
haServerPcsd = 2224; # pcsd cluster daemon (TCP)
|
||||
|
||||
# Opened on the docker host's firewall for the Traefik-fronted
|
||||
# container stack (docker-compose config lives in the separate
|
||||
# /home/debian/docker repo, not here): 80/443 are Traefik's own
|
||||
# HTTP/HTTPS listeners; 8080 is an additional exposed service whose
|
||||
# exact backend isn't declared in this repo (modules/build-types/docker.nix).
|
||||
# Docker host — Traefik HTTP/HTTPS listeners plus one additional exposed
|
||||
# service (modules/build-types/docker.nix)
|
||||
dockerHttp = 80;
|
||||
dockerHttps = 443;
|
||||
dockerExtra = 8080;
|
||||
|
||||
# Beszel monitoring hub, reachable at
|
||||
# http://<dockerHost>.<homeDomain>:<beszelHub> from every agent
|
||||
# (modules/beszel/enable-agent.nix, hosts/nixos/home.nix).
|
||||
# Docker Swarm inter-node ports (modules/build-types/ha-docker.nix).
|
||||
# Firewalled to haDockerSwarmCidr only — vmbr3 is an isolated bridge
|
||||
# with no physical uplink, so these ports are unreachable from LAN.
|
||||
dockerSwarmMgmt = 2377; # TCP — Raft consensus + cluster management
|
||||
dockerSwarmDisc = 7946; # TCP+UDP — Serf gossip (container network discovery)
|
||||
dockerSwarmVxlan = 4789; # UDP — VXLAN overlay data path
|
||||
|
||||
# Beszel monitoring hub on docker.sweet.home, reached by every agent
|
||||
# (modules/beszel/enable-agent.nix, hosts/nixos/home.nix)
|
||||
beszelHub = 8090;
|
||||
|
||||
# Proxmox VE and Proxmox Backup Server web UIs, opened as desktop
|
||||
# shortcuts on the gui build type (hosts/nixos/home.nix).
|
||||
# Proxmox VE and PBS web UIs — desktop shortcuts on the gui build type
|
||||
# (hosts/nixos/home.nix, modules/build-types/gui.nix)
|
||||
pveWeb = 8006;
|
||||
pbsWeb = 8007;
|
||||
|
||||
# Tor relay's ORPort — the port other Tor relays connect to for onion
|
||||
# routing traffic (modules/tor/enable-relay.nix). Tor's own conventional
|
||||
# default; opened via services.tor.openFirewall rather than
|
||||
# networking.firewall.allowedTCPPorts directly, but kept here anyway so
|
||||
# it's not a bare literal duplicated between the relay's settings and
|
||||
# anything else that ever needs to reference it.
|
||||
# Tor relay's ORPort (modules/tor/enable-relay.nix). Opened via
|
||||
# services.tor.openFirewall rather than allowedTCPPorts directly, but
|
||||
# kept here so it's not a bare literal if ever referenced elsewhere.
|
||||
torRelayOrPort = 9001;
|
||||
};
|
||||
|
||||
# ── Build / image settings ────────────────────────────────────────────────
|
||||
|
||||
# .raw disk image size for every proxmox-* host's standalone Disko image
|
||||
# build (modules/disko/proxmox.nix, config.system.build.diskoImagesScript
|
||||
# — see docs/proxmox-images.md). Root fills whatever's left after the ESP
|
||||
# and swap partitions within this total.
|
||||
# build (modules/disko/proxmox.nix — see docs/proxmox-images.md).
|
||||
proxmoxImageSize = "50G";
|
||||
|
||||
# nix-cache's Nix store garbage collection retention
|
||||
# (modules/nix-cache/server.nix).
|
||||
# nix-cache Nix store GC retention (modules/nix-cache/server.nix)
|
||||
nixCacheGcMaxAge = "30d";
|
||||
|
||||
# Traefik access log rotation, watched on the docker host at
|
||||
# nfsShares.dockerVolumes.mountpoint (modules/traefik/rotate-logs.nix).
|
||||
# nfsShares.dockerVolumes.mountpoint (modules/traefik/rotate-logs.nix)
|
||||
traefikLogRotate = {
|
||||
maxSize = "100M"; # rotate once a log file exceeds this size
|
||||
keep = 20; # number of rotated logs to retain before deleting the oldest
|
||||
keep = 20; # number of rotated logs to retain
|
||||
};
|
||||
|
||||
}
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIoCzxgfFjE3scsLzTitDcrmJ/Bh4MQ+3cEH8YOQ665z ha-docker-1
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"data": "ENC[AES256_GCM,data:fd++6Dyfyq7JfeGflnUXkmAKWhTG9Z0tvMCmVreczaoVtPN9+t+atIIa9EB3I2PWbOEYoIKelO0fDuK3ZrY1mrp8K8Stb3vJdB8lz97dBB/Gn5BonRAMYszMpfp9+rgf8pZqc89KWt3xz7z9EFKjTE7Kf8n08K1pHyqVjZ9W+bH9Zxh4WEWGeaKAyH4nlmVLjZ7rKch9ErY28bMVBwiiXIDFeXy/TIkSk7pVODFeCqMcuXFFnPememm7vXme+wtepqNc6WfrKTQOuQFe9SDK/A36Cl/92hSqjooTLU9XXjQNrEf1xAOkZhpTpbvsF32O103ixtqDY72ZMmC/RExAvaTT+Jf+Utfnq0S8Gna+d2rHyzbbvtdUC68vqA6QhL9qJe3HHwyAPbcGJ2YONgTsKot9DR3zYEWVwTjydBL8Z3/JTw62A+sLve4WuXgq9yooYGF4s/SVX1+a0e1MkQ2ahn7a6rIEyj+YK2bX71G5Kw+KBXwHzFW6duJs4mN23Yu9N+xrU7RdAT0O3hWGeBgmLfp8ko4uWOxm2Kin,iv:+5am0O58f/8tqrSEiFld4Xu2GZNDsEobjFPDgNyv768=,tag:GLERbrGo1rRKUXZBnUXEvw==,type:str]",
|
||||
"sops": {
|
||||
"age": [
|
||||
{
|
||||
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxcDErcVphYUFWUnN0WlBa\nVU9jUlZkMzArZFlYbENRNmZFQzN5Q1ZNa2k4CmlYa21yYXIrZG1QUVVNSy9oWE40\nK2t1dCtrR0R3cVQ5cVcwakphUFRvQ2cKLS0tIFdVL1dwNmMxR0xJOHh2ZlUzQmNm\nV1NNMldMRGd4NDAxbU1nbGZ0bWJKTDgKnyzQCimHCBke8dM3kdkjVu4lDcfvkvOi\n+k4GJVCuTQ60rwWXPsB8kiPMDp/+8tNO09LG/fR3vcGRgX65KcjMSA==\n-----END AGE ENCRYPTED FILE-----\n",
|
||||
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
|
||||
}
|
||||
],
|
||||
"lastmodified": "2026-07-30T09:22:17Z",
|
||||
"mac": "ENC[AES256_GCM,data:MjPZqcKTTO3uoZk/u7fdfQhgpcHsjKggzu+/pJanXUgIDWtCiKfFbRsQmwH5nYiRyjHOJdX4xMcFRmaYEN6oz/hzDR5ty+0mSuVNIemBz8OpShwjGY2E+pAkdxDgWLsc/Q+CyywGscLslbgHSAKuKPdPlyzhuRyy1QTyk/ty7Xk=,iv:MSQsNuENPgde3YitmHTyDjTYR5ZuhzwRgcWb5DAPY6g=,tag:Yo231g12u6aMPUauIf65hg==,type:str]",
|
||||
"version": "3.13.3"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPwfEcHZRbdI5GK9cn59SDrLruNOJJEjVfJXZve25nnE ha-docker-2
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"data": "ENC[AES256_GCM,data:3sWc8kfLREvxWRf0dtZFgubyxrAnfgiOT8b7nGtAQyQVaoy3moi3TGhPVdeihwbEOaEQX+YHvLWIzfxtLQxTPDxjvu3v2+dAYrjobIDHJaon3Th9vLfF9dxl/NVQB0wy/uL26z1Q4bhYj2yOU2V9wIzJ4G/94OvnttZKUub2iKmiUji36KsZkq6FVL0e0fUcv134dbo65yTIjmVB6U4SSB4nCrVKNg7z5EeXBBZodulK4kJ5zsspgTHLZbI43cW/tXmnNoW2NTFZen7Q1rLhlB0ISkXSXp0Sn6rIQsSZpywdc24cZ0fenqER38UI4PhwUPXd34Mv/LxQ6D5TnblANDQn1rsfxjjFY2JHXWO7kydccS5Us0vBHWMHbgZbC50+D1VIvvrK9SoEnZ7LW5WRJc1QaCvAz5eVbaB1V1UQ+eJjZToBQY1yvCg3vyj2nlX0A+BAPdZZMZgRazUp5CZaspcvcszIjTK8TDwqMeefOmUfNkC2Hgz83ij2lu+Kr9/Z1BurvIMotPDFER7qqYFodZJR4HMQ40rS9egO,iv:qh4tRPY6wZfOHdk1bDzQme/QqBVfMSfh2gZVTy1BEN0=,tag:8UcO2ia1nlzZPHNKuMYrrw==,type:str]",
|
||||
"sops": {
|
||||
"age": [
|
||||
{
|
||||
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBK1h5RUtPdjQwWm9vWGJK\nVzZHbU1GU2dVN3NKSWxqOEhtTUZwbDkzbTA4CjREek1WbnZrVkdQZGU3djFVVjg4\nOUNkNzk3eXIvN2hDMVNsc3dZS3hJb28KLS0tIFgxdlhwUG9PdUlsRFdRdmJkOGhh\nVU1uWkpwSVhKL1VRTktoQmxEQVk5bjgKQHofvV/y5tPG0AJUoC1/oXBqAIlX6FB/\nAHKB7k3JxbgrzkR40uKuA9y6qmokGkVPoD3p7N5fd5G3aZY3fsYtjw==\n-----END AGE ENCRYPTED FILE-----\n",
|
||||
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
|
||||
}
|
||||
],
|
||||
"lastmodified": "2026-07-30T09:22:26Z",
|
||||
"mac": "ENC[AES256_GCM,data:bICdSPh5zGwWW1MI4rZZgKOZWM6X2h2OEL+uK3EQNGsGoRnW2JicBdIjqBkQsin5Ke4a0i0n/LudIB/G90+NkQYXu8/ixve3vN6VvfnoBb53BuDp/Hn7hJkvvn/1iXDmYsg1S75IM6xTm4RkY8nbLTuBltgiZvrN6mYMrxcXc9s=,iv:rGnu8V9P3vOrHA06GDq3Ru9l3+x93hfrVVZtN+hR9e4=,tag:rYZ1h95YoZzLg2ZaRxJmQA==,type:str]",
|
||||
"version": "3.13.3"
|
||||
}
|
||||
}
|
||||
Reference in New Issue
Block a user