Compare commits

...
Author SHA1 Message Date
beatzaplentyandClaude Sonnet 4.6 2741451642 fix(deploy): remove sudo from docker commands; fix grep false-positive on inactive
nixos user is in the docker group so sudo is not needed (and breaks because
sudo requires a TTY for password input). Also grep -q 'active' matched
'inactive' causing ha-docker-2 to be seen as already joined.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
2026-07-30 20:11:58 +10:00
beatzaplenty 9f1fed26c2 secrets(ha-docker): refresh sops host-key registrations for new VM instances 2026-07-30 20:01:48 +10:00
beatzaplentyandClaude Sonnet 4.6 3b6ac30946 fix(deploy): use PROXMOX_REMOTE_REPO_DIR for remote repo path in phase 3.5
The phase 3.5 branch-switch step was hardcoded to /home/user/nixos, which
assumes the standalone nixos Gitea repo is checked out there. When the
infrastructure mono-repo is used instead, the path differs. Now reads
PROXMOX_REMOTE_REPO_DIR (same variable create-proxmox-resource.sh uses)
with graceful fallback if the branch switch fails.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
2026-07-30 19:43:21 +10:00
beatzaplentyandClaude Sonnet 4.6 cee33aa2d8 secrets(ha-docker): add clan SSH host keys and re-encrypt common secrets
Adds sops-encrypted SSH host key pairs for proxmox-ha-docker-1 and
proxmox-ha-docker-2 (clan vars), and re-encrypts secrets/common.yaml
to include both hosts as recipients so they can decrypt on first boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
2026-07-30 19:34:57 +10:00
beatzaplenty ac9099a1f2 secrets(ha-docker): add IPA keytabs for ha-docker-1 and ha-docker-2 2026-07-30 19:34:40 +10:00
beatzaplentyandClaude Sonnet 4.6 c96752e5d0 Add Docker Swarm HA cluster: ha-docker-1 and ha-docker-2
Two new NixOS Proxmox VMs (VMIDs 202/203) forming a dual-manager Docker
Swarm on dedicated vmbr3 (192.168.30.0/24, VLAN 30) for gossip and VXLAN,
with NFS via the storage-client network (vmbr2) from the existing HA cluster.

- nixos/variables.nix: add ha-docker IP/interface/port vars and swarm CIDR
- nixos/modules/build-types/ha-docker.nix: new build type — Docker 29,
  NFS mounts, beszel-agent, health monitoring, swarm firewall rules with
  checkReversePath = "loose" for VXLAN routing mesh
- nixos/hosts/ha-docker-{1,2}/host.nix: per-host identity — three NICs
  (LAN, storage, swarm), IPA dyndns pinned to LAN interface
- nixos/flake.nix: add proxmox-ha-docker-{1,2} targets; build-validated
  with nix build --dry-run (169 derivations, no errors)
- nixos/docs/ip-addressing.md: document VLAN 30 / swarm.home zone,
  ha-docker IP allocations across all three subnets
- nixos/scripts/docker-swarm/deploy.sh: 10-phase lifecycle script
  (bridge, keys, IPA, VMs, swarm init, DNS, verify); modelled on
  scripts/ha/deploy.sh with --destroy mode
- nixos/docs/internal/docker-swarm-cutover.md: service-by-service
  migration guide covering Traefik log rotation, Nextcloud cron sidecar,
  docker-health-to-gotify swarm awareness updates, Passbolt/Gitea steps,
  DNS cutover, and CT 105 decommission checklist

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
2026-07-30 19:01:17 +10:00
beatzaplenty 43314b6aa3 added gitea app secret 2026-07-30 17:49:57 +10:00
beatzaplenty db84a0bcc1 Add Gitea read-only token to nix access-tokens for private repo rebuilds 2026-07-30 17:47:22 +10:00
beatzaplenty 419b0f706b updated flake URL 2026-07-30 17:39:22 +10:00
beatzaplenty 89e1c5dc9a Fix netbootMinimalSystem: set stateVersion and forceImportRoot 2026-07-30 17:36:00 +10:00
beatzaplenty 77db51a88f Fix formatting in pxe-boot host.nix 2026-07-30 17:28:37 +10:00
beatzaplenty ad0b4a14e6 Fix codex-maintenance.sh path prefix when run from mono-repo subtree 2026-07-30 17:27:48 +10:00
beatzaplenty c9dbb03e31 Merge commit 'e812169e521abeccf3903c0ab6ac00538de8b9dc' as 'nixos' 2026-07-30 17:07:01 +10:00
beatzaplenty b6ed0c6007 Remove nixos scaffold placeholder to allow git subtree add 2026-07-30 17:06:30 +10:00
beatzaplentyandClaude Sonnet 4.6 78fdf3d94c Update network topology: Tailscale routing and DNS proxy, remove Pi-hole
- Document bidirectional LAN ↔ Tailscale routing via tailscale-router
  (static route on router for 100.64.0.0/10, subnet advertisement into Tailnet)
- Document DNS proxy chain: FreeIPA conditional forwarder for *.ts.net
  → tailscale-router → MagicDNS; reverse for *.sweet.home from Tailscale clients
- Remove stale pihole.sweet.home from IP table (decommissioned)
- Fix freeipa role: remove Pi-hole IP from ipa_dns_forwarder, document
  ts.net conditional forwarder as a post-install step
- Fix inventory: remove Pi-hole forwarder IP, add ts.net forwarder note

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvNjoxTWEDkhXsd1Dq2ETP
2026-07-30 07:10:03 +10:00
beatzaplentyandClaude Sonnet 4.6 f85c65870f Initial infrastructure mono-repo scaffold
Consolidates nixos, docker, raspi, and debian-configuration into a single
infrastructure-as-code repo. Includes:

- ansible/: full inventory + proxmox-hardening, freeipa, and raspberrypi
  roles (converted from debian-configuration bash scripts)
- terraform/: Proxmox VMs, Dynu DNS, Pi-hole (decommissioned stub),
  Docker container catalog — migrated from docker/infrastructure/terraform/
- stacks/docker/, stacks/raspi/, nixos/: placeholder READMEs pending
  git subtree population (see implementation plan)
- docs/: internal MkDocs site with architecture, network topology, runbooks,
  and drift-detection guide; external sanitized site
- scripts/: drift-detect.sh, docs-build.sh, install-hooks.sh, check-secrets.sh
- CI: secret-scan (push/PR), drift-detect (daily), docs-build (on change)
- Pi-hole removed throughout — DNS is FreeIPA, DHCP is router

See docs/internal/implementation-plan.md for the phased rollout after
pushing to Gitea.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvNjoxTWEDkhXsd1Dq2ETP
2026-07-30 07:07:47 +10:00
beatzaplenty 97c9353fa8 Merge pull request 'refactor: full repo sweep — variables, docs, and comment cleanup' (#108) from worktree-reflective-puzzling-frog into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #108
2026-07-29 20:29:33 +00:00
beatzaplentyandClaude Sonnet 4.6 cded77919d refactor: full repo sweep — variables, docs, and comment cleanup
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
- variables.nix: switch to rec {}, extract giteaDomain/giteaRepoPath,
  extraAdminSshKeys, haLanNfsFqdn, tailscaleResolverIp, ports.dhcp,
  ports.dns; ipaServer now derives from homeDomain ref; section headers
- modules: use new vars throughout (pxe-boot, ts-dns-forwarder,
  cluster-config, configuration.nix, mount-pxe-images) — eval unchanged
- docs: delete ephemeral planning docs (AUDIT_REPORT, ha-network-audit,
  network-cutover); add docs/ha.md; drop migration reference table from
  ip-addressing.md; remove stale server example from beszel.md
- CLAUDE.md/README.md/AGENTS.md: fix build types (tailscale-router,
  ha-server, drop server); document scripts/ha/, scripts/ipa/, and
  all previously undocumented top-level and lib scripts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-30 06:28:52 +10:00
beatzaplenty 71381ad990 update default editor
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
2026-07-30 05:32:23 +10:00
beatzaplenty 857c81f64a updated beszel key
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
2026-07-29 21:33:48 +10:00
beatzaplenty 334ffbda09 updated beszel token
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
2026-07-29 21:27:16 +10:00
beatzaplentyandClaude Sonnet 4.6 620a344d78 feat(pxe-boot): switch NFS mount to HA cluster LAN VIP
Check NixOS configurations / eval-hosts (push) Successful in 10m31s
Updates mount-pxe-images.nix to mount pxe-boot/images from
ha-vip-lan.sweet.home (Pacemaker vip-lan, 192.168.2.229) instead of
server.sweet.home. Storage root changes from /tank to haStorageRoot
(/srv/ha-data). NFSv3+nolock options for LXC are unchanged.

Removes nfsServerHost and storageRoot from variables.nix — all NFS
clients now reference the HA cluster directly (docker via haStorageNfsFqdn
on VLAN 20, pxe-boot via ha-vip-lan on VLAN 2). Updates the nfsShares
comment to reflect the new single-source-of-truth role without the
dead server.nix references.

DNS records for server.sweet.home and server.storage.home were already
cleaned up automatically by IPA dynamic DNS when the VM was deleted.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 20:53:46 +10:00
beatzaplentyandClaude Sonnet 4.6 24e5c9fe0c feat: decommission server.sweet.home
Check NixOS configurations / eval-hosts (push) Failing after 3m46s
Removes all flake artifacts for the retired NFS/ZFS file server:
- flake.nix: drop linode-server, proxmox-server, lxc-server targets
- hosts/server/: delete host identity
- modules/build-types/server.nix: delete build type (ZFS pool + NFS exports)
- vars/per-machine/{linode,lxc,proxmox}-server/: delete clan vars
- secrets/server.keytab: delete IPA keytab
- .sops.yaml: remove server key anchors, strip from common.yaml recipients,
  delete server.yaml and server.keytab creation rules
- variables.nix: remove serverIp, serverStorageIp (nfsServerHost/storageRoot
  retained — pxe-boot/mount-pxe-images.nix still references them)
- docs: mark server decommissioned in ip-addressing.md

Docker NFS mounts already pointed at nfs.storage.home (previous commit).
VM destruction and DNS cleanup (server.sweet.home, server.storage.home
A/PTR records) to follow once the VM is confirmed stopped.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 20:46:19 +10:00
beatzaplentyandClaude Sonnet 4.6 4ed2db906a feat(docker): switch NFS mounts to HA cluster storage VIP
Check NixOS configurations / eval-hosts (push) Failing after 16m43s
Updates mount-data.nix to mount all docker shares from
nfs.storage.home (Pacemaker vip-storage, 192.168.20.229) over docker's
eth1/vmbr2 interface instead of server.sweet.home over the LAN.
Adds haStorageNfsFqdn variable to variables.nix for the storage.home
zone FQDN so mounts survive a future VIP renumber without a rebuild.
Storage root changes from /tank (server ZFS pool) to /srv/ha-data
(HA cluster XFS-over-DRBD). Share subpaths are unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 20:27:15 +10:00
beatzaplentyandClaude Sonnet 4.6 944af9597d docs(dns): document DNS zone layout for all three networks
Check NixOS configurations / eval-hosts (push) Failing after 14m52s
Adds a DNS Zones section to ip-addressing.md covering the three FreeIPA
zones (sweet.home / cluster.home / storage.home) with all A records and PTR
records as configured. Updates Stage 6 in network-cutover.md to reference
nfs.storage.home by DNS name rather than the raw VIP IP, and marks DNS as a
completed prerequisite for the HA storage cutover.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 20:12:25 +10:00
beatzaplentyandClaude Sonnet 4.6 4f0b07031d fix(ha): add NFS port check on storage VIP in health.sh
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Storage VIP now serves both NFS and iSCSI. Add port 2049 check
alongside the existing 3260 check in the storage VIP section.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 19:09:27 +10:00
beatzaplentyandClaude Sonnet 4.6 8c86144694 docs(ha): update docs to reflect implemented network segmentation
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
ip-addressing.md:
  - Storage-client network purpose updated: NFS primary, iSCSI available
  - Protocol separation box updated: NFS on both VIPs (each CIDR-restricted);
    iSCSI on VLAN 20 only but not in active use
  - docker role updated to NFS client (not iSCSI initiator)
  - vip-storage description updated to NFS + iSCSI endpoint

ha-network-audit.md:
  - Implementation status header added noting all audit items resolved
  - Records decisions that diverged from original recommendations
    (VLAN renumbering, dual VIP, NFS-over-iSCSI for docker/swarm)
  - Notes deferred items (STONITH, iSCSI ACLs)

network-cutover.md:
  - New Stage 6: HA storage cutover — docker NFS remount from
    server.sweet.home to vip-storage (192.168.20.229), final rsync
    steps, and server.sweet.home decommission procedure
  - Former Stage 6 (final cleanup) renumbered to Stage 7

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 19:04:42 +10:00
beatzaplentyandClaude Sonnet 4.6 3c28d48bc7 feat(ha): add NFS exports on storage-client network (VLAN 20)
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
vip-storage (192.168.20.229) now serves NFS as well as iSCSI, firewalled
to haClientCidr (192.168.20.0/24) only. This allows docker and future
swarm nodes to NFS-mount shared volumes from the storage network rather
than the LAN, keeping storage traffic off vmbr0.

LAN NFS (vip-lan 192.168.2.229) remains for pxe-boot and other LAN
clients. Protocol and subnet boundaries enforced by firewall on both nodes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 18:55:25 +10:00
beatzaplentyandClaude Sonnet 4.6 e9a4913069 fix(ha): split health.sh VIP check into LAN (NFS) and storage (iSCSI)
Check NixOS configurations / eval-hosts (push) Successful in 10m43s
Storage VIP (192.168.20.229) is on vmbr2 — an internal-only bridge with no
physical uplink, unreachable from the LAN. Test it via SSH to the active node
(which has VLAN 20 connectivity) instead of directly. LAN VIP (192.168.2.229)
continues to be tested directly from wherever the script runs.

Also adds VIP_LAN variable to config section.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 18:33:34 +10:00
beatzaplentyandClaude Sonnet 4.6 bf4836efac refactor(ha): renumber cluster→VLAN10, storage→VLAN20; iSCSI/NFS protocol separation
Check NixOS configurations / eval-hosts (push) Failing after 17m31s
Renumber internal HA bridges so VLAN ID matches IP third octet:
  vmbr1 (cluster): 192.168.4.x → 192.168.10.x (VLAN 10, /29)
  vmbr2 (storage): 192.168.5.x → 192.168.20.x (VLAN 20, /24)

Enforce protocol separation at the firewall on both HA nodes:
  - iSCSI (3260/tcp): storage-client subnet (192.168.20.0/24) only
  - NFS (111,2049,20048 tcp+udp): LAN subnet (192.168.2.0/24) only
  - Cluster subnet (192.168.10.224/29) accepted wholesale (DRBD+Corosync)
Moves from allowedTCPPorts/allowedUDPPorts to source-restricted extraCommands
rules on the nixos-fw chain.

NFS exports reduced to LAN-only (drop haClientCidr lines from mkNfsExports).
Storage-client network clients use iSCSI only; no NFS on VLAN 20.

All HA script VIP/storage IP defaults updated to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 18:19:07 +10:00
beatzaplentyandClaude Sonnet 4.6 5eea38d3ca fix(flake): fix /etc/flake-target on ha-server-{1,2} so Switch-nix alias works
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
mkTarget computed flakeTarget as "${platform}-${buildType}" which gives
"proxmox-ha-server" for both nodes, but the actual nixosConfigurations
keys are "proxmox-ha-server-1" / "proxmox-ha-server-2".  This caused
Switch-nix to fail immediately with an "unknown flake attribute" error.

Add an optional nameSuffix parameter to mkTarget and pass "-1" / "-2"
for the ha-server entries so /etc/flake-target matches the real
attribute name on each node.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 16:25:51 +10:00
beatzaplentyandClaude Sonnet 4.6 6f38ad67e0 feat(ha): add dual VIP — retain 192.168.2.229 LAN alongside 192.168.5.229 storage
Check NixOS configurations / eval-hosts (push) Successful in 10m35s
Both VIPs fail over together in ha-group:
  vip-lan:     192.168.2.229/24 on ens18 (vmbr0) — LAN iSCSI + NFS access
  vip-storage: 192.168.5.229/24 on ens20 (vmbr2) — storage-client iSCSI + NFS

variables.nix: haServerLanVip = "192.168.2.229"
cluster-init.sh: both VIP primitives in ha-group XML; VIP_LAN var added
ip-addressing.md: both VIP entries documented

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 16:18:45 +10:00
beatzaplentyandClaude Sonnet 4.6 8d43b7039c feat(ha): add vmbr2 storage-client network and corosync ring1
Check NixOS configurations / eval-hosts (push) Successful in 10m39s
Infrastructure changes already applied to pve1:
- vmbr2 internal bridge created (192.168.5.0/24, no physical uplink)
- VM 200 (ha-server-1): net2 added → vmbr2 (ens20)
- VM 201 (ha-server-2): net2 added → vmbr2 (ens20)
- CT 105 (docker): net1 added → vmbr2 (eth1)
- VM 101 (server): net1 added → vmbr2 (ens19) — needs reboot to activate

NixOS config (deploy to ha nodes to complete; docker/server at cutover):
- ha-server-{1,2}/host.nix: ens20 with 192.168.5.{228,227}/24
- docker/host.nix: eth1 with 192.168.5.225/24
- server/host.nix: ens19 with 192.168.5.226/24
- cluster-config.nix: corosync ring1 on LAN IPs as backup heartbeat path
- cluster-config.nix: allow haClientCidr (192.168.5.0/24) in iptables
- ha-server.nix: NFS exports now allow both lanCidr and haClientCidr
- VIP moves from 192.168.2.229 (vmbr0/LAN) to 192.168.5.229 (vmbr2)
- iSCSI portal to be rebound from [::0] to 192.168.5.229 at cutover

variables.nix: haStorageCidr corrected to 192.168.4.224/29; new vars:
vmStorageClientInterface, lxcStorageInterface, haServer{1,2}ClientIp,
dockerStorageIp, serverStorageIp, haClientCidr/PrefixLength; haServerVip
updated to 192.168.5.229.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 16:02:53 +10:00
beatzaplentyandClaude Sonnet 4.6 8e58226d2b docs(ha): add network segmentation audit report
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Read-only discovery of pve1 bridges, HA cluster VMs (200/201), and
Docker CT (105) — current state, risk flags, recommended vmbr2 storage
bridge layout, gap list, and migration notes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 15:38:21 +10:00
beatzaplenty a5fb0404e2 Merge pull request 'fix(gc-hosts): filter discovered guests against flake to exclude non-NixOS hosts' (#107) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m19s
Reviewed-on: #107
2026-07-29 04:08:05 +00:00
beatzaplentyandClaude Sonnet 4.6 4ba9b141fe fix(gc-hosts): filter discovered guests against flake to exclude non-NixOS hosts
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m32s
pve1 runs non-NixOS guests (FreeIPA domain-controller, Proxmox Data Manager, etc.)
alongside NixOS ones. Validate each discovered hostname against the set of
hostnames defined in nixosConfigurations before adding it to the gc list.

Brings back the flake eval but uses it correctly: extract all hostname values
(not target-name keys) to build a filter set, then only include pve1 guests
whose name matches a flake-managed NixOS host.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 14:06:17 +10:00
beatzaplenty f3c2965f78 Merge pull request 'fix(gc-hosts): use pct/qm list output directly as hostnames' (#106) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
Reviewed-on: #106
2026-07-29 04:02:50 +00:00
beatzaplenty b9d3b51028 Merge pull request 'fix(ha): resolve data disk device via by-id symlink even in dry-run' (#105) from worktree-partitioned-swimming-pizza into main
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Reviewed-on: #105
2026-07-29 04:01:49 +00:00
beatzaplentyandClaude Sonnet 4.6 539bdf9833 fix(ha): resolve data disk device via by-id symlink even in dry-run
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m21s
The by-id lookup is read-only so it's safe to run in dry-run mode.
Previously it was gated behind `if ! $DRY_RUN`, which always triggered
the sdb fallback warning in dry-run — making it look like the device
path wasn't reliable when the symlink actually exists on both servers.

Now the lookup always runs and the script errors out with a clear message
if the by-id symlink is genuinely missing, instead of silently falling
back to a guessed /dev/sd* name.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:58:53 +10:00
beatzaplentyandClaude Sonnet 4.6 a5308a7ee5 fix(gc-hosts): use pct/qm list output directly as hostnames
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m32s
The flake hostname map approach was wrong: create-proxmox-resource.sh names
every guest after its NixOS hostname (pct create --hostname <nixos-hostname>,
qm create --name <nixos-hostname>), so pct/qm list already output the final
hostname directly. Looking those names up in a map keyed by flake target names
(lxc-docker, proxmox-server, ...) found nothing and the dynamic list was
always empty.

Drop the flake eval entirely; use the pct/qm list output as-is, filtering
only nix-cache and already-seen entries.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:58:45 +10:00
beatzaplenty dc6c37ed2e Merge pull request 'fix(ha): prefix qm commands with sudo for non-root Proxmox SSH user' (#104) from worktree-partitioned-swimming-pizza into main
Check NixOS configurations / eval-hosts (push) Failing after 4m22s
Reviewed-on: #104
2026-07-29 03:55:28 +00:00
beatzaplenty b5e61d62bd Merge pull request 'fix(gc-hosts): fix pve1 PATH and discovery sudo prompts' (#103) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Reviewed-on: #103
2026-07-29 03:53:50 +00:00
beatzaplentyandClaude Sonnet 4.6 ced1657407 fix(ha): prefix qm commands with sudo for non-root Proxmox SSH user
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m25s
qm lives at /usr/sbin/qm, which is not in the default PATH for
non-interactive SSH sessions as a non-root user.  Add PVE_SUDO (set to
"sudo" when PVE_SSH_USER != root, matching create-proxmox-resource.sh's
own sudo_prefix pattern) and prepend it to all three qm invocations in
the script.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:53:41 +10:00
beatzaplentyandClaude Sonnet 4.6 7b4794211d fix(gc-hosts): fix pve1 PATH and discovery sudo prompts
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m45s
Two bugs:

1. Dynamic discovery used BatchMode=yes for the pct/qm list SSH call, which
   silently suppressed any sudo password prompt and returned empty output.
   Fix: split SSH_OPTS into SSH_OPTS (BatchMode, for gc) and SSH_QUERY_OPTS
   (no BatchMode, for discovery) so sudo can prompt when needed.

2. pve1 gc used a non-login SSH session which doesn't source /etc/profile,
   so nix-collect-garbage was never on PATH. The Nix installer registers
   itself via /etc/profile.d/nix-daemon.sh which only runs in login shells.
   Fix: use "bash -l -c 'nix-collect-garbage -d'" instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:53:02 +10:00
beatzaplenty fdf41c659c Merge pull request 'Worktree crispy churning kernighan' (#102) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m50s
Reviewed-on: #102
2026-07-29 03:46:05 +00:00
beatzaplentyandClaude Sonnet 4.6 0ef8259225 fix(gc-hosts): source nix-daemon profile on pve1 before running gc
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
BatchMode SSH sessions don't source /etc/profile on non-NixOS hosts, so
nix-collect-garbage isn't on PATH for the wayne user. Source the nix-daemon
profile script explicitly, matching the pattern in scripts/lib/nix-bootstrap.sh.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:45:48 +10:00
beatzaplentyandClaude Sonnet 4.6 629c1457a9 refactor(gc-hosts): discover running pve1 guests dynamically each run
Replace the static host list with dynamic discovery: workstation (nixos)
and pve1 are hard-wired first and second; remaining hosts are discovered
on every run by SSHing to pve1, listing running VMs/containers via
pct/qm list, and resolving their NixOS hostnames from a single flake eval.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:44:26 +10:00
beatzaplenty 7ba603c005 Merge pull request 'feat(scripts): add gc-hosts.sh for parallel nix gc across all live hosts' (#101) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #101
2026-07-29 03:36:09 +00:00
beatzaplentyandClaude Sonnet 4.6 decf3ddff9 feat(scripts): add gc-hosts.sh for parallel nix gc across all live hosts
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m29s
Runs nix-collect-garbage -d on all deployed NixOS hosts and pve1 in
parallel, skipping nix-cache to avoid evicting shared cache paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:35:24 +10:00
beatzaplenty 4490dfab7d Merge pull request 'feat(nix-cache): authorize pve1's wayne key as remote builder client' (#100) from worktree-pve1-nix-cache-builder into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #100
2026-07-29 03:16:53 +00:00
beatzaplentyandClaude Sonnet 4.6 fa163b613d feat(nix-cache): authorize pve1's wayne key as remote builder client
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
Generated /home/wayne/.ssh/id_ed25519 on pve1 (single-user Nix install
owned by wayne, no daemon — builds run as wayne not root). Adds the
public key to remoteBuilderAuthorizedKeys so nix-cache accepts it.

After deploying nix-cache, re-run configure-nix-cache-client.sh as
wayne on pve1 to wire in the builder line and known_hosts entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:16:30 +10:00
beatzaplenty 15bc5bd369 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
2026-07-29 13:02:03 +10:00
beatzaplenty 5a4fbbf7ad enable beszel agent 2026-07-29 13:01:53 +10:00
beatzaplenty e720bbb018 Merge pull request 'fix(nfs): mask rpc-svcgssd on server and ha-server build types' (#99) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Failing after 9m46s
Reviewed-on: #99
2026-07-29 02:54:12 +00:00
beatzaplentyandClaude Sonnet 4.6 40cdf724b4 fix(nfs): mask rpc-svcgssd on server and ha-server build types
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m43s
nfs-utils changed ConditionPathExists from !/etc/krb5.keytab to
/etc/krb5.keytab, so on IPA-joined hosts the service now starts instead
of skipping. Exports use standard auth (no sec=krb5) so the nfs/ Kerberos
principal is never provisioned and the service fails.

enable = false masks the unit so nfs-server's Wants= can't pull it in.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 12:51:50 +10:00
beatzaplenty aaa193645b Merge pull request 'refactor(beszel): move KEY and TOKEN fully into enable-agent.nix' (#98) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Successful in 10m45s
Reviewed-on: #98
2026-07-29 02:38:19 +00:00
beatzaplentyandClaude Sonnet 4.6 f22ff7db79 refactor(beszel): move KEY and TOKEN fully into enable-agent.nix
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m59s
KEY (hub's SSH public key) is now stored in variables.nix as beszelHubKey
and set by enable-agent.nix — no host file ever needs to touch it.
TOKEN was already universal; this removes the last per-host beszel config.

Importing enable-agent.nix in a build type is now the only step needed to
add a new host to beszel monitoring.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 12:37:54 +10:00
beatzaplenty 0da3c3070f updated flake.lock and added beszel agent environment
Check NixOS configurations / eval-hosts (push) Failing after 16m56s
2026-07-29 12:18:38 +10:00
beatzaplenty e1551feefd enabled beszel-agent
Check NixOS configurations / eval-hosts (push) Successful in 10m34s
2026-07-29 12:14:49 +10:00
beatzaplentyandClaude Sonnet 4.6 b4bc30cb2c chore(secrets): remove per-host beszel-token secrets
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
Five files contained only beszel-token and are deleted entirely.
nix-cache.yaml retains cache-priv-key; beszel-token removed from it.
The universal token in secrets/common.yaml replaces all of these.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 12:00:38 +10:00
beatzaplenty 5c8d55bd78 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Failing after 4m33s
2026-07-29 11:50:44 +10:00
beatzaplenty 999c9a3151 add beszel universal token 2026-07-29 11:50:01 +10:00
beatzaplenty 1e0fff1b26 Merge pull request 'refactor(beszel): switch to universal token via secrets/common.yaml' (#97) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Failing after 13m26s
Reviewed-on: #97
2026-07-29 01:47:15 +00:00
beatzaplentyandClaude Sonnet 4.6 24c6469f10 refactor(beszel): switch to universal token via secrets/common.yaml
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m40s
Replace per-host host-token.nix imports with a single beszel-token secret
in secrets/common.yaml, wired once in enable-agent.nix. Host files now only
need services.beszel.agent.environment.KEY — no imports block required.

Delete modules/beszel/host-token.nix (no longer referenced anywhere).

Action needed: run `sops secrets/common.yaml` and add `beszel-token: <value>`
from the beszel hub UI before deploying.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 10:58:15 +10:00
beatzaplenty c9458ac8a6 Merge pull request 'docs: add beszel agent setup guide' (#96) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
Reviewed-on: #96
2026-07-29 00:41:22 +00:00
beatzaplentyandClaude Sonnet 4.6 99ba0ed52b docs: add beszel agent setup guide
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s
Documents how to wire beszel-agent into any flake build: the two-module
split (enable-agent.nix in build types, host-token.nix in host files),
sops secret creation, hub pairing flow, and optional EXTRA_FILESYSTEMS /
Docker monitoring knobs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 10:41:03 +10:00
beatzaplenty 1e4072029e update rmote-builder keys
Check NixOS configurations / eval-hosts (push) Successful in 10m38s
2026-07-29 10:39:53 +10:00
beatzaplenty 46649fc7e0 update remote-builder client keys
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
2026-07-29 10:29:40 +10:00
beatzaplenty 4cbcc3beb9 updated .gitignore and change default editor in home.nix
Check NixOS configurations / eval-hosts (push) Failing after 13m5s
2026-07-29 10:17:37 +10:00
beatzaplenty 35f696ccd5 Merge pull request 'feat(ha): add resize-data-disk.sh for online data disk expansion' (#95) from worktree-ha-resize-disk into main
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
Reviewed-on: #95
2026-07-29 00:01:49 +00:00
beatzaplentyandClaude Sonnet 4.6 e18b605706 feat(ha): add resize-data-disk.sh for online data disk expansion
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m27s
Three-phase online resize: Proxmox qm resize → guest block device
rescan → drbdadm resize + xfs_growfs. No downtime required.
Detects active node via crm_mon, pre-checks DRBD UpToDate/UpToDate,
auto-discovers VMIDs from qm list, and resolves the block device name
from the stable scsi-0QEMU... by-id symlink with a slot-index fallback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 09:59:39 +10:00
beatzaplenty ee2451d87c remove stray pihole backup
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
2026-07-29 09:50:50 +10:00
beatzaplentyandClaude Sonnet 4.6 71a6c4738c fix(ha): fix crm_mon active-node detection in all three scripts
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
crm_mon 2.x formats the Promoted line as "    * Promoted: [ node ]" — the
asterisk bullet means the previous grep -E '^\s*(Promoted|Masters):' never
matched, so active-node detection silently returned empty in health.sh,
failover.sh, and acceptance-tests.sh.

Fix: pipe through grep -v Unpromoted first, then grep -E '(Promoted|Masters):'
without anchoring to start-of-line.

Also: remove the SSH_OPTS=-i ~/.ssh/... variable pattern in health.sh and
failover.sh; tilde is not expanded inside double-quoted strings, so $SSH_OPTS
was passing a literal ~ to SSH.  Inline the key path in each function
definition instead (same as acceptance-tests.sh already did).

Also: drop the 2>/dev/null embedded in the crm_mon argument string — the
outer 2>/dev/null on the n1/n2 call already suppresses SSH stderr; the
embedded one was harmless but noisy to reason about.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 02:36:22 +10:00
beatzaplentyandClaude Sonnet 4.6 793a2b5924 feat(ha): add failover.sh and health.sh operator scripts
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
failover.sh: graceful active→standby resource migration with pre-checks
(DRBD UpToDate/UpToDate, quorum), spinner wait for XFS mount on target,
rollback on timeout, --to/--force/--dry-run/--timeout flags.

health.sh: read-only status panel — node reachability, quorum, DRBD
role/dstate/cs per node, Pacemaker resources, failure history, XFS mount
usage, and service port checks (NFS 2049, iSCSI 3260) via VIP.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 02:23:47 +10:00
beatzaplenty 98b5429fb9 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 02:04:51 +10:00
beatzaplentyandClaude Sonnet 4.6 f658bdbabc fix(ha/cluster-init): enable Pacemaker maintenance-mode during DRBD sync
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Even with both nodes in standby, Pacemaker's monitor operations keep running.
When the monitor sees DRBD is Primary on a standby node (that it didn't start),
it triggers a stop action — killing the initial sync after ~10 s.

Enable maintenance-mode after standby stops the existing resources but before
DRBD is manually brought up for the sync.  Maintenance-mode suspends all
start/stop/monitor actions so Pacemaker is completely hands-off during the
sync.  Disable it alongside crm_standby -v off once UpToDate/UpToDate is
confirmed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:49:15 +10:00
beatzaplentyandClaude Sonnet 4.6 d7e63cd80e fix(ha/cluster-init): fix DRBD metadata skip condition and StandAlone grace period
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
The metadata creation check used grep -q "UpToDate" which matches when the
resource is DOWN — drbdadm dstate reads metadata directly and returns just
"UpToDate" (no slash) for a down-but-previously-synced resource.  This caused
metadata creation to be silently skipped, leaving mismatched UUIDs from an
interrupted sync which cause DRBD to go WFConnection→StandAlone immediately.

Fix: require exact "UpToDate/UpToDate" (with slash, only possible when the
resource is UP and both disks are confirmed synced) before skipping.

Also give the StandAlone detection a 5-iteration (15 s) grace period before
dying, to avoid false-positive on the first few ticks while the peer is still
coming up.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:45:03 +10:00
beatzaplentyandClaude Sonnet 4.6 b88ea49880 fix(ha/cluster-init): keep Pacemaker in standby until DRBD sync completes
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Clearing crm_standby before the initial sync finished caused Pacemaker's OCF
DRBD agent to race with the manual drbdadm up/primary calls.  The agent saw
DRBD in WFConnection or SyncSource and tore it down, driving the resource back
to StandAlone and killing the sync in seconds.

Move the crm_standby -v off calls to immediately after the sync-complete
break, so Pacemaker only resumes once DRBD is UpToDate/UpToDate and safe
to hand back.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:41:15 +10:00
beatzaplentyandClaude Sonnet 4.6 c7268ade8e fix(ha/cluster-init): detect StandAlone DRBD state in sync wait loop
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
If the DRBD peer connection drops during the initial sync wait (cs:StandAlone),
the loop would spin forever printing "waiting for sync progress" with no
indication of what's wrong. Now parses cs: from /proc/drbd each tick and
dies with a clear error if StandAlone is detected. Also shows the cs: field
in the non-syncing message so the actual connection state is always visible.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:29:04 +10:00
beatzaplentyandClaude Sonnet 4.6 367158548e fix(ha): wrap crm-fence-peer.sh to set PATH for kernel UMH callout
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
When the DRBD kernel module invokes fence-peer via the User Mode Helper
mechanism it provides a minimal PATH that omits /run/current-system/sw/bin.
crm-fence-peer.sh calls cibadmin, crm_mon etc.; without those in PATH a
pipeline breaks with SIGPIPE.  A signal-killed process has WEXITSTATUS()==0,
so the kernel sees exit code 0, logs "fence-peer helper broken, returned 0",
and retries forever — blocking Pacemaker failover completely.

Fix: use pkgs.writeShellScript to create thin PATH-fixing wrappers in the
Nix store.  The wrappers prepend /run/current-system/sw/bin before exec-ing
the real crm-fence-peer.sh / crm-unfence-peer.sh, giving them a working
Pacemaker toolchain regardless of what PATH the kernel provides.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:22:46 +10:00
beatzaplenty 585126beee secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 01:11:23 +10:00
beatzaplentyandClaude Sonnet 4.6 ceb491b18d fix(ha/cluster-init): drop TTY detection, always use \r for sync progress
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
The TTY check ([[ -t 1 ]]) evaluated false under sudo, causing each update
to print on a new line via log(). Since \r worked correctly in the original
code (user confirmed), drop the branching and always use printf \r.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:55:31 +10:00
beatzaplentyandClaude Sonnet 4.6 6d16eaff89 fix(ha/cluster-init): fix /proc/drbd sync progress parsing and TTY display
Check NixOS configurations / eval-hosts (push) Successful in 10m29s
The sync progress grep was silently failing because /proc/drbd formats the
percentage line as "sync'ed:  5.2%" (two spaces after colon), but the
pattern matched only one.  Use [[:space:]]+ throughout to tolerate any
whitespace.

Also make the display TTY-aware: interactive sessions get the spinning
\r in-place update; piped/logged sessions get a plain log line printed
once per unique percentage point (avoiding scroll spam).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:51:35 +10:00
beatzaplentyandClaude Sonnet 4.6 e085d4707c fix(ha): add DRBD sync progress indicator and pre-flight sync gate
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
cluster-init.sh: replace the fixed 300-iteration sync wait with an
indefinite loop that prints an in-place progress line (%done, ETA, speed)
from /proc/drbd every 3 s. Clears the line with printf \r before logging
completion, so the output stays clean alongside the [cluster-init] log
lines.

acceptance-tests.sh: add a pre-flight check that hard-exits if
drbdadm dstate is not UpToDate/UpToDate, with a hint to the watch
command for monitoring progress. Tests cannot give accurate results
while the initial full-sync is in progress.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:46:57 +10:00
beatzaplenty c8d4440787 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:40:11 +10:00
beatzaplentyandClaude Sonnet 4.6 e7531b276e fix(ha): use crm_mon for Active node detection and add cleanup before VIP wait
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
acceptance-tests.sh: replace drbdadm-role-based Active node detection with
crm_mon-based detection. DRBD role briefly reads Secondary during Pacemaker
transitions, causing T2/T3/T4 to target the wrong node. crm_mon Promoted:/
Masters: lines are authoritative; wait up to 90 s for Pacemaker to settle
before falling back to NODE1.

cluster-init.sh: add crm_resource --cleanup before the VIP wait loop so
stale migration-threshold failure counters from previous cluster-init runs
are cleared before Pacemaker decides resource placement.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:24:47 +10:00
beatzaplenty 49a10d7cc5 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:16:51 +10:00
beatzaplentyandClaude Sonnet 4.6 ead4f55805 fix(ha): use stable by-id disk path for DRBD instead of /dev/sd*
Check NixOS configurations / eval-hosts (push) Failing after 9m44s
/dev/sda and /dev/sdb are assigned by the OS based on Proxmox disk-add
order, which is not consistent across VMs. Use the SCSI controller path
instead — drive-scsi1 is always the dedicated data disk on all HA nodes
regardless of which sda/sdb it gets assigned to.

variables.nix: replace per-node haServer{1,2}DrbdDisk with a single
haServerDrbdDisk using /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1.

cluster-config.nix: revert to shared volume block (per-node block was
needed for /dev/sd* but the by-id path is identical on both nodes).

cluster-init.sh:
- single DRBD_DISK variable (matching haServerDrbdDisk)
- robust /etc/drbd.conf patch: NixOS manages this file as a symlink to
  a read-only Nix store path; cp --remove-destination breaks the symlink
  before sed -i so the edit actually takes effect
- scp helper script to NODE2 rather than bash -c over SSH to avoid
  quoting complexity

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:11:13 +10:00
beatzaplenty cac5ec45cc secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:02:48 +10:00
beatzaplentyandClaude Sonnet 4.6 ec90753a09 fix(ha): per-node DRBD disk assignment (NODE2 data disk is /dev/sda not /dev/sdb)
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Proxmox VM disk ordering differs between the two HA nodes:
  ha-server-1: sda=OS (50G), sdb=DRBD data (32G)
  ha-server-2: sda=DRBD data (32G), sdb=OS (50G)

The DRBD resource config was using a shared disk=/dev/sdb which targeted
the OS disk on ha-server-2, causing drbdmeta and drbdadm up to operate
on the mounted root filesystem (hence "Device or resource busy").

Changes:
- variables.nix: add haServer1DrbdDisk/haServer2DrbdDisk
- cluster-config.nix: move volume block inside per-host on{} sections
  so each node uses the correct backing disk
- cluster-init.sh: use NODE1_DRBD_DISK/NODE2_DRBD_DISK variables;
  add runtime check that patches /etc/drbd.d/*.res on the running nodes
  if the deployed config points to the wrong disk (workaround for VMs
  built before this fix; redeploy with --force-rebuild to make permanent)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:57:07 +10:00
beatzaplenty df1ddee735 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:48:23 +10:00
beatzaplentyandClaude Sonnet 4.6 8e8261be31 fix(ha/cluster-init): drop bash -c wrapper for NODE2 drbdmeta calls
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
drbdmeta is in /run/current-system/sw/bin which is already on sudo's
PATH on the HA nodes; the bash -c wrapper with PATH manipulation was
adding complexity without benefit and introduced quoting hazards that
likely caused write-dev-uuid to fail or run in the wrong context.

Use direct n2_ssh calls instead — the same pattern that works when
tested manually on NODE2.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:42:36 +10:00
beatzaplenty 5e9541741f secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:30:12 +10:00
beatzaplentyandClaude Sonnet 4.6 07543d7d56 fix(ha/cluster-init): add drbdmeta to PATH and remove openssl dependency
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
drbdmeta lives in the same Nix store dir as drbdadm but sudo doesn't
inherit the full PATH, so drbdmeta was not found (exit 127) even though
drbdadm was.  Resolve drbdmeta's directory from drbdadm's location and
prepend it to PATH.

Replace openssl rand for UUID generation with /proc/sys/kernel/random/uuid
— openssl is not guaranteed to be on PATH in a minimal NixOS root
environment, but /proc/sys/kernel/random/uuid is always present.

Apply the same PATH fix on NODE2 inline in the bash -c invocations that
call drbdmeta over SSH.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:24:01 +10:00
beatzaplentyandClaude Sonnet 4.6 a7c4a24fc3 fix(ha): use drbdmeta --force directly; fix T4 grep-c arithmetic bug
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
cluster-init.sh: replace 'drbdadm create-md --force' with direct drbdmeta
calls using --force on both steps.  drbdadm create-md --force passes --force
to the create-md sub-call but NOT to write-dev-uuid, which then fails when
/dev/sdb is busy and stdin is not a TTY ("stdin not a TTY, not waiting for
confirmation" → exit 20).  write-dev-uuid failing means DRBD has no UUID,
so the peer can't identify the device → connection fails → no sync → wrong
Active node detected by acceptance tests.

acceptance-tests.sh T4: grep -c returns exit code 1 when the count is 0
(no matches), triggering '|| echo "0"' and producing "0\n0" which breaks
[[ "$IQN_COUNT" -ge 1 ]] with "arithmetic syntax error".  Fixed by running
the pipe inside bash -c with '|| true' on the grep to suppress the
non-zero exit code.  Same fix applied to T5's IQN_ON_STANDBY check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:09:06 +10:00
beatzaplenty a98955a8da secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:08:32 +10:00
beatzaplenty 7d60741e73 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:00:18 +10:00
beatzaplenty 427be2b287 secrets(ha): refresh sops host-key registrations for new VM instances 2026-07-28 22:55:02 +10:00
beatzaplentyandClaude Sonnet 4.6 684351b89b fix(ha): pre-seed SSH host key in disko image; fix DRBD init race
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
Root cause of recurring sops failures on new VM boots: disko builds raw
disk images, and create-proxmox-resource.sh only syncs the clan-var SSH
host key to the Proxmox node (for proxmox.nix to bake into the image) when
it actually builds — reusing a cached image skips sync_remote_host_keys, so
destroy+recreate reuses a stale image with the wrong or randomly-generated
key baked in.  On first boot the VM gets a different key than what .sops.yaml
was encrypted for, and sops fails permanently.

Fix 1 — deploy.sh Phase 3: always pass --force-rebuild so every VM creation
rebuilds the disko image fresh with the current clan-var key baked in via
NIXOS_HOST_KEYS_DIR (proxmox.nix already reads this under --impure).

Fix 2 — deploy.sh Phase 5.5: after VMs boot, scan their actual ed25519 host
keys and, if they drift from clan vars, update the clan var pub-key files,
rewrite the .sops.yaml age anchors, and re-encrypt all affected sops files.
Defence-in-depth: normally a no-op after Fix 1, but catches any residual
mismatch (e.g. --skip-create-vms reuse of an older image).

Fix 3 — cluster-init.sh: add crm_standby -v on for both nodes before DRBD
metadata init.  Without this, Pacemaker's OCF DRBD agent races: it sees
drbdadm down as a failure and immediately calls drbdadm up again, leaving
/dev/sdb busy when create-md / write-dev-uuid runs (drbdmeta exits 20 with
"stdin not a TTY, not waiting for confirmation").  Standby suppresses
resource scheduling during init; crm_standby -v off restores it after DRBD
is up on both nodes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 22:51:29 +10:00
beatzaplenty 327ff0b44d secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 22:27:29 +10:00
beatzaplenty 43b4cc6aa6 secrets(ha): regenerate IPA keytabs for ha-server-1 and ha-server-2
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
2026-07-28 21:54:00 +10:00
beatzaplentyandClaude Sonnet 4.6 dd984019a1 fix(ha): prevent drbd.service from starting alongside Pacemaker
Check NixOS configurations / eval-hosts (push) Failing after 9m47s
drbd.service runs drbdadm up all at activation time, but Pacemaker's
OCF drbd agent manages the resource lifecycle (up/down/promote/demote).
When both run simultaneously the device is busy, causing drbdmeta
apply-al to abort with exit 20.

Set wantedBy = [] so the kernel module and config are still present
(via services.drbd.enable = true) but systemd doesn't auto-start the
service. Pacemaker's OCF agent calls drbdadm directly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 21:48:43 +10:00
beatzaplenty cb8a51b2fd updated sops keys
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
2026-07-28 21:41:31 +10:00
beatzaplenty 18ab0ff254 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 21:19:22 +10:00
beatzaplenty 5c5f22f84a +x on scripts and remove beszel from ha for now
Check NixOS configurations / eval-hosts (push) Failing after 5m35s
2026-07-28 20:48:51 +10:00
beatzaplenty 094eaa752b secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 20:35:12 +10:00
beatzaplentyandClaude Sonnet 4.6 fe9fc7364b fix(ha/cluster-init): explicitly start pacemaker after corosync restart
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
At first boot, pacemaker fails with 'Dependency failed' because ipa-activation
fails (IPA not enrolled yet) before corosync/authkey are ready, causing a
systemd boot-ordering race.  The service recovers fine when started manually.
Add an explicit 'systemctl start pacemaker' on both nodes immediately after
restarting corosync so cluster-init doesn't time out waiting for it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:47:33 +10:00
beatzaplentyandClaude Sonnet 4.6 1ce4589830 fix(ha/deploy): remove _patch_targetctl — iscsi-target.nix now correct
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
The _patch_targetctl runtime dropin was a workaround for the wrong targetctl
binary path in the original iscsi-target.nix (used pkgs.targetcli-fb, but
targetctl is in rtslib-fb's python3 env).  The module is now fixed to use
\${python3}/bin/targetctl, so new VM images have the correct ExecStart/ExecStop
baked in and the patch finds nothing — failing with exit 1 and killing the deploy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:41:20 +10:00
beatzaplenty 7417cc1b0a Merge pull request 'feat(gui): add wayne's GUI Home Manager config matching the nixos user' (#94) from worktree-functional-mapping-russell into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #94
2026-07-28 09:29:28 +00:00
beatzaplentyandClaude Sonnet 4.6 ec44b7955b feat(gui): add wayne's GUI Home Manager config matching the nixos user
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
Adds nextcloud-client, chromium, git, vim, claude-code, fish, sops,
the Nextcloud tray service, SOPS_AGE_KEY_FILE, and Proxmox/PBS desktop
shortcuts to wayne's HM config on GUI hosts. Uses lib.mkDefault on the
EDITOR setting in ipa/client.nix so the GUI layer can override to vim.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 19:28:22 +10:00
beatzaplentyandClaude Sonnet 4.6 cf3d8ea9a5 fix(sops): resolve merge conflicts in ha-server keytab files
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
Both secrets/ha-server-{1,2}.keytab had unresolved conflict markers from
the stash/merge interaction.  Keep the upstream version (2 age recipients:
admin + the host's own key) so each host can decrypt its own keytab on boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:26:39 +10:00
beatzaplentyandClaude Sonnet 4.6 756e45743c fix(sops): resolve merge conflict in .sops.yaml keytab recipients
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
The HA branch merge left conflict markers in the ha-server-{1,2}.keytab
path_regex entries.  Keep the upstream side (adds *proxmox-ha-server-{1,2}
as recipients) so each host can decrypt its own IPA keytab on boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:23:07 +10:00
beatzaplenty 7e8755d141 added ipa keytabs for ha servers
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
2026-07-28 19:12:23 +10:00
beatzaplenty 8194707478 Merge pull request 'Worktree ha file server test' (#93) from worktree-ha-file-server-test into main
Check NixOS configurations / eval-hosts (push) Successful in 10m37s
Reviewed-on: #93
2026-07-28 09:09:04 +00:00
beatzaplentyandClaude Sonnet 4.6 d740064a35 fix(ha): all 7 acceptance tests pass — targetctl, fencing, failover, data integrity
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s
Deploy/init fixes:
- iscsi-target.nix: targetctl binary is in rtslib-fb (python3 env), not
  targetcli-fb — fixes ExecStart and ExecStop for the targetctl.service
- deploy.sh: _patch_targetctl() applies runtime dropin to both nodes before
  cluster-init so Pacemaker can manage the iSCSI target from first start
- cluster-init.sh: replace crm configure heredoc with cibadmin --replace XML
  (pacemaker-4.0 schema: globally-unique in meta_attributes, promoted-max/
  promoted-node-max, Promoted role in constraints); force_unmount=true on
  xfs-data; DRBD promote timeout 240s
- cluster-config.nix: add crm-fence-peer.sh/crm-unfence-peer.sh handlers;
  update fencing comment to reflect resource-only + Pacemaker-aware handler
  replacing STONITH during testing phase
- ha-server.nix: add openiscsi to systemPackages for T4 iscsiadm availability

Acceptance test fixes:
- acceptance-tests.sh: fix ((PASS++)) set -e bug → PASS=$((PASS+1));
  detect Active/Standby dynamically via drbdadm role (Pacemaker can promote
  either node); T4 bash TCP probe instead of iscsiadm; T5 timeout 120s;
  T6 echo|sudo tee for root-owned XFS write (bash -c redirect runs as nixos
  not sudo — permission denied); use ns cat / ns rm for root-owned reads

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:03:39 +10:00
beatzaplenty 164d14eb87 Merge pull request 'fix(docker): declaratively apply groupmod --non-unique for IPA GID conflict' (#92) from worktree-parsed-mapping-raven into main
Check NixOS configurations / eval-hosts (push) Successful in 10m36s
Reviewed-on: #92
2026-07-28 08:19:53 +00:00
beatzaplentyandClaude Sonnet 4.6 720399b00d fix(docker): declaratively apply groupmod --non-unique for IPA GID conflict
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
NixOS's group activation uses plain groupmod, which silently skips the
GID change when SSSD exposes the same GID (50010) via the IPA
docker-access group through NSS.  Add an activation script that runs
after the normal 'groups' step and applies groupmod --non-unique so the
local docker group can share GID 50010 with the SSSD-provided IPA group.
If the GID actually changes on a live system the script also restarts
docker.socket + docker.service so the socket is recreated with the new
GID.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 18:17:43 +10:00
beatzaplentyandClaude Sonnet 4.6 da4d808c6a fix(ha/cluster-init): use crm configure instead of cibadmin XML for resources
cibadmin raw XML fails schema validation under pacemaker-4.0: globally-unique
is not a valid direct <clone> attribute, and master-max/master-node-max are
renamed.  Switch to crm configure commands which are schema-version-aware:
- promotable clone with promoted-max/promoted-node-max
- order/colocation constraints using Promoted role (Pacemaker 4.0 naming)
- crm configure handles schema differences automatically

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:57:49 +10:00
beatzaplentyandClaude Sonnet 4.6 79cde50e27 fix(ha/cluster-init): use targetcli delete commands for LIO teardown
clearconfig does not reliably clear kernel LIO configfs state — the kernel
still holds backing-file references, blocking umount.  Replace clearconfig
with explicit targetcli delete commands (/iscsi delete, /backstores/fileio
delete) which do release kernel state.  Also make DRBD secondary demote
idempotent (skip if already Secondary).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:55:26 +10:00
beatzaplentyandClaude Sonnet 4.6 d31d9fa584 fix(ha/cluster-init): fix LIO teardown, mount idempotency, VIP portal
- Remove VIP-specific portal binding — the VIP doesn't exist until Pacemaker
  assigns it; the default all-IPs portal (::0:3260) is correct for Pacemaker
  to manage
- Clear existing LIO targets before re-running targetcli (idempotent on
  partial failures)
- Tear down LIO kernel objects after saveconfig so umount succeeds (LIO holds
  the backing file open otherwise)
- Guard mount with mountpoint check so re-runs don't fail when already mounted
- Use --replace for cibadmin constraints (idempotent vs --create)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:50:24 +10:00
beatzaplentyandClaude Sonnet 4.6 c76698efb7 fix(ha): add xfsprogs to system packages; fix cluster-init PATH and DRBD check
- ha-server.nix: add xfsprogs to systemPackages so mkfs.xfs is on PATH for
  root (needed by cluster-init.sh during initial setup)
- cluster-config.nix: create /var/lib/drbd via tmpfiles to silence
  lk_bdev_save warnings from drbd-utils
- cluster-init.sh: dynamically find xfsprogs in /nix/store if not on PATH
  (fallback for running VMs before xfsprogs is in the system profile)
- cluster-init.sh: fix DRBD metadata check on node2 — broken regex now uses
  grep -E for ERE alternation to correctly skip create-md when DRBD is already
  set up (previous regex would have triggered create-md on a live secondary)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:43:27 +10:00
beatzaplenty 601db79689 Merge pull request 'fix(ipa): restore IPA password login via LightDM and su' (#91) from worktree-parsed-mapping-raven into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #91
2026-07-28 07:31:46 +00:00
beatzaplentyandClaude Sonnet 4.6 acebbdbe26 fix(ha/deploy): use nixos+sudo instead of root SSH; temp key for inter-node comms
Root SSH was failing because only the RSA admin key was authorized but the
local dev box only has an ed25519 key. Fix:

- cluster-config.nix: add ed25519 keys to root (same set as nixos user) so
  future deployments work without the temp-key workaround
- deploy.sh/acceptance-tests.sh: SSH as nixos user with sudo instead of root@
- cluster-init.sh: HA_USER/HA_KEY env vars + n2_ssh()/n2_scp() helpers so
  inter-node SSH works regardless of whether root-to-root is available
- deploy.sh Phase 6: generate temp keypair, authorize on node2, place on node1
  for root to use during cluster-init, clean up afterward

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:30:53 +10:00
beatzaplentyandClaude Sonnet 4.6 fc8f7baf3e fix(ipa): restore IPA password login via LightDM and su
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
Two independent fixes:

1. hashedPassword = "!" on the IPA user stub
   Without any shadow entry, pam_unix returns PAM_AUTHINFO_UNAVAIL
   before prompting, so PAM_AUTHTOK is never set. The "!" marker
   (account locked, not a real hash) ensures a shadow entry exists so
   pam_unix prompts and sets PAM_AUTHTOK — which the subsequent
   pam_sss module can then use.

2. pam_sss try_first_pass instead of use_first_pass (login + su)
   use_first_pass silently fails when PAM_AUTHTOK is unset.
   try_first_pass prompts independently in that case, making IPA
   password auth work even if pam_unix returns early for any reason.

Root cause found during incident: wayne (uid 50002) was outside IPA's
auto-SID range (163800000+), so sidgen never assigned him an
ipaNTSecurityIdentifier. Without it, ipadb's handle_authdata step fails
with "Generic error" after SPAKE pre-auth succeeds. Fixed by manually
adding objectClass: ipaNTUserAttrs + ipaNTSecurityIdentifier RID 550002
to wayne's LDAP entry on domain-controller. Any future IPA user with a
manually-assigned uid outside the auto-range needs the same treatment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 17:25:37 +10:00
beatzaplentyandClaude Sonnet 4.6 3f9b968a41 fix(ha/deploy): use non-sudo SSH for /nix writability check
pve_check() always uses sudo, so "sudo test -w /nix" passes as root
regardless of whether the SSH user can actually write there.  Use a
direct non-sudo SSH command for the writability probe so the check
reflects wayne's own access, not root's.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:15:20 +10:00
beatzaplentyandClaude Sonnet 4.6 1263c7540c fix(ha/deploy): ensure remote clone is on correct branch before building
When create-proxmox-resource.sh clones the repo to pve1, it stays on
whatever branch was checked out. Add a pre-build phase that detects
branch mismatch and switches the remote clone to the current local
branch before building, so the Proxmox node always builds from the
same commits we're deploying.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:12:39 +10:00
beatzaplentyandClaude Sonnet 4.6 a8d8b1465c fix(ha/deploy): fix /nix ownership on pve1 before codex-setup
pve1 has a pre-existing /nix store owned by a different UID; wayne's IPA
UID (50002) can't write to it.  Add a pre-phase-3 check that uses wayne's
passwordless sudo to chown -R the store before create-proxmox-resource.sh
runs codex-setup.sh on the node.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:11:05 +10:00
beatzaplentyandClaude Sonnet 4.6 b76d54e702 fix(ha): fix deploy script bugs and correct cluster configuration
- Add root SSH key + passwordless sudo to ha-server nodes (needed for
  deploy script to run cluster-init.sh via SSH as root on node1)
- Fix cluster-init.sh: correct default IPs (228/227/229 per variables.nix),
  use \${VAR:-default} for all config so deploy.sh can override via env
- Fix acceptance-tests.sh: same IP corrections, add -i flag to SSH calls,
  use \${VAR:-default} pattern
- Fix deploy.sh dry-run bugs: pve_check() always runs SSH for read-only
  probes so bridge existence check is accurate; wait_for_ssh skips in
  dry-run instead of timing out
- Fix cluster-init invocation: upload script via scp and run via SSH as
  root on node1 (was incorrectly trying to run the script locally)
- Fix acceptance-tests invocation: pass IP env vars from deploy.sh

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:04:13 +10:00
beatzaplentyandClaude Sonnet 4.6 59854a0229 feat(ha): implement clan vars and fully encrypt all HA secrets
- Run sync-host-keys.sh for proxmox-ha-server-{1,2}: generates SSH host
  key pairs in vars/per-machine/, registers age anchors in .sops.yaml,
  adds both hosts as recipients for common.yaml, ha-corosync-authkey,
  and per-host secrets/keytab files
- Re-encrypt secrets/common.yaml with both new host keys
- Convert all stub secrets to real sops-encrypted files:
    secrets/ha-server-{1,2}.yaml    (YAML, beszel-token = PLACEHOLDER)
    secrets/ha-server-{1,2}.keytab  (binary, stub text encrypted)
    secrets/ha-corosync-authkey      (binary, stub text encrypted)
- Add scripts/ha/deploy.sh: full lifecycle script (bridge setup, VM
  creation, DRBD disk + storage NIC attachment, boot wait, cluster-init,
  acceptance tests, --destroy)

Bootstrap order (operator runs these before first deploy):
  1. bash scripts/ha/deploy.sh            # deploys, tests
  # Post-deploy secret replacement:
  2. sops secrets/ha-server-{1,2}.yaml   (set real beszel-token)
  3. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
  4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
  5. Set services.beszel.agent.environment.KEY in host.nix after hub pairing
  6. nixos-rebuild switch on both nodes to pick up real secrets

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 16:48:04 +10:00
beatzaplentyandClaude Sonnet 4.6 dc83333526 feat(ha): pre-stage IPA enrollment for ha-server-1 and ha-server-2
- Add stub keytab files (secrets/ha-server-{1,2}.keytab) so modules/ipa/client.nix
  activates for both nodes — IPA security.ipa.enable = true confirmed by eval
- Add .sops.yaml creation rules for both keytabs (admin-only until
  sync-host-keys.sh registers the host age keys)

Eval: both proxmox-ha-server-1 and proxmox-ha-server-2 build cleanly
with IPA enabled, beszel agent enabled, corosync authkey + beszel-token
+ ipa-host-keytab all wired via sops-nix.

Bootstrap order before first deploy:
  1. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
  2. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
  3. sops updatekeys secrets/common.yaml
  4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
  5. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
  6. sops secrets/ha-server-1.yaml  (set beszel-token)
  7. sops secrets/ha-server-2.yaml  (set beszel-token)
  8. On node1 after first boot: corosync-keygen, then
     sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey
  9. Set services.beszel.agent.environment.KEY in each host.nix after hub pairing
 10. git add/commit the real secrets, then nixos-rebuild switch on both nodes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 16:30:29 +10:00
beatzaplenty a960c662f0 Merge pull request 'feat(ipa/docker): grant docker access via IPA group membership' (#90) from worktree-docker-ipa-group into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #90
2026-07-28 05:48:45 +00:00
beatzaplenty 6f602b2245 Merge pull request 'chore: full sweep — docs sync, SSH key module extraction, NFS dedup, dead code removal' (#89) from worktree-full-sweep into main
Check NixOS configurations / eval-hosts (push) Successful in 10m39s
Reviewed-on: #89
2026-07-28 05:26:57 +00:00
beatzaplentyandClaude Sonnet 4.6 5beed2d75c feat(ipa/docker): grant docker access via IPA group membership
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m39s
Create an IPA group "docker-access" (GID 50010) and pin the local
"docker" group to that GID on all Docker hosts.  Any IPA user in the
docker-access group automatically gains docker socket access through
SSSD supplementary-group resolution — no per-host docker.members
entry needed.

Specific changes:
- variables.nix: add dockerAccessGid = 50010
- modules/docker/enable-service.nix: lib.mkForce docker GID to
  dockerAccessGid, removing the need to name individual IPA users
- modules/build-types/docker.nix: remove direct wayne docker.members
  entry (access now comes from IPA group)
- modules/ipa/client.nix: refactor repeated security.* / systemd.*
  top-level keys into merged attribute sets (fixes statix W20); add
  security.pam.services.lightdm.makeHomeDir so the GUI login path
  also creates the home dir on first login

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 15:16:05 +10:00
80 changed files with 4543 additions and 1736 deletions
+2
View File
@@ -23,3 +23,5 @@ host-keys/
# Temporary Milestone 1 audit checklist (remove-sensetive-info-refactor.md) # Temporary Milestone 1 audit checklist (remove-sensetive-info-refactor.md)
# - working notes only, never committed, deleted once every row is rotated. # - working notes only, never committed, deleted once every row is rotated.
secrets-inventory.md secrets-inventory.md
.claude/worktrees/
.claude/settings.local.json
+46 -22
View File
@@ -7,21 +7,22 @@ keys:
- &linode-gui age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs - &linode-gui age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs
- &linode-minimal age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx - &linode-minimal age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx
- &linode-nix-cache age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e - &linode-nix-cache age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e
- &linode-server age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
- &linode-tailscale-router age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m - &linode-tailscale-router age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- &lxc-docker age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th - &lxc-docker age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th
- &lxc-minimal age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k - &lxc-minimal age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k
- &lxc-nix-cache age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7 - &lxc-nix-cache age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7
- &lxc-pxe-boot age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt - &lxc-pxe-boot age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt
- &lxc-server age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
- &lxc-tailscale-router age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2 - &lxc-tailscale-router age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- &lxc-tor-relay age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2 - &lxc-tor-relay age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
- &proxmox-docker age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c - &proxmox-docker age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c
- &proxmox-gui age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn - &proxmox-gui age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn
- &proxmox-nix-cache age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68 - &proxmox-nix-cache age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
- &proxmox-pxe-boot age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz - &proxmox-pxe-boot age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
- &proxmox-server age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
- &proxmox-tailscale-router age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0 - &proxmox-tailscale-router age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
- &proxmox-ha-server-1 age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
- &proxmox-ha-server-2 age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
- &proxmox-ha-docker-1 age16y0vfts5v2k20e2rj23qa5lc5gm96gm64uwsqsr0y688xl0ne46qcfr0sm
- &proxmox-ha-docker-2 age1tm3zj5lp3elw2j832az4nj9xxmhqd66ag3cqcv3vskvmd7qdmqmsdpdcn0
creation_rules: creation_rules:
# Shared across every currently-deployed host: root/nixos password hash, # Shared across every currently-deployed host: root/nixos password hash,
@@ -39,21 +40,22 @@ creation_rules:
- *linode-gui - *linode-gui
- *linode-minimal - *linode-minimal
- *linode-nix-cache - *linode-nix-cache
- *linode-server
- *linode-tailscale-router - *linode-tailscale-router
- *lxc-docker - *lxc-docker
- *lxc-minimal - *lxc-minimal
- *lxc-nix-cache - *lxc-nix-cache
- *lxc-pxe-boot - *lxc-pxe-boot
- *lxc-server
- *lxc-tailscale-router - *lxc-tailscale-router
- *lxc-tor-relay - *lxc-tor-relay
- *proxmox-docker - *proxmox-docker
- *proxmox-gui - *proxmox-gui
- *proxmox-nix-cache - *proxmox-nix-cache
- *proxmox-pxe-boot - *proxmox-pxe-boot
- *proxmox-server
- *proxmox-tailscale-router - *proxmox-tailscale-router
- *proxmox-ha-server-1
- *proxmox-ha-server-2
- *proxmox-ha-docker-1
- *proxmox-ha-docker-2
- path_regex: secrets/nix-cache\.yaml$ - path_regex: secrets/nix-cache\.yaml$
key_groups: key_groups:
@@ -63,14 +65,6 @@ creation_rules:
- *lxc-nix-cache - *lxc-nix-cache
- *proxmox-nix-cache - *proxmox-nix-cache
- path_regex: secrets/server\.yaml$
key_groups:
- age:
- *admin
- *linode-server
- *lxc-server
- *proxmox-server
- path_regex: secrets/tor-relay\.yaml$ - path_regex: secrets/tor-relay\.yaml$
key_groups: key_groups:
- age: - age:
@@ -93,12 +87,14 @@ creation_rules:
key_groups: key_groups:
- age: - age:
- *admin - *admin
- *proxmox-ha-server-1
# proxmox-ha-server-1 added by sync-host-keys.sh # proxmox-ha-server-1 added by sync-host-keys.sh
- path_regex: secrets/ha-server-2\.yaml$ - path_regex: secrets/ha-server-2\.yaml$
key_groups: key_groups:
- age: - age:
- *admin - *admin
- *proxmox-ha-server-2
# proxmox-ha-server-2 added by sync-host-keys.sh # proxmox-ha-server-2 added by sync-host-keys.sh
# Shared HA cluster corosync authkey (binary sops file). # Shared HA cluster corosync authkey (binary sops file).
@@ -108,6 +104,8 @@ creation_rules:
key_groups: key_groups:
- age: - age:
- *admin - *admin
- *proxmox-ha-server-1
- *proxmox-ha-server-2
# proxmox-ha-server-1 added by sync-host-keys.sh # proxmox-ha-server-1 added by sync-host-keys.sh
# proxmox-ha-server-2 added by sync-host-keys.sh # proxmox-ha-server-2 added by sync-host-keys.sh
@@ -166,14 +164,6 @@ creation_rules:
- *proxmox-gui - *proxmox-gui
- *linode-gui - *linode-gui
- path_regex: secrets/server\.keytab$
key_groups:
- age:
- *admin
- *linode-server
- *lxc-server
- *proxmox-server
- path_regex: secrets/docker\.keytab$ - path_regex: secrets/docker\.keytab$
key_groups: key_groups:
- age: - age:
@@ -195,3 +185,37 @@ creation_rules:
- *lxc-minimal - *lxc-minimal
- *proxmox-minimal - *proxmox-minimal
- *linode-minimal - *linode-minimal
# Host keytab for ha-server-1 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/ha-server-1\.keytab$
key_groups:
- age:
- *admin
- *proxmox-ha-server-1
# proxmox-ha-server-1 added by sync-host-keys.sh
# Host keytab for ha-server-2 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/ha-server-2\.keytab$
key_groups:
- age:
- *admin
- *proxmox-ha-server-2
# proxmox-ha-server-2 added by sync-host-keys.sh
# Host keytab for ha-docker-1 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/ha-docker-1\.keytab$
key_groups:
- age:
- *admin
- *proxmox-ha-docker-1
# Host keytab for ha-docker-2 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/ha-docker-2\.keytab$
key_groups:
- age:
- *admin
- *proxmox-ha-docker-2
+1 -1
View File
@@ -7,7 +7,7 @@ servers and workstation.
The flake exposes NixOS configurations named `<platform>-<buildtype>` The flake exposes NixOS configurations named `<platform>-<buildtype>`
(platforms: `linode`, `proxmox`, `lxc`, `baremetal`; build types: `minimal`, (platforms: `linode`, `proxmox`, `lxc`, `baremetal`; build types: `minimal`,
`nix-cache`, `server`, `docker`, `gui`, `pxe-boot`, `tailscale-router`, `nix-cache`, `docker`, `gui`, `pxe-boot`, `tailscale-router`,
`tor-relay`, `ha-server`), generated from `modules/platforms/*` and `tor-relay`, `ha-server`), generated from `modules/platforms/*` and
`modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not every `modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not every
combination is built — `pxe-boot` has no `linode` variant, `ha-server` only combination is built — `pxe-boot` has no `linode` variant, `ha-server` only
-150
View File
@@ -1,150 +0,0 @@
# Flake End-to-End Audit Report
**Date:** 2026-07-21
**Scope:** Full static lint/eval sweep + live build/deploy/interrogate/destroy testing of every `lxc-*` and `proxmox-*` flake target against `pve.sweet.home`, plus an audit of the operator's ability to manage the flake/secrets tooling.
**Branch:** `worktree-flake-e2e-audit` (this session's isolated worktree)
## Executive Summary
The flake itself is in good shape: `nixpkgs-fmt`, `statix`, and a full eval + dry-run build of every host and package are all clean. Every `lxc-*`/`proxmox-*` target's NixOS configuration builds successfully — no target has a broken derivation graph.
The issues found are **operational, not code-level**:
1. **pve.sweet.home is critically low on disk space** (91-95% full during this session) and cannot currently build the two largest closures (`gui`, `pxe-boot`) to completion — this actively blocks deploying/redeploying those hosts via the documented workflow.
2. **A real, reproducible secrets-decryption failure** was caught live: a stale cached container image (built before a same-day sops-key fix) boots with sshd never starting and every secret failing to decrypt. This is a **general hazard in `create-proxmox-resource.sh`'s "reuse the cached image if present" default**, not a one-off.
3. **sops key/anchor drift**: `proxmox-minimal` has a `.sops.yaml` recipient anchor with no corresponding private key anywhere in this environment; several `lxc-*`/`proxmox-*` targets have no sops registration at all yet.
4. One concrete script bug was found and **fixed in this session**: `create-proxmox-resource.sh` never enabled the QEMU guest agent channel on VMs it creates, despite the guest OS already running it.
5. A management-surface audit (of the operator's ability to run this repo day to day) found 5 process gaps, detailed below.
Nothing here required or received a `nixos-rebuild switch/boot/test`, `nixos-install`, or any disk-formatting command — all validation was `nix build`/`nix eval`, plus disposable `pct`/`qm` create-then-destroy cycles via the repo's own `create-proxmox-resource.sh`.
---
## 1. Static Analysis Results — all clean
`bash scripts/codex-maintenance.sh --full-check --dry-run` (whole-tree sweep, not just changed files):
| Check | Result |
|---|---|
| Secret grep | Clean — only the documented exceptions (installer's own hashed passwords, `access-tokens` comment references) |
| `nixpkgs-fmt --check` | 0/53 files would be reformatted |
| `statix` | No lint warnings |
| nix-cache host key drift check | Up to date |
| Full eval of every host's `system.build.toplevel` | All 19 `nixosConfigurations` targets evaluate cleanly |
| Dry-run build of every host + package | All succeed, no derivation errors |
No drift, no formatting issues, no lint findings anywhere in the tree.
---
## 2. Per-Target Test Results
Legend: **LIVE** = built on pve, `pct`/`qm` create → interrogated → destroyed. **BUILD-ONLY** = `nix build` validated the config (mostly `.config.system.build.toplevel`, occasionally `.tarball`), no resource created on pve.
| Target | Test type | Result | Notes |
|---|---|---|---|
| `lxc-docker` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT105 is already running this identity in production; `--allow-duplicate-host` would have destroyed it. |
| `lxc-minimal` | **LIVE** | ✅ PASS (after retry) | First attempt reused a stale cached tarball predating a same-day sops-key commit → activation failed, sshd never started (see Finding #2). Redeployed with `--force-rebuild`: clean boot, `systemctl is-system-running` = `running`, secrets decrypted, sshd listening, users correct. |
| `lxc-nix-cache` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — CT101 is already running this identity. First local build attempt appeared to hang on a remote-builder handoff to nix-cache; killed and retried with `--builders ""` (local-only), succeeded. |
| `lxc-gui` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Registered a fresh sops key (no prior registration existed), built successfully through the full NixOS system closure, then **failed packaging the tarball**: `No space left on device` on pve's root filesystem. Not a flake defect. |
| `lxc-pxe-boot` | **LIVE (attempted)** | ⚠️ BLOCKED by pve disk space | Same failure as `lxc-gui` — this target additionally builds a full nested installer/netboot image (`stage-installer-artifacts.nix`), making it similarly large. Failed with the same `No space left on device` error, immediately after the gui attempt had already consumed pve's remaining headroom. |
| `lxc-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; live deploy also would have hit `boot.zfs.extraPools` trying to import a real ZFS pool that doesn't exist in an isolated test container — an expected limitation of testing this build type outside its real hardware, not a bug. |
| `lxc-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
| `lxc-tor-relay` | BUILD-ONLY | ✅ PASS | Live redeploy skipped — CT106 already holds this identity in production. |
| `proxmox-docker` | BUILD-ONLY | ✅ PASS (after retry) | Live redeploy skipped — both CT105 *and* VM103 already hold `docker` identities. Combined `toplevel` + `diskoImagesScript` build crashed with a **Nix-internal assertion failure** (`worker.cc:360`) under this session's memory pressure (see Finding #6) — not a flake bug. Retried with `toplevel` alone: clean. |
| `proxmox-minimal` | **LIVE (attempted)** | ⚠️ BLOCKED by key drift → BUILD-ONLY | `.sops.yaml` has a registered `&proxmox-minimal` anchor but **no corresponding private key exists anywhere in this environment** — the script correctly refused to generate a mismatched replacement. Fell back to `toplevel` build: ✅ PASS. |
| `proxmox-nix-cache` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
| `proxmox-gui` | BUILD-ONLY | ⚠️ Killed after ~40min (resource-limited) | This session's local build machine has only 2GB RAM; swap filled completely (2.0/2.0GB) and the build stalled, so it was killed rather than risk destabilizing the session further. **Not a flake defect** — the equivalent `gui` NixOS configuration already proved fully buildable during the `lxc-gui` live attempt above (it built the entire system closure successfully and only failed at the pve-side tarball-packaging step due to disk space, not the config). |
| `proxmox-pxe-boot` | BUILD-ONLY | ⚠️ Killed after ~35min (resource-limited) | Was deep into building the nested installer's kernel initrd (this build type bundles a full netboot installer image via `stage-installer-artifacts.nix`) when killed to keep the audit moving. **Not a flake defect** — this target's own module logic was already effectively validated via the earlier *live* pve deploy attempt (`lxc-pxe-boot` above), which built the complete image and only failed at the final tarball-packaging step due to pve's disk space (Finding 1). |
| `proxmox-server` | BUILD-ONLY | ✅ PASS | No sops key registered yet; same ZFS-pool caveat as `lxc-server` would apply to a live deploy. |
| `proxmox-tailscale-exit-node` | BUILD-ONLY | ✅ PASS | No sops key registered yet. |
**Not tested at all:** `linode-*` targets (not deployable to Proxmox) and `installer` (not a normal host) — both were still covered by the static eval/dry-run-build sweep above.
---
## 3. Findings, Ranked by Severity
### Finding 1 — pve.sweet.home is critically low on disk space (blocks real deployments)
At session start: `/dev/mapper/pve-root` was **95% full, 5.3GB free** (of 94GB). After two failed large builds it recovered slightly to **91% full, 8.2GB free** (nix cleans up its own failed-build scratch space). `/nix/store` alone is 26GB; `nix-store --gc --print-dead` reports **zero** reclaimable garbage — everything currently in the store is a live GC root, so `nix-collect-garbage` won't help without first removing old roots.
**Why it matters:** `create-proxmox-resource.sh` builds every VM/CT image **directly on pve**, not on a build machine and transferred over. With <10GB headroom, any closure approaching a few GB (the `gui` build type: full Cinnamon desktop + Firefox + LibreOffice + GIMP + VS Code + xrdp; the `pxe-boot` build type: nginx/atftpd *plus* an entire nested installer/netboot image) cannot currently be built there at all. Both `lxc-gui` and `lxc-pxe-boot` failed live with `No space left on device` during this audit.
**Recommended action:** Expand `pve-root`'s LV, or free space by pruning old container templates in `/var/lib/vz/template/cache` (1.5GB) / old backups in `/var/lib/vz/dump` (306MB) / auditing what's pinning 26GB of `/nix/store` as live GC roots (likely `result-*` symlinks — see below). This is real production disk state; **not something this session touched or fixed** — it needs the operator's judgment on what's safe to remove.
**Secondary, smaller finding:** every `create-proxmox-resource.sh` run leaves a `result-<target>` symlink in the node's repo checkout as a permanent GC root (`ls /root/nixos/result-*` on pve showed 3 from this session alone: `lxc-docker`, `lxc-minimal`, `lxc-nix-cache`). These accumulate forever and pin their entire closures in the store. Consider having the script clean up its own `result-*` link after staging the built artifact (or use a temp `--out-link` under `/tmp`), so `nix-collect-garbage` can actually reclaim old build outputs.
### Finding 2 — Stale cached images can silently ship broken secrets (reproduced live)
`create-proxmox-resource.sh`'s default behavior is: if the node already has `<target>.tar.xz`/`.raw` staged, **reuse it** — only `--force-rebuild` forces a fresh build. This session hit exactly the failure mode `docs/auto-installer.md` already warns about: `lxc-minimal`'s cached tarball (built 2026-07-20T15:57Z) predated a same-day sops-key fix commit (2026-07-20T17:49Z, "clean up in ailse 3"). The deployed container booted with:
```
sops-install-secrets: failed to decrypt '.../common.yaml': Error getting data key: 0 successful groups required, got 0
Activation script snippet 'setupSecrets' failed (1)
```
— every secret permanently failed to decrypt, `sshd` never started (though the container otherwise looked "running"). This was **not a code bug**: the currently-committed `secrets/common.yaml` decrypts fine for that host's key when checked independently; the *cached artifact on pve* simply reflected an older commit's ciphertext. Redeploying with `--force-rebuild` fixed it immediately.
**Why it matters:** this is silent and easy to trigger by accident — any operator who redeploys a host without remembering `--force-rebuild` after a secrets change gets a container that looks like it started (`pct start` succeeds, `pct status` = running) but is completely inaccessible.
**Recommended action:** Have `create-proxmox-resource.sh` compare the cached image's build timestamp (or embed the source commit hash in the staged filename) against current HEAD, and warn (or refuse without `--force-rebuild`) if they differ — rather than silently trusting presence alone.
### Finding 3 — sops key/anchor drift
Two concrete instances hit live during this session:
- **`proxmox-minimal`**: `.sops.yaml` already has a registered `&proxmox-minimal` age recipient, but this environment's `host-keys/` directory has no corresponding private key file. `sync-host-keys.sh` correctly refused to generate a replacement (it would silently mismatch whatever's already registered/deployed) — but this means **no environment currently has this host's private key**, unless it exists on some other machine that was never backed up here.
- **`lxc-gui`**, and by the same logic `lxc-server`/`lxc-tailscale-exit-node`/most `proxmox-*` targets, have **no sops registration at all yet** — expected for undeployed hosts per `docs/auto-installer.md`, but this session's live-testing needed to register `lxc-gui`'s key on the fly, which immediately hit **Finding 3b**: registering a key locally does nothing for pve's build until it's pushed to `origin/main` (pve builds via `git pull`, not from this uncommitted worktree). This is exactly gap #4 the management-surface audit (below) already flagged in the abstract — this session hit it concretely.
**Recommended action:** for `proxmox-minimal`, decide whether to regenerate its key (destroying old-key decrypt access, if anything still holds it) or track down wherever the original private key lives and back it up here. For the general pattern, see the management-surface audit's recommendation to pre-flight-check key registration before building.
### Finding 4 — QEMU guest agent never wired up (found and fixed this session)
`modules/common/configuration.nix:44` sets `services.qemuGuest.enable = true` on every host — the guest-side agent daemon is correctly enabled everywhere. But `scripts/proxmox/create-proxmox-resource.sh`'s `qm create` call never passed `--agent 1`, so **Proxmox never created the virtio-serial channel** the agent needs. Every `proxmox-*` VM this script ever created was silently missing `qm guest exec`/IP-address reporting in the Proxmox UI, despite the guest daemon actually running.
**Status: fixed in this session's worktree** (`scripts/proxmox/create-proxmox-resource.sh`, `qm create` now includes `--agent enabled=1`) — see the diff, included in the PR from this session.
### Finding 5 — Orphaned container on pve (CT102)
`pve.sweet.home` has a stopped LXC container, **VMID 102**, with an essentially empty config (`lock: create` and nothing else — no hostname, no rootfs, no network) — the leftover of a `pct create` that started and never finished. It predates this session (not created by any of this audit's activity) and wasn't touched. **Recommend the operator confirm it's abandoned and remove it** (`pct destroy 102 --purge 1`) — left as-is it may be someone's genuine in-progress work, so it wasn't assumed safe to delete autonomously.
### Finding 6 — Nix-internal crash under memory pressure (tooling, not flake)
Building `proxmox-docker`'s `toplevel` and `diskoImagesScript` together crashed with a Nix-internal assertion failure (`Assertion '!awake.empty()' failed ... worker.cc:360`, a known class of bug in Nix's multi-goal build scheduler) while this session's 2GB-RAM build container was under heavy swap pressure (1.8-2.0/2GB swap in use) from a separate concurrent build. Retrying the same target alone (no concurrency) succeeded cleanly. **Not a flake defect** — purely an artifact of this session's constrained build environment; noted for completeness since it looked alarming in isolation.
### Finding 7 — Management-surface audit: 5 operability gaps
A focused audit of "can the operator actually run this repo day to day" (flake, home-manager, sops, related scripts) found:
1. **No documented recovery path if the `&admin` sops age key is lost without a backup.** `scripts/secrets/backup-admin-key.sh` exists and works but is referenced nowhere in `README.md`/`docs/` — no forcing function ensures a backup was ever taken. `rotate-admin-key.sh` requires the *old* key to re-key; there's no bootstrap-from-nothing path documented (the real fallback — deriving an age identity from any still-live host's own SSH key — isn't written down anywhere).
2. **home-manager has no standalone iteration path.** It's wired only inside `nixosConfigurations` (`flake.nix`) — no `homeConfigurations` output. The fastest real shortcut (`nix build .#nixosConfigurations.<target>.config.home-manager.users.nixos.home.activationPackage`) isn't documented anywhere, so the practical workflow is a full host rebuild to test one HM tweak.
3. **Gitea's flake-lock-update workflow pushes straight to `main` with no pre-merge validation.** `.gitea/workflows/update-flake-lock.yml` commits and pushes `nix flake update`'s result directly; `codex-maintenance.sh` only runs *after*, on the resulting push — a genuinely broken lockfile bump lands on `main` before anything catches it. (The GitHub-side workflow is safer — PR-based — but has the opposite gap: nothing alerts if the PR sits unmerged.)
4. **No pre-flight check that a build target has a registered sops key before building it.** `docs/auto-installer.md` documents the failure mode (silent, total secrets-decrypt failure) but nothing in `create-proxmox-resource.sh` refuses to proceed when it's about to build a target with no `.sops.yaml` anchor — it's on the operator to remember. This session's `lxc-gui` test hit close to this exact gap (needed the key added on the fly, mid-session).
5. **`vars.remoteBuilderAuthorizedKeys` has the same drift risk as `vars.nixCacheHostKey`, but no checker script.** `sync-nix-cache-host-key.sh --check` guards the latter; the former (and `vars.pxeServerIp`/`vars.pbsIp`) has no equivalent — a rotated/revoked client key just silently stops working with no diagnostic pointing back here.
---
## 4. Action Plan (priority order)
1. **Free up disk space on pve.sweet.home** (or expand `pve-root`). Blocking: `lxc-gui`, `proxmox-gui`, `lxc-pxe-boot`, `proxmox-pxe-boot` cannot currently be built/redeployed on this node at all.
2. **Decide on `proxmox-minimal`'s orphaned sops key**: locate the original private key and back it up here, or accept regenerating it (breaks decrypt access for whoever/whatever currently holds the old one).
3. **Merge this session's PR** (see below) to get the `--agent 1` fix and `lxc-gui`'s new sops registration onto `main` — required before `lxc-gui` can be live-redeployed with working secrets.
4. **Add a staleness guard to `create-proxmox-resource.sh`'s cache-reuse path** (Finding 2) — highest-leverage fix, since it silently produces a broken-but-"running" host.
5. **Add a pre-flight sops-anchor check to `create-proxmox-resource.sh`** (management-surface gap #4) — same root cause class as #4 above, catch it before building instead of at first boot.
6. Investigate/clean up **CT102** on pve (Finding 5) — confirm abandoned, then remove.
7. Document `backup-admin-key.sh` in `README.md`'s Security Notes and add the live-host-key bootstrap-recovery procedure to `docs/` (management-surface gap #1).
8. Add pre-push validation to the Gitea flake-lock-update workflow (management-surface gap #3).
9. Lower-priority: document the home-manager `activationPackage` shortcut (gap #2); extend `sync-nix-cache-host-key.sh`'s drift-check pattern to `remoteBuilderAuthorizedKeys` (gap #5).
10. Follow-up session: finish build-validating `proxmox-gui` and `proxmox-pxe-boot` (both killed here after 35-40min on this session's 2GB-RAM machine — not failures, just unfinished) once pve has headroom (item 1) — ideally from a machine with more RAM. `proxmox-server` and `proxmox-tailscale-exit-node` already passed build-only validation in this session, no follow-up needed.
---
## 5. Uncommitted Changes From This Session
This worktree (`worktree-flake-e2e-audit`) currently has:
- `scripts/proxmox/create-proxmox-resource.sh` — the `--agent enabled=1` fix (Finding 4).
- `.sops.yaml` / `secrets/common.yaml``lxc-gui`'s new age key registered as a recipient (generated live during this session's testing).
Per this session's standard workflow, these will be committed, pushed, and opened as a draft PR rather than pushed to `main` directly — merging it is the operator's call, and is also **prerequisite to live-redeploying `lxc-gui` successfully** (its build will keep hitting the sops-staleness failure from Finding 2 on pve until this registration is on `origin/main`).
+108 -19
View File
@@ -21,9 +21,7 @@ machines when deployed.
`modules/installer/common.nix` (the auto-installer's own root/nixos login — `modules/installer/common.nix` (the auto-installer's own root/nixos login —
a deliberate, documented choice, see `docs/auto-installer.md`, not a deliberate, documented choice, see `docs/auto-installer.md`, not
accidental tech debt) and **SSH public keys** in `variables.nix` accidental tech debt) and **SSH public keys** in `variables.nix`
(`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`) plus a couple of (`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`, `vars.beszelHubKey`). Don't use the installer's hardcoded hash as a
per-host `KEY` values for beszel-agent auth (`hosts/server/host.nix`,
`hosts/nix-cache/host.nix`). Don't use the installer's hardcoded hash as a
template for a *real* host — every other host uses sops-nix template for a *real* host — every other host uses sops-nix
(`hashedPasswordFile`, see "Security Notes" in `README.md`). Flag any *new* (`hashedPasswordFile`, see "Security Notes" in `README.md`). Flag any *new*
secret-like string you encounter instead of committing it. secret-like string you encounter instead of committing it.
@@ -254,6 +252,14 @@ instead of copying it.
silent skip rather than a failure) only reports drift; the no-flags form silent skip rather than a failure) only reports drift; the no-flags form
updates both files in place. Declarative clients still need a rebuild to updates both files in place. Declarative clients still need a rebuild to
pick up the fix. pick up the fix.
- `scripts/secrets/push-host-keys.sh [--all | <target>] [--dry-run]
[--skip-git-check]` — pushes newly-generated SSH host keys from
`host-keys/` to already-running NixOS hosts, so they can decrypt sops
secrets after a rebuild following `sync-host-keys.sh
--regenerate-all-keys`. Verifies that `.sops.yaml` and `secrets/*.yaml`
are committed and pushed to the remote first (hosts rebuild from the
remote Gitea flake, so recipient changes must land there before any key
push).
### `scripts/proxmox/` ### `scripts/proxmox/`
@@ -275,6 +281,16 @@ instead of copying it.
failure just falls back to building from source / `cache.nixos.org`) so failure just falls back to building from source / `cache.nixos.org`) so
the node substitutes from and can offload builds to nix-cache on every the node substitutes from and can offload builds to nix-cache on every
subsequent run, not just this one. subsequent run, not just this one.
- `scripts/proxmox/clone-pve1-to-pve-test.sh <vmid> [--new-vmid <id>]
[--mode snapshot|suspend|stop] [--dry-run]` — ad-hoc clone of a single
VM or CT from pve1 (production) to pve-test (sandbox) via vzdump +
qmrestore/pct restore. Streams the archive directly between nodes (no
local staging copy). Always restores with `--unique 1` (fresh MAC
addresses) since the original is still running on the LAN. Cleans up
the vzdump archive from both nodes after a successful restore. The
script's own default is pve1 → pve-test, matching CLAUDE.md's policy
(unlike `create-proxmox-resource.sh`, which defaults to production for
the operator's own unqualified use).
- `scripts/proxmox/configure-nix-cache-client.sh [--dry-run] - `scripts/proxmox/configure-nix-cache-client.sh [--dry-run]
[--no-remote-builder] [--no-restart]` — the non-NixOS equivalent of [--no-remote-builder] [--no-restart]` — the non-NixOS equivalent of
`modules/nix-cache/client.nix`/`remote-builder-client.nix`, for a plain `modules/nix-cache/client.nix`/`remote-builder-client.nix`, for a plain
@@ -290,6 +306,50 @@ instead of copying it.
marked block rather than duplicating it); restarts `nix-daemon` by marked block rather than duplicating it); restarts `nix-daemon` by
default so the change takes effect immediately. default so the change takes effect immediately.
### `scripts/ha/`
HA cluster lifecycle and operational scripts. All mutate real cluster state
when run for real — always run against pve-test first unless the operator
explicitly targets pve1.
- `scripts/ha/deploy.sh [--skip-*] [--destroy] [--dry-run]` — full
lifecycle manager: phases through bridge creation, key sync, VM creation
(via `create-proxmox-resource.sh`), NIC/disk attachment, and cluster
initialisation. `--destroy` tears it back down. Safe to rerun
idempotently; each phase can be individually skipped.
- `scripts/ha/cluster-init.sh` — one-time cluster bootstrap run **as root
on ha-server-1** after both VMs are booted. Generates/distributes the
Corosync authkey, initialises DRBD metadata, creates XFS on `/dev/drbd0`,
configures LIO iSCSI, and registers all Pacemaker resources (DRBD → XFS
→ iSCSI → NFS → VIPs).
- `scripts/ha/health.sh` — read-only cluster health snapshot: SSH
reachability, quorum, DRBD state, Pacemaker resources, and VIP port
reachability. Safe to run from the workstation at any time.
- `scripts/ha/failover.sh [--to node1|node2] [--force] [--timeout <s>]
[--dry-run]` — graceful failover by putting the active node into
Pacemaker standby and waiting for resources to appear on the target.
- `scripts/ha/acceptance-tests.sh` — T1T7 acceptance tests (failover,
NFS/iSCSI connectivity, DRBD sync, etc.) that must all pass before the
cluster is considered production-ready.
- `scripts/ha/resize-data-disk.sh --size +NNg [--force] [--dry-run]` —
online data-disk resize: `qm resize` on both VMs, guest block-device
rescan, `drbdadm resize`, `xfs_growfs`. No downtime required.
- `scripts/ha/cluster-enable-stonith.sh` — enables the `fence_pve_ssh`
STONITH resource after the fence SSH key is deployed to both nodes and
authorised on the Proxmox host. Run once after `cluster-init.sh`.
- `scripts/ha/fence-pve-ssh.py` — Python STONITH fence agent for Pacemaker.
Deploy to `/etc/pacemaker/fence_pve_ssh` on both HA nodes (`chmod +x`).
SSHes to the Proxmox host and runs `qm stop/start <vmid>`.
### `scripts/ipa/`
- `scripts/ipa/create-nixos-ipa-host-account.sh [options] <hostname>` —
adds a NixOS host to the FreeIPA domain and produces a sops-encrypted
keytab at `secrets/<hostname>.keytab`, ready for `modules/ipa/client.nix`.
Replaces three error-prone manual steps: `ipa host-add`, `ipa-getkeytab`
(run on the DC, SCP'd back), and `sops encrypt` in the correct location
(must be at `secrets/<hostname>.keytab` for the creation rule to match).
### `scripts/lib/` ### `scripts/lib/`
Sourced by the scripts above, never run directly: Sourced by the scripts above, never run directly:
@@ -299,6 +359,15 @@ Sourced by the scripts above, never run directly:
`create-proxmox-resource.sh` runs over SSH. `create-proxmox-resource.sh` runs over SSH.
- `nix-eval.sh` — `NIX_EVAL_FLAGS` plus `list_flake_targets`/ - `nix-eval.sh` — `NIX_EVAL_FLAGS` plus `list_flake_targets`/
`flake_target_hostname` flake-introspection helpers. `flake_target_hostname` flake-introspection helpers.
- `nix-parallel.sh` — `run_nix_parallel`: fans out independent `nix eval`/
`nix build --dry-run` calls across up to `NIX_PARALLEL_JOBS` processes,
capped by available memory (~1 GB/job) rather than raw `nproc` to avoid
OOM on constrained CI runners. Used by `codex-maintenance.sh`.
- `clan-vars.sh` — helpers for reading/writing SSH host keys stored as clan
vars (`vars/per-machine/<target>/openssh/`, sops-encrypted) instead of
the gitignored `host-keys/` directory. Sourced by
`create-proxmox-resource.sh` and `sync-host-keys.sh`; depends on
`sops-age.sh` and `ssh-host-keys.sh` being sourced first.
- `ssh-host-keys.sh` — `generate_host_ed25519_key`/`ssh_pubkey_to_age`, - `ssh-host-keys.sh` — `generate_host_ed25519_key`/`ssh_pubkey_to_age`,
shared by `sync-host-keys.sh` and `prepare-host-key.sh`. shared by `sync-host-keys.sh` and `prepare-host-key.sh`.
- `sops-age.sh` — `age_pubkey_from_identity_file`/`sops_yaml_admin_pubkey`/ - `sops-age.sh` — `age_pubkey_from_identity_file`/`sops_yaml_admin_pubkey`/
@@ -318,6 +387,17 @@ Sourced by the scripts above, never run directly:
default cores/memory, `NIX_CACHE_HOST`, `LAN_DOMAIN`) sourced by default cores/memory, `NIX_CACHE_HOST`, `LAN_DOMAIN`) sourced by
`create-proxmox-resource.sh` and `scripts/installer/auto-install.sh`. Add `create-proxmox-resource.sh` and `scripts/installer/auto-install.sh`. Add
new cross-script config here instead of duplicating it per-script. new cross-script config here instead of duplicating it per-script.
- `scripts/recover-hosts.sh [<hostname> ...]` — fixes sops/SSH-key/GitHub-token
issues on deployed NixOS hosts and triggers a `Switch-nix` rebuild on each.
With no args discovers every known hostname; with args checks only those.
Fixes applied automatically (prompts before rebuilding): SSH host key drift
(restores the registered key) and stale GitHub access tokens (empties the
rendered `nix-github-token.conf` so Nix falls back to unauthenticated requests
until sops-nix re-renders the correct token after the next successful rebuild).
- `scripts/gc-hosts.sh [--dry-run]` — runs `nix-collect-garbage -d` on all live
NixOS hosts (workstation first, then pve1, then all Proxmox guests). Excludes
`nix-cache` (gc-ing the shared binary cache evicts store paths other hosts
depend on). Uses passwordless sudo where available; falls back to user-level gc.
- `scripts/bump-nixpkgs-release.sh` — bumps `flake.nix`'s `nixpkgs.url`/ - `scripts/bump-nixpkgs-release.sh` — bumps `flake.nix`'s `nixpkgs.url`/
`home-manager.url` in place. Exists because flake input URLs can't `home-manager.url` in place. Exists because flake input URLs can't
reference `variables.nix` (confirmed empirically — `nix flake metadata` reference `variables.nix` (confirmed empirically — `nix flake metadata`
@@ -356,12 +436,13 @@ nixosSystem {
``` ```
Platforms: `linode`, `proxmox`, `lxc`, `baremetal`. Build types: `minimal`, Platforms: `linode`, `proxmox`, `lxc`, `baremetal`. Build types: `minimal`,
`nix-cache`, `server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`, `nix-cache`, `docker`, `gui`, `pxe-boot`, `tailscale-router`, `tor-relay`,
`tor-relay`. Not every combination is built — e.g. `pxe-boot` has no `linode` `ha-server`. Not every combination is built — e.g. `pxe-boot` has no `linode`
variant (PXE/DHCP/TFTP need LAN L2 adjacency a Linode VPS doesn't have), variant (PXE/DHCP/TFTP need LAN L2 adjacency a Linode VPS doesn't have),
`tor-relay` currently only exists as `lxc-tor-relay`, and `baremetal` `tor-relay` only exists as `lxc-tor-relay`, `ha-server` only exists as
currently only exists as `baremetal-gui` (the real gui-host hardware — `proxmox-ha-server-{1,2}`, and `baremetal` only exists as `baremetal-gui`
see `hosts/nixos/host.nix` and `modules/platforms/baremetal.nix`). Treat (the real gui-host hardware — see `hosts/nixos/host.nix` and
`modules/platforms/baremetal.nix`). Treat
`flake.nix`'s `flake.nix`'s
`generatedTargets` as the source `generatedTargets` as the source
of truth for which hosts exist — `README.md`, `AGENTS.md`, of truth for which hosts exist — `README.md`, `AGENTS.md`,
@@ -375,9 +456,8 @@ removing a host.
- `hosts/<name>/host.nix` — per-machine identity **only**: hostname, hostId, - `hosts/<name>/host.nix` — per-machine identity **only**: hostname, hostId,
per-machine secrets, `system.stateVersion`. These files carry no `imports` per-machine secrets, `system.stateVersion`. These files carry no `imports`
of their own beyond narrow parameterized helpers (see of their own — all shared behavior comes from the platform/build-type modules
`modules/beszel/host-token.nix` below) — all shared behavior comes from the composed in `flake.nix`, not from the host file.
platform/build-type modules composed in `flake.nix`, not from the host file.
- `modules/platforms/{linode,proxmox,lxc,baremetal}.nix` — platform-specific - `modules/platforms/{linode,proxmox,lxc,baremetal}.nix` — platform-specific
config: boot method, guest tooling, and the hardware config, imported config: boot method, guest tooling, and the hardware config, imported
directly by the platform module itself — **not** wired in from directly by the platform module itself — **not** wired in from
@@ -394,7 +474,7 @@ removing a host.
`vzdump` backup-archive metadata this doesn't have), no install step — `vzdump` backup-archive metadata this doesn't have), no install step —
see `docs/auto-installer.md`. see `docs/auto-installer.md`.
- `modules/build-types/*.nix` — what a system is for: - `modules/build-types/*.nix` — what a system is for:
minimal/server/docker/gui/pxe-boot/nix-cache/tailscale-exit-node/tor-relay. minimal/docker/gui/pxe-boot/nix-cache/tailscale-router/tor-relay/ha-server.
- `modules/common/configuration.nix` — base NixOS config imported by every - `modules/common/configuration.nix` — base NixOS config imported by every
host: locale, users, nix settings, git. host: locale, users, nix settings, git.
- `modules/common/home.nix` / `hosts/nixos/home.nix` — Home Manager config for - `modules/common/home.nix` / `hosts/nixos/home.nix` — Home Manager config for
@@ -420,7 +500,7 @@ removing a host.
`modules/platforms/baremetal.nix` also imports `modules/platforms/baremetal.nix` also imports
`modules/services/zfs/enable-service.nix` for this (the `zfs_unstable` `modules/services/zfs/enable-service.nix` for this (the `zfs_unstable`
package, autoScrub/autoSnapshot/trim) — the only other importer today is package, autoScrub/autoSnapshot/trim) — the only other importer today is
`server`'s NFS data pool, an unrelated non-root ZFS use. `ha-server`'s NFS data pool, an unrelated non-root ZFS use.
- `modules/boot/efi.nix` — systemd-boot + EFI vars, paired with the disko module. - `modules/boot/efi.nix` — systemd-boot + EFI vars, paired with the disko module.
- `modules/installer/` — the auto-installer environment (ISO, also served as - `modules/installer/` — the auto-installer environment (ISO, also served as
PXE netboot): `common.nix` (shared config + the generated PXE netboot): `common.nix` (shared config + the generated
@@ -434,14 +514,21 @@ removing a host.
substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the
full design (per-host local stores, no shared `/nix/store`, and how the full design (per-host local stores, no shared `/nix/store`, and how the
`nixremote` signing/SSH keys fit together). `nixremote` signing/SSH keys fit together).
- `modules/beszel/host-token.nix` — parameterized helper module - `modules/ha/` — HA cluster NixOS modules: `cluster-config.nix` (DRBD,
(`{ name, sopsFile }`) that wires a host's beszel-agent sops secret/template Corosync, Pacemaker, firewall rules, cluster-wide NFS/iSCSI port
and `environmentFile`; used by `hosts/server/host.nix` and authorisation — shared by both ha-server nodes), `pacemaker-stack.nix`
`hosts/nix-cache/host.nix` to avoid duplicating that boilerplate. (Pacemaker + Corosync service enablement), and supporting modules. See
`docs/ha.md` for the cluster operational guide.
- `modules/ipa/client.nix` — FreeIPA client enrollment: sssd, Kerberos keytab,
and IPA host registration; imported by every real host via
`modules/common/configuration.nix`.
- `modules/beszel/enable-agent.nix` — enables beszel-agent, sets `HUB_URL`,
fixes the upstream `StateDirectory` bug, and wires the universal
`beszel-token` sops secret (from `secrets/common.yaml`) into the agent's
`environmentFile`; see `docs/beszel.md` for the full setup guide.
- `modules/tailscale/`, `modules/docker/`, `modules/networking/`, - `modules/tailscale/`, `modules/docker/`, `modules/networking/`,
`modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose, `modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose,
single-host single-host feature modules (e.g. `docker/enable-service.nix`,
feature modules (e.g. `docker/enable-service.nix`,
`services/zfs/enable-service.nix`). Grep `modules/build-types/*.nix` for `services/zfs/enable-service.nix`). Grep `modules/build-types/*.nix` for
each build type's `imports` list to see which modules apply where. each build type's `imports` list to see which modules apply where.
@@ -465,3 +552,5 @@ duplicating config.
- `docs/flake-lock-automation.md` — how `flake.lock` updates flow through CI - `docs/flake-lock-automation.md` — how `flake.lock` updates flow through CI
(scheduled `nix flake update` PR + host-eval-on-PR workflow) and why hosts (scheduled `nix flake update` PR + host-eval-on-PR workflow) and why hosts
should track the committed lock file rather than `nixos-rebuild --upgrade-all`. should track the committed lock file rather than `nixos-rebuild --upgrade-all`.
- `docs/ha.md` — HA file-server cluster: DRBD + XFS + LIO iSCSI + NFS managed
by Corosync + Pacemaker; network topology; lifecycle scripts in `scripts/ha/`.
+2 -3
View File
@@ -9,8 +9,8 @@ Targets are named `<platform>-<buildtype>`, generated from two orthogonal
pieces composed in `flake.nix`: pieces composed in `flake.nix`:
- **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`, `baremetal` - **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`, `baremetal`
- **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`, - **Build types** (what it's for): `minimal`, `nix-cache`, `docker`, `gui`,
`gui`, `pxe-boot`, `tailscale-router`, `tor-relay`, `ha-server` `pxe-boot`, `tailscale-router`, `tor-relay`, `ha-server`
Not every combination exists — `pxe-boot` has no `linode` variant, since Not every combination exists — `pxe-boot` has no `linode` variant, since
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have, PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have,
@@ -24,7 +24,6 @@ hardware). The full list:
| `proxmox-minimal` | Minimal NixOS host profile on Proxmox — previously the flat `nix-minimal` target | | `proxmox-minimal` | Minimal NixOS host profile on Proxmox — previously the flat `nix-minimal` target |
| `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container | | `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container |
| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder — previously the flat `nix-cache` target | | `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder — previously the flat `nix-cache` target |
| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host — previously the flat `server` target |
| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack — previously the flat `docker` target | | `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack — previously the flat `docker` target |
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target | | `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target |
| `baremetal-gui` | Same Cinnamon desktop workstation, on the real gui-host hardware — ZFS RAID0 root, systemd-boot | | `baremetal-gui` | Same Cinnamon desktop workstation, on the real gui-host hardware — ZFS RAID0 root, systemd-boot |
+104
View File
@@ -0,0 +1,104 @@
# Beszel agent
[Beszel](https://github.com/henrygd/beszel) is the monitoring dashboard used
in this LAN. The hub runs as a Docker container on `docker.sweet.home` (port
`vars.ports.beszelHub`, 8090). Each monitored NixOS host runs a
`beszel-agent` that connects back to the hub.
---
## How it works
Everything is handled by a single module:
**`modules/beszel/enable-agent.nix`** — imported by a build type. It:
- Enables `beszel-agent`
- Sets `HUB_URL` to `docker.sweet.home:8090`
- Sets `KEY` from `vars.beszelHubKey` (`variables.nix`) — the hub's SSH
public key, shared by every agent. Update `beszelHubKey` if the docker
host is ever rebuilt and the hub generates a new keypair.
- Reads the universal `beszel-token` from `secrets/common.yaml` via sops
and passes it to the agent as `TOKEN` in an env file
- Fixes an upstream bug where the agent couldn't persist its hub-pairing
fingerprint across restarts (adds a real `StateDirectory`)
A host file needs no beszel configuration at all — just import the module
in the build type and add the system in the hub UI.
---
## Adding beszel to a new build type
Add `../beszel/enable-agent.nix` to the `imports` list in
`modules/build-types/<type>.nix`:
```nix
imports = [
../beszel/enable-agent.nix
# ... other imports
];
```
That's the only change required. The host file needs nothing.
---
## Adding a new system to the hub
1. Rebuild and deploy the host with its build type importing `enable-agent.nix`.
2. Open the beszel hub (`http://docker.sweet.home:8090`).
3. Go to **Systems → Add system**, enter the host's IP and the default port
(45876). The agent will connect and the system will appear as active.
---
## One-time setup: add the token to `secrets/common.yaml`
The universal token is stored once in the common secrets file, shared by all
agents. Only needed once, not per-host:
```sh
sops secrets/common.yaml
```
Add:
```yaml
beszel-token: <token from the beszel hub Settings → Keys>
```
`secrets/common.yaml` is already a sops recipient for every host via their
SSH host keys, so no additional sops recipient setup is needed.
---
## Optional: monitoring extra filesystems
To report disk usage for a mount beyond the root filesystem, add
`EXTRA_FILESYSTEMS` in the host file:
```nix
services.beszel.agent.environment = {
EXTRA_FILESYSTEMS = "/mnt/data"; # colon-separated for multiple paths
};
```
---
## Optional: monitoring Docker containers
`enable-agent.nix` has a commented-out line for Docker monitoring:
```nix
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
```
Uncomment it if the host runs docker-socket-proxy and you want per-container
stats. Hosts without Docker should leave it commented out.
---
## If the hub key changes
If the docker host is ever rebuilt and beszel generates a new SSH keypair,
update `beszelHubKey` in `variables.nix` and rebuild all beszel-enabled hosts.
The new key is visible in the beszel hub under **Settings → Keys**.
+160
View File
@@ -0,0 +1,160 @@
# HA File-Server Cluster
Two `proxmox-ha-server-{1,2}` VMs form an active/passive file-server cluster:
DRBD replicates a block device between nodes; Corosync + Pacemaker manage
failover; XFS, LIO iSCSI, and NFS are brought up as a collocated resource
group on whichever node holds the DRBD Primary role.
NixOS modules: `modules/ha/`. Lifecycle scripts: `scripts/ha/`.
Cluster-wide constants: `variables.nix` (`haServer*` vars).
---
## Network layout
Three subnets — all internal to pve1 (`vmbr0`/`vmbr1`/`vmbr2`):
| Subnet | VLAN | CIDR | Bridge | Purpose |
|---|---|---|---|---|
| LAN | 2 | `192.168.2.0/24` | `vmbr0` | Management, LAN NFS |
| Cluster | 10 | `192.168.10.224/29` | `vmbr1` | Corosync ring0 + DRBD replication |
| Storage-client | 20 | `192.168.20.0/24` | `vmbr2` | NFS + iSCSI for docker/swarm |
Each HA VM has three NICs: `ens18` (LAN/vmbr0), `ens19` (cluster/vmbr1),
`ens20` (storage-client/vmbr2). See `docs/ip-addressing.md` for all IPs.
Corosync ring0 uses the cluster NIC; ring1 (backup heartbeat) uses the LAN
NIC. DRBD replicates over the cluster NIC. No storage traffic crosses the LAN.
---
## Pacemaker resources
All resources run collocated on whichever node is Primary, in this order:
```
ms-drbd0 (promotable DRBD clone)
→ xfs-data (XFS mount on /dev/drbd0 → /srv/ha-data)
→ iscsi-target (targetctl)
→ nfs-server (nfs-server.service)
→ vip-lan (192.168.2.229/24 on vmbr0 — NFS for LAN clients)
→ vip-storage (192.168.20.229/24 on vmbr2 — NFS + iSCSI for VLAN 20)
```
`vip-lan` serves pxe-boot and other LAN-only NFS clients.
`vip-storage` serves docker and any future swarm nodes; iSCSI is available on
VLAN 20 but NFS is preferred for multi-host volume sharing.
---
## DRBD fencing
`fencing resource-only` with `crm-fence-peer.sh`/`crm-unfence-peer.sh`
wrappers (`modules/ha/cluster-config.nix`). The DRBD kernel module invokes
these via the User Mode Helper with a minimal PATH; the wrappers prepend
`/run/current-system/sw/bin` before exec-ing the real handlers so Pacemaker
tools (`cibadmin`, `crm_mon`, etc.) are found.
STONITH is initially disabled (`stonith-enabled: false`,
`no-quorum-policy: ignore`). Enable it once the `fence_pve_ssh` fence agent
(`scripts/ha/fence-pve-ssh.py`) is deployed and authorised:
```bash
scripts/ha/cluster-enable-stonith.sh # run as root on ha-server-1
```
---
## Deploying the cluster from scratch
Use `scripts/ha/deploy.sh` — it orchestrates all phases:
```bash
# Against pve-test (safe — Claude's default target):
scripts/ha/deploy.sh --node "$PVE_TEST_HOST" [--dry-run]
# Against pve1 (production — requires explicit operator go-ahead):
scripts/ha/deploy.sh --node "$PVE1_HOST"
```
Phases (each skippable with `--skip-<phase>`):
1. `ensure-bridge` — creates `vmbr1`/`vmbr2` on the Proxmox node if absent
2. `sync-keys` — generates SSH host keys for both nodes; registers sops recipients
3. `create-vms` — builds disk images, creates VMs via `create-proxmox-resource.sh`
4. `add-hardware` — attaches storage NIC and DRBD data disk to each VM
5. `init-cluster` — runs `scripts/ha/cluster-init.sh` on ha-server-1
`--destroy` runs the teardown sequence.
---
## Day-to-day operations
```bash
# Read-only health check (safe from workstation):
scripts/ha/health.sh
# Graceful failover (prompts for confirmation):
scripts/ha/failover.sh [--to node1|node2]
# Online data-disk growth (no downtime):
scripts/ha/resize-data-disk.sh --size +20G
# Acceptance tests (run after any significant change):
scripts/ha/acceptance-tests.sh
```
---
## Adding FreeIPA host accounts
IPA host registration is automated:
```bash
scripts/ipa/create-nixos-ipa-host-account.sh <hostname>
```
This runs `ipa host-add`, fetches a keytab from the domain controller, and
writes a sops-encrypted `secrets/<hostname>.keytab` in one step. The module
`modules/ipa/client.nix` (imported by every host via
`modules/common/configuration.nix`) consumes the keytab via sops-nix.
---
## Storage layout
```
/srv/ha-data/
docker/
config/ NFS → docker:/mnt/docker/config
databases/ NFS → docker:/mnt/docker/databases
volumes/ NFS → docker:/mnt/docker/volumes
nextcloud-data/ NFS → docker:/mnt/docker/nextcloud-data
proxmox/
iso/ NFS → pve1 ISO storage
lxc/ NFS → pve1 CT template storage
pxe-boot/
images/ NFS → pxe-boot:/srv/pxe/http/images (PXE assets)
raspi/
volumes/ NFS → raspi NFS mounts
iscsi-lun.img iSCSI fileio backstore (VLAN 20 only, not in active use)
```
All shares are defined in `variables.nix` (`vars.nfsShares.*`). The NFS
export list lives in `modules/ha/nfs-exports.nix`.
---
## Key variables
| Variable | Description |
|---|---|
| `vars.haServer1Ip` / `vars.haServer2Ip` | LAN management IPs |
| `vars.haServer1StorageIp` / `vars.haServer2StorageIp` | Cluster NIC IPs (DRBD/Corosync ring0) |
| `vars.haServerLanVip` | Pacemaker `vip-lan` — NFS for LAN (192.168.2.229) |
| `vars.haServerVip` | Pacemaker `vip-storage` — NFS + iSCSI for VLAN 20 (192.168.20.229) |
| `vars.haLanNfsFqdn` | FQDN of `vip-lan`: `ha-vip-lan.sweet.home` |
| `vars.haStorageRoot` | XFS mount point: `/srv/ha-data` |
| `vars.haServerDrbdDisk` | Block device for DRBD backing store |
| `vars.haStorageCidr` | Cluster subnet CIDR (`192.168.10.224/29`) |
| `vars.haClientCidr` | Storage-client subnet CIDR (`192.168.20.0/24`) |
+330
View File
@@ -0,0 +1,330 @@
# Docker Swarm Cutover Plan
Migration guide for moving containerised services from the existing single-host
Docker LXC container (CT 105, `docker.sweet.home`, 192.168.2.225) to the new
Docker Swarm cluster (`ha-docker-1` / `ha-docker-2`, 192.168.2.230231).
CT 105 stays running throughout. Services migrate one stack at a time.
Roll back any stack by restarting it on CT 105 if anything goes wrong.
---
## Prerequisites
- Swarm cluster deployed and healthy (`scripts/docker-swarm/deploy.sh`).
- Both nodes show `Ready / Active / Manager` in `docker node ls`.
- NFS mounts healthy on both swarm nodes (`/mnt/docker/config`, `/mnt/docker/databases`, `/mnt/docker/volumes`).
- Access to FreeIPA DNS admin to update A records during cutover.
---
## 1. Traefik — switch to Docker log rotation
**Current state (CT 105):** Traefik writes access logs to the NFS volume at
`/mnt/docker/volumes/traefik-data/logs/`. `modules/traefik/rotate-logs.nix`
rotates those files via `logrotate`.
**Swarm approach:** Remove file-based access logging from Traefik's static
config and rely on Docker's json-file log driver with built-in rotation.
Traefik container logs (including access events) then live under
`/var/lib/docker/containers/<id>/` on the node running Traefik.
### Steps
**1a.** In the Traefik stack definition, add logging config to the service:
```yaml
services:
traefik:
logging:
driver: "json-file"
options:
max-size: "100m"
max-file: "20"
```
**1b.** In `traefik.yml` (Traefik's static config), remove the `accessLog`
file path if present. To keep structured access logs, use Traefik's
`accessLog.format: json` with no `filePath` — logs then go to stdout and are
captured by the json-file driver above.
**1c.** Deploy Traefik to the swarm:
```bash
# On either swarm manager:
docker stack deploy -c /mnt/docker/config/traefik/docker-compose.yml traefik
```
Traefik should be deployed as a **global mode** service so it runs on all
swarm nodes and handles ingress on whichever node a request arrives at:
```yaml
services:
traefik:
deploy:
mode: global
placement:
constraints:
- node.role == manager
```
**1d.** After confirming Traefik works on the swarm, remove
`traefik/rotate-logs.nix` from the `docker` build type in
`modules/build-types/docker.nix` and rebuild CT 105.
**DNS:** Update `docker.sweet.home` and any service FQDNs that point at
192.168.2.225 to a swarm VIP or round-robin A records once Traefik is running
on the swarm. See section 8 (DNS cutover).
---
## 2. Nextcloud — migrate cron job to sidecar container
**Current state (CT 105):** `modules/docker/nextcloud-cron-job.nix` runs a
systemd timer every 5 minutes that calls:
```bash
docker exec nextcloud-webapp php ./cron.php
```
**Swarm problem:** `docker exec` only works against the local daemon. If
Nextcloud is scheduled on the other swarm node, the exec fails silently and
cron never runs.
**Swarm approach:** Add a `nextcloud-cron` sidecar container to the Nextcloud
stack definition, pinned to the same node as the main Nextcloud container via
placement constraints.
### Steps
**2a.** Choose which swarm node will host Nextcloud (e.g. `ha-docker-1`).
Label that node:
```bash
# On either swarm manager:
docker node update --label-add nextcloud=true ha-docker-1
```
**2b.** In the Nextcloud stack compose file, add the sidecar and pin both
services to the labelled node:
```yaml
services:
nextcloud-webapp:
image: nextcloud:production # pin same version as CT 105
deploy:
replicas: 1
placement:
constraints:
- node.labels.nextcloud == true
# ... existing volumes, env, networks ...
nextcloud-cron:
image: nextcloud:production # same image, different entrypoint
entrypoint: /cron.sh
deploy:
replicas: 1
placement:
constraints:
- node.labels.nextcloud == true # must co-locate with webapp
volumes:
# Same data volume as nextcloud-webapp so cron sees the same files.
- nextcloud-data:/var/www/html
# No ports exposed — cron only runs PHP inside the container.
```
`/cron.sh` is Nextcloud's built-in cron entrypoint. It runs
`php -f /var/www/html/cron.php` in a loop, sleeping for 5 minutes between
runs — identical to the current systemd timer.
**2c.** Migrate Nextcloud's data volume to the swarm:
```
/mnt/docker/volumes/nextcloud-data/ → already on NFS, no migration needed
/mnt/docker/databases/nextcloud/ → already on NFS, no migration needed
```
The NFS paths are identical on the swarm nodes (`mount-data.nix` mounts the
same shares from the same VIP). Stop Nextcloud on CT 105, deploy on the
swarm, confirm it starts cleanly.
**2d.** Remove `nextcloud-cron-job.nix` from `modules/build-types/docker.nix`
and rebuild CT 105 after confirming Nextcloud works on the swarm.
---
## 3. docker-health-to-gotify — update for swarm awareness
**Current state (CT 105):** The script at
`/home/nixos/docker/monitoring/gotify/docker-health-to-gotify.sh` runs every
minute, calls `docker ps --filter health=unhealthy`, and notifies Gotify.
**Swarm behaviour:** The same script runs on both swarm nodes independently,
each monitoring its own local Docker daemon. This gives per-node coverage
across the swarm.
**Changes needed in the script** (edit the copy on the NFS volume — it takes
effect on both nodes simultaneously on the next timer fire):
### 3a. Strip the Swarm task suffix from service names
In swarm mode, `docker ps --format '{{.Names}}'` returns names like
`nextcloud-webapp.1.abc123xyz`. The notification should show `nextcloud-webapp`,
not the full task name.
```bash
# Before:
CONTAINER_NAME=$(docker ps --format '{{.Names}}' ...)
# After:
CONTAINER_NAME=$(docker ps --format '{{.Names}}' ... | cut -d. -f1)
```
### 3b. Include the reporting node in the Gotify message
Add `$(hostname)` to the notification payload so you know which swarm node
detected the problem:
```bash
MESSAGE="[$(hostname)] ${CONTAINER_NAME} is unhealthy"
```
### 3c. Extend to catch swarm service replica failures
`docker ps` only shows what's running locally. If a service has zero healthy
replicas (task crash-looping) it may not show up on either node's `docker ps`
at the same moment. Add a swarm-level check:
```bash
# Run only on managers (both ha-docker nodes are managers):
if docker info --format '{{.Swarm.ControlAvailable}}' 2>/dev/null | grep -q true; then
# Find services where running replicas < desired replicas
docker service ls --format '{{.Name}}\t{{.Replicas}}' | \
awk -F'\t' '$2 !~ /^[0-9]+\/[0-9]+$/ || split($2,a,"/") && a[1] < a[2] { print $1, $2 }' | \
while read -r svc_name replicas; do
# Send Gotify notification for degraded service
curl -s -X POST "${GOTIFY_URL}/message" \
-H "X-Gotify-Key: ${GOTIFY_TOKEN}" \
-d "title=Swarm service degraded" \
-d "message=[$(hostname)] ${svc_name}: ${replicas} replicas"
done
fi
```
This catches the case where a service's desired replicas are not running
(e.g. OOM kill, image pull failure) — a failure mode that doesn't produce a
Docker health event on any node.
---
## 4. Passbolt migration
Passbolt has strict data integrity requirements. Migrate with care:
1. **Backup first**`docker exec passbolt-webapp php /usr/share/php/passbolt/bin/cake passbolt export_keys` and a database dump.
2. Database is on NFS (`/mnt/docker/databases/passbolt/`) — no data copy needed.
3. Pin Passbolt to a specific node: `docker node update --label-add passbolt=true ha-docker-1`
4. Add placement constraint `node.labels.passbolt == true` to the Passbolt stack.
5. Stop on CT 105, deploy on swarm, verify login works.
6. Test email delivery and 2FA.
---
## 5. Gitea migration
Gitea's data directory is on NFS (`/mnt/docker/volumes/gitea-data/`).
1. Stop Gitea on CT 105: `docker stop gitea`
2. Deploy to swarm with placement constraint (pin to `ha-docker-1` initially).
3. Verify web UI and SSH clone/push work.
4. Update DNS: `gitea.lan.ddnsgeek.com` → swarm Traefik endpoint.
5. Update the flake remote URL in `variables.nix` (`giteaDomain`) if the address changes.
---
## 6. Other services
Deploy remaining services (Grafana, InfluxDB, NodeRed, Prometheus, etc.)
as swarm stacks. Most have no special migration concern — they use NFS
volumes already on the shared storage.
Services with stateful databases (PostgreSQL, MariaDB) should follow the
pattern: stop on CT 105, confirm NFS database directory is intact, deploy on
swarm, verify.
---
## 7. Monitoring — Beszel
The Beszel hub runs on CT 105 (`docker.sweet.home:8090`). Both swarm nodes
run `beszel-agent` (from `modules/beszel/enable-agent.nix`), pointing at the
existing hub URL.
No migration needed for Beszel itself during the container migration. Once
all services are on the swarm, you may wish to move the Beszel hub too (as a
swarm service with a placement constraint) but this is optional.
---
## 8. DNS cutover
When a service is confirmed working on the swarm, update the FreeIPA DNS
A record from the CT 105 IP (192.168.2.225) to a swarm node IP or, when a
shared Traefik frontend is in place, to a round-robin record across both nodes.
**Recommended approach — Traefik as the single entry point:**
```
service.lan.ddnsgeek.com → Traefik on swarm (global mode)
docker.sweet.home → keep as 192.168.2.225 (CT 105) until fully decommissioned
```
For LAN-only services using `*.sweet.home` names, update FreeIPA directly:
```bash
# On domain-controller (or via SSH):
ipa dnsrecord-mod sweet.home nextcloud --a-rec=192.168.2.230
# Add 192.168.2.231 as a second A record for round-robin (optional):
ipa dnsrecord-add sweet.home nextcloud --a-rec=192.168.2.231
```
Services behind Traefik don't need their own DNS updates — only Traefik's
own entry point IPs need to change.
---
## 9. NixOS cleanup — CT 105
Once all services are migrated:
**Remove from `modules/build-types/docker.nix`:**
- `../docker/nextcloud-cron-job.nix` — replaced by sidecar container
- `../traefik/rotate-logs.nix` — replaced by Docker log driver
**Keep in `modules/build-types/docker.nix` until CT 105 is decommissioned:**
- `../docker/docker-health-to-gotify.nix` — still monitors CT 105's own daemon
- Everything else
**When decommissioning CT 105:**
1. Confirm all NFS volumes are in use only by swarm services (not CT 105).
2. Stop CT 105: `pct stop 105` on pve1.
3. Archive/remove the `lxc-docker` and `proxmox-docker` targets from `flake.nix`.
4. Remove `hosts/docker/`, `modules/build-types/docker.nix`, and `modules/docker/`.
5. Update `variables.nix` to remove `dockerIp`, `dockerStorageIp`, `dockerHost`
(or reassign `dockerHost` to point at a swarm node for Beszel hub resolution).
---
## Rollback
Any stack can be rolled back to CT 105 independently:
```bash
# On CT 105:
docker start <service-name>
# Update DNS A record back to 192.168.2.225
ipa dnsrecord-mod sweet.home <service> --a-rec=192.168.2.225
```
CT 105 remains running throughout the cutover. Only decommission it after
every service is confirmed stable on the swarm and you have run one full
backup cycle from the new hosts.
+148 -48
View File
@@ -2,16 +2,92 @@
## Subnets ## Subnets
| Subnet | CIDR | Purpose | Routed? | | Subnet | VLAN | CIDR | Purpose | Routed? |
|---|---|---|---| |---|---|---|---|---|
| LAN | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) | | LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
| Storage | `192.168.4.0/29` | HA file server DRBD replication | No — internal `vmbr1` only, no uplink | | Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
| Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink |
| Swarm cluster | 30 | `192.168.30.0/24` | Docker Swarm gossip (TCP/UDP 7946) + VXLAN overlay (UDP 4789) | No — internal `vmbr3` only, no uplink |
The storage subnet never leaves pve1. `vmbr1` is a Proxmox Linux bridge with no physical port When expanded to a second Proxmox node, VLAN 10 (cluster), VLAN 20 (storage-client), and VLAN 30 (swarm) all share
attached; traffic between the two HA file server VMs stays in-kernel. the same inter-node trunk NIC via 802.1q VLAN tagging — different VLAN IDs, same physical cable.
The host octet is consistent across subnets for any host that has multiple interfaces — e.g. The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
ha-node1 is always `.228` (LAN: `192.168.2.228`, storage: `192.168.4.228`). bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
VLAN IDs match the third octet of each subnet (VLAN 2 → 192.168.**2**.x, VLAN 10 → 192.168.**10**.x,
VLAN 20 → 192.168.**20**.x). The host octet is consistent across all subnets — e.g. ha-node1
is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20.228` (storage client).
**Protocol separation** (enforced by firewall on HA nodes):
- NFS (ports 111, 2049, 20048): both subnets, each restricted to its own CIDR
- VLAN 2 only → `vip-lan` (192.168.2.229) — pxe-boot and other LAN clients
- VLAN 20 only → `vip-storage` (192.168.20.229) — docker, future swarm nodes
- iSCSI (port 3260): VLAN 20 only — available but not in active use; NFS is preferred
for multi-host access (shared volumes across a Docker Swarm require a shared filesystem,
not per-host block devices)
---
## DNS Zones
FreeIPA (domain-controller.sweet.home) is authoritative for all zones.
Four zones correspond to the four subnets. All zones are internal only; no external delegation.
### sweet.home — VLAN 2 (192.168.2.x)
General LAN zone. All infrastructure hostnames live here.
| Hostname | A record | Notes |
|---|---|---|
| `domain-controller.sweet.home` | `192.168.2.253` | FreeIPA / KDC / DNS |
| `ha-vip-lan.sweet.home` | `192.168.2.229` | Pacemaker `vip-lan` — NFS for LAN clients |
| `ha-server-1.sweet.home` | `192.168.2.228` | HA node 1 management NIC |
| `ha-server-2.sweet.home` | `192.168.2.227` | HA node 2 management NIC |
| `server.sweet.home` | `192.168.2.226` | Current ZFS/NFS server (retiring) |
| `docker.sweet.home` | `192.168.2.225` | Docker/Traefik host |
| `nix-cache.sweet.home` | `192.168.2.224` | Nix binary cache + remote builder |
| `pxe-boot.sweet.home` | `192.168.2.223` | PXE / TFTP / HTTP netboot |
| `tailscale-router.sweet.home` | `192.168.2.222` | Tailscale exit node |
| `tor-relay.sweet.home` | `192.168.2.221` | Tor relay |
| `pdm.sweet.home` | `192.168.2.220` | Proxmox Deploy Manager |
| `nixos.sweet.home` | `192.168.2.39` | Bare-metal workstation (DHCP) |
| `pve1.sweet.home` | `192.168.2.245` | Proxmox VE hypervisor |
| `pbs.sweet.home` | `192.168.2.244` | Proxmox Backup Server |
PTR records exist for all static hosts. The workstation (`nixos.sweet.home`) is
DHCP-assigned; its PTR is omitted.
### cluster.home — VLAN 10 (192.168.10.x)
Internal only — Corosync ring0 heartbeat and DRBD replication between HA nodes.
No VIP exists on this subnet (DRBD/Corosync endpoints are static per-node IPs).
| Hostname | A record | Notes |
|---|---|---|
| `ha-server-1.cluster.home` | `192.168.10.228` | HA node 1 cluster NIC (ens19 / vmbr1) |
| `ha-server-2.cluster.home` | `192.168.10.227` | HA node 2 cluster NIC (ens19 / vmbr1) |
PTR records exist for both. DNS here is for debugging convenience — DRBD and
Corosync use the IPs from the NixOS config directly, not DNS.
### storage.home — VLAN 20 (192.168.20.x)
Internal only — NFS (and iSCSI) client access to the HA storage VIP. NFS clients
mount from **`nfs.storage.home`** (the Pacemaker floating VIP) so mounts survive
failover transparently without reconfiguration.
| Hostname | A record | Notes |
|---|---|---|
| `nfs.storage.home` | `192.168.20.229` | Pacemaker `vip-storage` — NFS + iSCSI VIP |
| `ha-server-1.storage.home` | `192.168.20.228` | HA node 1 storage-client NIC (ens20 / vmbr2) |
| `ha-server-2.storage.home` | `192.168.20.227` | HA node 2 storage-client NIC (ens20 / vmbr2) |
| `docker.storage.home` | `192.168.20.225` | Docker host storage-client NIC (eth1 / vmbr2) |
| `server.storage.home` | `192.168.20.226` | server VM storage-client NIC (decommissioned — remove DNS record after VM is destroyed) |
PTR records exist for all five. Remove `server.storage.home`, `server.sweet.home`,
and their PTRs from FreeIPA DNS once the server VM is destroyed.
--- ---
@@ -56,16 +132,18 @@ All VMs and LXC containers run on pve1.
| IP | Hostname | Role | Status | | IP | Hostname | Role | Status |
|---|---|---|---| |---|---|---|---|
| `192.168.2.229` | ha-vip | HA file server iSCSI floating VIP (Pacemaker) | Future | | `192.168.2.229` | ha-vip-lan | HA file server LAN floating VIP (Pacemaker `vip-lan`) — LAN iSCSI + NFS | Active |
| `192.168.2.228` | ha-node1 | HA file server node 1 (DRBD + XFS + iSCSI) | Future | | `192.168.2.228` | ha-node1 | HA file server node 1 — management NIC | Active |
| `192.168.2.227` | ha-node2 | HA file server node 2 (DRBD + XFS + iSCSI) | Future | | `192.168.2.227` | ha-node2 | HA file server node 2 — management NIC | Active |
| `192.168.2.226` | server | Current NFS/ZFS file server — retires when HA is live | Retiring | | `192.168.2.226` | server | Former NFS/ZFS file server — decommissioned | Removed from flake |
| `192.168.2.225` | docker | Docker / Traefik stack | Active | | `192.168.2.225` | docker | Docker / Traefik stack (CT 105 — existing single-host) | Active |
| `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active | | `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
| `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active | | `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active |
| `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active | | `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active |
| `192.168.2.221` | tor-relay | Tor relay | Active | | `192.168.2.221` | tor-relay | Tor relay | Active |
| `192.168.2.220` | pdm | Proxmox Deploy Manager | Active | | `192.168.2.220` | pdm | Proxmox Deploy Manager | Active |
| `192.168.2.231` | ha-docker-2 | Docker Swarm node 2 — management NIC | Active |
| `192.168.2.230` | ha-docker-1 | Docker Swarm node 1 — management NIC | Active |
### Client DHCP pool (.10.59) ### Client DHCP pool (.10.59)
@@ -78,51 +156,73 @@ MAC address churn.
--- ---
## Storage network — 192.168.4.0/29 ## Cluster network — VLAN 10 — 192.168.10.224/29
Internal to pve1 only. Proxmox bridge `vmbr1`, no physical NIC attached. Internal to pve1 only. Proxmox bridge `vmbr1`, no physical NIC attached.
| IP | Hostname | Interface role | | IP | Hostname | Interface role |
|---|---|---| |---|---|---|
| `192.168.4.228` | ha-node1 | DRBD replication NIC | | `192.168.10.228` | ha-node1 | DRBD replication + Corosync ring0 (primary heartbeat) |
| `192.168.4.227` | ha-node2 | DRBD replication NIC | | `192.168.10.227` | ha-node2 | DRBD replication + Corosync ring0 (primary heartbeat) |
| — | no gateway | Isolated — not routed to LAN or internet | | — | no gateway | Isolated — not routed to LAN or internet |
Corosync ring1 (backup heartbeat only) uses the LAN IPs (`192.168.2.228` / `192.168.2.227`)
over `vmbr0` — no additional bridge needed, and DRBD traffic never crosses ring1.
--- ---
## Migration reference ## Storage-client network — VLAN 20 — 192.168.20.0/24
Current → target IP for every host being renumbered. Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached.
| Host | Current IP | New IP | Config location | | IP | Hostname | Interface / role |
|---|---|---|---| |---|---|---|
| router | `192.168.2.254` | `192.168.2.254` | unchanged | | `192.168.20.229` | ha-vip-storage | Pacemaker floating VIP — NFS + iSCSI endpoint |
| domain-controller | `192.168.2.138` | `192.168.2.253` | `/etc/sysconfig/network-scripts/ifcfg-eth0` on guest | | `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
| pve1 | `192.168.2.250` | `192.168.2.245` | `/etc/network/interfaces` on Proxmox host | | `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
| pbs | `192.168.2.108` | `192.168.2.244` | static config on PBS host | | `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — decommissioned |
| nixos workstation | `192.168.2.119` | `192.168.2.243` | `networking.interfaces` / NetworkManager on guest | | `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client (CT 105) |
| ha-node1 | — | `192.168.2.228` | future | | `192.168.20.231` | ha-docker-2 | Storage-client NIC (ens19 / vmbr2) — NFS client |
| ha-node2 | — | `192.168.2.227` | future | | `192.168.20.230` | ha-docker-1 | Storage-client NIC (ens19 / vmbr2) — NFS client |
| ha-vip | — | `192.168.2.229` | future (Pacemaker resource) | | — | no gateway | Isolated — not routed to LAN or internet |
| server | `192.168.2.252` | `192.168.2.226` | static config on guest |
| docker | `192.168.2.249` | `192.168.2.225` | static config on guest |
| nix-cache | `192.168.2.120` | `192.168.2.224` | static config on guest |
| pxe-boot | `192.168.2.247` | `192.168.2.223` | static config on guest; update `vars.pxeServerIp` in `variables.nix` ✓ |
| tailscale-router | `192.168.2.121` | `192.168.2.222` | static config on guest |
| tor-relay | `192.168.2.107` | `192.168.2.221` | static config on guest |
| pdm | `192.168.2.248` | `192.168.2.220` | static config on guest |
### Cutover notes NFS clients mount from `192.168.20.229` (surviving failover transparently via the VIP).
Firewall on each HA node restricts NFS and iSCSI ports to `192.168.20.0/24` — LAN hosts
cannot reach either service on this VIP. The `vip-storage` endpoint is not reachable
from the workstation directly (internal bridge only); health checks proxy through the
active HA node.
---
## Swarm cluster network — VLAN 30 — 192.168.30.0/24
Internal to pve1 only. Proxmox bridge `vmbr3`, no physical NIC attached.
Carries Docker Swarm inter-node traffic only: Raft consensus (TCP 2377),
Serf gossip (TCP/UDP 7946), and VXLAN overlay data path (UDP 4789).
Docker Swarm is initialised with `--advertise-addr` and `--data-path-addr`
both pointing to this subnet so all cluster traffic stays on `vmbr3` and
never crosses the LAN.
| IP | Hostname | Interface / role |
|---|---|---|
| `192.168.30.231` | ha-docker-2 | Swarm cluster NIC (ens20 / vmbr3) |
| `192.168.30.230` | ha-docker-1 | Swarm cluster NIC (ens20 / vmbr3) |
| — | no gateway | Isolated — not routed to LAN or internet |
### DNS zone: `swarm.home` — VLAN 30 (192.168.30.x)
| Hostname | A record | Notes |
|---|---|---|
| `ha-docker-1.swarm.home` | `192.168.30.230` | Swarm NIC — debugging only |
| `ha-docker-2.swarm.home` | `192.168.30.231` | Swarm NIC — debugging only |
Operators reach the Docker API on the LAN IPs (`192.168.2.230`/`.231`), not these addresses.
The `swarm.home` records exist for diagnostic convenience (e.g. confirming `vmbr3` routing).
### Multi-node Proxmox expansion
When a second Proxmox node (pve2) is added, VLAN 10 (cluster), VLAN 20 (storage-client),
and VLAN 30 (swarm) all extend to pve2 via 802.1q VLAN tagging on the inter-node trunk
link. All three internal networks share the same physical NIC between hypervisors —
VLAN tags provide the logical separation.
- **Do domain-controller first** — it becomes the DNS server; everything else depends on it
having its new IP and FreeIPA DNS configured before Pi-hole is retired.
- **pve1 last among physical hosts** — changing the Proxmox management IP drops the web UI
briefly; all guests keep running.
- **Update Pi-hole custom.list / FreeIPA DNS A records** to new IPs before flipping any host,
so name resolution stays valid throughout the migration.
- **variables.nix already updated** for `pxeServerIp` (.247→.223), `pbsIp` (.108→.244), and
new `domainControllerIp` (.253). Rebuild affected hosts after renumbering.
- **Router DHCP**: once domain-controller is at .253 and FreeIPA DNS is serving `sweet.home`,
switch router DHCP on with pool .10.59 and DNS option pointing to .253; retire Pi-hole CT.
- **Pi-hole's iPXE dnsmasq config** (`99-ipxe-chainload.conf`) moves to the pxe-boot CT as a
dnsmasq proxy-mode config before Pi-hole is decommissioned.
-366
View File
@@ -1,366 +0,0 @@
# Network Cutover Plan
Moves the LAN from the current flat/Pi-hole-managed state to the new IP scheme
defined in `docs/ip-addressing.md`. Works in five independent stages — each
stage is safe to pause after and resume later. Rollback steps are given at
every point where something can break.
**Before starting anything:** confirm you have
- SSH access to `192.168.2.138` (domain-controller, current IP)
- SSH access to `192.168.2.250` (pve1)
- Browser access to Pi-hole admin at `http://192.168.2.253`
- Browser access to router admin at `http://192.168.2.254`
- The FreeIPA `admin` password to hand
---
## Stage 1 — Prepare FreeIPA DNS (zero downtime)
Everything here is additive. Pi-hole keeps running. Nothing breaks if you stop
mid-stage.
### 1a. Add NextDNS forwarders
```bash
ssh wayne@192.168.2.138
kinit admin # enter FreeIPA admin password when prompted
ipa dnsconfig-mod \
--forwarder=45.90.28.142 \
--forwarder=45.90.30.142 \
--forward-policy=only
```
**Verify external resolution works through FreeIPA before continuing:**
```bash
dig @127.0.0.1 google.com +short # must return an IP, not SERVFAIL
```
### 1b. Add A records for every host at their CURRENT IPs
These represent the live state now. You'll update each record to the new IP
when you renumber that host in Stage 5.
```bash
ipa dnsrecord-add sweet.home pve1 --a-rec 192.168.2.250
ipa dnsrecord-add sweet.home pbs --a-rec 192.168.2.108
ipa dnsrecord-add sweet.home nixos --a-rec 192.168.2.119
ipa dnsrecord-add sweet.home server --a-rec 192.168.2.252
ipa dnsrecord-add sweet.home docker --a-rec 192.168.2.249
ipa dnsrecord-add sweet.home nix-cache --a-rec 192.168.2.120
ipa dnsrecord-add sweet.home pxe-boot --a-rec 192.168.2.247
ipa dnsrecord-add sweet.home tailscale-router --a-rec 192.168.2.121
ipa dnsrecord-add sweet.home tor-relay --a-rec 192.168.2.107
ipa dnsrecord-add sweet.home pdm --a-rec 192.168.2.248
ipa dnsrecord-add sweet.home router --a-rec 192.168.2.254
```
### 1c. Clean up stale reverse-zone PTR records
FreeIPA already has PTR records from an earlier import but some are wrong.
Fix them now so reverse DNS is accurate from day one.
```bash
# Remove stale "win11" entry at .250 (should be pve1)
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec win11.
ipa dnsrecord-add 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
# Fix unqualified PTR records (missing .sweet.home. suffix)
ipa dnsrecord-mod 2.168.192.in-addr.arpa 108 --ptr-rec pbs.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 248 --ptr-rec pdm.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 249 --ptr-rec docker.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 252 --ptr-rec server.sweet.home.
# Add any missing PTR records
ipa dnsrecord-add 2.168.192.in-addr.arpa 119 --ptr-rec nixos.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 120 --ptr-rec nix-cache.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 121 --ptr-rec tailscale-router.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 247 --ptr-rec pxe-boot.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 254 --ptr-rec router.sweet.home.
```
### 1d. Point domain-controller's own DNS at itself
```bash
sudo nmcli connection modify "System eth0" ipv4.dns "127.0.0.1"
sudo nmcli connection up "System eth0"
```
**Verify:**
```bash
dig pve1.sweet.home +short # must return 192.168.2.250
dig google.com +short # must return an IP (NextDNS forwarding)
```
**Rollback 1d:** `sudo nmcli connection modify "System eth0" ipv4.dns "192.168.2.253" && sudo nmcli connection up "System eth0"`
---
## Stage 2 — Move pxe-boot DHCP options off Pi-hole (zero downtime)
Pi-hole's dnsmasq currently serves the iPXE boot options via
`99-ipxe-chainload.conf`. Before Pi-hole is retired, that config must move to
the pxe-boot CT running dnsmasq in proxy mode so PXE boot keeps working.
### 2a. Add dnsmasq proxy config to the pxe-boot NixOS module
In `modules/build-types/pxe-boot.nix`, add:
```nix
services.dnsmasq = {
enable = true;
settings = {
# Proxy mode: respond only to PXE DHCP requests, leave normal leases to router
dhcp-range = [ "192.168.2.0,proxy" ];
# iPXE client detection
dhcp-match = [
"set:ipxe,175"
"set:efi64,option:client-arch,7"
"set:efi64,option:client-arch,9"
];
dhcp-userclass = "set:ipxe,iPXE";
# Boot file selection
dhcp-boot = [
"tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
"tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
"tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
"undionly.kpxe,,${vars.pxeServerIp}"
];
};
};
```
### 2b. Rebuild and deploy the pxe-boot CT
```bash
# On pve1 — build the new tarball
nix build .#lxc-pxe-boot.config.system.build.tarball
# Verify dnsmasq starts correctly in the CT after deploy
ssh nixos@192.168.2.247 systemctl status dnsmasq
```
### 2c. Remove the iPXE config from Pi-hole
In the Pi-hole CT, remove `/etc/dnsmasq.d/99-ipxe-chainload.conf` and
restart the FTL service:
```bash
ssh wayne@pve1.sweet.home \
"sudo pct exec 100 -- bash -c 'rm /etc/dnsmasq.d/99-ipxe-chainload.conf && systemctl restart pihole-FTL'"
```
**Verify:** PXE boot a test machine — it should still get an iPXE response and
reach the boot menu.
**Rollback 2c:** restore the file from the Pi-hole config backup at
`/etc/pihole/config_backups/` and restart pihole-FTL.
---
## Stage 3 — DHCP migration: Pi-hole → router (brief maintenance window)
**Do this in the evening.** Existing DHCP leases stay valid during the
switchover so connected devices don't drop — only new lease requests fail
during the gap, which is under 60 seconds if you follow the steps in order.
The key: configure the router's DHCP DNS option to point at `.253` (Pi-hole's
current IP). This way, all new leases issued by the router still get the same
DNS server address — clients never need to change their DNS config. When Pi-hole
is retired and the DC takes `.253` in Stage 4, `.253` just starts answering
differently. No client reconfiguration.
### 3a. Pre-configure router DHCP (do not enable yet)
Log into `http://192.168.2.254`, find the DHCP settings and fill in — but
leave DHCP **disabled** until step 3b:
| Setting | Value |
|---|---|
| Start IP | 192.168.2.10 |
| End IP | 192.168.2.59 |
| Subnet mask | 255.255.255.0 |
| Gateway | 192.168.2.254 |
| Primary DNS | 192.168.2.253 |
| Secondary DNS | *(leave blank)* |
| Lease time | 24h |
Save without enabling.
### 3b. Switchover (do steps in quick succession)
1. **Disable Pi-hole DHCP:** Pi-hole admin UI → Settings → DHCP → uncheck
"DHCP server enabled" → Save
2. **Enable router DHCP** immediately after step 1
### 3c. Verify router DHCP is working
On a phone or laptop, disconnect from WiFi and reconnect (or run
`sudo dhclient -r && sudo dhclient` on a Linux host):
```bash
ip addr show # IP should be in 192.168.2.1059 range
dig google.com # should resolve (Pi-hole DNS still running at .253)
dig pve1.sweet.home # should resolve via FreeIPA at .138 (relayed via Pi-hole)
```
Wait 1015 minutes for the most active devices to renew their leases. There's
no need to wait for all leases to expire before proceeding.
**Rollback 3b:** Re-enable Pi-hole DHCP. Disable router DHCP. Done — existing
leases remain valid so most devices are unaffected.
---
## Stage 4 — Move domain-controller from .138 to .253
Pi-hole lives at `.253`. The DC must take `.253` the moment Pi-hole stops so
clients that still have `.253` as their DNS server don't notice the change.
Script these commands in advance and run them in rapid succession.
**Pre-stage: have this SSH command ready before running step 4a:**
```bash
ssh wayne@192.168.2.138 "
sudo nmcli connection modify 'System eth0' \
ipv4.addresses '192.168.2.253/24' \
ipv4.gateway '192.168.2.254' \
ipv4.dns '127.0.0.1' \
ipv4.method manual && \
sudo nmcli connection up 'System eth0'
"
```
**Also update the Proxmox VM config to match (run from pve1):**
```bash
sudo qm set 108 \
--ipconfig0 ip=192.168.2.253/24,gw=192.168.2.254 \
--nameserver 192.168.2.253
```
### 4a. Stop Pi-hole
```bash
ssh wayne@pve1.sweet.home "sudo pct stop 100"
```
### 4b. Immediately: change DC's IP to .253
Run the pre-staged SSH command from above. You have ~30 seconds before any
client notices Pi-hole is gone. If SSH to `.138` refuses (the IP is already
changing), open a Proxmox console to VM 108 and run the `nmcli` commands
there.
### 4c. Update Proxmox VM config
Run the pre-staged `qm set 108` command from above.
### 4d. Verify
```bash
ssh wayne@192.168.2.253 # must connect (new DC IP)
dig @192.168.2.253 pve1.sweet.home +short # must return 192.168.2.250
dig @192.168.2.253 google.com +short # must return an IP
```
From a client device that renewed its DHCP lease in Stage 3:
```bash
cat /etc/resolv.conf # should show 192.168.2.253
dig pve1.sweet.home # should resolve
```
**Rollback 4:** `ssh wayne@pve1.sweet.home "sudo pct start 100"`. Change DC IP
back to .138 via Proxmox console. This restores full Pi-hole DNS/DHCP service.
Leave Pi-hole CT stopped-but-intact for 48 hours before deleting it.
---
## Stage 5 — Host renumbering (one at a time, any order)
For each host:
1. Update FreeIPA DNS A record and PTR record to the new IP
2. Change the static IP on the host itself
3. Verify SSH to new IP
4. Update `variables.nix` if that host has an IP variable (pxe-boot, pbs — already done in this PR)
**FreeIPA record update template** (run as admin on domain-controller):
```bash
ipa dnsrecord-mod sweet.home <hostname> --a-rec <new-ip>
ipa dnsrecord-del 2.168.192.in-addr.arpa <old-last-octet> --ptr-rec <hostname>.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa <new-last-octet> --ptr-rec <hostname>.sweet.home.
```
### Renumbering order
| # | Host | Old IP | New IP | How to change IP |
|---|---|---|---|---|
| 1 | nixos workstation | .119 | .243 | NetworkManager on guest; or `nmcli connection modify` |
| 2 | nix-cache | .120 | .224 | `pct set 102 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.224/24,gw=192.168.2.254` then `pct reboot 102` |
| 3 | tailscale-router | .121 | .222 | Static config on guest; check Tailscale ACLs if IP is referenced there |
| 4 | tor-relay | .107 | .221 | `pct set 104 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.221/24,gw=192.168.2.254` then `pct reboot 104` |
| 5 | pdm | .248 | .220 | `pct set 106 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.220/24,gw=192.168.2.254` then `pct reboot 106` |
| 6 | pxe-boot | .247 | .223 | `pct set 103 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.223/24,gw=192.168.2.254` then rebuild NixOS (already updated in variables.nix) |
| 7 | server | .252 | .226 | Static config on guest; NFS clients (docker) lose mounts briefly — they remount automatically |
| 8 | docker | .249 | .225 | Static config on guest; do this after server is at .226 |
| 9 | pbs | .108 | .244 | Static config on PBS host itself; update in `pbsIp` already done in variables.nix |
| 10 | pve1 | .250 | .245 | Edit `/etc/network/interfaces` on the Proxmox host — see below |
### pve1 renumber (step 10 — do last)
All guests keep running; only the Proxmox web UI is briefly unreachable.
```bash
ssh wayne@pve1.sweet.home
# Edit /etc/network/interfaces: change address from .250 to .245
sudo nano /etc/network/interfaces
# Change: address 192.168.2.250/24
# To: address 192.168.2.245/24
sudo systemctl restart networking
# SSH will drop here — reconnect to new IP
```
```bash
ssh wayne@192.168.2.245 # verify
```
Update FreeIPA DNS:
```bash
ipa dnsrecord-mod sweet.home pve1 --a-rec 192.168.2.245
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 245 --ptr-rec pve1.sweet.home.
```
**Rollback any step 5 host:** change the IP back on the guest and update the
FreeIPA record back to the old IP. The old IP is unoccupied so you can
temporarily use either.
---
## Stage 6 — Final cleanup
Once all hosts are at their new IPs and verified:
```bash
# Delete the Pi-hole CT (already stopped since Stage 4)
ssh wayne@pve1.sweet.home "sudo pct destroy 100"
# Remove stale FreeIPA records for retired addresses
ipa dnsrecord-del sweet.home pihole --del-all
ipa dnsrecord-del 2.168.192.in-addr.arpa 253 --ptr-rec pihole.sweet.home.
# Rebuild any NixOS hosts that reference pbsIp or pxeServerIp to pick up
# the updated variables.nix values (pxe-boot mandatory; others as convenient)
```
---
## Rollback summary
| What broke | How to roll back |
|---|---|
| FreeIPA DNS not resolving | Check `systemctl status named` on DC; restart if failed |
| FreeIPA DNS unreachable | `pct start 100` on pve1 (restores Pi-hole) |
| Router DHCP not handing out leases | Re-enable Pi-hole DHCP; disable router DHCP |
| DC unreachable after IP change | Proxmox console on VM 108 → `nmcli connection up "System eth0"` with old IP |
| Host unreachable after renumber | Proxmox console → revert IP; or `pct set <id> --net0 ...` old IP and reboot CT |
| pve1 web UI gone after renumber | SSH to .245 and check `/etc/network/interfaces`; if wrong, fix and restart networking |
Generated
+3 -3
View File
@@ -259,11 +259,11 @@
}, },
"nixpkgs_2": { "nixpkgs_2": {
"locked": { "locked": {
"lastModified": 1785104993, "lastModified": 1785133411,
"narHash": "sha256-eKbrvPoAOFutbYMdbB3r5EQVmFxKv24iKqHPPUXA0gM=", "narHash": "sha256-Yjv0WEg39KRYS0rBdTbu6Fc/or/ihAKk13W9sQ6VWd0=",
"owner": "NixOS", "owner": "NixOS",
"repo": "nixpkgs", "repo": "nixpkgs",
"rev": "8623c4c20aa4ca2f5fb81510d2944066c3fb0d96", "rev": "2f5a153c270b70cb0f8c11f46d96d6d3bc39f4e3",
"type": "github" "type": "github"
}, },
"original": { "original": {
+12 -8
View File
@@ -45,9 +45,9 @@
# (hostName, hostId, per-machine secrets). Every build type except # (hostName, hostId, per-machine secrets). Every build type except
# nix-cache itself consumes the nix-cache substituter and remote # nix-cache itself consumes the nix-cache substituter and remote
# builder. # builder.
mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix }: mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix, nameSuffix ? "" }:
let let
flakeTarget = "${platform}-${buildType}"; flakeTarget = "${platform}-${buildType}${nameSuffix}";
in in
nixpkgs.lib.nixosSystem { nixpkgs.lib.nixosSystem {
inherit system; inherit system;
@@ -109,9 +109,6 @@
proxmox-nix-cache = mkTarget { platform = "proxmox"; buildType = "nix-cache"; hostPath = ./hosts/nix-cache/host.nix; }; proxmox-nix-cache = mkTarget { platform = "proxmox"; buildType = "nix-cache"; hostPath = ./hosts/nix-cache/host.nix; };
lxc-nix-cache = mkTarget { platform = "lxc"; buildType = "nix-cache"; hostPath = ./hosts/nix-cache/host.nix; }; lxc-nix-cache = mkTarget { platform = "lxc"; buildType = "nix-cache"; hostPath = ./hosts/nix-cache/host.nix; };
linode-server = mkTarget { platform = "linode"; buildType = "server"; hostPath = ./hosts/server/host.nix; };
proxmox-server = mkTarget { platform = "proxmox"; buildType = "server"; hostPath = ./hosts/server/host.nix; };
lxc-server = mkTarget { platform = "lxc"; buildType = "server"; hostPath = ./hosts/server/host.nix; };
linode-docker = mkTarget { platform = "linode"; buildType = "docker"; hostPath = ./hosts/docker/host.nix; }; linode-docker = mkTarget { platform = "linode"; buildType = "docker"; hostPath = ./hosts/docker/host.nix; };
proxmox-docker = mkTarget { platform = "proxmox"; buildType = "docker"; hostPath = ./hosts/docker/host.nix; }; proxmox-docker = mkTarget { platform = "proxmox"; buildType = "docker"; hostPath = ./hosts/docker/host.nix; };
@@ -131,8 +128,11 @@
lxc-tor-relay = mkTarget { platform = "lxc"; buildType = "tor-relay"; hostPath = ./hosts/tor-relay/host.nix; }; lxc-tor-relay = mkTarget { platform = "lxc"; buildType = "tor-relay"; hostPath = ./hosts/tor-relay/host.nix; };
proxmox-ha-server-1 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-1/host.nix; }; proxmox-ha-server-1 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-1/host.nix; nameSuffix = "-1"; };
proxmox-ha-server-2 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-2/host.nix; }; proxmox-ha-server-2 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-2/host.nix; nameSuffix = "-2"; };
proxmox-ha-docker-1 = mkTarget { platform = "proxmox"; buildType = "ha-docker"; hostPath = ./hosts/ha-docker-1/host.nix; nameSuffix = "-1"; };
proxmox-ha-docker-2 = mkTarget { platform = "proxmox"; buildType = "ha-docker"; hostPath = ./hosts/ha-docker-2/host.nix; nameSuffix = "-2"; };
}; };
# Auto-install environments (migrated from the former nix-auto-installer # Auto-install environments (migrated from the former nix-auto-installer
@@ -203,7 +203,11 @@
(modulesPath + "/installer/netboot/netboot-minimal.nix") (modulesPath + "/installer/netboot/netboot-minimal.nix")
]; ];
}) })
{ networking.hostName = "nixos-minimal"; } {
networking.hostName = "nixos-minimal";
system.stateVersion = "26.05";
boot.zfs.forceImportRoot = false;
}
]; ];
}; };
+4 -4
View File
@@ -5,10 +5,10 @@
hostName = "docker"; hostName = "docker";
hostId = "007f0200"; hostId = "007f0200";
useDHCP = false; useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{ interfaces = {
address = vars.dockerIp; ${vars.vmLanInterface}.ipv4.addresses = [{ address = vars.dockerIp; prefixLength = vars.lanPrefixLength; }];
prefixLength = vars.lanPrefixLength; ${vars.lxcStorageInterface}.ipv4.addresses = [{ address = vars.dockerStorageIp; prefixLength = vars.haClientPrefixLength; }];
}]; };
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; }; defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ]; nameservers = [ vars.domainControllerIp ];
}; };
+34
View File
@@ -0,0 +1,34 @@
{ vars, ... }:
{
networking = {
hostName = vars.haDocker1Host;
hostId = "a1d0c4e1";
useDHCP = false;
interfaces = {
# ens18 — LAN management NIC (vmbr0, 192.168.2.0/24)
${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.haDocker1Ip;
prefixLength = vars.lanPrefixLength;
}];
# ens19 — storage-client NIC (vmbr2, 192.168.20.0/24) — NFS from HA cluster
${vars.haDockerStorageInterface}.ipv4.addresses = [{
address = vars.haDocker1StorageIp;
prefixLength = vars.haClientPrefixLength;
}];
# ens20 — swarm cluster NIC (vmbr3, 192.168.30.0/24) — Docker gossip + VXLAN
${vars.haDockerSwarmInterface}.ipv4.addresses = [{
address = vars.haDocker1SwarmIp;
prefixLength = vars.haDockerSwarmPrefixLength;
}];
};
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# Only register the LAN IP with IPA DNS. Without this, sssd dyndns
# would also register Docker bridge IPs (172.x.x.x) and the storage/swarm
# NIC IPs as A records for ha-docker-1.sweet.home.
security.ipa.dyndns.interface = vars.vmLanInterface;
system.stateVersion = "26.05";
}
+32
View File
@@ -0,0 +1,32 @@
{ vars, ... }:
{
networking = {
hostName = vars.haDocker2Host;
hostId = "a2d0c4e2";
useDHCP = false;
interfaces = {
# ens18 — LAN management NIC (vmbr0, 192.168.2.0/24)
${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.haDocker2Ip;
prefixLength = vars.lanPrefixLength;
}];
# ens19 — storage-client NIC (vmbr2, 192.168.20.0/24) — NFS from HA cluster
${vars.haDockerStorageInterface}.ipv4.addresses = [{
address = vars.haDocker2StorageIp;
prefixLength = vars.haClientPrefixLength;
}];
# ens20 — swarm cluster NIC (vmbr3, 192.168.30.0/24) — Docker gossip + VXLAN
${vars.haDockerSwarmInterface}.ipv4.addresses = [{
address = vars.haDocker2SwarmIp;
prefixLength = vars.haDockerSwarmPrefixLength;
}];
};
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# Only register the LAN IP with IPA DNS — same reasoning as ha-docker-1.
security.ipa.dyndns.interface = vars.vmLanInterface;
system.stateVersion = "26.05";
}
+5 -18
View File
@@ -1,30 +1,17 @@
{ vars, ... }: { vars, ... }:
{ {
imports = [
(import ../../modules/beszel/host-token.nix {
name = "ha-server-1";
sopsFile = ../../secrets/ha-server-1.yaml;
})
];
networking = { networking = {
hostName = vars.haServer1Host; hostName = vars.haServer1Host;
hostId = "3a4b5c6d"; hostId = "3a4b5c6d";
useDHCP = false; useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{ interfaces = {
address = vars.haServer1Ip; ${vars.vmLanInterface}.ipv4.addresses = [{ address = vars.haServer1Ip; prefixLength = vars.lanPrefixLength; }];
prefixLength = vars.lanPrefixLength; ${vars.vmStorageInterface}.ipv4.addresses = [{ address = vars.haServer1StorageIp; prefixLength = vars.haStoragePrefixLength; }];
}]; ${vars.vmStorageClientInterface}.ipv4.addresses = [{ address = vars.haServer1ClientIp; prefixLength = vars.haClientPrefixLength; }];
interfaces.${vars.vmStorageInterface}.ipv4.addresses = [{ };
address = vars.haServer1StorageIp;
prefixLength = vars.haStoragePrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; }; defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ]; nameservers = [ vars.domainControllerIp ];
}; };
# Set KEY after pairing this host with the beszel hub; the token is sops-managed.
services.beszel.agent.environment.KEY = "";
system.stateVersion = "26.05"; system.stateVersion = "26.05";
} }
+5 -18
View File
@@ -1,30 +1,17 @@
{ vars, ... }: { vars, ... }:
{ {
imports = [
(import ../../modules/beszel/host-token.nix {
name = "ha-server-2";
sopsFile = ../../secrets/ha-server-2.yaml;
})
];
networking = { networking = {
hostName = vars.haServer2Host; hostName = vars.haServer2Host;
hostId = "7e8f9a0b"; hostId = "7e8f9a0b";
useDHCP = false; useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{ interfaces = {
address = vars.haServer2Ip; ${vars.vmLanInterface}.ipv4.addresses = [{ address = vars.haServer2Ip; prefixLength = vars.lanPrefixLength; }];
prefixLength = vars.lanPrefixLength; ${vars.vmStorageInterface}.ipv4.addresses = [{ address = vars.haServer2StorageIp; prefixLength = vars.haStoragePrefixLength; }];
}]; ${vars.vmStorageClientInterface}.ipv4.addresses = [{ address = vars.haServer2ClientIp; prefixLength = vars.haClientPrefixLength; }];
interfaces.${vars.vmStorageInterface}.ipv4.addresses = [{ };
address = vars.haServer2StorageIp;
prefixLength = vars.haStoragePrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; }; defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ]; nameservers = [ vars.domainControllerIp ];
}; };
# Set KEY after pairing this host with the beszel hub; the token is sops-managed.
services.beszel.agent.environment.KEY = "";
system.stateVersion = "26.05"; system.stateVersion = "26.05";
} }
-11
View File
@@ -1,13 +1,6 @@
{ vars, ... }: { vars, ... }:
{ {
imports = [
(import ../../modules/beszel/host-token.nix {
name = "nix-cache";
sopsFile = ../../secrets/nix-cache.yaml;
})
];
networking = { networking = {
hostName = vars.nixCacheHost; hostName = vars.nixCacheHost;
useDHCP = false; useDHCP = false;
@@ -19,10 +12,6 @@
nameservers = [ vars.domainControllerIp ]; nameservers = [ vars.domainControllerIp ];
}; };
services.beszel.agent.environment = {
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
};
# Preserved from the pre-refactor `nix-cache` target — stateVersion must # Preserved from the pre-refactor `nix-cache` target — stateVersion must
# never be bumped on an already-installed machine. # never be bumped on an already-installed machine.
system.stateVersion = "25.05"; system.stateVersion = "25.05";
+1 -1
View File
@@ -26,7 +26,7 @@
# Optional: set environment vars # Optional: set environment vars
sessionVariables = { sessionVariables = {
EDITOR = "vim"; EDITOR = "nano";
SOPS_AGE_KEY_FILE = "${config.home.homeDirectory}/.config/sops/age/keys.txt"; SOPS_AGE_KEY_FILE = "${config.home.homeDirectory}/.config/sops/age/keys.txt";
}; };
+1 -1
View File
@@ -11,7 +11,7 @@
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; }; defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
nameservers = [ vars.domainControllerIp ]; nameservers = [ vars.domainControllerIp ];
}; };
services.beszel.agent.environment = { };
# Preserved from the pre-refactor `pxe-boot` target — stateVersion must # Preserved from the pre-refactor `pxe-boot` target — stateVersion must
# never be bumped on an already-installed machine. # never be bumped on an already-installed machine.
system.stateVersion = "25.05"; system.stateVersion = "25.05";
-32
View File
@@ -1,32 +0,0 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "server";
sopsFile = ../../secrets/server.yaml;
})
];
networking = {
hostName = vars.nfsServerHost;
hostId = "6689f93e";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.serverIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
services.beszel.agent.environment = {
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
LOG_LEVEL = "debug";
};
# Preserved from the pre-refactor `server` target — stateVersion must never
# be bumped on an already-installed machine.
system.stateVersion = "25.05";
}
-11
View File
@@ -1,13 +1,6 @@
{ vars, ... }: { vars, ... }:
{ {
imports = [
(import ../../modules/beszel/host-token.nix {
name = "tailscale-router";
sopsFile = ../../secrets/tailscale-router.yaml;
})
];
networking = { networking = {
hostName = "tailscale-router"; hostName = "tailscale-router";
useDHCP = false; useDHCP = false;
@@ -19,10 +12,6 @@
nameservers = [ vars.domainControllerIp ]; nameservers = [ vars.domainControllerIp ];
}; };
services.beszel.agent.environment = {
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
};
# No networking.hostId: only ZFS-touching hosts (server, docker) need one # No networking.hostId: only ZFS-touching hosts (server, docker) need one
# for pool-import safety, and this host does neither. # for pool-import safety, and this host does neither.
-11
View File
@@ -1,13 +1,6 @@
{ vars, ... }: { vars, ... }:
{ {
imports = [
(import ../../modules/beszel/host-token.nix {
name = "tor-relay";
sopsFile = ../../secrets/tor-relay.yaml;
})
];
networking = { networking = {
hostName = "tor-relay"; hostName = "tor-relay";
useDHCP = false; useDHCP = false;
@@ -22,10 +15,6 @@
# No networking.hostId: only ZFS-touching hosts need one for pool-import # No networking.hostId: only ZFS-touching hosts need one for pool-import
# safety, and this host does neither. # safety, and this host does neither.
services.beszel.agent.environment = {
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
};
# A genuinely new host (not a pre-refactor carry-over), so it tracks the # A genuinely new host (not a pre-refactor carry-over), so it tracks the
# flake's current nixpkgs release rather than being pinned to an older one. # flake's current nixpkgs release rather than being pinned to an older one.
system.stateVersion = "26.05"; system.stateVersion = "26.05";
+18 -5
View File
@@ -1,10 +1,23 @@
{ vars, ... }: { config, vars, ... }:
{ {
services.beszel.agent.enable = true; # Universal token shared by all beszel agents. Add to secrets/common.yaml:
services.beszel.agent.environment = { # sops secrets/common.yaml
#DOCKER_HOST = "tcp://docker-socket-proxy:2375"; # beszel-token: <value from the beszel hub UI>
HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}"; sops.secrets."beszel-token" = { };
sops.templates."beszel.env".content = ''
TOKEN=${config.sops.placeholder."beszel-token"}
'';
services.beszel.agent = {
enable = true;
environmentFile = config.sops.templates."beszel.env".path;
environment = {
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
KEY = vars.beszelHubKey;
};
}; };
# The upstream module runs beszel-agent under DynamicUser with # The upstream module runs beszel-agent under DynamicUser with
-11
View File
@@ -1,11 +0,0 @@
{ name, sopsFile }:
{ config, ... }:
{
sops.secrets."beszel-token".sopsFile = sopsFile;
sops.templates."${name}-beszel.env".content = ''
TOKEN=${config.sops.placeholder."beszel-token"}
'';
services.beszel.agent.environmentFile = config.sops.templates."${name}-beszel.env".path;
}
-3
View File
@@ -33,9 +33,6 @@
]; ];
users.users.${vars.primaryUser}.extraGroups = [ "docker" ]; users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
# Grant the IPA domain user docker access via the local group so that
# `wayne` can manage containers without sudo.
users.groups.docker.members = [ "wayne" ];
services.openssh.settings.PermitRootLogin = "yes"; services.openssh.settings.PermitRootLogin = "yes";
networking.firewall.allowedTCPPorts = [ networking.firewall.allowedTCPPorts = [
+66
View File
@@ -81,4 +81,70 @@
programs.firefox.enable = true; programs.firefox.enable = true;
nixpkgs.config.allowUnfree = true; nixpkgs.config.allowUnfree = true;
# GUI-specific Home Manager additions for the IPA primary user, extending
# the baseline in modules/ipa/client.nix with desktop apps and services
# that only make sense on a graphical workstation.
home-manager.users.${vars.ipaUser} = { pkgs, ... }: {
home = {
packages = with pkgs; [
git
vim
nextcloud-client
chromium
claude-code
fish
sops
];
sessionVariables = {
EDITOR = "nano";
SOPS_AGE_KEY_FILE = "/home/${vars.ipaUser}/.config/sops/age/keys.txt";
};
file = {
".local/share/applications/proxmox-chromium-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox (Chromium)
Exec=chromium --app=https://pve.${vars.homeDomain}:${toString vars.ports.pveWeb} --window-size=1920,1080 --window-position=0,0
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=Hypervisor;
StartupWMClass=PVE
'';
".local/share/applications/pbs-chromium-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox Backup Server (Chromium)
Exec=chromium --app=https://${vars.pbsIp}:${toString vars.ports.pbsWeb} --window-size=1920,1080 --window-position=0,0
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=backup;
'';
".local/share/applications/proxmox-firefox-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox (Firefox)
Exec=firefox --new-instance https://pve.${vars.homeDomain}:${toString vars.ports.pveWeb} --profile ProxmoxWebApp --window-size=1920,1080 --class ProxmoxWebApp
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=Hypervisor;
StartupWMClass=PVE
'';
".local/share/applications/pbs-firefox-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox Backup Server (Firefox)
Exec=firefox --new-window https://${vars.pbsIp}:${toString vars.ports.pbsWeb} --profile PbsWebApp --window-size=1920,1080 --class PbsWebApp
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=backup;
StartupWMClass=PBS
'';
};
};
services.nextcloud-client = {
enable = true;
startInBackground = true;
};
};
} }
+84
View File
@@ -0,0 +1,84 @@
# Docker Swarm node build type.
#
# Produces NixOS hosts that form a Docker Swarm manager cluster. Two nodes
# (ha-docker-1, ha-docker-2) are both managers so either can accept Docker
# API and `docker stack` commands.
#
# Key differences from the existing `docker` build type (used by CT 105):
# - nextcloud-cron-job.nix is EXCLUDED — `docker exec` breaks in swarm
# because the target container may be on the other node. The cron job
# is replaced by a nextcloud-cron sidecar in the Nextcloud stack.
# See docs/internal/docker-swarm-cutover.md.
# - traefik/rotate-logs.nix is EXCLUDED — log rotation moves to Docker's
# json-file log driver (max-size/max-file on the Traefik service
# definition). See docs/internal/docker-swarm-cutover.md.
# - raspi/mount-data.nix is EXCLUDED — specific to CT 105's backup role.
# - Swarm firewall ports (2377/tcp, 7946/tcp+udp, 4789/udp) are opened
# on the swarm NIC (ens20/vmbr3) only.
# - checkReversePath = "loose" is required for the Swarm ingress routing
# mesh: VXLAN return traffic is asymmetric (arrives ens20, exits ens18).
# - beszel-agent is enabled for host-level monitoring.
{ pkgs, vars, ... }:
{
# Pin Docker Engine to version 29, matching CT 105, so image layers cached
# on NFS volumes remain compatible across old and new hosts.
nixpkgs.overlays = [
(final: prev: {
docker = prev.docker_29;
docker_cli = prev.docker_29;
})
];
imports = [
../docker/enable-service.nix
../docker/mount-data.nix
../docker/docker-health-to-gotify.nix
../beszel/enable-agent.nix
../services/enable-rpcbind.nix
];
environment.systemPackages = with pkgs; [
nfs-utils
];
boot.supportedFilesystems = [ "nfs" ];
systemd.tmpfiles.rules = [
# Symlink ~/docker → NFS config mount so the docker-health-to-gotify
# script (and operator convenience) resolves ~/docker/... correctly.
"L+ /home/${vars.primaryUser}/docker - - - - ${vars.nfsShares.dockerConfig.mountpoint}"
"d /mnt/docker 0755 ${vars.primaryUser} users -"
];
users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
networking.firewall = {
# LAN-facing service ports — same as the existing docker build type.
allowedTCPPorts = [
vars.ports.dockerHttp
vars.ports.dockerHttps
vars.ports.dockerExtra
vars.ports.beszelHub
];
# Swarm inter-node ports restricted to the swarm NIC (ens20/vmbr3).
# vmbr3 is an isolated internal bridge — no LAN reachability.
interfaces.${vars.haDockerSwarmInterface} = {
allowedTCPPorts = [
vars.ports.dockerSwarmMgmt # 2377 — Raft + cluster management
vars.ports.dockerSwarmDisc # 7946 — Serf gossip (TCP half)
];
allowedUDPPorts = [
vars.ports.dockerSwarmDisc # 7946 — Serf gossip (UDP half)
vars.ports.dockerSwarmVxlan # 4789 — VXLAN overlay data path
];
};
# Docker Swarm ingress routing mesh creates asymmetric routes: a request
# arrives on ens18 (LAN) for a container that lives on ens20's VXLAN
# overlay; the return path differs from the incoming interface. Strict
# rp_filter drops these packets. "loose" allows them.
checkReversePath = "loose";
};
}
+15 -5
View File
@@ -14,15 +14,18 @@
# the Beszel hub) is not set yet — add it to hosts/ha-server-{1,2}/host.nix # the Beszel hub) is not set yet — add it to hosts/ha-server-{1,2}/host.nix
# under services.beszel.agent.environment.KEY once the hub accepts the # under services.beszel.agent.environment.KEY once the hub accepts the
# new agents, following the pattern in hosts/server/host.nix. # new agents, following the pattern in hosts/server/host.nix.
{ lib, vars, ... }: { lib, pkgs, vars, ... }:
let let
# Generates /etc/exports lines for all nfsShares data entries. Shared # Generates /etc/exports lines for all nfsShares data entries.
# pattern with modules/build-types/server.nix — both export the same # LAN (VLAN 2): NFS via vip-lan (192.168.2.229) for pxe-boot and other LAN clients.
# set of shares, differing only in the storage root they serve from. # Storage-client (VLAN 20): NFS via vip-storage (192.168.20.229) for docker and
# future swarm nodes; firewall restricts these ports to haClientCidr only.
mkNfsExports = storageRoot: mkNfsExports = storageRoot:
lib.concatMapStrings lib.concatMapStrings
(share: " ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n") (share:
" ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n" +
" ${storageRoot}/${share.subpath} ${vars.haClientCidr}${vars.nfsShares.options}\n")
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares)); (lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares));
in in
{ {
@@ -33,6 +36,10 @@ in
../beszel/enable-agent.nix ../beszel/enable-agent.nix
]; ];
# xfsprogs: mkfs.xfs/xfs_info needed by cluster-init.sh.
# openiscsi: iscsiadm needed by acceptance-tests.sh T4 (iSCSI discovery check).
environment.systemPackages = [ pkgs.xfsprogs pkgs.openiscsi ];
services.nfs.server = { services.nfs.server = {
enable = true; enable = true;
exports = mkNfsExports vars.haStorageRoot; exports = mkNfsExports vars.haStorageRoot;
@@ -41,4 +48,7 @@ in
# Pacemaker controls nfs-server — prevent systemd from starting it at boot # Pacemaker controls nfs-server — prevent systemd from starting it at boot
# on both nodes (only the Active node should be serving NFS). # on both nodes (only the Active node should be serving NFS).
systemd.services.nfs-server.wantedBy = lib.mkForce [ ]; systemd.services.nfs-server.wantedBy = lib.mkForce [ ];
# Same reason as server.nix: exports use standard auth, not Kerberos.
systemd.services.rpc-svcgssd.enable = false;
} }
+23 -20
View File
@@ -6,6 +6,10 @@ let
tftpRoot = "${pxeRoot}/tftp"; tftpRoot = "${pxeRoot}/tftp";
pxeBaseUrl = "http://${vars.pxeServerIp}"; pxeBaseUrl = "http://${vars.pxeServerIp}";
# Base network address extracted from lanCidr (e.g. "192.168.2.0" from
# "192.168.2.0/24") — used by dnsmasq's proxy DHCP range directive.
lanBaseAddr = lib.head (lib.splitString "/" vars.lanCidr);
bootIpxe = pkgs.writeText "boot.ipxe" '' bootIpxe = pkgs.writeText "boot.ipxe" ''
#!ipxe #!ipxe
@@ -73,16 +77,16 @@ let
boot boot
''; '';
# Kickstart file for domain-controller.sweet.home. # Kickstart file for ${vars.ipaServer}.
# Installs Rocky Linux 9, sets a static IP, creates wayne with the # Installs Rocky Linux 9, sets a static IP, creates ${vars.ipaUser} with
# admin SSH key, then on first reboot runs ipa-server-install via a # the admin SSH key, then on first reboot runs ipa-server-install via a
# systemd oneshot service. Passwords are generated at %post time, # systemd oneshot service. Passwords are generated at %post time, written
# written to /root/ipa-credentials.txt (chmod 600), and read back by # to /root/ipa-credentials.txt (chmod 600), and read back by the
# the first-boot script — never hardcoded here or in the repo. # first-boot script — never hardcoded here or in the repo.
rockyFreeIpaKs = pkgs.writeText "rocky-freeipa.ks" '' rockyFreeIpaKs = pkgs.writeText "rocky-freeipa.ks" ''
#version=RHEL9 #version=RHEL9
# Unattended Rocky Linux 9 + FreeIPA install # Unattended Rocky Linux 9 + FreeIPA install
# Target: domain-controller.${vars.homeDomain} ${vars.domainControllerIp} # Target: ${vars.ipaServer} ${vars.domainControllerIp}
url --url=${rockyMirror}/BaseOS/${rockyArch}/os/ url --url=${rockyMirror}/BaseOS/${rockyArch}/os/
repo --name=appstream --baseurl=${rockyMirror}/AppStream/${rockyArch}/os/ repo --name=appstream --baseurl=${rockyMirror}/AppStream/${rockyArch}/os/
@@ -93,14 +97,14 @@ let
# DHCP during install; static IP configured in %post via NM config file # DHCP during install; static IP configured in %post via NM config file
network --bootproto=dhcp --device=link --activate network --bootproto=dhcp --device=link --activate
network --hostname=domain-controller.sweet.home network --hostname=${vars.ipaServer}
selinux --enforcing selinux --enforcing
firewall --enabled --service=ssh firewall --enabled --service=ssh
rootpw --lock rootpw --lock
user --name=wayne --groups=wheel --shell=/bin/bash user --name=${vars.ipaUser} --groups=wheel --shell=/bin/bash
sshkey --username=wayne "${vars.adminSshKey}" sshkey --username=${vars.ipaUser} "${vars.adminSshKey}"
zerombr zerombr
clearpart --all --initlabel --drives=sda clearpart --all --initlabel --drives=sda
@@ -148,7 +152,7 @@ let
# -- /etc/hosts: FQDN must resolve to the real IP (not loopback) for IPA -- # -- /etc/hosts: FQDN must resolve to the real IP (not loopback) for IPA --
sed -i '/domain-controller/d' /etc/hosts sed -i '/domain-controller/d' /etc/hosts
echo '${vars.domainControllerIp} domain-controller.${vars.homeDomain} domain-controller' >> /etc/hosts echo '${vars.domainControllerIp} ${vars.ipaServer} domain-controller' >> /etc/hosts
# -- Generate IPA passwords and store securely -- # -- Generate IPA passwords and store securely --
DM_PASS=$(openssl rand -base64 24 | tr -dc 'A-Za-z0-9' | head -c 24) DM_PASS=$(openssl rand -base64 24 | tr -dc 'A-Za-z0-9' | head -c 24)
@@ -168,13 +172,13 @@ let
ADMIN_PASS=$(grep '^IPA Admin:' /root/ipa-credentials.txt | awk '{print $NF}') ADMIN_PASS=$(grep '^IPA Admin:' /root/ipa-credentials.txt | awk '{print $NF}')
ipa-server-install \ ipa-server-install \
--realm=SWEET.HOME \ --realm=${lib.strings.toUpper vars.homeDomain} \
--domain=sweet.home \ --domain=${vars.homeDomain} \
--hostname=domain-controller.sweet.home \ --hostname=${vars.ipaServer} \
--ds-password="$DM_PASS" \ --ds-password="$DM_PASS" \
--admin-password="$ADMIN_PASS" \ --admin-password="$ADMIN_PASS" \
--setup-dns \ --setup-dns \
--forwarder=192.168.2.253 \ --forwarder=${vars.domainControllerIp} \
--no-dnssec-validation \ --no-dnssec-validation \
--no-ntp \ --no-ntp \
--unattended --unattended
@@ -314,6 +318,7 @@ in
imports = [ imports = [
../pxe-boot/stage-installer-artifacts.nix ../pxe-boot/stage-installer-artifacts.nix
../pxe-boot/mount-pxe-images.nix ../pxe-boot/mount-pxe-images.nix
../beszel/enable-agent.nix
]; ];
environment.systemPackages = with pkgs; [ environment.systemPackages = with pkgs; [
@@ -340,9 +345,7 @@ in
atftpd = { atftpd = {
enable = true; enable = true;
root = tftpRoot; root = tftpRoot;
extraOptions = [ extraOptions = [ "--verbose=5" ];
"--verbose=5"
];
}; };
openssh.settings.PermitRootLogin = "yes"; openssh.settings.PermitRootLogin = "yes";
@@ -427,7 +430,7 @@ in
# Without this dnsmasq tries to bind port 53 which systemd-resolved # Without this dnsmasq tries to bind port 53 which systemd-resolved
# already owns, causing startup failure. # already owns, causing startup failure.
port = 0; port = 0;
dhcp-range = [ "192.168.2.0,proxy" ]; dhcp-range = [ "${lanBaseAddr},proxy" ];
dhcp-match = [ dhcp-match = [
"set:ipxe,175" "set:ipxe,175"
"set:efi64,option:client-arch,7" "set:efi64,option:client-arch,7"
@@ -444,5 +447,5 @@ in
}; };
networking.firewall.allowedTCPPorts = [ vars.ports.pxeBootHttp ]; networking.firewall.allowedTCPPorts = [ vars.ports.pxeBootHttp ];
networking.firewall.allowedUDPPorts = [ vars.ports.pxeBootTftp 67 ]; networking.firewall.allowedUDPPorts = [ vars.ports.pxeBootTftp vars.ports.dhcp ];
} }
-115
View File
@@ -1,115 +0,0 @@
{ vars, lib, pkgs, ... }:
let
poolName = lib.removePrefix "/" vars.storageRoot;
# For each NFS share subpath, generate every ancestor path so ZFS datasets
# are created parent-first. e.g. "docker/config" → ["docker" "docker/config"]
ancestors = path:
let parts = lib.splitString "/" path;
in lib.imap1 (i: _: lib.concatStringsSep "/" (lib.take i parts)) parts;
poolDatasets = lib.unique (
lib.concatMap (share: ancestors share.subpath)
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares))
);
# Generates /etc/exports lines for all nfsShares data entries (every
# attrset value — excludes the bare `options` string). Both server and
# ha-server export the same share set from different storage roots, so
# this helper is the single source of truth for the export line format.
mkNfsExports = storageRoot:
lib.concatMapStrings
(share: " ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n")
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares));
in
{
imports = [
../beszel/enable-agent.nix
../services/zfs/enable-service.nix
];
boot.zfs.extraPools = [ poolName ];
# On a fresh image deploy the data disk (scsi1) starts blank — no pool
# exists yet, so zfs-import-tank.service would spin for 60 s and fail.
# This service runs first: if the pool is already present it exits instantly;
# otherwise it creates it (with all required datasets) so the standard
# import service finds it ready on the very first boot.
systemd.services."zfs-init-${poolName}" = {
description = "Initialize '${poolName}' ZFS pool on first boot if not present";
wantedBy = [ "zfs-import-${poolName}.service" ];
before = [ "zfs-import-${poolName}.service" ];
after = [ "systemd-udev-settle.service" ];
unitConfig.DefaultDependencies = false;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
};
path = [ pkgs.zfs_unstable ];
script = ''
# Already imported nothing to do.
if zpool list "${poolName}" >/dev/null 2>&1; then
exit 0
fi
# Locate the data disk first used for both the fallback import
# attempt and, only if the disk is genuinely blank, pool creation.
DATA_DISK=""
for candidate in /dev/disk/by-id/scsi-*drive-scsi1; do
[[ "$candidate" == *-part* ]] && continue
[ -b "$candidate" ] && DATA_DISK="$candidate" && break
done
if [ -z "$DATA_DISK" ]; then
echo "zfs-init-${poolName}: no data disk found (expected /dev/disk/by-id/scsi-*drive-scsi1)" >&2
exit 1
fi
# Try importing via the by-id symlink directory first (normal path),
# then fall back to scanning the disk directly. The two-step exists
# because of a udev race: systemd-udev-settle.service can clear before
# /dev/disk/by-id/ entries are fully populated, causing the first
# import to fail even when the pool is intact on the disk.
if zpool import -d /dev/disk/by-id -N "${poolName}" 2>/dev/null; then
exit 0
fi
if zpool import -d "$DATA_DISK" -N "${poolName}" 2>/dev/null; then
exit 0
fi
# Both import attempts failed. Before creating a new pool, verify the
# disk is genuinely blank if ZFS label metadata is present the import
# failed for some other reason and we must not clobber existing data.
if zdb -l "$DATA_DISK" 2>/dev/null | grep -q "name: '${poolName}'"; then
echo "zfs-init-${poolName}: $DATA_DISK has ZFS pool '${poolName}' metadata but import failed refusing to overwrite existing data. Run 'zpool import -d $DATA_DISK ${poolName}' manually to investigate." >&2
exit 1
fi
# Disk is genuinely blank: create the pool. -f is intentionally
# omitted so that if we somehow reach this point with an existing pool
# on the disk, zpool refuses rather than silently destroying data.
echo "zfs-init-${poolName}: creating pool on $DATA_DISK"
zpool create "${poolName}" "$DATA_DISK"
${lib.concatMapStrings (ds: ''
zfs create "${poolName}/${ds}"
'') poolDatasets}
'';
};
systemd.services.nfs-server = {
after = [ "zfs-mount.service" ];
requires = [ "zfs-mount.service" ];
};
services.nfs.server = {
enable = true;
exports = mkNfsExports vars.storageRoot;
};
# mountd (20048) is needed for showmount/NFSv3 mount protocol — without it
# clients can reach portmapper (111) and get the mountd port back, then
# time out trying to connect to it. All three ports need TCP and UDP.
networking.firewall.allowedTCPPorts = [ vars.ports.nfsRpcbind vars.ports.nfsd vars.ports.nfsMountd ];
networking.firewall.allowedUDPPorts = [ vars.ports.nfsRpcbind vars.ports.nfsd vars.ports.nfsMountd ];
}
+23 -43
View File
@@ -5,13 +5,13 @@ let
sudo nixos-rebuild switch \ sudo nixos-rebuild switch \
--no-write-lock-file \ --no-write-lock-file \
--refresh \ --refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target) --flake git+https://${vars.giteaDomain}/${vars.giteaRepoPath}.git?dir=${vars.giteaRepoFlakePath}#$(cat /etc/flake-target)
''; '';
testCmd = '' testCmd = ''
sudo nixos-rebuild test \ sudo nixos-rebuild test \
--no-write-lock-file \ --no-write-lock-file \
--refresh \ --refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target) --flake git+https://${vars.giteaDomain}/${vars.giteaRepoPath}.git?dir=${vars.giteaRepoFlakePath}#$(cat /etc/flake-target)
''; '';
buildImageFn = '' buildImageFn = ''
buildImage() { buildImage() {
@@ -39,21 +39,18 @@ in
}; };
interactiveShellInit = buildImageFn; interactiveShellInit = buildImageFn;
}; };
networking.networkmanager.enable = true; networking.networkmanager.enable = true;
# Recommended over the true default (bypasses ZFS's own import safeguards) # Recommended over the true default (bypasses ZFS's own import safeguards)
# per the option's own docs; matches hosts/docker/host.nix and # per the option's own docs; matches hosts/docker/host.nix and
# modules/services/zfs/enable-service.nix, which already set this # modules/services/zfs/enable-service.nix. Harmless no-op on hosts without ZFS.
# explicitly. Harmless no-op on hosts that don't use ZFS at all.
boot.zfs.forceImportRoot = false; boot.zfs.forceImportRoot = false;
# Set your time zone.
time.timeZone = vars.timeZone; time.timeZone = vars.timeZone;
# Enable QEMU agent
services.qemuGuest.enable = true; services.qemuGuest.enable = true;
# Enable docker-compose
environment.systemPackages = with pkgs; [ environment.systemPackages = with pkgs; [
vim vim
btop btop
@@ -63,11 +60,10 @@ in
]; ];
# Secrets shared by every host, decrypted at activation via each host's # Secrets shared by every host, decrypted at activation via each host's
# existing SSH host key (sops-nix derives the age key from # SSH host key (sops-nix derives the age key from
# /etc/ssh/ssh_host_ed25519_key automatically — see modules/common/README # /etc/ssh/ssh_host_ed25519_key automatically). hashedPassword secrets need
# or docs/ for the sops workflow). hashedPassword/hashedPasswordFile need
# neededForUsers so they're available before the normal secret-activation # neededForUsers so they're available before the normal secret-activation
# step, since user creation happens very early in boot. # step user creation happens very early in boot.
sops = { sops = {
defaultSopsFile = ../../secrets/common.yaml; defaultSopsFile = ../../secrets/common.yaml;
@@ -75,27 +71,27 @@ in
"root-hashedPassword".neededForUsers = true; "root-hashedPassword".neededForUsers = true;
"nixos-hashedPassword".neededForUsers = true; "nixos-hashedPassword".neededForUsers = true;
"nix-github-token" = { }; "nix-github-token" = { };
"nix-gitea-token" = { };
}; };
# nix.conf doesn't support a *File-style option for access-tokens, so the # nix.conf has no *File-style option for access-tokens, so tokens are
# token is rendered into a runtime-only file (never touches the Nix store) # rendered into a runtime-only file (never touches the Nix store) and
# and pulled in via nix.conf's native !include directive. # pulled in via nix.conf's native !include directive.
templates."nix-github-token.conf".content = '' templates."nix-access-tokens.conf".content = ''
access-tokens = github.com=${config.sops.placeholder."nix-github-token"} access-tokens = github.com=${config.sops.placeholder."nix-github-token"} ${vars.giteaDomain}=${config.sops.placeholder."nix-gitea-token"}
''; '';
}; };
nix.extraOptions = '' nix.extraOptions = ''
!include ${config.sops.templates."nix-github-token.conf".path} !include ${config.sops.templates."nix-access-tokens.conf".path}
''; '';
users = { users = {
# With mutableUsers = false, update-users-groups.pl enforces hashedPasswordFile # mutableUsers = false makes update-users-groups.pl enforce hashedPasswordFile
# on every activation regardless of whether the account already exists in # on every activation, not just on newly-created accounts. Without this, a
# /etc/shadow. The default (true) only applies hashedPasswordFile to newly- # freshly-built proxmox disk image (activation runs without a usable sops key,
# created accounts — which means a freshly-built proxmox disk image (where # so both accounts land in shadow with '!') will never have its passwords fixed
# activation runs without a usable sops key, so both accounts land in shadow # by subsequent boots.
# with !) will never have its passwords fixed by subsequent boots.
mutableUsers = false; mutableUsers = false;
users.root = { users.root = {
@@ -104,39 +100,23 @@ in
users.${vars.primaryUser} = { users.${vars.primaryUser} = {
isNormalUser = true; isNormalUser = true;
extraGroups = [ "wheel" ]; # Enable sudo for the user. extraGroups = [ "wheel" ];
packages = with pkgs; [ packages = with pkgs; [ tree ];
tree
];
hashedPasswordFile = config.sops.secrets."nixos-hashedPassword".path; hashedPasswordFile = config.sops.secrets."nixos-hashedPassword".path;
openssh.authorizedKeys.keys = [ openssh.authorizedKeys.keys = [ vars.adminSshKey ] ++ vars.extraAdminSshKeys;
vars.adminSshKey
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGygkCljN6uKpdJbHTOQtn8ZnH+wKXDLAwrDFbLrE/65 nixos@nixos"
];
}; };
}; };
# Enable the OpenSSH daemon.
services.openssh.enable = true; services.openssh.enable = true;
#Enable flakes
nix.settings = { nix.settings = {
experimental-features = [ "nix-command" "flakes" ]; experimental-features = [ "nix-command" "flakes" ];
auto-optimise-store = true; auto-optimise-store = true;
}; };
programs.git = { programs.git = {
enable = true; enable = true;
package = pkgs.git; package = pkgs.git;
config = { config.credential.helper = "store";
credential.helper = "store";
};
}; };
} }
+33 -1
View File
@@ -1,13 +1,45 @@
{ pkgs, vars, ... }: { lib, pkgs, vars, ... }:
let
gid = toString vars.dockerAccessGid;
in
{ {
virtualisation.docker = { virtualisation.docker = {
enable = true; enable = true;
package = pkgs.docker; package = pkgs.docker;
}; };
# Pin the docker group GID to match the IPA "docker-access" group so that
# IPA group membership alone grants access to the Docker socket. Any user
# whose supplementary groups (resolved by SSSD from IPA) include GID
# vars.dockerAccessGid will pass the socket group-permission check without
# any per-host users.groups.docker.members entry.
users.groups.docker.gid = lib.mkForce vars.dockerAccessGid;
users.users.${vars.primaryUser}.extraGroups = [ "docker" ]; users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
environment.systemPackages = with pkgs; [ environment.systemPackages = with pkgs; [
docker-compose docker-compose
docker-buildx docker-buildx
]; ];
# NixOS's group activation uses plain `groupmod` without --non-unique.
# When SSSD is active it exposes the IPA "docker-access" group at
# vars.dockerAccessGid via NSS, so groupmod sees that GID as already in
# use and silently skips the change (warning: "not applying GID change").
# This script runs after the normal "groups" step and applies the change
# with --non-unique (which lets the local docker group share the GID with
# the SSSD-provided IPA group). If the GID actually changed it also
# restarts docker.socket so the socket is recreated with the new GID.
system.activationScripts.docker-group-gid = {
deps = [ "groups" ];
text = ''
current=$(grep "^docker:" /etc/group | cut -d: -f3)
if [ "$current" != "${gid}" ]; then
${pkgs.shadow}/bin/groupmod --non-unique -g ${gid} docker
if ${pkgs.systemd}/bin/systemctl is-active --quiet docker.socket; then
${pkgs.systemd}/bin/systemctl stop docker.service docker.socket
rm -f /var/run/docker.sock
${pkgs.systemd}/bin/systemctl start docker.socket docker.service
fi
fi
'';
};
} }
+13 -18
View File
@@ -10,24 +10,19 @@ let
# non-blocking behavior, so they don't need `nofail` too). # non-blocking behavior, so they don't need `nofail` too).
automountOpts = if config.boot.isContainer then [ "nofail" ] else [ "x-systemd.automount" ]; automountOpts = if config.boot.isContainer then [ "nofail" ] else [ "x-systemd.automount" ];
# A bare hostname here never resolves reliably: systemd-resolved only # FQDN in the storage.home zone — resolves to the Pacemaker vip-storage
# ever tries LLMNR for single-label names (never DNS, regardless of any # (192.168.20.229) on docker's eth1/vmbr2 interface. Using the DNS name
# configured search domain), and a *global* search domain (the first fix # rather than the raw IP means a future VIP renumber only requires a DNS
# attempted here) backfires worse -- confirmed live on lxc-docker, adding # update, not a NixOS rebuild. The storage.home zone is served by the same
# `networking.search` made systemd-resolved prioritize its domain-matched # FreeIPA nameserver (domainControllerIp) that docker already uses, so
# but server-less global scope over eth0's correctly-configured one for # resolution reaches it over eth0 without any extra routing.
# every "*.sweet.home" query, silently sending them to public fallback nfsServer = vars.haStorageNfsFqdn;
# DNS instead. `resolvectl query --interface=eth0 server.sweet.home` storageRoot = vars.haStorageRoot;
# resolved fine throughout, proving the LAN DNS server was never the
# problem -- only the ambient, unqualified device string was. Using the
# FQDN directly sidesteps all of that, matching the pattern
# ../raspi/mount-data.nix already uses for the same reason.
nfsServer = "${vars.nfsServerHost}.${vars.homeDomain}";
in in
{ {
fileSystems = { fileSystems = {
${vars.nfsShares.dockerConfig.mountpoint} = { ${vars.nfsShares.dockerConfig.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath}"; device = "${nfsServer}:${storageRoot}/${vars.nfsShares.dockerConfig.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
@@ -38,7 +33,7 @@ in
}; };
${vars.nfsShares.dockerDatabases.mountpoint} = { ${vars.nfsShares.dockerDatabases.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath}"; device = "${nfsServer}:${storageRoot}/${vars.nfsShares.dockerDatabases.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
@@ -49,7 +44,7 @@ in
}; };
${vars.nfsShares.dockerVolumes.mountpoint} = { ${vars.nfsShares.dockerVolumes.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}"; device = "${nfsServer}:${storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
@@ -60,7 +55,7 @@ in
}; };
${vars.nfsShares.nextcloudData.mountpoint} = { ${vars.nfsShares.nextcloudData.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath}"; device = "${nfsServer}:${storageRoot}/${vars.nfsShares.nextcloudData.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
@@ -71,7 +66,7 @@ in
}; };
${vars.nfsShares.raspiVolumes.mountpoint} = { ${vars.nfsShares.raspiVolumes.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath}"; device = "${nfsServer}:${storageRoot}/${vars.nfsShares.raspiVolumes.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
+77 -20
View File
@@ -11,14 +11,53 @@
# Both host keys must be registered via sync-host-keys.sh first so both nodes can decrypt it. # Both host keys must be registered via sync-host-keys.sh first so both nodes can decrypt it.
# #
# DRBD fencing: # DRBD fencing:
# Production setting is resource-only: DRBD waits for the STONITH fence # resource-only with crm-fence-peer.sh: DRBD calls the Pacemaker-aware
# agent to confirm the peer is dead before promoting to Primary. This # crm-fence-peer.sh handler before promoting. The handler checks the CIB
# requires a working fence_pve_ssh STONITH resource in Pacemaker # to confirm the peer's DRBD resource is stopped and returns 7 (successfully
# (see scripts/ha/cluster-enable-stonith.sh). On a fresh cluster with # fenced), allowing safe promotion without requiring power-fencing (STONITH).
# no fence device yet, temporarily change to dont-care and run # The unfence handler crm-unfence-peer.sh clears the outdate flag when the
# cluster-enable-stonith.sh once the fence key is deployed. # peer reconnects. This is the correct setting for Pacemaker+DRBD clusters
{ lib, vars, ... }: # with STONITH disabled; crm-fence-peer.sh replaces the need for a separate
# STONITH device during the testing phase. Switch to resource-and-stonith
# once the fence_pve_ssh STONITH resource is active (see
# scripts/ha/cluster-enable-stonith.sh).
#
# PATH wrapper: when the DRBD kernel module invokes the fence-peer handler
# via the UMH (User Mode Helper) mechanism it provides a minimal PATH that
# omits /run/current-system/sw/bin. crm-fence-peer.sh calls cibadmin,
# crm_mon etc.; if those aren't found a pipeline in the script breaks with
# SIGPIPE. A process killed by signal has WEXITSTATUS() == 0, so the kernel
# sees exit code 0 and logs "fence-peer helper broken, returned 0", looping
# forever. The writeShellScript wrappers below prepend the NixOS sw path
# before exec-ing the real handler, giving it a working Pacemaker toolchain.
{ lib, pkgs, vars, ... }:
let
fencePeerWrapper = pkgs.writeShellScript "drbd-fence-peer" ''
export PATH="/run/current-system/sw/bin:/run/current-system/sw/sbin:$PATH"
exec /run/current-system/sw/lib/drbd/crm-fence-peer.sh "$@"
'';
unfencePeerWrapper = pkgs.writeShellScript "drbd-unfence-peer" ''
export PATH="/run/current-system/sw/bin:/run/current-system/sw/sbin:$PATH"
exec /run/current-system/sw/lib/drbd/crm-unfence-peer.sh "$@"
'';
in
{ {
# Root SSH access — same key set as the nixos user so all admin keys can reach root.
users.users.root.openssh.authorizedKeys.keys = [ vars.adminSshKey ] ++ vars.extraAdminSshKeys;
# Passwordless sudo for wheel — operator SSHes as nixos and uses sudo for
# cluster management commands (drbdadm, crm*, pcs, etc.)
security.sudo.wheelNeedsPassword = lib.mkForce false;
# DRBD lock-file directory (drbd-utils checks for it; missing → harmless but noisy warnings).
systemd.tmpfiles.rules = [ "d /var/lib/drbd 0750 root root -" ];
# Prevent drbd.service from auto-starting at boot / nixos-rebuild switch.
# Pacemaker's OCF drbd agent calls drbdadm up/down directly when managing
# the resource. If drbd.service also runs drbdadm up all while DRBD is
# already Primary under Pacemaker, apply-al fails with "device busy" (exit 20).
systemd.services.drbd.wantedBy = lib.mkForce [ ];
services.drbd = { services.drbd = {
enable = true; enable = true;
config = '' config = ''
@@ -37,12 +76,16 @@
disk { disk {
fencing resource-only; fencing resource-only;
} }
handlers {
fence-peer "${fencePeerWrapper}";
unfence-peer "${unfencePeerWrapper}";
}
} }
resource ha-data { resource ha-data {
volume 0 { volume 0 {
device /dev/drbd0; device /dev/drbd0;
disk /dev/sdb; disk ${vars.haServerDrbdDisk};
meta-disk internal; meta-disk internal;
} }
@@ -75,33 +118,47 @@
services.corosync = { services.corosync = {
clusterName = "ha-cluster"; clusterName = "ha-cluster";
nodelist = [ nodelist = [
{ nodeid = 1; name = vars.haServer1Host; ring_addrs = [ vars.haServer1StorageIp ]; } # ring0: cluster-internal vmbr1 (primary heartbeat + DRBD path)
{ nodeid = 2; name = vars.haServer2Host; ring_addrs = [ vars.haServer2StorageIp ]; } # ring1: LAN vmbr0 (backup heartbeat only — never carries DRBD)
{ nodeid = 1; name = vars.haServer1Host; ring_addrs = [ vars.haServer1StorageIp vars.haServer1Ip ]; }
{ nodeid = 2; name = vars.haServer2Host; ring_addrs = [ vars.haServer2StorageIp vars.haServer2Ip ]; }
]; ];
}; };
networking.firewall = { networking.firewall = {
allowedTCPPorts = [ allowedTCPPorts = [
vars.ports.haServerIscsi
vars.ports.haServerPacemakerRemoted vars.ports.haServerPacemakerRemoted
vars.ports.haServerPcsd vars.ports.haServerPcsd
vars.ports.haServerDrbd vars.ports.haServerDrbd
vars.ports.nfsRpcbind
vars.ports.nfsd
vars.ports.nfsMountd
]; ];
allowedUDPPorts = [ allowedUDPPorts = [
vars.ports.haServerCorosync1 vars.ports.haServerCorosync1
vars.ports.haServerCorosync2 vars.ports.haServerCorosync2
vars.ports.haServerCorosyncCrypto vars.ports.haServerCorosyncCrypto
vars.ports.nfsRpcbind
vars.ports.nfsd
vars.ports.nfsMountd
]; ];
# Protocol separation: iSCSI (VLAN 20 / storage clients only),
# NFS (VLAN 2 / LAN only). Cluster-internal subnets accepted wholesale
# since they are isolated bridges with no external uplink.
extraCommands = '' extraCommands = ''
iptables -A INPUT -s ${vars.haServer1Ip}/32 -j ACCEPT iptables -A nixos-fw -s ${vars.haServer1Ip}/32 -j nixos-fw-accept
iptables -A INPUT -s ${vars.haServer2Ip}/32 -j ACCEPT iptables -A nixos-fw -s ${vars.haServer2Ip}/32 -j nixos-fw-accept
iptables -A INPUT -s ${vars.haStorageCidr} -j ACCEPT iptables -A nixos-fw -s ${vars.haStorageCidr} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.haClientCidr} --dport ${toString vars.ports.haServerIscsi} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsRpcbind} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsRpcbind} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsd} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsd} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsMountd} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsMountd} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsRpcbind} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsRpcbind} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsd} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsd} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsMountd} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsMountd} -j nixos-fw-accept
''; '';
}; };
} }
+1 -1
View File
@@ -25,7 +25,7 @@
let let
python3 = pkgs.python3.withPackages (ps: [ ps.rtslib-fb ]); python3 = pkgs.python3.withPackages (ps: [ ps.rtslib-fb ]);
targetctl = "${pkgs.targetcli-fb}/bin/targetctl"; targetctl = "${python3}/bin/targetctl";
targetctlStop = pkgs.writeScript "targetctl-stop" '' targetctlStop = pkgs.writeScript "targetctl-stop" ''
#!${python3}/bin/python3 #!${python3}/bin/python3
+116 -94
View File
@@ -38,47 +38,122 @@ lib.mkIf enabled {
networking.domain = lib.mkDefault vars.homeDomain; networking.domain = lib.mkDefault vars.homeDomain;
networking.nameservers = lib.mkDefault [ vars.domainControllerIp ]; networking.nameservers = lib.mkDefault [ vars.domainControllerIp ];
security.ipa = { security = {
enable = true; ipa = {
domain = vars.homeDomain; enable = true;
inherit realm; domain = vars.homeDomain;
server = vars.ipaServer; inherit realm;
certificate = caCertPkg; server = vars.ipaServer;
inherit basedn; certificate = caCertPkg;
ipaHostname = fqdn; inherit basedn;
offlinePasswords = true; ipaHostname = fqdn;
cacheCredentials = true; offlinePasswords = true;
cacheCredentials = true;
};
# Create the home directory on first login if it doesn't exist yet.
# IPA users have no pre-created home on the host; without this sshd
# opens a session to a non-existent directory and resets the connection.
# lightdm also needs this so the GUI login path can create the home dir
# if it was not pre-seeded by the tmpfiles rule above (e.g. on first boot
# before SSSD has resolved the user).
pam.services = {
sshd.makeHomeDir = true;
lightdm.makeHomeDir = true;
# pam_unix returns PAM_AUTHINFO_UNAVAIL without prompting when the local
# stub has "!" in shadow (account locked), so PAM_AUTHTOK is never set
# and pam_sss's use_first_pass fails with "No authentication token".
# Changing to try_first_pass makes pam_sss prompt independently when no
# prior module has set the token, restoring IPA password login via
# LightDM and su.
login.rules.auth.sss.settings = lib.mkForce { try_first_pass = true; };
su.rules.auth.sss.settings = lib.mkForce { try_first_pass = true; };
};
# HM with useUserPackages = true (flake.nix) sets users.users.${ipaUser}.packages,
# which forces the stub into /etc/passwd. pam_sss.so with the "localusers" flag
# (added by NixOS when SSSD is enabled) then skips SSSD for any user it finds in
# local /etc/passwd — including this stub — falling through to pam_unix, which has
# no password for the stub → sudo auth always fails.
#
# Fix: NOPASSWD for the IPA user. The IPA user already authenticated to reach a
# shell (SSH public key from IPA or Kerberos), so re-prompting via a broken PAM
# path is security theater on a single-admin homelab.
sudo.extraRules = [{
users = [ vars.ipaUser ];
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
}];
}; };
# Fetch SSH public keys from IPA so users can log in with the key stored systemd = {
# in their IPA profile rather than needing ~/.ssh/authorized_keys on every # Fetch SSH public keys from IPA so users can log in with the key stored
# host. sss_ssh_authorizedkeys queries SSSD (which queries IPA LDAP). # in their IPA profile rather than needing ~/.ssh/authorized_keys on every
# # host. sss_ssh_authorizedkeys queries SSSD (which queries IPA LDAP).
# /nix/store is 1775 (group-writable by nixbld). OpenSSH 10.0+ rejects #
# AuthorizedKeysCommand binaries whose path contains any group-writable # /nix/store is 1775 (group-writable by nixbld). OpenSSH 10.0+ rejects
# component, silently skipping the command. Copy to /usr/local/bin (all # AuthorizedKeysCommand binaries whose path contains any group-writable
# components root-owned, 755) so the path passes sshd's safety check. # component, silently skipping the command. Copy to /usr/local/bin (all
systemd.tmpfiles.rules = [ # components root-owned, 755) so the path passes sshd's safety check.
"d /usr/local 0755 root root - -" tmpfiles.rules = [
"d /usr/local/bin 0755 root root - -" "d /usr/local 0755 root root - -"
"C+ /usr/local/bin/sss_ssh_authorizedkeys 0555 root root - ${pkgs.sssd}/bin/sss_ssh_authorizedkeys" "d /usr/local/bin 0755 root root - -"
# Pre-create the IPA user's home dir so Home Manager activation succeeds "C+ /usr/local/bin/sss_ssh_authorizedkeys 0555 root root - ${pkgs.sssd}/bin/sss_ssh_authorizedkeys"
# even before their first login. On a fresh system SSSD may not have # Pre-create the IPA user's home dir so Home Manager activation succeeds
# resolved the user yet — tmpfiles warns and skips in that case (non-fatal), # even before their first login. On a fresh system SSSD may not have
# and pam_mkhomedir covers the first-login path as a fallback. # resolved the user yet — tmpfiles warns and skips in that case (non-fatal),
"d /home/${vars.ipaUser} 0700 ${vars.ipaUser} ${vars.ipaUser} - -" # and pam_mkhomedir covers the first-login path as a fallback.
]; "d /home/${vars.ipaUser} 0700 ${vars.ipaUser} ${vars.ipaUser} - -"
];
# security.ipa enables Kerberos (security.krb5) which causes systemd to
# start auth-rpcgss-module.service and rpc-gssd.service for Kerberos NFS
# authentication. LXC containers can't load the auth_rpcgss kernel module
# and don't have /var/lib/nfs/rpc_pipefs, so both services fail.
#
# The NixOS IPA module already adds a drop-in for auth-rpcgss-module.service
# with ConditionPathExists=/etc/krb5.keytab. We use lib.mkForce to win the
# text conflict and add ConditionVirtualization=!container alongside it so
# the service is skipped (not failed) in containers that do have a keytab.
# Same fix for rpc-gssd.service which also fails in containers.
units = lib.mkIf config.boot.isContainer {
"auth-rpcgss-module.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
# rpc-gssd also has ConditionPathExists from the NixOS IPA module (and an
# X-Restart-Triggers store path from systemd.nix). Use mkForce to win;
# omit X-Restart-Triggers since this service is skipped in containers anyway.
"rpc-gssd.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
};
# home-manager-<user>.service fails on first enrollment because /home/wayne
# doesn't exist until the user's first login (pam_mkhomedir creates it then).
# ConditionPathExists makes systemd skip the service (exit 0, condition not
# met) instead of failing. After first login the dir exists and subsequent
# rebuilds activate HM normally.
services."home-manager-${vars.ipaUser}".unitConfig.ConditionPathExists =
"/home/${vars.ipaUser}";
};
services.openssh.extraConfig = '' services.openssh.extraConfig = ''
AuthorizedKeysCommand /usr/local/bin/sss_ssh_authorizedkeys %u AuthorizedKeysCommand /usr/local/bin/sss_ssh_authorizedkeys %u
AuthorizedKeysCommandUser nobody AuthorizedKeysCommandUser nobody
''; '';
# Create the home directory on first login if it doesn't exist yet.
# IPA users have no pre-created home on the host; without this sshd
# opens a session to a non-existent directory and resets the connection.
security.pam.services.sshd.makeHomeDir = true;
# Host keytab: pre-provisioned on the IPA server, sops-encrypted binary. # Host keytab: pre-provisioned on the IPA server, sops-encrypted binary.
# Placed at /etc/krb5.keytab before SSSD starts so the host authenticates # Placed at /etc/krb5.keytab before SSSD starts so the host authenticates
# to IPA without running ipa-client-install. # to IPA without running ipa-client-install.
@@ -92,44 +167,6 @@ lib.mkIf enabled {
restartUnits = [ "sssd.service" ]; restartUnits = [ "sssd.service" ];
}; };
# security.ipa enables Kerberos (security.krb5) which causes systemd to
# start auth-rpcgss-module.service and rpc-gssd.service for Kerberos NFS
# authentication. LXC containers can't load the auth_rpcgss kernel module
# and don't have /var/lib/nfs/rpc_pipefs, so both services fail.
#
# The NixOS IPA module already adds a drop-in for auth-rpcgss-module.service
# with ConditionPathExists=/etc/krb5.keytab. We use lib.mkForce to win the
# text conflict and add ConditionVirtualization=!container alongside it so
# the service is skipped (not failed) in containers that do have a keytab.
# Same fix for rpc-gssd.service which also fails in containers.
systemd.units = lib.mkIf config.boot.isContainer {
"auth-rpcgss-module.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
# rpc-gssd also has ConditionPathExists from the NixOS IPA module (and an
# X-Restart-Triggers store path from systemd.nix). Use mkForce to win;
# omit X-Restart-Triggers since this service is skipped in containers anyway.
"rpc-gssd.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
};
# Home Manager config for the IPA primary user, applied on every enrolled
# host. Manages what IPA doesn't: dotfiles, user-scoped packages, session
# variables. Switch-nix/Test-nix/buildImage are system-wide (configuration.nix)
# so they don't need to be repeated here.
# NixOS requires isNormalUser/isSystemUser + group on any entry in # NixOS requires isNormalUser/isSystemUser + group on any entry in
# users.users. HM with useUserPackages = true (set in flake.nix) adds a stub # users.users. HM with useUserPackages = true (set in flake.nix) adds a stub
# entry for each HM user so it can install packages to # entry for each HM user so it can install packages to
@@ -137,35 +174,20 @@ lib.mkIf enabled {
# With security.ipa setting "passwd: sss files" in nsswitch, SSSD's IPA entry # With security.ipa setting "passwd: sss files" in nsswitch, SSSD's IPA entry
# takes priority for NSS lookups — this local stub is only a fallback when # takes priority for NSS lookups — this local stub is only a fallback when
# SSSD is unreachable (at which point auth fails anyway). # SSSD is unreachable (at which point auth fails anyway).
# HM with useUserPackages = true (flake.nix) sets users.users.${ipaUser}.packages,
# which forces the stub into /etc/passwd. pam_sss.so with the "localusers" flag
# (added by NixOS when SSSD is enabled) then skips SSSD for any user it finds in
# local /etc/passwd — including this stub — falling through to pam_unix, which has
# no password for the stub → sudo auth always fails.
#
# Fix: NOPASSWD for the IPA user. The IPA user already authenticated to reach a
# shell (SSH public key from IPA or Kerberos), so re-prompting via a broken PAM
# path is security theater on a single-admin homelab.
users.users.${vars.ipaUser} = { users.users.${vars.ipaUser} = {
isNormalUser = true; isNormalUser = true;
group = "users"; group = "users";
extraGroups = [ "wheel" ]; extraGroups = [ "wheel" ];
createHome = false; createHome = false;
# "!" is not a password hash — it is the standard "account locked" marker.
# It cannot authenticate anyone locally. It exists solely so NixOS generates
# a shadow entry for this stub user; without one pam_unix returns
# PAM_AUTHINFO_UNAVAIL before prompting, which means PAM_AUTHTOK is never
# set and the subsequent pam_sss use_first_pass call has nothing to work
# with — blocking LightDM and su logins even when IPA/SSSD auth succeeds.
hashedPassword = "!";
}; };
# home-manager-<user>.service fails on first enrollment because /home/wayne
# doesn't exist until the user's first login (pam_mkhomedir creates it then).
# ConditionPathExists makes systemd skip the service (exit 0, condition not
# met) instead of failing. After first login the dir exists and subsequent
# rebuilds activate HM normally.
systemd.services."home-manager-${vars.ipaUser}".unitConfig.ConditionPathExists =
"/home/${vars.ipaUser}";
security.sudo.extraRules = [{
users = [ vars.ipaUser ];
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
}];
# Home Manager config for the IPA primary user, applied on every enrolled # Home Manager config for the IPA primary user, applied on every enrolled
# host. Manages what IPA doesn't: dotfiles, user-scoped packages, session # host. Manages what IPA doesn't: dotfiles, user-scoped packages, session
# variables. Switch-nix/Test-nix/buildImage are system-wide (configuration.nix) # variables. Switch-nix/Test-nix/buildImage are system-wide (configuration.nix)
@@ -180,7 +202,7 @@ lib.mkIf enabled {
homeDirectory = lib.mkForce "/home/${vars.ipaUser}"; homeDirectory = lib.mkForce "/home/${vars.ipaUser}";
stateVersion = "26.05"; stateVersion = "26.05";
packages = with pkgs; [ tmux sshfs ]; packages = with pkgs; [ tmux sshfs ];
sessionVariables.EDITOR = "nano"; sessionVariables.EDITOR = lib.mkDefault "nano";
}; };
programs.home-manager.enable = true; programs.home-manager.enable = true;
programs.bash.enable = true; programs.bash.enable = true;
+5 -5
View File
@@ -1,14 +1,14 @@
{ config, lib, vars, ... }: { config, lib, vars, ... }:
let let
# Use the same FQDN approach as docker/mount-data.nix — a bare hostname is # FQDN of the LAN NFS VIP (Pacemaker vip-lan, 192.168.2.229). Defined in
# unreliable: systemd-resolved only tries LLMNR for single-label names, and # variables.nix as haLanNfsFqdn; using the FQDN avoids systemd-resolved
# a global search domain causes it to skip the interface-scoped LAN DNS. # LLMNR quirks and survives a future VIP renumber via a DNS-only update.
nfsServer = "${vars.nfsServerHost}.${vars.homeDomain}"; nfsServer = vars.haLanNfsFqdn;
in in
{ {
fileSystems.${vars.nfsShares.pxebootImages.mountpoint} = { fileSystems.${vars.nfsShares.pxebootImages.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.pxebootImages.subpath}"; device = "${nfsServer}:${vars.haStorageRoot}/${vars.nfsShares.pxebootImages.subpath}";
fsType = "nfs"; fsType = "nfs";
options = [ options = [
"_netdev" "_netdev"
+18 -25
View File
@@ -3,12 +3,11 @@
{ {
# Run dnsmasq on the LAN interface as a forwarding-only resolver for # Run dnsmasq on the LAN interface as a forwarding-only resolver for
# *.ts.net (Tailscale MagicDNS names). FreeIPA's bind-dyndb-ldap # *.ts.net (Tailscale MagicDNS names). FreeIPA's bind-dyndb-ldap
# cannot reach 100.100.100.100 (Tailscale's internal resolver) directly # cannot reach vars.tailscaleResolverIp directly because the DC is not a
# because the DC is not a Tailscale node. This host IS a Tailscale node # Tailscale node. This host IS a Tailscale node and can reach it via
# and can reach 100.100.100.100 via its tailscale0 interface, so it # tailscale0, so it acts as an intermediary: FreeIPA has a conditional
# acts as an intermediary: FreeIPA has a conditional forward zone for # forward zone for ts.net pointing here (vars.tailscaleRouterIp), and this
# ts.net pointing here (vars.tailscaleRouterIp), and this dnsmasq # dnsmasq instance forwards those queries onward to Tailscale's resolver.
# instance forwards those queries onward to Tailscale's resolver.
# #
# Configure FreeIPA once after deploying this host: # Configure FreeIPA once after deploying this host:
# kinit admin # kinit admin
@@ -29,32 +28,26 @@
resolveLocalQueries = false; resolveLocalQueries = false;
settings = { settings = {
# Listen only on the LAN interface — not tailscale0 or loopback. # Listen only on the LAN interface — not tailscale0 or loopback.
# bind-interfaces prevents dnsmasq from binding to 0.0.0.0 and # bind-interfaces prevents dnsmasq from binding to 0.0.0.0 and then
# then filtering by interface later; combined with `interface` this # filtering by interface later; combined with `interface` this ensures
# ensures it genuinely listens only on eth0. # it genuinely listens only on eth0.
bind-interfaces = true; bind-interfaces = true;
interface = [ vars.lxcLanInterface ]; interface = [ vars.lxcLanInterface ];
# Forward-only: no local /etc/hosts or /etc/resolv.conf reading, # Forward-only: no local /etc/hosts or /etc/resolv.conf reading, no
# no negative caching of NXDOMAIN for names this instance doesn't # negative caching of NXDOMAIN for names this instance doesn't serve.
# serve. All ts.net queries come from FreeIPA's conditional forwarder # All ts.net queries come from FreeIPA's conditional forwarder and must
# and must be answered by Tailscale's resolver. # be answered by Tailscale's resolver.
no-hosts = true; no-hosts = true;
no-resolv = true; no-resolv = true;
# Tailscale's internal "Quad100" resolver — reachable from any # Forward *.tailnetDomain to Tailscale's internal resolver, scoped to
# Tailscale node via the tailscale0 interface. Scoped to the # the tailnet-specific subdomain rather than all of ts.net (FreeIPA
# specific tailnet subdomain (vars.tailnetDomain) rather than # refuses to shadow ts.net, a real public TLD).
# all of ts.net: FreeIPA refuses to shadow ts.net (a real public server = [ "/${vars.tailnetDomain}/${vars.tailscaleResolverIp}" ];
# TLD with DNSimple nameservers) so the conditional forward zone
# in FreeIPA must use the tailnet-specific subdomain instead:
# ipa dnsforwardzone-add ${vars.tailnetDomain} \
# --forwarder=${vars.tailscaleRouterIp} \
# --forward-policy=only
server = [ "/${vars.tailnetDomain}/100.100.100.100" ];
}; };
}; };
networking.firewall.allowedUDPPorts = [ 53 ]; networking.firewall.allowedUDPPorts = [ vars.ports.dns ];
networking.firewall.allowedTCPPorts = [ 53 ]; networking.firewall.allowedTCPPorts = [ vars.ports.dns ];
} }
Binary file not shown.
+12 -1
View File
@@ -37,6 +37,17 @@ source "${script_dir}/lib/nix-parallel.sh"
repo_root="$(cd "${script_dir}/.." && pwd)" repo_root="$(cd "${script_dir}/.." && pwd)"
cd "$repo_root" cd "$repo_root"
# When this repo is a subdirectory of a larger git repo (e.g. a mono-repo
# subtree), `git diff --name-only` outputs paths relative to the outer git
# root, not this directory. Compute a prefix to strip so pattern matching
# below works correctly regardless of nesting depth.
_git_root="$(git rev-parse --show-toplevel 2>/dev/null || echo "$repo_root")"
if [[ "$repo_root" != "$_git_root" ]]; then
_subtree_prefix="${repo_root#"$_git_root"/}/"
else
_subtree_prefix=""
fi
full_check=false full_check=false
dry_run=false dry_run=false
@@ -120,7 +131,7 @@ if ! $full_check; then
base_ref="$(resolve_base_ref)" base_ref="$(resolve_base_ref)"
echo echo
echo "Changed-files scope: diffing against ${base_ref}" echo "Changed-files scope: diffing against ${base_ref}"
mapfile -t changed_files < <(git diff --name-only --diff-filter=ACMR "$base_ref" -- . | sort -u) mapfile -t changed_files < <(git diff --name-only --diff-filter=ACMR "$base_ref" -- . | sort -u | sed "s|^${_subtree_prefix}||")
if [[ ${#changed_files[@]} -eq 0 ]]; then if [[ ${#changed_files[@]} -eq 0 ]]; then
echo "No changed files detected." echo "No changed files detected."
+595
View File
@@ -0,0 +1,595 @@
#!/usr/bin/env bash
# deploy.sh — Full lifecycle management for the Docker Swarm HA cluster.
#
# Provisions two NixOS Proxmox VMs (ha-docker-1, ha-docker-2) as dual-manager
# Docker Swarm nodes sharing NFS storage from the existing HA file-server
# cluster. Both nodes are managers so either can accept Docker API and
# `docker stack` commands.
#
# Usage:
# scripts/docker-swarm/deploy.sh [options]
# scripts/docker-swarm/deploy.sh --destroy [options]
#
# Phases (all run by default; skip any with --skip-<phase>):
# 1. ensure-bridge Create vmbr3 (swarm cluster bridge) on the Proxmox node.
# 2. sync-keys Generate SSH host keys for both nodes (clan vars).
# 3. ipa-hosts Create IPA host objects + sops-encrypted keytabs.
# 4. create-vms Build NixOS disk images and create VMs via create-proxmox-resource.sh.
# 5. add-hardware Attach vmbr2 (storage) and vmbr3 (swarm) NICs; start VMs.
# 6. boot-wait Wait for SSH on both LAN IPs.
# 7. refresh-sops-keys Detect disko key drift; re-encrypt secrets; commit.
# 8. init-swarm docker swarm init on node1; manager join on node2; label nodes.
# 9. dns Register storage.home and swarm.home A records in FreeIPA.
# 10. verify docker node ls; NFS mount check; swarm health.
#
# Options:
# --node <host> Proxmox host (default: pve1.sweet.home)
# --vmid1 <n> VMID for ha-docker-1 (default: 202)
# --vmid2 <n> VMID for ha-docker-2 (default: 203)
# --storage <pool> Proxmox storage pool (default: local-zfs)
# --swarm-bridge <br> Bridge for Docker Swarm cluster network (default: vmbr3)
# --storage-bridge <br> Bridge for NFS storage network (default: vmbr2)
# --memory <MB> RAM per node (default: 4096)
# --cores <n> vCPUs per node (default: 4)
# --skip-ensure-bridge Skip vmbr3 creation/check
# --skip-sync-keys Skip sync-host-keys.sh (clan vars already exist)
# --skip-ipa-hosts Skip IPA host account creation (keytabs already exist)
# --skip-create-vms Skip VM creation (VMs already exist)
# --skip-add-hardware Skip NIC attachment (already attached)
# --skip-boot-wait Skip boot/SSH wait (VMs already running)
# --skip-refresh-sops-keys Skip sops host-key drift fix
# --skip-init-swarm Skip swarm initialisation (already initialised)
# --skip-dns Skip FreeIPA DNS record creation
# --skip-verify Skip post-deploy health checks
# --force-rebuild Pass --force-rebuild to create-proxmox-resource.sh
# --destroy Stop and delete both VMs (skip all other phases)
# --dry-run Print what would run without executing
# -h|--help Show this message
#
# Prerequisites:
# - SSH access to the Proxmox node as $PROXMOX_SSH_USER (wayne).
# - sops age key in the standard location (used by sync-host-keys.sh).
# - SSH access to domain-controller.sweet.home as $PROXMOX_SSH_USER for DNS phase.
# - For --skip-sync-keys: clan vars already in vars/per-machine/proxmox-ha-docker-{1,2}/.
# - For --skip-ipa-hosts: secrets/ha-docker-{1,2}.keytab already exist and are committed.
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
# shellcheck source=../env.sh
source "${REPO_ROOT}/scripts/env.sh"
# ── Defaults ──────────────────────────────────────────────────────────────────
NODE="${PVE1_HOST}" # deploy.sh targets pve1 by default (authorised for this cluster)
VMID1=202
VMID2=203
STORAGE="${PROXMOX_STORAGE:-local-zfs}"
SWARM_BRIDGE="vmbr3"
STORAGE_BRIDGE="vmbr2"
MEMORY_MB=4096
CORES=4
SKIP_ENSURE_BRIDGE=false
SKIP_SYNC_KEYS=false
SKIP_IPA_HOSTS=false
SKIP_CREATE_VMS=false
SKIP_ADD_HARDWARE=false
SKIP_BOOT_WAIT=false
SKIP_REFRESH_SOPS_KEYS=false
SKIP_INIT_SWARM=false
SKIP_DNS=false
SKIP_VERIFY=false
FORCE_REBUILD=false
DESTROY=false
DRY_RUN=false
# ── Variables from repo (mirrors variables.nix) ───────────────────────────────
NODE1_HOST="ha-docker-1"
NODE2_HOST="ha-docker-2"
NODE1_LAN_IP="192.168.2.230"
NODE2_LAN_IP="192.168.2.231"
NODE1_SWARM_IP="192.168.30.230"
NODE2_SWARM_IP="192.168.30.231"
NODE1_STORAGE_IP="192.168.20.230"
NODE2_STORAGE_IP="192.168.20.231"
SWARM_CIDR="192.168.30.0/24"
STORAGE_CIDR="192.168.20.0/24"
STORAGE_ZONE="storage.home"
SWARM_ZONE="swarm.home"
SSH_USER="${PROXMOX_SSH_USER:-wayne}"
DC_HOST="${IPA_SERVER:-domain-controller.sweet.home}"
# ── Argument parsing ──────────────────────────────────────────────────────────
usage() {
sed -n '/^# Usage:/,/^[^#]/{ /^#/{ s/^# \?//; p } }' "$0"
exit "${1:-0}"
}
while [[ $# -gt 0 ]]; do
case "$1" in
--node) NODE="$2"; shift 2 ;;
--vmid1) VMID1="$2"; shift 2 ;;
--vmid2) VMID2="$2"; shift 2 ;;
--storage) STORAGE="$2"; shift 2 ;;
--swarm-bridge) SWARM_BRIDGE="$2"; shift 2 ;;
--storage-bridge) STORAGE_BRIDGE="$2"; shift 2 ;;
--memory) MEMORY_MB="$2"; shift 2 ;;
--cores) CORES="$2"; shift 2 ;;
--skip-ensure-bridge) SKIP_ENSURE_BRIDGE=true; shift ;;
--skip-sync-keys) SKIP_SYNC_KEYS=true; shift ;;
--skip-ipa-hosts) SKIP_IPA_HOSTS=true; shift ;;
--skip-create-vms) SKIP_CREATE_VMS=true; shift ;;
--skip-add-hardware) SKIP_ADD_HARDWARE=true; shift ;;
--skip-boot-wait) SKIP_BOOT_WAIT=true; shift ;;
--skip-refresh-sops-keys) SKIP_REFRESH_SOPS_KEYS=true; shift ;;
--skip-init-swarm) SKIP_INIT_SWARM=true; shift ;;
--skip-dns) SKIP_DNS=true; shift ;;
--skip-verify) SKIP_VERIFY=true; shift ;;
--force-rebuild) FORCE_REBUILD=true; shift ;;
--destroy) DESTROY=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage 0 ;;
*) echo "Unknown option: $1" >&2; usage 1 ;;
esac
done
# ── Helpers ───────────────────────────────────────────────────────────────────
log() { echo "==> $*"; }
logn() { echo " $*"; }
err() { echo "ERROR: $*" >&2; exit 1; }
run() {
if $DRY_RUN; then
echo "[dry-run] $*"
else
"$@"
fi
}
pve() {
if $DRY_RUN; then
echo "[dry-run] ssh ${SSH_USER}@${NODE} sudo $*"
else
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
fi
}
pve_check() {
# Read-only probe — always executes even in dry-run.
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
}
SWARM_USER="nixos"
n1() {
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${SWARM_USER}@${NODE1_LAN_IP}" "$@" 2>/dev/null
}
n2() {
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${SWARM_USER}@${NODE2_LAN_IP}" "$@" 2>/dev/null
}
dc() {
# Run ipa commands on domain-controller as $SSH_USER.
if $DRY_RUN; then
echo "[dry-run] ssh ${SSH_USER}@${DC_HOST} $*"
return 0
fi
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${DC_HOST}" "$@"
}
wait_for_ssh() {
local ip="$1" label="$2"
if $DRY_RUN; then
logn "[dry-run] Skipping SSH wait for ${label} (${ip})"
return 0
fi
local deadline=$(( $(date +%s) + 300 ))
log "Waiting for SSH on ${label} (${ip}) — up to 5 min..."
while [[ $(date +%s) -lt $deadline ]]; do
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=3 \
-o BatchMode=yes "${SWARM_USER}@${ip}" true 2>/dev/null; then
logn "${label} is up."
return 0
fi
sleep 5
done
err "Timed out waiting for SSH on ${label} (${ip})"
}
# ── Destroy mode ──────────────────────────────────────────────────────────────
if $DESTROY; then
log "Destroying Docker Swarm VMs (${VMID1}=${NODE1_HOST}, ${VMID2}=${NODE2_HOST}) on ${NODE}"
for vmid in "$VMID1" "$VMID2"; do
STATUS=$(pve "qm status ${vmid} 2>/dev/null" 2>/dev/null || true)
if echo "$STATUS" | grep -q "running"; then
log "Stopping VMID ${vmid}..."
pve "qm stop ${vmid} --skiplock 1"
sleep 5
fi
if $DRY_RUN || pve "qm config ${vmid} >/dev/null 2>&1"; then
log "Deleting VMID ${vmid}..."
run pve "qm destroy ${vmid} --purge 1"
else
logn "VMID ${vmid} not found — already gone."
fi
done
log "Done — swarm VMs destroyed."
exit 0
fi
# ── Phase 1: Ensure swarm bridge ──────────────────────────────────────────────
if ! $SKIP_ENSURE_BRIDGE; then
log "Phase 1: Ensuring swarm bridge ${SWARM_BRIDGE} on ${NODE}"
if pve_check "test -d /sys/class/net/${SWARM_BRIDGE}" &>/dev/null; then
logn "${SWARM_BRIDGE} already exists — skipping."
else
logn "Creating isolated internal bridge ${SWARM_BRIDGE} (no upstream port, ${SWARM_CIDR})"
BRIDGE_CONF="auto ${SWARM_BRIDGE}
iface ${SWARM_BRIDGE} inet manual
bridge-ports none
bridge-stp off
bridge-fd 0"
if $DRY_RUN; then
echo "[dry-run] Would write /etc/network/interfaces.d/${SWARM_BRIDGE}.conf and ifup it"
else
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"echo '${BRIDGE_CONF}' | sudo tee /etc/network/interfaces.d/${SWARM_BRIDGE}.conf > /dev/null && sudo ifup ${SWARM_BRIDGE}"
logn "${SWARM_BRIDGE} created and brought up."
fi
fi
fi
# ── Phase 2: Sync host keys ───────────────────────────────────────────────────
if ! $SKIP_SYNC_KEYS; then
log "Phase 2: Syncing SSH host keys for both swarm targets"
for target in proxmox-ha-docker-1 proxmox-ha-docker-2; do
CLAN_DIR="${REPO_ROOT}/vars/per-machine/${target}/openssh"
if [[ -d "$CLAN_DIR" ]]; then
logn "Clan vars for ${target} already exist — skipping."
else
logn "Generating host keys for ${target}..."
run bash "${REPO_ROOT}/scripts/secrets/sync-host-keys.sh" "$target"
fi
done
fi
# ── Phase 3: IPA host accounts ────────────────────────────────────────────────
if ! $SKIP_IPA_HOSTS; then
log "Phase 3: Creating IPA host accounts and keytabs"
IPA_SCRIPT="${REPO_ROOT}/scripts/ipa/create-nixos-ipa-host-account.sh"
for host in "${NODE1_HOST}" "${NODE2_HOST}"; do
KEYTAB="${REPO_ROOT}/secrets/${host}.keytab"
if [[ -f "$KEYTAB" ]]; then
logn "Keytab for ${host} already exists — skipping."
else
logn "Creating IPA host account and keytab for ${host}..."
run bash "$IPA_SCRIPT" "$host"
fi
done
if ! $DRY_RUN; then
# Keytabs must be committed and pushed before VMs rebuild from Gitea.
CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"
logn "Committing keytabs and pushing to Gitea (branch: ${CURRENT_BRANCH})..."
(cd "${REPO_ROOT}" && \
git add secrets/ha-docker-1.keytab secrets/ha-docker-2.keytab .sops.yaml && \
git commit -m "secrets(ha-docker): add IPA keytabs for ha-docker-1 and ha-docker-2" || true && \
git push origin "${CURRENT_BRANCH}")
logn "Pushed."
fi
fi
# ── Phase 3.5: Prepare Proxmox node for building ─────────────────────────────
if ! $SKIP_CREATE_VMS && ! $DRY_RUN; then
CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"
local_ssh() { ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "$*"; }
if local_ssh "test -d /nix" &>/dev/null && ! local_ssh "test -w /nix" &>/dev/null; then
logn "/nix exists but not writable by ${SSH_USER} — fixing ownership with sudo..."
local_ssh "sudo chown -R ${SSH_USER} /nix"
logn "Done."
fi
unset -f local_ssh
# Use PROXMOX_REMOTE_REPO_DIR (from env.sh) so the build path is consistent
# with what create-proxmox-resource.sh will use. The default is
# /home/<user>/nixos (the standalone nixos repo clone on pve1), but can be
# overridden to e.g. /home/<user>/infrastructure/nixos when the infrastructure
# mono-repo is checked out on pve1 instead.
REMOTE_REPO="${PROXMOX_REMOTE_REPO_DIR:-/home/${SSH_USER}/nixos}"
if pve_check "test -d ${REMOTE_REPO}/.git" &>/dev/null; then
REMOTE_BRANCH=$(ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"cd ${REMOTE_REPO} && git rev-parse --abbrev-ref HEAD 2>/dev/null")
if [[ "$REMOTE_BRANCH" != "$CURRENT_BRANCH" ]]; then
logn "Remote repo (${REMOTE_REPO}) is on '${REMOTE_BRANCH}', switching to '${CURRENT_BRANCH}'..."
if ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"cd ${REMOTE_REPO} && git fetch origin && git checkout '${CURRENT_BRANCH}' && git pull --ff-only" 2>&1; then
logn "Done."
else
logn "WARNING: branch switch failed — proceeding anyway (create-proxmox-resource.sh will retry)"
fi
fi
else
logn "Remote repo ${REMOTE_REPO} not found on ${NODE} — create-proxmox-resource.sh will clone it."
fi
fi
# ── Phase 4: Create VMs ───────────────────────────────────────────────────────
if ! $SKIP_CREATE_VMS; then
log "Phase 4: Building and creating swarm VMs on ${NODE}"
CREATE="${REPO_ROOT}/scripts/proxmox/create-proxmox-resource.sh"
REBUILD_FLAG=""
$FORCE_REBUILD && REBUILD_FLAG="--force-rebuild"
for spec in "${VMID1}:${NODE1_HOST}:proxmox-ha-docker-1" "${VMID2}:${NODE2_HOST}:proxmox-ha-docker-2"; do
IFS=: read -r vmid host_name flake_target <<< "$spec"
log "Creating ${flake_target} (VMID ${vmid}) on ${NODE}..."
# --force-rebuild is always passed: create-proxmox-resource.sh only calls
# sync_remote_host_keys (which bakes the clan-var SSH key into the disk
# image) when it actually builds. Reusing a cached image skips that step
# and leaves the VM unable to decrypt sops secrets on first boot.
run bash "$CREATE" \
--type vm \
--host "$host_name" \
--vmid "$vmid" \
--node "$NODE" \
--storage "$STORAGE" \
--memory "$MEMORY_MB" \
--cores "$CORES" \
--force-rebuild
done
fi
# ── Phase 5: Add NICs and start VMs ──────────────────────────────────────────
if ! $SKIP_ADD_HARDWARE; then
log "Phase 5: Attaching storage (${STORAGE_BRIDGE}) and swarm (${SWARM_BRIDGE}) NICs"
for vmid in "$VMID1" "$VMID2"; do
logn "VMID ${vmid}: stopping to add NICs..."
pve "qm stop ${vmid} --skiplock 1 2>/dev/null; sleep 3" || true
logn "Adding net1 (${STORAGE_BRIDGE} — NFS storage)..."
pve "qm set ${vmid} --net1 virtio,bridge=${STORAGE_BRIDGE},firewall=0"
logn "Adding net2 (${SWARM_BRIDGE} — Docker Swarm)..."
pve "qm set ${vmid} --net2 virtio,bridge=${SWARM_BRIDGE},firewall=0"
logn "Starting VMID ${vmid}..."
pve "qm start ${vmid}"
done
fi
# ── Phase 6: Wait for SSH ─────────────────────────────────────────────────────
if ! $SKIP_BOOT_WAIT; then
log "Phase 6: Waiting for both nodes to come up on LAN IPs"
wait_for_ssh "$NODE1_LAN_IP" "$NODE1_HOST"
wait_for_ssh "$NODE2_LAN_IP" "$NODE2_HOST"
logn "Both nodes are SSHable."
sleep 10 # let systemd finish activation
fi
# ── Phase 7: Refresh sops host-key registrations ─────────────────────────────
#
# Disko builds raw disk images: each new VM boots with a freshly-generated SSH
# host key, not the one pre-seeded in clan vars. Scan the running VMs; if
# their ed25519 keys differ from the clan var, update the clan var, rewrite
# the .sops.yaml anchor, and re-encrypt all affected sops files.
if ! $SKIP_REFRESH_SOPS_KEYS; then
if $DRY_RUN; then
logn "[dry-run] Would scan VM host keys and refresh .sops.yaml / secrets if needed"
else
log "Phase 7: Refreshing sops host-key registrations (disko key drift fix)"
SOPS_UPDATED=false
for spec in \
"${NODE1_LAN_IP}:proxmox-ha-docker-1:${NODE1_HOST}" \
"${NODE2_LAN_IP}:proxmox-ha-docker-2:${NODE2_HOST}"; do
IFS=: read -r node_ip flake_target host_name <<< "$spec"
CLAN_PUB="${REPO_ROOT}/vars/per-machine/${flake_target}/openssh/ssh_host_ed25519_key.pub/value"
logn "Scanning ed25519 host key from ${host_name} (${node_ip})..."
RAW=$(ssh-keyscan -t ed25519 "${node_ip}" 2>/dev/null | grep -v "^#") || true
if [[ -z "$RAW" ]]; then
logn "WARNING: no ed25519 key returned for ${node_ip} — skipping"
continue
fi
SCANNED_TYPE=$(awk '{print $2}' <<< "$RAW")
SCANNED_KEY=$(awk '{print $3}' <<< "$RAW")
SCANNED_PUBKEY="${SCANNED_TYPE} ${SCANNED_KEY} ${host_name}"
CURRENT=$(tr -d '\n' < "$CLAN_PUB" 2>/dev/null || true)
if [[ "$SCANNED_PUBKEY" == "$CURRENT" ]]; then
logn "${host_name}: clan var matches running key — no update needed"
continue
fi
logn "${host_name}: key drift detected — updating clan var"
logn " old: ${CURRENT}"
logn " new: ${SCANNED_PUBKEY}"
echo "$SCANNED_PUBKEY" > "$CLAN_PUB"
SOPS_UPDATED=true
ANCHOR="${flake_target}"
NEW_AGE=$(echo "$SCANNED_PUBKEY" | \
nix run --quiet --no-warn-dirty nixpkgs#ssh-to-age 2>/dev/null)
[[ -z "$NEW_AGE" ]] && err "ssh-to-age produced no output for ${host_name}"
logn " new age key: ${NEW_AGE}"
sed -i "/&${ANCHOR} /s| age[a-z0-9]*$| ${NEW_AGE}|" "${REPO_ROOT}/.sops.yaml"
done
if $SOPS_UPDATED; then
logn "Running sops updatekeys on affected secrets..."
SOPS="nix run --quiet --no-warn-dirty nixpkgs#sops --"
(cd "${REPO_ROOT}" && \
$SOPS updatekeys -y secrets/common.yaml && \
$SOPS updatekeys -y secrets/ha-docker-1.keytab && \
$SOPS updatekeys -y secrets/ha-docker-2.keytab)
logn "Committing refreshed host keys and re-encrypted secrets..."
(cd "${REPO_ROOT}" && \
git add \
vars/per-machine/proxmox-ha-docker-1/openssh/ssh_host_ed25519_key.pub/value \
vars/per-machine/proxmox-ha-docker-2/openssh/ssh_host_ed25519_key.pub/value \
.sops.yaml \
secrets/common.yaml \
secrets/ha-docker-1.keytab \
secrets/ha-docker-2.keytab && \
git commit -m "secrets(ha-docker): refresh sops host-key registrations for new VM instances" || true)
logn "Sops keys refreshed and committed."
fi
fi
fi
# ── Phase 8: Initialise Docker Swarm ─────────────────────────────────────────
if ! $SKIP_INIT_SWARM; then
log "Phase 8: Initialising Docker Swarm"
if $DRY_RUN; then
logn "[dry-run] Would run: docker swarm init --advertise-addr ${NODE1_SWARM_IP} --data-path-addr ${NODE1_SWARM_IP} on ${NODE1_HOST}"
logn "[dry-run] Would join ${NODE2_HOST} as manager"
logn "[dry-run] Would label both nodes"
else
# Check if node1 is already a swarm manager.
if n1 "docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null | grep -qx "active"; then
logn "${NODE1_HOST} is already in a swarm — skipping init."
else
logn "Initialising swarm on ${NODE1_HOST} (advertise: ${NODE1_SWARM_IP})..."
n1 "docker swarm init \
--advertise-addr ${NODE1_SWARM_IP} \
--data-path-addr ${NODE1_SWARM_IP}"
logn "Swarm initialised on ${NODE1_HOST}."
fi
# Check if node2 is already joined.
if n2 "docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null | grep -qx "active"; then
logn "${NODE2_HOST} is already in the swarm — skipping join."
else
logn "Fetching manager join token from ${NODE1_HOST}..."
JOIN_TOKEN=$(n1 "docker swarm join-token manager -q")
[[ -z "$JOIN_TOKEN" ]] && err "Failed to get swarm manager join token from ${NODE1_HOST}"
logn "Joining ${NODE2_HOST} as manager (advertise: ${NODE2_SWARM_IP})..."
n2 "docker swarm join \
--token ${JOIN_TOKEN} \
--advertise-addr ${NODE2_SWARM_IP} \
--data-path-addr ${NODE2_SWARM_IP} \
${NODE1_SWARM_IP}:2377"
logn "${NODE2_HOST} joined as manager."
fi
# Label nodes for service placement constraints.
logn "Labelling swarm nodes..."
n1 "docker node update --label-add node=${NODE1_HOST} ${NODE1_HOST}" || true
n1 "docker node update --label-add node=${NODE2_HOST} ${NODE2_HOST}" || true
logn "Labels applied."
fi
fi
# ── Phase 9: DNS registration ─────────────────────────────────────────────────
if ! $SKIP_DNS; then
log "Phase 9: Registering DNS records in FreeIPA"
if $DRY_RUN; then
logn "[dry-run] Would create/verify ${SWARM_ZONE} zone and add A records"
else
# Check for and create the swarm.home zone if absent.
if ! dc "ipa dnszone-show ${SWARM_ZONE}" >/dev/null 2>&1; then
logn "Creating ${SWARM_ZONE} DNS zone..."
dc "ipa dnszone-add ${SWARM_ZONE} \
--name-server=${DC_HOST}. \
--admin-email=hostmaster@${SWARM_ZONE}"
# Reverse zone for 192.168.30.x
dc "ipa dnszone-add 30.168.192.in-addr.arpa \
--name-server=${DC_HOST}. \
--admin-email=hostmaster@${SWARM_ZONE}" 2>/dev/null || \
logn " (reverse zone 30.168.192.in-addr.arpa already exists or skipped)"
else
logn "${SWARM_ZONE} zone already exists."
fi
# storage.home A records (zone already exists from HA cluster setup).
for spec in "${NODE1_HOST}:${NODE1_STORAGE_IP}" "${NODE2_HOST}:${NODE2_STORAGE_IP}"; do
IFS=: read -r hostname ip <<< "$spec"
logn "Adding ${hostname}.${STORAGE_ZONE}${ip}"
dc "ipa dnsrecord-add ${STORAGE_ZONE} ${hostname} --a-rec=${ip} --a-create-reverse" 2>/dev/null || \
logn " (record already exists or reverse zone missing — continuing)"
done
# swarm.home A records.
for spec in "${NODE1_HOST}:${NODE1_SWARM_IP}" "${NODE2_HOST}:${NODE2_SWARM_IP}"; do
IFS=: read -r hostname ip <<< "$spec"
logn "Adding ${hostname}.${SWARM_ZONE}${ip}"
dc "ipa dnsrecord-add ${SWARM_ZONE} ${hostname} --a-rec=${ip} --a-create-reverse" 2>/dev/null || \
logn " (record already exists — continuing)"
done
fi
fi
# ── Phase 10: Verify ──────────────────────────────────────────────────────────
if ! $SKIP_VERIFY; then
log "Phase 10: Verifying swarm health"
if $DRY_RUN; then
logn "[dry-run] Would verify swarm node list and NFS mounts"
else
logn "Swarm node list:"
n1 "docker node ls" || err "docker node ls failed on ${NODE1_HOST}"
logn "Checking swarm state on both nodes..."
for spec in "${NODE1_LAN_IP}:${NODE1_HOST}" "${NODE2_LAN_IP}:${NODE2_HOST}"; do
IFS=: read -r ip hostname <<< "$spec"
STATE=$(ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"${SWARM_USER}@${ip}" "docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null)
if [[ "$STATE" != "active" ]]; then
err "${hostname} swarm state is '${STATE}', expected 'active'"
fi
logn " ${hostname}: swarm=${STATE}"
done
logn "Checking NFS mounts on both nodes..."
for spec in "${NODE1_LAN_IP}:${NODE1_HOST}" "${NODE2_LAN_IP}:${NODE2_HOST}"; do
IFS=: read -r ip hostname <<< "$spec"
NFS_OK=$(ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"${SWARM_USER}@${ip}" "df -h /mnt/docker/config 2>/dev/null | grep -c nfs || echo 0" 2>/dev/null)
if [[ "$NFS_OK" -ge 1 ]]; then
logn " ${hostname}: /mnt/docker/config NFS mount ✓"
else
logn " WARNING: ${hostname}: /mnt/docker/config does not appear to be NFS-mounted"
logn " (automount may still be pending — try: ssh nixos@${ip} 'ls /mnt/docker/config')"
fi
done
logn "Checking overlay network..."
NETWORKS=$(n1 "docker network ls --filter driver=overlay --format '{{.Name}}'")
if echo "$NETWORKS" | grep -q "ingress"; then
logn " ingress overlay network present ✓"
else
logn " WARNING: ingress overlay network not found — swarm may not be fully initialised"
fi
fi
fi
log "Deploy complete. Both nodes are ready for 'docker stack deploy'."
log "Connect to either manager:"
log " ssh nixos@${NODE1_LAN_IP} (${NODE1_HOST})"
log " ssh nixos@${NODE2_LAN_IP} (${NODE2_HOST})"
+237
View File
@@ -0,0 +1,237 @@
#!/usr/bin/env bash
# gc-hosts.sh — Run nix-collect-garbage -d on all live NixOS hosts.
#
# The host list is rebuilt on every run:
# 1. This workstation (nixos) — always first
# 2. pve1 — always second (non-NixOS Proxmox node with Nix installed)
# 3. Every NixOS guest currently running on pve1 (discovered via pct/qm list)
#
# nix-cache is excluded: gc-ing the shared binary cache evicts store paths
# that other hosts depend on for substitution.
#
# NixOS hosts: tries "sudo -n nix-collect-garbage -d" first (works when
# wheelNeedsPassword = false, e.g. the HA cluster). Falls back to user-level
# "nix-collect-garbage -d" if sudo needs a password — still collects
# unreferenced store paths and old nixos-user profile generations, but leaves
# old system generations in place.
# pve1: runs "bash -l -c nix-collect-garbage -d" as the login user so
# /etc/profile is sourced and the Nix daemon's PATH is set up automatically.
#
# Usage (from repo root):
# bash scripts/gc-hosts.sh [--dry-run]
set -euo pipefail
cd "$(dirname "$0")/.."
source scripts/env.sh 2>/dev/null || true
source scripts/lib/nix-eval.sh 2>/dev/null || true
# ── config ────────────────────────────────────────────────────────────────────
: "${MAX_JOBS:=8}"
: "${NIXOS_USER:=nixos}"
: "${PVE1_SSH_USER:=${PROXMOX_SSH_USER:-wayne}}"
# GC connections use BatchMode — no interactive prompts, just succeed or fail.
SSH_OPTS=(-o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=10)
# Discovery connections do NOT use BatchMode so that sudo can prompt if needed
# (pct/qm list require root access on Proxmox).
SSH_QUERY_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=10)
DRY_RUN=0
for arg in "$@"; do
case "$arg" in
--dry-run) DRY_RUN=1 ;;
*) echo "Unknown option: $arg" >&2; exit 1 ;;
esac
done
# ── build the host list ───────────────────────────────────────────────────────
# ORDERED_HOSTS: names in display/execution order.
# HOST_TARGET[name]: SSH target string (user@host).
# HOST_TYPE[name]: "nixos" (try sudo gc, fallback user) | "nix" (login-shell gc).
declare -a ORDERED_HOSTS=()
declare -A HOST_TARGET=()
declare -A HOST_TYPE=()
declare -A _SEEN_HOSTNAMES=() # dedup tracker
_add_host() {
local name="$1" target="$2" type="$3"
if [[ -n "${_SEEN_HOSTNAMES[$name]+_}" ]]; then return; fi
_SEEN_HOSTNAMES[$name]=1
ORDERED_HOSTS+=("$name")
HOST_TARGET[$name]="$target"
HOST_TYPE[$name]="$type"
}
# 1. Workstation (hard-wired first)
_add_host "nixos" "${NIXOS_USER}@nixos" "nixos"
# 2. pve1 (hard-wired second; non-NixOS, no system generations)
_add_host "pve1" "${PVE1_SSH_USER}@${PVE1_HOST}" "nix"
# 3. Dynamically discover running NixOS guests on pve1
#
# create-proxmox-resource.sh names every guest after its NixOS hostname:
# pct create ... --hostname <nixos-hostname> (LXC)
# qm create ... --name <nixos-hostname> (VM)
# So pct/qm list output already contains the NixOS hostname directly.
# We validate against the flake to filter out non-NixOS guests on pve1
# (e.g. FreeIPA, Proxmox Backup Server) that share the same Proxmox node.
echo "Discovering running guests on ${PVE1_HOST}..."
# Eval the flake once to get the set of hostnames that are actually NixOS.
# Values are NixOS hostnames (e.g. "docker"); keys are flake targets ("lxc-docker").
nixos_hostnames=""
nixos_hostnames="$(
nix eval --json "${NIX_EVAL_FLAGS[@]}" .#nixosConfigurations \
--apply 'cfgs: builtins.attrValues (builtins.mapAttrs (_: cfg: cfg.config.networking.hostName) cfgs)' \
2>/dev/null | jq -r '.[]' | sort -u
)" || { echo " warning: flake eval failed — non-NixOS guests will not be filtered" >&2; }
# SSH_QUERY_OPTS (no BatchMode) so sudo can prompt if needed for pct/qm.
if ssh "${SSH_QUERY_OPTS[@]}" "${PVE1_SSH_USER}@${PVE1_HOST}" "true" 2>/dev/null; then
running_guests="$(
ssh "${SSH_QUERY_OPTS[@]}" "${PVE1_SSH_USER}@${PVE1_HOST}" bash -s <<'DISCOVER'
sudo pct list 2>/dev/null | awk 'NR>1 && $2=="running" { print $NF }'
sudo qm list 2>/dev/null | awk 'NR>1 && $3=="running" { print $2 }'
DISCOVER
)" || running_guests=""
while IFS= read -r hostname; do
[[ -z "$hostname" ]] && continue
# Exclude nix-cache.
case "$hostname" in *nix-cache*) continue ;; esac
# Skip if not a flake-managed NixOS host (filters non-NixOS pve1 guests).
if [[ -n "$nixos_hostnames" ]] && ! grep -qxF "$hostname" <<< "$nixos_hostnames"; then
continue
fi
# Skip if already in the list (e.g. a proxmox-gui guest whose hostname is nixos).
if [[ -n "${_SEEN_HOSTNAMES[$hostname]+_}" ]]; then continue; fi
echo " + $hostname"
_add_host "$hostname" "${NIXOS_USER}@${hostname}" "nixos"
done <<< "$(echo "$running_guests" | sort -u)"
else
echo " warning: ${PVE1_HOST} unreachable — skipping dynamic host discovery" >&2
fi
echo ""
echo "Hosts: ${ORDERED_HOSTS[*]}"
echo ""
# ── dry-run ───────────────────────────────────────────────────────────────────
if [[ "$DRY_RUN" -eq 1 ]]; then
echo "[dry-run] commands that would run:"
for host in "${ORDERED_HOSTS[@]}"; do
target="${HOST_TARGET[$host]}"
type="${HOST_TYPE[$host]}"
if [[ "$type" == "nixos" ]]; then
echo " ssh ${SSH_OPTS[*]} $target 'sudo -n nix-collect-garbage -d'"
echo " # fallback: ssh ... $target 'nix-collect-garbage -d'"
else
echo " ssh ${SSH_OPTS[*]} $target 'bash -l -c nix-collect-garbage -d'"
fi
done
exit 0
fi
# ── gc worker ─────────────────────────────────────────────────────────────────
gc_one() {
local host="$1" target="${HOST_TARGET[$1]}" type="${HOST_TYPE[$1]}" logfile="$2"
if ! ssh "${SSH_OPTS[@]}" "$target" "true" 2>>"$logfile"; then
echo "unreachable"; return
fi
if [[ "$type" == "nixos" ]]; then
if ssh "${SSH_OPTS[@]}" "$target" "sudo -n nix-collect-garbage -d" \
>>"$logfile" 2>>"$logfile"; then
echo "ok(sudo)"; return
fi
echo "[sudo needs password — falling back to user-level gc]" >>"$logfile"
if ssh "${SSH_OPTS[@]}" "$target" "nix-collect-garbage -d" \
>>"$logfile" 2>>"$logfile"; then
echo "ok(user)"; return
fi
else
# Non-NixOS node: use a login shell so /etc/profile is sourced and the
# Nix daemon's bin dir is on PATH (set up by /etc/profile.d/nix-daemon.sh
# which the Nix installer adds to /etc/profile).
if ssh "${SSH_OPTS[@]}" "$target" "bash -l -c 'nix-collect-garbage -d'" \
>>"$logfile" 2>>"$logfile"; then
echo "ok"; return
fi
fi
echo "failed:$?"
}
# ── parallel execution ────────────────────────────────────────────────────────
echo "Running gc on ${#ORDERED_HOSTS[@]} hosts (up to ${MAX_JOBS} parallel)..."
echo ""
TMPDIR_GC="$(mktemp -d)"
trap 'rm -rf "$TMPDIR_GC"' EXIT
declare -A LOGS=()
job_count=0
for host in "${ORDERED_HOSTS[@]}"; do
logfile="${TMPDIR_GC}/${host}.log"
resultfile="${TMPDIR_GC}/${host}.result"
LOGS[$host]="$logfile"
: > "$logfile"
( result="$(gc_one "$host" "$logfile")"; echo "$result" > "$resultfile" ) &
(( job_count++ )) || true
if [[ "$job_count" -ge "$MAX_JOBS" ]]; then
wait -n 2>/dev/null || wait
(( job_count-- )) || true
fi
done
wait
# ── summary ───────────────────────────────────────────────────────────────────
echo "Results:"
echo "──────────────────────────────"
ok_hosts=()
warn_hosts=()
fail_hosts=()
for host in "${ORDERED_HOSTS[@]}"; do
result="$(cat "${TMPDIR_GC}/${host}.result" 2>/dev/null || echo "failed:missing")"
case "$result" in
ok|"ok(sudo)"|"ok(user)")
printf " %-22s %s\n" "$host" "$result"
ok_hosts+=("$host") ;;
unreachable)
printf " %-22s UNREACHABLE\n" "$host"
warn_hosts+=("$host") ;;
*)
printf " %-22s FAILED (%s)\n" "$host" "$result"
fail_hosts+=("$host") ;;
esac
done
echo ""
echo " ${#ok_hosts[@]} succeeded, ${#warn_hosts[@]} unreachable, ${#fail_hosts[@]} failed"
for host in "${warn_hosts[@]+"${warn_hosts[@]}"}" "${fail_hosts[@]+"${fail_hosts[@]}"}"; do
logfile="${LOGS[$host]}"
if [[ -s "$logfile" ]]; then
echo ""
echo "── $host ──"
cat "$logfile"
fi
done
echo ""
[[ "${#fail_hosts[@]}" -eq 0 ]]
Regular → Executable
+124 -60
View File
@@ -7,49 +7,109 @@
set -euo pipefail set -euo pipefail
# ── Configuration ───────────────────────────────────────────────────────── # ── Configuration ─────────────────────────────────────────────────────────
NODE1="ha-server-1" # All values override-able via environment variables; defaults match variables.nix.
NODE2="ha-server-2" NODE1="${NODE1:-ha-server-1}"
NODE1_IP="192.168.2.200" # vars.haServer1Ip NODE2="${NODE2:-ha-server-2}"
NODE2_IP="192.168.2.201" # vars.haServer2Ip NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
VIP="192.168.2.202" # vars.haServerVip NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
XFS_MOUNT="/srv/ha-data" # vars.haStorageRoot VIP="${VIP:-192.168.20.229}" # vars.haServerVip
ISCSI_IQN="iqn.2026-01.home.sweet:ha-storage" # vars.haIscsiIqn XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
ISCSI_IQN="${ISCSI_IQN:-iqn.2026-01.home.sweet:ha-storage}" # vars.haIscsiIqn
# ────────────────────────────────────────────────────────────────────────── # ──────────────────────────────────────────────────────────────────────────
PASS=0 PASS=0
FAIL=0 FAIL=0
RESULTS=() RESULTS=()
pass() { echo " PASS: $1"; ((PASS++)); RESULTS+=("PASS $1"); } # Use PASS=$((PASS+1)) instead of ((PASS++)) — the latter evaluates to 0 when
fail() { echo " FAIL: $1"; ((FAIL++)); RESULTS+=("FAIL $1"); } # PASS=0, which triggers set -e and kills the script after the very first PASS.
pass() { echo " PASS: $1"; PASS=$((PASS+1)); RESULTS+=("PASS $1"); }
fail() { echo " FAIL: $1"; FAIL=$((FAIL+1)); RESULTS+=("FAIL $1"); }
n1() { ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 "root@${NODE1_IP}" "$@" 2>/dev/null; } HA_USER="nixos"
n2() { ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 "root@${NODE2_IP}" "$@" 2>/dev/null; } n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
echo "════════════════════════════════════════════════════" echo "════════════════════════════════════════════════════"
echo " HA Cluster Acceptance Tests — $(date '+%Y-%m-%d %H:%M:%S')" echo " HA Cluster Acceptance Tests — $(date '+%Y-%m-%d %H:%M:%S')"
echo "════════════════════════════════════════════════════" echo "════════════════════════════════════════════════════"
# ── Pre-flight: DRBD sync must be complete ────────────────────────────────
# Tests that check disk state, XFS mount, and iSCSI will fail or give false
# results while the initial full sync is in progress. Block until done.
echo ""
echo "Pre-flight: verifying DRBD sync is complete..."
DRBD_PREFLIGHT=$(n1 "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_PREFLIGHT" | grep -q "^UpToDate/UpToDate$"; then
echo ""
echo " ERROR: DRBD initial sync not complete."
echo " Current dstate on $NODE1: $DRBD_PREFLIGHT"
echo ""
echo " Monitor progress:"
echo " ssh nixos@$NODE1_IP 'sudo watch -n3 cat /proc/drbd'"
echo ""
echo " Re-run this script once dstate shows UpToDate/UpToDate."
exit 1
fi
echo " dstate: $DRBD_PREFLIGHT — ready."
# ── Detect Active/Standby nodes ────────────────────────────────────────────
# Use crm_mon to detect which node holds the Promoted (Primary) DRBD resource.
# Pacemaker is authoritative; DRBD role can briefly read as Secondary while
# Pacemaker is mid-transition, giving a false Active/Standby swap.
# Wait up to 90 s for Pacemaker to settle before giving up.
echo ""
echo "Detecting Active/Standby nodes (waiting for Pacemaker to settle)..."
ACTIVE_NODE=""
for i in $(seq 1 30); do
# crm_mon -1 output contains "Promoted: [ <node> ]" for the DRBD master.
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || n2 "crm_mon -1" 2>/dev/null || true)
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" — the * bullet
# means ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | grep -E '(Promoted|Masters):' | grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
[[ -n "$ACTIVE_NODE" ]] && break
sleep 3
done
if [[ -z "$ACTIVE_NODE" ]]; then
echo " WARNING: could not determine Active node from crm_mon after 90 s — defaulting to $NODE1"
ACTIVE_NODE="$NODE1"
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
STANDBY_NODE="$NODE2"; STANDBY_IP="$NODE2_IP"
na() { n1 "$@"; }
ns() { n2 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
STANDBY_NODE="$NODE1"; STANDBY_IP="$NODE1_IP"
na() { n2 "$@"; }
ns() { n1 "$@"; }
fi
echo " Active: $ACTIVE_NODE ($ACTIVE_IP)"
echo " Standby: $STANDBY_NODE ($STANDBY_IP)"
# ── T1: Corosync quorum established ────────────────────────────────────── # ── T1: Corosync quorum established ──────────────────────────────────────
echo "" echo ""
echo "[T1] Corosync quorum" echo "[T1] Corosync quorum"
if n1 "corosync-quorumtool -s" 2>/dev/null | grep -q "Quorate:.*Yes"; then if na "corosync-quorumtool -s" 2>/dev/null | grep -q "Quorate:.*Yes"; then
pass "cluster has quorum" pass "cluster has quorum"
else else
fail "cluster does not have quorum — check corosync on both nodes" fail "cluster does not have quorum — check corosync on both nodes"
fi fi
# ── T2: DRBD Primary on node1, Secondary on node2 ──────────────────────── # ── T2: DRBD Primary on Active node, Secondary on Standby ────────────────
echo "" echo ""
echo "[T2] DRBD roles" echo "[T2] DRBD roles"
DRBD_ROLE=$(n1 "drbdadm role ha-data" 2>/dev/null || echo "unknown") DRBD_ROLE=$(na "drbdadm role ha-data" 2>/dev/null || echo "unknown")
if [[ "$DRBD_ROLE" == "Primary/Secondary" || "$DRBD_ROLE" == "Primary" ]]; then if [[ "$DRBD_ROLE" == "Primary/Secondary" || "$DRBD_ROLE" == "Primary" ]]; then
pass "DRBD Primary on $NODE1 ($DRBD_ROLE)" pass "DRBD Primary on $ACTIVE_NODE ($DRBD_ROLE)"
else else
fail "unexpected DRBD role on $NODE1: $DRBD_ROLE (expected Primary/Secondary)" fail "unexpected DRBD role on $ACTIVE_NODE: $DRBD_ROLE (expected Primary/Secondary)"
fi fi
DRBD_DSTATE=$(n1 "drbdadm dstate ha-data" 2>/dev/null || echo "unknown") DRBD_DSTATE=$(na "drbdadm dstate ha-data" 2>/dev/null || echo "unknown")
if echo "$DRBD_DSTATE" | grep -q "UpToDate"; then if echo "$DRBD_DSTATE" | grep -q "UpToDate"; then
pass "DRBD disk state UpToDate ($DRBD_DSTATE)" pass "DRBD disk state UpToDate ($DRBD_DSTATE)"
else else
@@ -59,44 +119,45 @@ fi
# ── T3: XFS mounted at haStorageRoot on the Active node ────────────────── # ── T3: XFS mounted at haStorageRoot on the Active node ──────────────────
echo "" echo ""
echo "[T3] XFS mount" echo "[T3] XFS mount"
if n1 "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then if na "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
pass "XFS mounted at ${XFS_MOUNT} on $NODE1" pass "XFS mounted at ${XFS_MOUNT} on $ACTIVE_NODE"
else else
fail "XFS not mounted at ${XFS_MOUNT} on $NODE1" fail "XFS not mounted at ${XFS_MOUNT} on $ACTIVE_NODE"
fi fi
if n2 "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then if ns "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
fail "XFS unexpectedly mounted on $NODE2 (should only be on Active node)" fail "XFS unexpectedly mounted on $STANDBY_NODE (should only be on Active node)"
else else
pass "XFS not mounted on $NODE2 (correct — Secondary)" pass "XFS not mounted on $STANDBY_NODE (correct — Standby)"
fi fi
# ── T4: iSCSI target visible on both nodes ──────────────────────────────── # ── T4: iSCSI target visible on Active node ───────────────────────────────
echo "" echo ""
echo "[T4] iSCSI target" echo "[T4] iSCSI target"
IQN_COUNT=$(n1 "ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn" || echo "0") IQN_COUNT=$(na "bash -c 'ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn || true'" 2>/dev/null || echo "0")
if [[ "$IQN_COUNT" -ge 1 ]]; then if [[ "$IQN_COUNT" -ge 1 ]]; then
pass "iSCSI IQN active on $NODE1 ($IQN_COUNT target(s))" pass "iSCSI IQN active on $ACTIVE_NODE ($IQN_COUNT target(s))"
else else
fail "no iSCSI IQN active on $NODE1" fail "no iSCSI IQN active on $ACTIVE_NODE"
fi fi
# iSCSI discovery from node2 via VIP # iSCSI port reachable from Standby node via VIP.
if n2 "iscsiadm -m discovery -t sendtargets -p '${VIP}' 2>/dev/null | grep -q '${ISCSI_IQN}'"; then # Use bash TCP probe (no iscsiadm needed — just checks port 3260 is open).
pass "iSCSI target discoverable from $NODE2 via VIP ${VIP}" if ns "bash -c 'echo >/dev/tcp/${VIP}/3260' 2>/dev/null"; then
pass "iSCSI port 3260 reachable from $STANDBY_NODE via VIP ${VIP}"
else else
fail "iSCSI target not discoverable from $NODE2 via ${VIP}" fail "iSCSI port 3260 not reachable from $STANDBY_NODE via ${VIP}"
fi fi
# ── T5: Failover — standby node1, verify resources move to node2 ────────── # ── T5: Failover — standby Active node, verify resources move to Standby ──
echo "" echo ""
echo "[T5] Failover (standby $NODE1)" echo "[T5] Failover (standby $ACTIVE_NODE)"
MYNODE=$(n1 "crm_node -n" 2>/dev/null || echo "") ACTIVE_CRMD_NAME=$(na "crm_node -n" 2>/dev/null || echo "")
n1 "crm_standby -N '${MYNODE}' -v on" 2>/dev/null || true na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v on" 2>/dev/null || true
echo " Waiting up to 30 s for resources to move to $NODE2..." echo " Waiting up to 120 s for resources to move to $STANDBY_NODE..."
MOVED=false MOVED=false
for i in $(seq 1 30); do for i in $(seq 1 120); do
if n2 "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then if ns "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
MOVED=true MOVED=true
echo " Resources moved in ${i}s" echo " Resources moved in ${i}s"
break break
@@ -105,49 +166,52 @@ for i in $(seq 1 30); do
done done
if $MOVED; then if $MOVED; then
pass "XFS mounted on $NODE2 after failover" pass "XFS mounted on $STANDBY_NODE after failover"
IQN_ON_N2=$(n2 "ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn" || echo "0") IQN_ON_STANDBY=$(ns "bash -c 'ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn || true'" 2>/dev/null || echo "0")
[[ "$IQN_ON_N2" -ge 1 ]] \ [[ "$IQN_ON_STANDBY" -ge 1 ]] \
&& pass "iSCSI target active on $NODE2 after failover" \ && pass "iSCSI target active on $STANDBY_NODE after failover" \
|| fail "iSCSI target NOT active on $NODE2 after failover" || fail "iSCSI target NOT active on $STANDBY_NODE after failover"
else else
fail "XFS did not mount on $NODE2 within 30 s — failover incomplete" fail "XFS did not mount on $STANDBY_NODE within 120 s — failover incomplete"
fi fi
# ── T6: Data integrity — file written pre-failover readable post-failover # ── T6: Data integrity — file written post-failover readable ────────────
echo "" echo ""
echo "[T6] Data integrity" echo "[T6] Data integrity"
# Write a test file on node2 (now Active) and verify its content # Write a test file on the new Active (former Standby) and verify it.
# Use `echo | sudo tee` for the write: "echo ... > file" via bash -c has the
# redirect interpreted by the remote nixos shell (not sudo), so the file open
# runs as nixos and fails with EACCES on the root-owned XFS mount. Piping
# through sudo tee lets tee (running as root) open the file instead.
TEST_FILE="${XFS_MOUNT}/.acceptance-test-$$" TEST_FILE="${XFS_MOUNT}/.acceptance-test-$$"
TEST_CONTENT="ha-acceptance-test-$(date +%s)" TEST_CONTENT="ha-acceptance-test-$(date +%s)"
n2 "echo '${TEST_CONTENT}' > '${TEST_FILE}'" 2>/dev/null || true echo "${TEST_CONTENT}" | ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${STANDBY_IP}" sudo tee "${TEST_FILE}" > /dev/null 2>/dev/null || true
READBACK=$(n2 "cat '${TEST_FILE}' 2>/dev/null" || echo "") READBACK=$(ns cat "${TEST_FILE}" 2>/dev/null || echo "")
if [[ "$READBACK" == "$TEST_CONTENT" ]]; then if [[ "$READBACK" == "$TEST_CONTENT" ]]; then
pass "test file written and read back correctly on $NODE2" pass "test file written and read back correctly on $STANDBY_NODE"
else else
fail "data integrity check failed (wrote: '$TEST_CONTENT', read: '$READBACK')" fail "data integrity check failed (wrote: '$TEST_CONTENT', read: '$READBACK')"
fi fi
n2 "rm -f '${TEST_FILE}'" 2>/dev/null || true ns rm -f "${TEST_FILE}" 2>/dev/null || true
# ── T7: Node rejoin — un-standby node1, verify cluster is healthy ──────── # ── T7: Node rejoin — un-standby original Active, verify cluster is healthy ─
echo "" echo ""
echo "[T7] Node rejoin" echo "[T7] Node rejoin"
n1 "crm_standby -N '${MYNODE}' -v off" 2>/dev/null || true na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
n1 "crm_resource --cleanup" 2>/dev/null || true na "crm_resource --cleanup" 2>/dev/null || true
sleep 5 sleep 5
ONLINE_NODES=$(n2 "crm_mon -1 2>/dev/null | grep -c 'Online:'" || echo "0") if na "corosync-quorumtool -s 2>/dev/null | grep -q 'Quorate:.*Yes'"; then
if n1 "corosync-quorumtool -s 2>/dev/null | grep -q 'Quorate:.*Yes'"; then pass "$ACTIVE_NODE rejoined — cluster has quorum"
pass "$NODE1 rejoined — cluster has quorum"
else else
fail "$NODE1 did not rejoin with quorum" fail "$ACTIVE_NODE did not rejoin with quorum"
fi fi
DRBD_ROLE_AFTER=$(n1 "drbdadm role ha-data" 2>/dev/null || echo "unknown") DRBD_ROLE_AFTER=$(na "drbdadm role ha-data" 2>/dev/null || echo "unknown")
if echo "$DRBD_ROLE_AFTER" | grep -q "Secondary"; then if echo "$DRBD_ROLE_AFTER" | grep -q "Secondary"; then
pass "$NODE1 is DRBD Secondary after rejoin ($DRBD_ROLE_AFTER)" pass "$ACTIVE_NODE is DRBD Secondary after rejoin ($DRBD_ROLE_AFTER)"
else else
fail "unexpected DRBD role on $NODE1 after rejoin: $DRBD_ROLE_AFTER" fail "unexpected DRBD role on $ACTIVE_NODE after rejoin: $DRBD_ROLE_AFTER"
fi fi
# ── Summary ─────────────────────────────────────────────────────────────── # ── Summary ───────────────────────────────────────────────────────────────
View File
Regular → Executable
+299 -100
View File
@@ -21,22 +21,33 @@
set -euo pipefail set -euo pipefail
# ── Configuration ───────────────────────────────────────────────────────── # ── Configuration ─────────────────────────────────────────────────────────
# These must match variables.nix haServer* values and the Proxmox VMID # All values override-able via environment variables; defaults match variables.nix.
# assignments. Update before running. NODE1="${NODE1:-ha-server-1}"
NODE1="ha-server-1" NODE2="${NODE2:-ha-server-2}"
NODE2="ha-server-2" NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE1_IP="192.168.2.200" # vars.haServer1Ip NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
NODE2_IP="192.168.2.201" # vars.haServer2Ip VIP="${VIP:-192.168.20.229}" # vars.haServerVip (storage-client, vmbr2, VLAN 20)
VIP="192.168.2.202" # vars.haServerVip VIP_LAN="${VIP_LAN:-192.168.2.229}" # vars.haServerLanVip (LAN, vmbr0)
XFS_MOUNT="/srv/ha-data" # vars.haStorageRoot XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
ISCSI_IQN="iqn.2026-01.home.sweet:ha-storage" # vars.haIscsiIqn ISCSI_IQN="${ISCSI_IQN:-iqn.2026-01.home.sweet:ha-storage}" # vars.haIscsiIqn
ISCSI_LUN_FILE="${XFS_MOUNT}/iscsi-lun.img" ISCSI_LUN_FILE="${XFS_MOUNT}/iscsi-lun.img"
ISCSI_LUN_SIZE="10G" ISCSI_LUN_SIZE="10G"
DRBD_DEVICE="/dev/drbd0" DRBD_DEVICE="/dev/drbd0"
VMID_NODE1="" # FILL IN: Proxmox VMID for ha-server-1 # DRBD backing disk — by-id path that resolves correctly on both nodes
VMID_NODE2="" # FILL IN: Proxmox VMID for ha-server-2 # regardless of whether the OS-level name is sda or sdb (Proxmox VM disk
PVE_HOST="pve1.sweet.home" # ordering is not guaranteed). Matches haServerDrbdDisk in variables.nix.
PVE_USER="wayne" # Override DRBD_DISK if your hardware uses a different controller/slot path.
DRBD_DISK="${DRBD_DISK:-/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1}"
VMID_NODE1="${VMID_NODE1:-}" # set by deploy.sh; needed for STONITH
VMID_NODE2="${VMID_NODE2:-}"
PVE_HOST="${PVE_HOST:-pve1.sweet.home}"
PVE_USER="${PVE_USER:-wayne}"
# Inter-node SSH: HA_USER is the user to SSH as on NODE2; HA_KEY is the private
# key to use. Default is root-to-root (no key arg). deploy.sh sets HA_USER=nixos
# and HA_KEY=/tmp/cluster-init-key so the script works even when root-to-root SSH
# is not available.
HA_USER="${HA_USER:-root}"
HA_KEY="${HA_KEY:-}"
# NFS dataset subdirectories to create under XFS_MOUNT. # NFS dataset subdirectories to create under XFS_MOUNT.
# Must mirror vars.nfsShares subpath values in variables.nix. # Must mirror vars.nfsShares subpath values in variables.nix.
@@ -59,6 +70,44 @@ warn() { echo "[cluster-init] WARNING: $*" >&2; }
[[ $(id -u) -eq 0 ]] || die "must run as root" [[ $(id -u) -eq 0 ]] || die "must run as root"
[[ "$(hostname)" == "$NODE1" ]] || die "must run on $NODE1" [[ "$(hostname)" == "$NODE1" ]] || die "must run on $NODE1"
# NixOS may not include xfsprogs in root's PATH even when it's in the store.
# If mkfs.xfs is missing, search the Nix store for it.
if ! command -v mkfs.xfs &>/dev/null; then
_xfs_bin=$(find /nix/store -maxdepth 3 -name mkfs.xfs 2>/dev/null | head -1 | xargs dirname 2>/dev/null || true)
[[ -n "$_xfs_bin" ]] && export PATH="$_xfs_bin:$PATH" \
|| die "mkfs.xfs not found — add xfsprogs to ha-server.nix environment.systemPackages and rebuild"
fi
# drbdmeta lives alongside drbdadm but may not be in PATH when run via sudo.
if ! command -v drbdmeta &>/dev/null; then
_drbd_bin=$(dirname "$(command -v drbdadm)" 2>/dev/null || true)
[[ -n "$_drbd_bin" ]] && export PATH="$_drbd_bin:$PATH" \
|| die "drbdmeta not found — is drbd-utils in ha-server environment.systemPackages?"
fi
# Portable 16-hex-char UUID generator (no openssl required).
_rand_uuid() {
cat /proc/sys/kernel/random/uuid 2>/dev/null | tr -d '-' | cut -c1-16 | tr '[:lower:]' '[:upper:]'
}
# Inter-node SSH/SCP helpers — abstract over root-to-root vs nixos+sudo.
_SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=10"
[[ -n "$HA_KEY" ]] && _SSH_OPTS="-i $HA_KEY $_SSH_OPTS"
if [[ "$HA_USER" == "root" ]]; then
n2_ssh() { ssh $_SSH_OPTS "root@${NODE2_IP}" "$@"; }
n2_scp() { scp $_SSH_OPTS "$1" "root@${NODE2_IP}:$2"; }
else
# Non-root user with passwordless sudo; wrap each command with sudo.
n2_ssh() { ssh $_SSH_OPTS "${HA_USER}@${NODE2_IP}" sudo "$@"; }
n2_scp() {
# SCP to a tmp path, then sudo-move to the real destination as the remote user.
local src="$1" dst="$2"
local tmp="/tmp/_cluster_init_scp_$$"
scp $_SSH_OPTS "$src" "${HA_USER}@${NODE2_IP}:${tmp}"
ssh $_SSH_OPTS "${HA_USER}@${NODE2_IP}" sudo mv "${tmp}" "${dst}"
}
fi
# ── 0. Corosync authkey ─────────────────────────────────────────────────── # ── 0. Corosync authkey ───────────────────────────────────────────────────
AUTHKEY="/etc/corosync/authkey" AUTHKEY="/etc/corosync/authkey"
mkdir -p /etc/corosync mkdir -p /etc/corosync
@@ -68,15 +117,20 @@ if [[ ! -f "$AUTHKEY" ]]; then
chmod 0400 "$AUTHKEY" chmod 0400 "$AUTHKEY"
fi fi
log "Distributing authkey to $NODE2..." log "Distributing authkey to $NODE2..."
ssh "root@${NODE2_IP}" "mkdir -p /etc/corosync" n2_ssh "mkdir -p /etc/corosync"
scp -q "$AUTHKEY" "root@${NODE2_IP}:${AUTHKEY}" n2_scp "$AUTHKEY" "$AUTHKEY"
ssh "root@${NODE2_IP}" "chmod 0400 '${AUTHKEY}'" n2_ssh "chmod 0400 '${AUTHKEY}'"
log "Restarting corosync on both nodes..." log "Restarting corosync and pacemaker on both nodes..."
systemctl restart corosync systemctl restart corosync
ssh "root@${NODE2_IP}" "systemctl restart corosync" n2_ssh "systemctl restart corosync"
sleep 3 sleep 3
log "Starting pacemaker on both nodes (may have failed at boot before authkey was placed)..."
systemctl start pacemaker 2>/dev/null || systemctl restart pacemaker 2>/dev/null || true
n2_ssh "systemctl start pacemaker 2>/dev/null || systemctl restart pacemaker 2>/dev/null || true"
sleep 2
# ── 1. Corosync quorum ──────────────────────────────────────────────────── # ── 1. Corosync quorum ────────────────────────────────────────────────────
log "Waiting for corosync quorum..." log "Waiting for corosync quorum..."
for i in $(seq 1 30); do for i in $(seq 1 30); do
@@ -99,36 +153,141 @@ for i in $(seq 1 30); do
done done
# ── 2. DRBD initialisation ──────────────────────────────────────────────── # ── 2. DRBD initialisation ────────────────────────────────────────────────
# Put both nodes in Pacemaker standby first so it stops managed resources
# cleanly, then enable maintenance-mode so Pacemaker's monitor operations are
# suspended. Without maintenance-mode, Pacemaker keeps monitoring: when it
# sees DRBD Primary on a standby node (that it didn't start), it triggers a
# stop action — killing the initial sync after ~10 s. Maintenance-mode
# disables all start/stop/monitor actions for the duration of the sync; it is
# cleared after UpToDate/UpToDate is confirmed.
log "Setting both nodes to Pacemaker standby for DRBD metadata init..."
crm_standby -N "$NODE1" -v on 2>/dev/null || true
crm_standby -N "$NODE2" -v on 2>/dev/null || true
# Wait for Pacemaker to actually stop DRBD (if it was managing it).
log "Waiting for DRBD to stop under Pacemaker control..."
for i in $(seq 1 30); do
n1_role=$(drbdadm role ha-data 2>/dev/null || echo "Unconfigured")
n2_role=$(n2_ssh "drbdadm role ha-data 2>/dev/null" 2>/dev/null || echo "Unconfigured")
if [[ "$n1_role" == "Unconfigured" ]] && [[ "$n2_role" == "Unconfigured" ]]; then
log "DRBD stopped on both nodes"
break
fi
[[ $i -eq 30 ]] && warn "DRBD still active after 60s standby — forcing down anyway"
sleep 2
done
log "Enabling Pacemaker maintenance-mode (suspends monitor/start/stop during sync)..."
crm_attribute -t crm_config -n maintenance-mode -v true 2>/dev/null || true
log "Detaching DRBD on $NODE1 (belt-and-suspenders after standby)..."
drbdadm down ha-data 2>/dev/null || true
log "Detaching DRBD on $NODE2..."
n2_ssh "drbdadm down ha-data 2>/dev/null || true"
sleep 2
# Ensure /etc/drbd.conf on both nodes points to DRBD_DISK (the stable by-id
# path). VMs built before this fix may have /dev/sda or /dev/sdb hardcoded.
# NixOS makes /etc/drbd.conf a symlink into the read-only Nix store, so
# sed -i on the symlink target would fail — we break the symlink first with
# cp --remove-destination, creating a regular writable copy.
# Rebuild+redeploy (--force-rebuild) to make this permanent.
_PATCH_DRBD=$(mktemp)
cat > "$_PATCH_DRBD" << 'PATCHEOF'
#!/bin/bash
WANT="$1"
conf=/etc/drbd.conf
if [[ -L "$conf" ]]; then
cp --remove-destination "$(readlink -f "$conf")" "$conf"
fi
cur=$(drbdadm sh-ll-dev ha-data 2>/dev/null | head -1 || true)
if [[ -n "$cur" && "$cur" != "$WANT" ]]; then
echo "[cluster-init] WARNING: patching $conf: $cur → $WANT (rebuild to make permanent)"
sed -i "s,${cur},${WANT},g" "$conf"
fi
PATCHEOF
chmod +x "$_PATCH_DRBD"
bash "$_PATCH_DRBD" "$DRBD_DISK"
n2_scp "$_PATCH_DRBD" "/tmp/patch-drbd-disk.sh"
n2_ssh "bash /tmp/patch-drbd-disk.sh ${DRBD_DISK}"
n2_ssh "rm -f /tmp/patch-drbd-disk.sh"
rm -f "$_PATCH_DRBD"
log "Initialising DRBD metadata on $NODE1..." log "Initialising DRBD metadata on $NODE1..."
if ! drbdadm dstate ha-data 2>/dev/null | grep -q "UpToDate\|Inconsistent\|Diskless"; then # Use drbdmeta --force directly for BOTH create-md and write-dev-uuid.
drbdadm create-md ha-data --force # drbdadm create-md --force passes --force to drbdmeta create-md but NOT to
# the write-dev-uuid sub-call it makes internally, so write-dev-uuid fails when
# the backing disk is still busy and stdin is not a TTY:
# "stdin not a TTY, not waiting for confirmation" → exit 20.
# Calling drbdmeta --force directly bypasses the exclusive-open confirmation on
# both steps without needing a TTY, regardless of whether the device is busy.
# Skip metadata creation only if DRBD is UP and fully synced (UpToDate/UpToDate).
# When the resource is down, drbdadm dstate reads metadata and returns just
# "UpToDate" (no slash) — that must not be treated as "already synced".
# Mismatched UUIDs from an interrupted sync cause instant WFConnection→StandAlone,
# so we always recreate metadata unless the sync is genuinely complete.
if [[ "$(drbdadm dstate ha-data 2>/dev/null)" != "UpToDate/UpToDate" ]]; then
UUID1=$(_rand_uuid)
drbdmeta --force 0 v08 "${DRBD_DISK}" internal create-md
drbdmeta --force 0 v08 "${DRBD_DISK}" internal write-dev-uuid "$UUID1"
fi fi
log "Initialising DRBD metadata on $NODE2..." log "Initialising DRBD metadata on $NODE2..."
ssh "root@${NODE2_IP}" " if [[ "$(n2_ssh "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null)" != "UpToDate/UpToDate" ]]; then
if ! drbdadm dstate ha-data 2>/dev/null | grep -q 'UpToDate\|Inconsistent\|Diskless'; then UUID2=$(n2_ssh "cat /proc/sys/kernel/random/uuid 2>/dev/null | tr -d '-' | cut -c1-16 | tr '[:lower:]' '[:upper:]'")
drbdadm create-md ha-data --force n2_ssh "drbdmeta --force 0 v08 ${DRBD_DISK} internal create-md"
fi n2_ssh "drbdmeta --force 0 v08 ${DRBD_DISK} internal write-dev-uuid ${UUID2}"
" fi
log "Bringing up DRBD on both nodes..." log "Bringing up DRBD on both nodes..."
drbdadm up ha-data 2>/dev/null || true drbdadm up ha-data 2>/dev/null || true
ssh "root@${NODE2_IP}" "drbdadm up ha-data 2>/dev/null" || true n2_ssh "drbdadm up ha-data" 2>/dev/null || true
log "Forcing $NODE1 to DRBD Primary for initial sync..." log "Forcing $NODE1 to DRBD Primary for initial sync..."
drbdadm primary ha-data --force drbdadm primary ha-data --force
# NOTE: Pacemaker standby is intentionally kept ON until after the sync
# completes. Clearing it here races with the OCF DRBD agent: Pacemaker
# sees DRBD in WFConnection/SyncSource and may call drbdadm-down thinking
# something went wrong, killing the sync. Standby is cleared below, after
# UpToDate/UpToDate is confirmed.
log "Waiting for DRBD to finish initial sync (this may take several minutes)..." log "Waiting for DRBD initial sync to complete (32 GB may take 1020 min)..."
for i in $(seq 1 300); do log " (monitor with: watch -n3 cat /proc/drbd)"
state=$(drbdadm dstate ha-data 2>/dev/null || echo "unknown") _sync_chars=('|' '/' '-' $'\\')
if echo "$state" | grep -q "UpToDate/UpToDate"; then _sync_iter=0
log "DRBD sync complete: $state" while true; do
_dstate=$(drbdadm dstate ha-data 2>/dev/null || echo "unknown")
if echo "$_dstate" | grep -q "UpToDate/UpToDate"; then
printf "\r%-80s\r" ""
log "DRBD initial sync complete (dstate: $_dstate)"
break break
fi fi
[[ $i -eq 300 ]] && warn "DRBD not UpToDate after 300 s — continuing anyway (check drbdadm status)" # Parse connection state from /proc/drbd (cs:SyncSource, cs:Connected, cs:StandAlone …)
sleep 1 _cs=$(grep -oE 'cs:[A-Za-z]+' /proc/drbd 2>/dev/null | head -1 | sed 's/cs://' || echo "unknown")
# /proc/drbd uses variable whitespace: "sync'ed: 5.2%" (two spaces).
_pct=$(grep -oE "sync'ed:[[:space:]]+[0-9.]+" /proc/drbd 2>/dev/null | grep -oE "[0-9.]+" | head -1 || echo "")
_eta=$(grep -oE "finish:[[:space:]]+[0-9:]+" /proc/drbd 2>/dev/null | grep -oE "[0-9:]+$" | head -1 || echo "")
_spd=$(grep -oE "speed:[[:space:]]+[0-9,]+" /proc/drbd 2>/dev/null | grep -oE "[0-9,]+$" | head -1 || echo "")
_sync_iter=$(( _sync_iter + 1 ))
_sc="${_sync_chars[$_sync_iter % 4]}"
if [[ "$_cs" == "StandAlone" && $_sync_iter -gt 5 ]]; then
printf "\r%-80s\r" ""
die "DRBD is StandAlone after 15 s — peer connection lost (dstate: $_dstate). " \
"Check corosync/network and re-run cluster-init."
elif [[ -n "$_pct" ]]; then
printf "\r [%s] syncing: %s%% done — ETA %s @ %s K/s " \
"$_sc" "$_pct" "${_eta:-??:??:??}" "${_spd:-?}"
else
printf "\r [%s] cs:%s dstate:%s — waiting for sync to start " "$_sc" "$_cs" "$_dstate"
fi
sleep 3
done done
log "Disabling Pacemaker maintenance-mode and clearing standby — handing DRBD back to Pacemaker..."
crm_attribute -t crm_config -n maintenance-mode -v false 2>/dev/null || true
crm_standby -N "$NODE1" -v off 2>/dev/null || true
crm_standby -N "$NODE2" -v off 2>/dev/null || true
# ── 3. XFS filesystem ───────────────────────────────────────────────────── # ── 3. XFS filesystem ─────────────────────────────────────────────────────
log "Creating XFS on ${DRBD_DEVICE}..." log "Creating XFS on ${DRBD_DEVICE}..."
if ! xfs_info "${DRBD_DEVICE}" &>/dev/null; then if ! xfs_info "${DRBD_DEVICE}" &>/dev/null; then
@@ -137,7 +296,7 @@ fi
log "Mounting ${DRBD_DEVICE} at ${XFS_MOUNT}..." log "Mounting ${DRBD_DEVICE} at ${XFS_MOUNT}..."
mkdir -p "${XFS_MOUNT}" mkdir -p "${XFS_MOUNT}"
mount "${DRBD_DEVICE}" "${XFS_MOUNT}" mountpoint -q "${XFS_MOUNT}" || mount "${DRBD_DEVICE}" "${XFS_MOUNT}"
# ── 4. NFS dataset directories ──────────────────────────────────────────── # ── 4. NFS dataset directories ────────────────────────────────────────────
log "Creating NFS dataset directories..." log "Creating NFS dataset directories..."
@@ -153,116 +312,154 @@ fi
# ── 6. LIO iSCSI target ─────────────────────────────────────────────────── # ── 6. LIO iSCSI target ───────────────────────────────────────────────────
log "Configuring LIO iSCSI target via targetcli..." log "Configuring LIO iSCSI target via targetcli..."
# Note: do NOT bind portal to ${VIP} here — the VIP isn't assigned yet (Pacemaker
# creates it). The default portal (all IPs, port 3260) is correct; Pacemaker's
# VIP resource will make the target reachable at the VIP address.
#
# Clear any existing LIO state first (idempotent: re-run after a partial failure).
# Use specific delete commands — clearconfig does not reliably clear kernel state.
if ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -q "${ISCSI_IQN}"; then
log "Clearing existing LIO target ${ISCSI_IQN} before reconfiguration..."
targetcli "/iscsi delete ${ISCSI_IQN}" 2>/dev/null || true
fi
if ls /sys/kernel/config/target/core/ 2>/dev/null | grep -q "fileio"; then
log "Clearing existing LIO backstore ha-lun0 before reconfiguration..."
targetcli "/backstores/fileio delete ha-lun0" 2>/dev/null || true
fi
targetcli <<EOF targetcli <<EOF
/backstores/fileio create name=ha-lun0 file_or_dev=${ISCSI_LUN_FILE} size=0 write_back=false /backstores/fileio create name=ha-lun0 file_or_dev=${ISCSI_LUN_FILE} size=0 write_back=false
/iscsi create ${ISCSI_IQN} /iscsi create ${ISCSI_IQN}
/iscsi/${ISCSI_IQN}/tpg1/luns create /backstores/fileio/ha-lun0 /iscsi/${ISCSI_IQN}/tpg1/luns create /backstores/fileio/ha-lun0
/iscsi/${ISCSI_IQN}/tpg1/portals create ${VIP}
/iscsi/${ISCSI_IQN}/tpg1 set attribute authentication=0 /iscsi/${ISCSI_IQN}/tpg1 set attribute authentication=0
/iscsi/${ISCSI_IQN}/tpg1 set attribute demo_mode_write_protect=0 /iscsi/${ISCSI_IQN}/tpg1 set attribute demo_mode_write_protect=0
saveconfig /etc/target/saveconfig.json saveconfig /etc/target/saveconfig.json
EOF EOF
log "Tearing down LIO kernel objects — Pacemaker will restore via targetctl on the Active node..."
# LIO holds the backing file open; clear kernel state now so the XFS unmount succeeds.
# Use specific delete commands (clearconfig does not reliably clear kernel configfs state).
targetcli "/iscsi delete ${ISCSI_IQN}" 2>/dev/null || warn "LIO iscsi delete failed — umount may fail"
targetcli "/backstores/fileio delete ha-lun0" 2>/dev/null || warn "LIO backstore delete failed"
log "Distributing iSCSI saveconfig to $NODE2..." log "Distributing iSCSI saveconfig to $NODE2..."
scp -q /etc/target/saveconfig.json "root@${NODE2_IP}:/etc/target/saveconfig.json" n2_scp /etc/target/saveconfig.json /etc/target/saveconfig.json
log "Unmounting ${XFS_MOUNT} — Pacemaker manages it..." log "Unmounting ${XFS_MOUNT} — Pacemaker manages it..."
umount "${XFS_MOUNT}" umount "${XFS_MOUNT}" || { sync; umount -l "${XFS_MOUNT}"; }
log "Demoting DRBD to Secondary — Pacemaker manages primary role..." log "Demoting DRBD to Secondary — Pacemaker manages primary role..."
drbdadm secondary ha-data drbdadm role ha-data 2>/dev/null | grep -q "^Primary" && drbdadm secondary ha-data || true
# ── 7. Pacemaker resources ──────────────────────────────────────────────── # ── 7. Pacemaker resources ────────────────────────────────────────────────
log "Configuring Pacemaker cluster properties..." log "Configuring Pacemaker cluster properties..."
crm_attribute -t crm_config -n stonith-enabled -v false crm_attribute -t crm_config -n stonith-enabled -v false
crm_attribute -t crm_config -n no-quorum-policy -v ignore crm_attribute -t crm_config -n no-quorum-policy -v ignore
log "Creating DRBD promotable clone resource..." log "Creating Pacemaker resources via cibadmin..."
cibadmin --replace --scope resources --xml-text " # Use cibadmin --replace with pacemaker-4.0-compatible XML.
<resources> # Key schema rules for pacemaker-4.0:
<clone id=\"ms-drbd0\" globally-unique=\"false\"> # - globally-unique must be in <meta_attributes>, not a direct <clone> attribute
<meta_attributes id=\"ms-drbd0-meta\"> # - promoted-max / promoted-node-max (not master-max / master-node-max)
<nvpair id=\"ms-drbd0-promotable\" name=\"promotable\" value=\"true\"/> # - constraint with-rsc-role="Promoted" (not "Master")
<nvpair id=\"ms-drbd0-master-max\" name=\"master-max\" value=\"1\"/> cibadmin --replace --scope resources --xml-text '<resources>
<nvpair id=\"ms-drbd0-master-node-max\" name=\"master-node-max\" value=\"1\"/> <clone id="ms-drbd0">
<nvpair id=\"ms-drbd0-clone-max\" name=\"clone-max\" value=\"2\"/> <meta_attributes id="ms-drbd0-meta">
<nvpair id=\"ms-drbd0-clone-node-max\" name=\"clone-node-max\" value=\"1\"/> <nvpair id="ms-drbd0-globally-unique" name="globally-unique" value="false"/>
<nvpair id=\"ms-drbd0-notify\" name=\"notify\" value=\"true\"/> <nvpair id="ms-drbd0-promotable" name="promotable" value="true"/>
<nvpair id=\"ms-drbd0-interleave\" name=\"interleave\" value=\"true\"/> <nvpair id="ms-drbd0-promoted-max" name="promoted-max" value="1"/>
<nvpair id="ms-drbd0-promoted-node-max" name="promoted-node-max" value="1"/>
<nvpair id="ms-drbd0-clone-max" name="clone-max" value="2"/>
<nvpair id="ms-drbd0-clone-node-max" name="clone-node-max" value="1"/>
<nvpair id="ms-drbd0-notify" name="notify" value="true"/>
<nvpair id="ms-drbd0-interleave" name="interleave" value="true"/>
</meta_attributes> </meta_attributes>
<primitive id=\"drbd0\" class=\"ocf\" type=\"drbd\" provider=\"linbit\"> <primitive id="drbd0" class="ocf" type="drbd" provider="linbit">
<instance_attributes id=\"drbd0-attrs\"> <instance_attributes id="drbd0-attrs">
<nvpair id=\"drbd0-resource\" name=\"drbd_resource\" value=\"ha-data\"/> <nvpair id="drbd0-resource" name="drbd_resource" value="ha-data"/>
</instance_attributes> </instance_attributes>
<operations> <operations>
<op id=\"drbd0-start\" name=\"start\" interval=\"0\" timeout=\"240s\"/> <op id="drbd0-start" name="start" interval="0" timeout="240s"/>
<op id=\"drbd0-stop\" name=\"stop\" interval=\"0\" timeout=\"120s\"/> <op id="drbd0-stop" name="stop" interval="0" timeout="120s"/>
<op id=\"drbd0-promote\" name=\"promote\" interval=\"0\" timeout=\"90s\"/> <op id="drbd0-promote" name="promote" interval="0" timeout="240s"/>
<op id=\"drbd0-demote\" name=\"demote\" interval=\"0\" timeout=\"90s\"/> <op id="drbd0-demote" name="demote" interval="0" timeout="90s"/>
<op id=\"drbd0-monitor-master\" name=\"monitor\" interval=\"20s\" timeout=\"20s\" role=\"Promoted\"/> <op id="drbd0-monitor-promoted" name="monitor" interval="20s" timeout="20s" role="Promoted"/>
<op id=\"drbd0-monitor-slave\" name=\"monitor\" interval=\"30s\" timeout=\"20s\" role=\"Unpromoted\"/> <op id="drbd0-monitor-unpromoted" name="monitor" interval="30s" timeout="20s" role="Unpromoted"/>
</operations> </operations>
</primitive> </primitive>
</clone> </clone>
<group id=\"ha-group\"> <group id="ha-group">
<primitive id=\"xfs-data\" class=\"ocf\" type=\"Filesystem\" provider=\"heartbeat\"> <primitive id="xfs-data" class="ocf" type="Filesystem" provider="heartbeat">
<instance_attributes id=\"xfs-data-attrs\"> <instance_attributes id="xfs-data-attrs">
<nvpair id=\"xfs-data-device\" name=\"device\" value=\"${DRBD_DEVICE}\"/> <nvpair id="xfs-data-device" name="device" value="/dev/drbd0"/>
<nvpair id=\"xfs-data-directory\" name=\"directory\" value=\"${XFS_MOUNT}\"/> <nvpair id="xfs-data-directory" name="directory" value="/srv/ha-data"/>
<nvpair id=\"xfs-data-fstype\" name=\"fstype\" value=\"xfs\"/> <nvpair id="xfs-data-fstype" name="fstype" value="xfs"/>
<nvpair id=\"xfs-data-options\" name=\"options\" value=\"defaults\"/> <nvpair id="xfs-data-options" name="options" value="defaults"/>
<nvpair id=\"xfs-data-force_unmount\" name=\"force_unmount\" value=\"false\"/> <nvpair id="xfs-data-force_unmount" name="force_unmount" value="true"/>
</instance_attributes> </instance_attributes>
<operations> <operations>
<op id=\"xfs-data-start\" name=\"start\" interval=\"0\" timeout=\"60s\"/> <op id="xfs-data-start" name="start" interval="0" timeout="60s"/>
<op id=\"xfs-data-stop\" name=\"stop\" interval=\"0\" timeout=\"60s\"/> <op id="xfs-data-stop" name="stop" interval="0" timeout="60s"/>
<op id=\"xfs-data-monitor\" name=\"monitor\" interval=\"20s\" timeout=\"40s\"/> <op id="xfs-data-monitor" name="monitor" interval="20s" timeout="40s"/>
</operations> </operations>
</primitive> </primitive>
<primitive id=\"iscsi-target\" class=\"systemd\" type=\"targetctl\"> <primitive id="iscsi-target" class="systemd" type="targetctl">
<operations> <operations>
<op id=\"iscsi-start\" name=\"start\" interval=\"0\" timeout=\"60s\"/> <op id="iscsi-start" name="start" interval="0" timeout="60s"/>
<op id=\"iscsi-stop\" name=\"stop\" interval=\"0\" timeout=\"60s\"/> <op id="iscsi-stop" name="stop" interval="0" timeout="60s"/>
<op id=\"iscsi-monitor\" name=\"monitor\" interval=\"20s\" timeout=\"40s\"/> <op id="iscsi-monitor" name="monitor" interval="20s" timeout="40s"/>
</operations> </operations>
</primitive> </primitive>
<primitive id=\"nfs-server\" class=\"systemd\" type=\"nfs-server\"> <primitive id="nfs-server" class="systemd" type="nfs-server">
<operations> <operations>
<op id=\"nfs-start\" name=\"start\" interval=\"0\" timeout=\"60s\"/> <op id="nfs-start" name="start" interval="0" timeout="60s"/>
<op id=\"nfs-stop\" name=\"stop\" interval=\"0\" timeout=\"60s\"/> <op id="nfs-stop" name="stop" interval="0" timeout="60s"/>
<op id=\"nfs-monitor\" name=\"monitor\" interval=\"30s\" timeout=\"40s\"/> <op id="nfs-monitor" name="monitor" interval="30s" timeout="40s"/>
</operations> </operations>
</primitive> </primitive>
<primitive id=\"vip\" class=\"ocf\" type=\"IPaddr2\" provider=\"heartbeat\"> <primitive id="vip-storage" class="ocf" type="IPaddr2" provider="heartbeat">
<instance_attributes id=\"vip-attrs\"> <instance_attributes id="vip-storage-attrs">
<nvpair id=\"vip-ip\" name=\"ip\" value=\"${VIP}\"/> <nvpair id="vip-storage-ip" name="ip" value="192.168.20.229"/>
<nvpair id=\"vip-cidr\" name=\"cidr_netmask\" value=\"24\"/> <nvpair id="vip-storage-cidr" name="cidr_netmask" value="24"/>
<nvpair id="vip-storage-nic" name="nic" value="ens20"/>
</instance_attributes> </instance_attributes>
<operations> <operations>
<op id=\"vip-start\" name=\"start\" interval=\"0\" timeout=\"20s\"/> <op id="vip-storage-start" name="start" interval="0" timeout="20s"/>
<op id=\"vip-stop\" name=\"stop\" interval=\"0\" timeout=\"20s\"/> <op id="vip-storage-stop" name="stop" interval="0" timeout="20s"/>
<op id=\"vip-monitor\" name=\"monitor\" interval=\"10s\" timeout=\"20s\"/> <op id="vip-storage-monitor" name="monitor" interval="10s" timeout="20s"/>
</operations>
</primitive>
<primitive id="vip-lan" class="ocf" type="IPaddr2" provider="heartbeat">
<instance_attributes id="vip-lan-attrs">
<nvpair id="vip-lan-ip" name="ip" value="192.168.2.229"/>
<nvpair id="vip-lan-cidr" name="cidr_netmask" value="24"/>
<nvpair id="vip-lan-nic" name="nic" value="ens18"/>
</instance_attributes>
<operations>
<op id="vip-lan-start" name="start" interval="0" timeout="20s"/>
<op id="vip-lan-stop" name="stop" interval="0" timeout="20s"/>
<op id="vip-lan-monitor" name="monitor" interval="10s" timeout="20s"/>
</operations> </operations>
</primitive> </primitive>
</group> </group>
</resources> </resources>'
"
log "Adding ordering and colocation constraints..." log "Adding Pacemaker ordering and colocation constraints..."
cibadmin --create --scope constraints --xml-text " cibadmin --replace --scope constraints --xml-text '<constraints>
<constraints> <rsc_order id="order-drbd-group" first="ms-drbd0" first-action="promote" then="ha-group" then-action="start" kind="Mandatory"/>
<rsc_order id=\"order-drbd-group\" first=\"ms-drbd0\" first-action=\"promote\" then=\"ha-group\" then-action=\"start\"/> <rsc_colocation id="coloc-group-with-drbd" score="INFINITY" rsc="ha-group" with-rsc="ms-drbd0" with-rsc-role="Promoted"/>
<rsc_colocation id=\"coloc-group-with-drbd\" rsc=\"ha-group\" with-rsc=\"ms-drbd0\" with-rsc-role=\"Master\" score=\"INFINITY\"/> </constraints>'
</constraints>
" log "Clearing stale Pacemaker failure history..."
crm_resource --cleanup 2>/dev/null || true
log "Waiting for resources to start..." log "Waiting for resources to start..."
for i in $(seq 1 60); do for i in $(seq 1 60); do
if crm_resource -r vip --locate 2>/dev/null | grep -q "running on"; then if crm_resource -r vip-storage --locate 2>/dev/null | grep -q "running on"; then
log "VIP is up: $(crm_resource -r vip --locate)" log "VIPs are up: $(crm_resource -r vip-storage --locate)"
break break
fi fi
[[ $i -eq 60 ]] && { warn "VIP not up after 120 s — check: crm_mon -1"; break; } [[ $i -eq 60 ]] && { warn "VIPs not up after 120 s — check: crm_mon -1"; break; }
sleep 2 sleep 2
done done
@@ -270,9 +467,11 @@ log ""
log "═══════════════════════════════════════════════════════════════" log "═══════════════════════════════════════════════════════════════"
log " HA cluster initialised." log " HA cluster initialised."
log "" log ""
log " crm_mon -1 — cluster status" log " crm_mon -1 — cluster status"
log " iscsiadm -m discovery -t st -p ${VIP} — verify iSCSI target" log " iscsiadm -m discovery -t st -p ${VIP} — verify iSCSI (storage net)"
log " showmount -e ${VIP} — verify NFS exports" log " iscsiadm -m discovery -t st -p ${VIP_LAN} — verify iSCSI (LAN)"
log " showmount -e ${VIP} — verify NFS exports (storage net)"
log " showmount -e ${VIP_LAN} — verify NFS exports (LAN)"
log "" log ""
log " To enable STONITH (after deploying fence SSH key):" log " To enable STONITH (after deploying fence SSH key):"
log " 1. Fill in VMID_NODE1 / VMID_NODE2 in cluster-enable-stonith.sh" log " 1. Fill in VMID_NODE1 / VMID_NODE2 in cluster-enable-stonith.sh"
+499
View File
@@ -0,0 +1,499 @@
#!/usr/bin/env bash
# deploy.sh — Full lifecycle management for the HA file-server cluster.
#
# Handles everything from zero (no VMs, no secrets) through a running,
# tested cluster, and optionally tears it back down.
#
# Usage:
# scripts/ha/deploy.sh [options]
# scripts/ha/deploy.sh --destroy [options]
#
# Phases (all run by default; skip any with --skip-*):
# 1. ensure-bridge Create storage bridge (vmbr1) on the Proxmox node if absent.
# 2. sync-keys Generate SSH host keys and register age keys for both nodes.
# 3. create-vms Build disk images and create both VMs via create-proxmox-resource.sh.
# 4. add-hardware Attach storage NIC (vmbr1) and DRBD data disk to each VM.
# 5. boot-wait Start VMs, wait for SSH on both nodes.
# 6. cluster-init Form the cluster: DRBD, corosync, Pacemaker, NFS, VIP.
# Also encrypts the generated corosync authkey into the repo.
# 7. run-tests Run acceptance tests (T1T7).
#
# Options:
# --node <host> Proxmox host to deploy on (default: pve1.sweet.home)
# --vmid1 <n> VMID for ha-server-1 (default: 200)
# --vmid2 <n> VMID for ha-server-2 (default: 201)
# --storage <pool> Proxmox storage pool (default: local-zfs)
# --storage-bridge <br> Bridge for HA storage network (default: vmbr1)
# --drbd-disk-gb <n> DRBD data disk size in GB (default: 32)
# --memory <MB> RAM per node (default: 4096)
# --cores <n> vCPUs per node (default: 4)
# --skip-ensure-bridge Skip storage bridge creation/check
# --skip-sync-keys Skip sync-host-keys.sh (clan vars already exist)
# --skip-create-vms Skip VM creation (VMs already exist)
# --skip-add-hardware Skip net1/scsi1 attachment (already attached)
# --skip-boot-wait Skip boot/SSH wait (VMs already running)
# --skip-refresh-sops-keys Skip scanning running VMs for fresh SSH host keys
# --skip-cluster-init Skip cluster formation (cluster already configured)
# --skip-tests Skip acceptance tests
# --force-rebuild Pass --force-rebuild to create-proxmox-resource.sh
# --destroy Stop and delete both VMs (skip all other phases)
# --dry-run Print what would run without executing
# -h|--help Show this message
#
# Prerequisites:
# - SSH access to the Proxmox node as $PROXMOX_SSH_USER (wayne).
# - sops age key in the standard location (used by sync-host-keys.sh).
# - For --skip-sync-keys: clan vars already in vars/per-machine/proxmox-ha-server-{1,2}/.
# - For full tests: secrets/common.yaml decryptable on both nodes (run
# `sops updatekeys secrets/common.yaml` after sync-keys).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# shellcheck source=../env.sh
source "${REPO_ROOT}/scripts/env.sh"
# ── Defaults ──────────────────────────────────────────────────────────────────
NODE="${PROXMOX_HOST:-$PVE1_HOST}"
VMID1=200
VMID2=201
STORAGE="${PROXMOX_STORAGE:-local-zfs}"
STORAGE_BRIDGE="vmbr1"
DRBD_DISK_GB=32
MEMORY_MB=4096
CORES=4
SKIP_ENSURE_BRIDGE=false
SKIP_SYNC_KEYS=false
SKIP_CREATE_VMS=false
SKIP_ADD_HARDWARE=false
SKIP_BOOT_WAIT=false
SKIP_REFRESH_SOPS_KEYS=false
SKIP_CLUSTER_INIT=false
SKIP_TESTS=false
FORCE_REBUILD=false
DESTROY=false
DRY_RUN=false
# ── Variables from repo ───────────────────────────────────────────────────────
NODE1_HOST="ha-server-1"
NODE2_HOST="ha-server-2"
NODE1_IP="192.168.2.228"
NODE2_IP="192.168.2.227"
STORAGE_IP1="192.168.10.228"
STORAGE_IP2="192.168.10.227"
STORAGE_CIDR="192.168.10.224/29"
SSH_USER="${PROXMOX_SSH_USER:-wayne}"
# ── Argument parsing ──────────────────────────────────────────────────────────
usage() {
sed -n '/^# Usage:/,/^[^#]/{ /^#/{ s/^# \?//; p } }' "$0"
exit "${1:-0}"
}
while [[ $# -gt 0 ]]; do
case "$1" in
--node) NODE="$2"; shift 2 ;;
--vmid1) VMID1="$2"; shift 2 ;;
--vmid2) VMID2="$2"; shift 2 ;;
--storage) STORAGE="$2"; shift 2 ;;
--storage-bridge) STORAGE_BRIDGE="$2"; shift 2 ;;
--drbd-disk-gb) DRBD_DISK_GB="$2"; shift 2 ;;
--memory) MEMORY_MB="$2"; shift 2 ;;
--cores) CORES="$2"; shift 2 ;;
--skip-ensure-bridge) SKIP_ENSURE_BRIDGE=true; shift ;;
--skip-sync-keys) SKIP_SYNC_KEYS=true; shift ;;
--skip-create-vms) SKIP_CREATE_VMS=true; shift ;;
--skip-add-hardware) SKIP_ADD_HARDWARE=true; shift ;;
--skip-boot-wait) SKIP_BOOT_WAIT=true; shift ;;
--skip-refresh-sops-keys) SKIP_REFRESH_SOPS_KEYS=true; shift ;;
--skip-cluster-init) SKIP_CLUSTER_INIT=true; shift ;;
--skip-tests) SKIP_TESTS=true; shift ;;
--force-rebuild) FORCE_REBUILD=true; shift ;;
--destroy) DESTROY=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage 0 ;;
*) echo "Unknown option: $1" >&2; usage 1 ;;
esac
done
# ── Helpers ───────────────────────────────────────────────────────────────────
log() { echo "==> $*"; }
logn() { echo " $*"; }
err() { echo "ERROR: $*" >&2; exit 1; }
run() {
if $DRY_RUN; then
echo "[dry-run] $*"
else
"$@"
fi
}
pve() {
# Run a command on the Proxmox node via SSH.
if $DRY_RUN; then
echo "[dry-run] ssh ${SSH_USER}@${NODE} sudo $*"
else
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
fi
}
pve_check() {
# Run a read-only probe on the Proxmox node — always executes even in dry-run.
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
}
HA_USER="nixos"
n1() {
# Run a command on ha-server-1 via SSH as nixos with sudo.
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null
}
n2() {
# Run a command on ha-server-2 via SSH as nixos with sudo.
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null
}
wait_for_ssh() {
local ip="$1" label="$2"
if $DRY_RUN; then
logn "[dry-run] Skipping SSH wait for ${label} (${ip})"
return 0
fi
local deadline=$(( $(date +%s) + 300 ))
log "Waiting for SSH on ${label} (${ip}) — up to 5 min..."
while [[ $(date +%s) -lt $deadline ]]; do
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=3 \
-o BatchMode=yes "${HA_USER}@${ip}" true 2>/dev/null; then
logn "${label} is up."
return 0
fi
sleep 5
done
err "Timed out waiting for SSH on ${label} (${ip})"
}
# ── Destroy mode ─────────────────────────────────────────────────────────────
if $DESTROY; then
log "Destroying HA cluster VMs (${VMID1}=${NODE1_HOST}, ${VMID2}=${NODE2_HOST}) on ${NODE}"
for vmid in "$VMID1" "$VMID2"; do
STATUS=$(pve "qm status ${vmid} 2>/dev/null" 2>/dev/null || true)
if echo "$STATUS" | grep -q "running"; then
log "Stopping VMID ${vmid}..."
pve "qm stop ${vmid} --skiplock 1"
sleep 5
fi
if $DRY_RUN || pve "qm config ${vmid} >/dev/null 2>&1"; then
log "Deleting VMID ${vmid}..."
run pve "qm destroy ${vmid} --purge 1"
else
logn "VMID ${vmid} not found — already gone."
fi
done
log "Done — cluster VMs destroyed."
exit 0
fi
# ── Phase 1: Storage bridge ───────────────────────────────────────────────────
if ! $SKIP_ENSURE_BRIDGE; then
log "Phase 1: Ensuring storage bridge ${STORAGE_BRIDGE} on ${NODE}"
if pve_check "test -d /sys/class/net/${STORAGE_BRIDGE}" &>/dev/null; then
logn "${STORAGE_BRIDGE} already exists — skipping."
else
logn "Creating isolated internal bridge ${STORAGE_BRIDGE} (no upstream port, ${STORAGE_CIDR})"
BRIDGE_CONF="auto ${STORAGE_BRIDGE}
iface ${STORAGE_BRIDGE} inet manual
bridge-ports none
bridge-stp off
bridge-fd 0"
if $DRY_RUN; then
echo "[dry-run] Would write /etc/network/interfaces.d/${STORAGE_BRIDGE}.conf and ifup it"
else
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"echo '${BRIDGE_CONF}' | sudo tee /etc/network/interfaces.d/${STORAGE_BRIDGE}.conf > /dev/null && sudo ifup ${STORAGE_BRIDGE}"
logn "${STORAGE_BRIDGE} created and brought up."
fi
fi
fi
# ── Phase 2: Sync host keys ───────────────────────────────────────────────────
if ! $SKIP_SYNC_KEYS; then
log "Phase 2: Syncing SSH host keys for both HA targets"
for target in proxmox-ha-server-1 proxmox-ha-server-2; do
CLAN_DIR="${REPO_ROOT}/vars/per-machine/${target}/openssh"
if [[ -d "$CLAN_DIR" ]]; then
logn "Clan vars for ${target} already exist — skipping."
else
logn "Generating host keys for ${target}..."
run bash "${REPO_ROOT}/scripts/secrets/sync-host-keys.sh" "$target"
fi
done
fi
# ── Phase 2.5: Prepare Proxmox node for building ─────────────────────────────
if ! $SKIP_CREATE_VMS && ! $DRY_RUN; then
CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"
# Fix /nix ownership if it exists but belongs to a different UID.
# pve1's IPA-enrolled wayne (UID 50002) can't write to a store created by
# another UID — passwordless sudo corrects it once.
# Use direct SSH (no sudo) for the writability check so we test wayne's own
# access, not root's.
local_ssh() { ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "$*"; }
if local_ssh "test -d /nix" &>/dev/null && ! local_ssh "test -w /nix" &>/dev/null; then
logn "/nix exists but not writable by ${SSH_USER} — fixing ownership with sudo (one-time)..."
local_ssh "sudo chown -R ${SSH_USER} /nix"
logn "Done."
fi
unset -f local_ssh
# Ensure the remote clone is on the correct branch so create-proxmox-resource.sh
# builds from the same commits we're deploying.
REMOTE_REPO="/home/${SSH_USER}/nixos"
if pve_check "test -d ${REMOTE_REPO}/.git" &>/dev/null; then
REMOTE_BRANCH=$(ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"cd ${REMOTE_REPO} && git rev-parse --abbrev-ref HEAD 2>/dev/null")
if [[ "$REMOTE_BRANCH" != "$CURRENT_BRANCH" ]]; then
logn "Remote clone is on '${REMOTE_BRANCH}', switching to '${CURRENT_BRANCH}'..."
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"cd ${REMOTE_REPO} && git fetch origin && git checkout '${CURRENT_BRANCH}' && git pull --ff-only"
logn "Done."
fi
fi
fi
# ── Phase 3: Create VMs ───────────────────────────────────────────────────────
if ! $SKIP_CREATE_VMS; then
log "Phase 3: Building and creating VMs on ${NODE}"
CREATE="${REPO_ROOT}/scripts/proxmox/create-proxmox-resource.sh"
for spec in "${VMID1}:ha-server-1:proxmox-ha-server-1" "${VMID2}:ha-server-2:proxmox-ha-server-2"; do
IFS=: read -r vmid host_name flake_target <<< "$spec"
log "Creating ${flake_target} (VMID ${vmid}) on ${NODE}..."
# Always --force-rebuild: create-proxmox-resource.sh only calls
# sync_remote_host_keys (which populates host-keys/ for proxmox.nix to
# bake the clan-var SSH key into the disko image) when it actually builds.
# Reusing a cached image skips that step, so destroy+recreate would reuse
# an image with a stale/random key baked in → sops fails on first boot.
run bash "$CREATE" \
--type vm \
--host "$host_name" \
--vmid "$vmid" \
--node "$NODE" \
--storage "$STORAGE" \
--memory "$MEMORY_MB" \
--cores "$CORES" \
--force-rebuild
done
fi
# ── Phase 4: Add storage NIC and DRBD disk ────────────────────────────────────
if ! $SKIP_ADD_HARDWARE; then
log "Phase 4: Attaching storage NIC (${STORAGE_BRIDGE}) and DRBD disk (${DRBD_DISK_GB}G) to each VM"
for vmid in "$VMID1" "$VMID2"; do
log " VMID ${vmid}: stopping to add hardware..."
pve "qm stop ${vmid} --skiplock 1 2>/dev/null; sleep 3" || true
logn "Adding net1 (${STORAGE_BRIDGE})..."
pve "qm set ${vmid} --net1 virtio,bridge=${STORAGE_BRIDGE},firewall=0"
logn "Adding scsi1 (${STORAGE}:${DRBD_DISK_GB}G for DRBD)..."
pve "qm set ${vmid} --scsi1 ${STORAGE}:${DRBD_DISK_GB},format=raw"
logn "Starting VMID ${vmid}..."
pve "qm start ${vmid}"
done
fi
# ── Phase 5: Wait for SSH ─────────────────────────────────────────────────────
if ! $SKIP_BOOT_WAIT; then
log "Phase 5: Waiting for both nodes to come up"
wait_for_ssh "$NODE1_IP" "$NODE1_HOST"
wait_for_ssh "$NODE2_IP" "$NODE2_HOST"
logn "Both nodes are SSHable."
# Give systemd a few seconds to settle after activation
sleep 10
fi
# ── Phase 5.5: Refresh sops host-key registrations ───────────────────────────
#
# Disko builds raw disk images; each new VM boots with a freshly-generated SSH
# host key rather than the one pre-seeded in clan vars. This phase scans the
# actual running VMs, and if their ed25519 host keys differ from what clan vars
# record: updates the clan var pub-key files, rewrites the .sops.yaml age-key
# anchors, and re-encrypts all affected sops files so the nodes can decrypt
# secrets on the next nixos-rebuild. Safe no-op when keys haven't changed.
if ! $SKIP_REFRESH_SOPS_KEYS; then
if $DRY_RUN; then
logn "[dry-run] Would scan VM host keys and refresh .sops.yaml / secrets if needed"
else
log "Phase 5.5: Refreshing sops host-key registrations (disko key drift fix)"
SOPS_UPDATED=false
for spec in \
"${NODE1_IP}:proxmox-ha-server-1:${NODE1_HOST}" \
"${NODE2_IP}:proxmox-ha-server-2:${NODE2_HOST}"; do
IFS=: read -r node_ip flake_target host_name <<< "$spec"
CLAN_PUB="${REPO_ROOT}/vars/per-machine/${flake_target}/openssh/ssh_host_ed25519_key.pub/value"
logn "Scanning ed25519 host key from ${host_name} (${node_ip})..."
RAW=$(ssh-keyscan -t ed25519 "${node_ip}" 2>/dev/null | grep -v "^#") || true
if [[ -z "$RAW" ]]; then
logn "WARNING: no ed25519 key returned by ssh-keyscan for ${node_ip} — skipping"
continue
fi
# ssh-keyscan returns: <ip> ssh-ed25519 <b64key>
SCANNED_TYPE=$(awk '{print $2}' <<< "$RAW")
SCANNED_KEY=$(awk '{print $3}' <<< "$RAW")
SCANNED_PUBKEY="${SCANNED_TYPE} ${SCANNED_KEY} ${host_name}"
CURRENT=$(tr -d '\n' < "$CLAN_PUB" 2>/dev/null || true)
if [[ "$SCANNED_PUBKEY" == "$CURRENT" ]]; then
logn "${host_name}: clan var matches running key — no update needed"
continue
fi
logn "${host_name}: key drift detected — updating clan var"
logn " old: ${CURRENT}"
logn " new: ${SCANNED_PUBKEY}"
echo "$SCANNED_PUBKEY" > "$CLAN_PUB"
SOPS_UPDATED=true
# Rewrite the .sops.yaml anchor for this host with the new age key.
ANCHOR="${flake_target}" # e.g. proxmox-ha-server-1
NEW_AGE=$(echo "$SCANNED_PUBKEY" | \
nix run --quiet --no-warn-dirty nixpkgs#ssh-to-age 2>/dev/null)
if [[ -z "$NEW_AGE" ]]; then
err "ssh-to-age produced no output for ${host_name} — check nixpkgs#ssh-to-age"
fi
logn " new age key: ${NEW_AGE}"
sed -i "/&${ANCHOR} /s| age[a-z0-9]*$| ${NEW_AGE}|" "${REPO_ROOT}/.sops.yaml"
done
if $SOPS_UPDATED; then
logn "Running sops updatekeys on affected secrets..."
SOPS="nix run --quiet --no-warn-dirty nixpkgs#sops --"
(cd "${REPO_ROOT}" && \
$SOPS updatekeys -y secrets/common.yaml && \
$SOPS updatekeys -y secrets/ha-server-1.yaml && \
$SOPS updatekeys -y secrets/ha-server-2.yaml && \
$SOPS updatekeys -y secrets/ha-server-1.keytab && \
$SOPS updatekeys -y secrets/ha-server-2.keytab)
# Note: ha-corosync-authkey is re-generated and re-encrypted by cluster-init below.
logn "Committing refreshed host keys and re-encrypted secrets..."
(cd "${REPO_ROOT}" && \
git add \
vars/per-machine/proxmox-ha-server-1/openssh/ssh_host_ed25519_key.pub/value \
vars/per-machine/proxmox-ha-server-2/openssh/ssh_host_ed25519_key.pub/value \
.sops.yaml \
secrets/common.yaml \
secrets/ha-server-1.yaml \
secrets/ha-server-2.yaml \
secrets/ha-server-1.keytab \
secrets/ha-server-2.keytab && \
git commit -m "secrets(ha): refresh sops host-key registrations for new VM instances" || true)
logn "Sops keys refreshed and committed."
fi
fi
fi
# ── Phase 6: Cluster init ─────────────────────────────────────────────────────
if ! $SKIP_CLUSTER_INIT; then
log "Phase 6: Initialising HA cluster"
CLUSTER_INIT="${REPO_ROOT}/scripts/ha/cluster-init.sh"
[[ -x "$CLUSTER_INIT" ]] || chmod +x "$CLUSTER_INIT"
if $DRY_RUN; then
logn "[dry-run] Would generate temp key, authorise on ${NODE2_HOST}, scp cluster-init.sh to ${NODE1_HOST}, and run it as root via sudo"
else
# Generate a temp keypair so cluster-init.sh can SSH node1→node2 as ${HA_USER}.
# Root on node1 has no keys; a temp key bridging node1→node2 nixos solves this.
TEMP_KEY="${REPO_ROOT}/.tmp-cluster-init-key"
TEMP_KEY_PUB="${TEMP_KEY}.pub"
rm -f "$TEMP_KEY" "$TEMP_KEY_PUB"
ssh-keygen -t ed25519 -f "$TEMP_KEY" -N "" -C "cluster-init-temp-$(date +%s)" -q
TEMP_PUBKEY=$(cat "$TEMP_KEY_PUB")
logn "Authorising temp key on ${NODE2_HOST} for ${HA_USER}..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE2_IP}" \
"mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo '${TEMP_PUBKEY}' >> ~/.ssh/authorized_keys"
logn "Placing temp key on ${NODE1_HOST} for root..."
scp -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"$TEMP_KEY" "${HA_USER}@${NODE1_IP}:/tmp/cluster-init-key"
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo mkdir -p /root/.ssh && sudo cp /tmp/cluster-init-key /root/.ssh/cluster-init-key && \
sudo chmod 600 /root/.ssh/cluster-init-key && rm -f /tmp/cluster-init-key"
logn "Uploading cluster-init.sh to ${NODE1_HOST}..."
scp -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"$CLUSTER_INIT" "${HA_USER}@${NODE1_IP}:/tmp/cluster-init.sh"
logn "Running cluster-init.sh on ${NODE1_HOST}..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo env NODE1=${NODE1_HOST} NODE2=${NODE2_HOST} \
NODE1_IP=${NODE1_IP} NODE2_IP=${NODE2_IP} \
VIP=192.168.20.229 XFS_MOUNT=/srv/ha-data \
ISCSI_IQN=iqn.2026-01.home.sweet:ha-storage \
VMID_NODE1=${VMID1} VMID_NODE2=${VMID2} \
HA_USER=${HA_USER} HA_KEY=/root/.ssh/cluster-init-key \
bash /tmp/cluster-init.sh"
logn "Cleaning up temp key from both nodes..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE2_IP}" \
"sed -i '/cluster-init-temp/d' ~/.ssh/authorized_keys" 2>/dev/null || true
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo rm -f /root/.ssh/cluster-init-key" 2>/dev/null || true
rm -f "$TEMP_KEY" "$TEMP_KEY_PUB"
# Encrypt the corosync authkey generated by cluster-init and commit it.
log " Encrypting corosync authkey into secrets/ha-corosync-authkey..."
AUTHKEY_TMP="${REPO_ROOT}/secrets/ha-corosync-authkey.tmp"
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo cat /etc/corosync/authkey" > "$AUTHKEY_TMP"
if [[ ! -s "$AUTHKEY_TMP" ]]; then
err "corosync authkey on node1 is empty — cluster-init may have failed."
fi
mv "$AUTHKEY_TMP" "${REPO_ROOT}/secrets/ha-corosync-authkey"
(cd "${REPO_ROOT}" && nix run nixpkgs#sops -- -e --input-type binary -i secrets/ha-corosync-authkey)
logn "Authkey encrypted. Committing..."
(cd "${REPO_ROOT}" && git add secrets/ha-corosync-authkey && \
git commit -m "secrets(ha): encrypt corosync authkey generated by cluster-init")
logn "Committed."
fi
fi
# ── Phase 7: Acceptance tests ─────────────────────────────────────────────────
if ! $SKIP_TESTS; then
log "Phase 7: Running acceptance tests (T1T7)"
if $DRY_RUN; then
logn "[dry-run] Would run acceptance-tests.sh against ${NODE1_HOST}/${NODE2_HOST}"
else
NODE1="$NODE1_HOST" NODE2="$NODE2_HOST" \
NODE1_IP="$NODE1_IP" NODE2_IP="$NODE2_IP" \
VIP="192.168.20.229" \
bash "${REPO_ROOT}/scripts/ha/acceptance-tests.sh"
fi
fi
log "Deploy complete."
+256
View File
@@ -0,0 +1,256 @@
#!/usr/bin/env bash
# failover.sh — graceful HA cluster failover
#
# Detects which node is active and moves all resources to the other node by
# putting the active node into Pacemaker standby. Waits for the XFS mount to
# appear on the target before returning.
#
# Usage:
# scripts/ha/failover.sh [--to node1|node2] [--force] [--timeout <s>] [--dry-run]
#
# --to node1|node2 target node (default: the node that is NOT currently active)
# --force skip the interactive confirmation prompt
# --timeout <s> seconds to wait for resources to move (default: 120)
# --dry-run show what would be done without changing anything
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}"
NODE2_IP="${NODE2_IP:-192.168.2.227}"
VIP="${VIP:-192.168.20.229}"
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"
HA_USER="${HA_USER:-nixos}"
# ──────────────────────────────────────────────────────────────────────────
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
# ── Argument parsing ───────────────────────────────────────────────────────
TARGET_NODE=""
FORCE=false
DRY_RUN=false
TIMEOUT=120
while [[ $# -gt 0 ]]; do
case "$1" in
--to)
shift
case "${1:-}" in
node1|ha-server-1) TARGET_NODE="$NODE1" ;;
node2|ha-server-2) TARGET_NODE="$NODE2" ;;
*) echo "ERROR: --to must be node1, node2, ha-server-1, or ha-server-2"; exit 1 ;;
esac
;;
--force) FORCE=true ;;
--dry-run) DRY_RUN=true ;;
--timeout) shift; TIMEOUT="${1:?--timeout requires a value}" ;;
*) echo "Unknown argument: $1"; echo "Usage: $0 [--to node1|node2] [--force] [--timeout <s>] [--dry-run]"; exit 1 ;;
esac
shift
done
DRY_PREFIX=""
$DRY_RUN && DRY_PREFIX="[dry-run] "
echo "════════════════════════════════════════════════════"
echo " HA Cluster Failover — $(date '+%Y-%m-%d %H:%M:%S')"
$DRY_RUN && echo " MODE: dry-run — no changes will be made"
echo "════════════════════════════════════════════════════"
# ── Detect active node ─────────────────────────────────────────────────────
echo ""
echo "Detecting active node..."
CRM_OUT=""
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" true 2>/dev/null; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
fi
if [[ -z "$CRM_OUT" ]]; then
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" true 2>/dev/null; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
fi
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" — the * bullet
# means ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -z "$ACTIVE_NODE" ]]; then
echo ""
echo "ERROR: could not determine active node from crm_mon."
echo " Is Pacemaker still settling? Try running scripts/ha/health.sh first."
echo " If Pacemaker is down on both nodes, manual recovery is required."
exit 1
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
STANDBY_NODE="$NODE2"
STANDBY_IP="$NODE2_IP"
na() { n1 "$@"; }
ns() { n2 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
STANDBY_NODE="$NODE1"
STANDBY_IP="$NODE1_IP"
na() { n2 "$@"; }
ns() { n1 "$@"; }
fi
echo " Active: $ACTIVE_NODE ($ACTIVE_IP)"
echo " Standby: $STANDBY_NODE ($STANDBY_IP)"
# ── Validate target ────────────────────────────────────────────────────────
if [[ -n "$TARGET_NODE" ]]; then
if [[ "$TARGET_NODE" == "$ACTIVE_NODE" ]]; then
echo ""
echo "ERROR: $TARGET_NODE is already the active node — nothing to do."
exit 1
fi
echo " Target: $TARGET_NODE (as requested)"
else
echo " Target: $STANDBY_NODE (auto — the other node)"
fi
# ── Pre-checks ─────────────────────────────────────────────────────────────
echo ""
echo "Pre-checks..."
DRBD_DSTATE=$(na "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_DSTATE" | grep -q "UpToDate/UpToDate"; then
echo ""
echo " WARNING: DRBD dstate is '$DRBD_DSTATE' (not UpToDate/UpToDate)."
echo " Failing over with a partially-synced disk risks split-brain."
if ! $FORCE; then
echo " Use --force to proceed anyway (not recommended)."
exit 1
fi
echo " --force specified — proceeding despite non-ideal DRBD state."
else
echo " DRBD dstate: $DRBD_DSTATE — OK"
fi
QUORUM_OK=$(na "corosync-quorumtool -s 2>/dev/null | grep -c 'Quorate:.*Yes'" 2>/dev/null || echo "0")
if [[ "$QUORUM_OK" -lt 1 ]]; then
echo " ERROR: cluster does not have quorum — failover would be unsafe."
exit 1
fi
echo " Quorum: OK"
# ── Confirm ────────────────────────────────────────────────────────────────
if ! $FORCE && ! $DRY_RUN; then
echo ""
echo " This will move all resources from $ACTIVE_NODE$STANDBY_NODE."
echo " VIP and services will be unreachable for ~1030 seconds."
printf " Proceed? [y/N] "
read -r ANSWER
[[ "${ANSWER,,}" == "y" || "${ANSWER,,}" == "yes" ]] || { echo "Aborted."; exit 0; }
fi
# ── Capture active node's crm_node name ───────────────────────────────────
# crm_node -n returns the node name as registered in Pacemaker (may differ
# from hostname if Pacemaker was configured with explicit node names).
ACTIVE_CRMD_NAME=$(na "crm_node -n 2>/dev/null" 2>/dev/null || echo "$ACTIVE_NODE")
# ── Perform failover ───────────────────────────────────────────────────────
echo ""
echo "${DRY_PREFIX}Putting $ACTIVE_NODE into standby (resources will migrate to $STANDBY_NODE)..."
if ! $DRY_RUN; then
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v on" 2>/dev/null || true
fi
# ── Wait for resources to move ─────────────────────────────────────────────
echo "${DRY_PREFIX}Waiting up to ${TIMEOUT}s for XFS to mount on $STANDBY_NODE..."
MOVED=false
SPIN_CHARS=('|' '/' '-' '\')
SPIN_I=0
if $DRY_RUN; then
echo " [dry-run] would wait for mountpoint $XFS_MOUNT on $STANDBY_NODE"
MOVED=true
else
for i in $(seq 1 "$TIMEOUT"); do
if ns "mountpoint -q '${XFS_MOUNT}' 2>/dev/null" 2>/dev/null; then
printf "\r%-80s\r" ""
echo " Resources moved in ${i}s"
MOVED=true
break
fi
SPIN_I=$(( SPIN_I + 1 ))
SC="${SPIN_CHARS[$((SPIN_I % 4))]}"
printf "\r [%s] waiting... (%ds) " "$SC" "$i"
sleep 1
done
fi
if ! $MOVED; then
echo ""
echo "ERROR: XFS did not mount on $STANDBY_NODE within ${TIMEOUT}s."
echo ""
echo " Current resource state:"
na "crm_mon -1 2>/dev/null" 2>/dev/null | grep -E 'Started|Stopped|Promoted|Unpromoted|FAILED' | sed 's/^/ /' || true
echo ""
echo " Clearing standby to restore $ACTIVE_NODE (undo the failover attempt)..."
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
na "crm_resource --cleanup" 2>/dev/null || true
exit 1
fi
# ── Clear failure history ──────────────────────────────────────────────────
echo "${DRY_PREFIX}Clearing Pacemaker failure history..."
if ! $DRY_RUN; then
ns "crm_resource --cleanup 2>/dev/null" 2>/dev/null || true
fi
# ── Re-enable original active node as standby ─────────────────────────────
echo "${DRY_PREFIX}Re-enabling $ACTIVE_NODE (now standby — will not claim resources)..."
if ! $DRY_RUN; then
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
fi
# ── Wait briefly for DRBD resync to begin ─────────────────────────────────
if ! $DRY_RUN; then
sleep 5
fi
# ── Final state ────────────────────────────────────────────────────────────
echo ""
echo "Failover complete. Final state:"
echo ""
CRM_OUT_AFTER=""
if ! $DRY_RUN; then
CRM_OUT_AFTER=$(ns "crm_mon -1" 2>/dev/null || na "crm_mon -1" 2>/dev/null || true)
else
CRM_OUT_AFTER="$CRM_OUT"
fi
NEW_ACTIVE=$(echo "$CRM_OUT_AFTER" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -n "$NEW_ACTIVE" ]]; then
if [[ "$NEW_ACTIVE" == "$ACTIVE_NODE" ]]; then
echo " WARNING: $ACTIVE_NODE is still showing as active in crm_mon."
echo " Pacemaker may still be settling — check again in a few seconds."
else
echo " Active: $NEW_ACTIVE"
echo " Standby: $ACTIVE_NODE"
fi
fi
echo ""
RESOURCES_AFTER=$(echo "$CRM_OUT_AFTER" | awk '/Full List of Resources/,0' | tail -n +2 || true)
[[ -z "$RESOURCES_AFTER" ]] && RESOURCES_AFTER=$(echo "$CRM_OUT_AFTER" | \
grep -E 'Started|Stopped|Promoted|Unpromoted|FAILED|Master|Slave' || true)
[[ -n "$RESOURCES_AFTER" ]] && echo "$RESOURCES_AFTER" | sed 's/^/ /'
echo ""
echo " (DRBD resync of $ACTIVE_NODE may take a moment; monitor with:"
echo " ssh nixos@${ACTIVE_IP} 'sudo watch -n3 cat /proc/drbd')"
echo ""
echo "════════════════════════════════════════════════════"
Regular → Executable
View File
+206
View File
@@ -0,0 +1,206 @@
#!/usr/bin/env bash
# health.sh — HA cluster health snapshot (read-only, non-destructive)
#
# Prints a compact status panel across both nodes: SSH reachability, quorum,
# DRBD state, Pacemaker resources, and service ports via the VIP.
# Run from any host with SSH access to the HA nodes.
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.20.229}" # vars.haServerVip (storage-client, VLAN 20 — internal only)
VIP_LAN="${VIP_LAN:-192.168.2.229}" # vars.haServerLanVip (LAN, VLAN 2 — reachable from workstation)
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
HA_USER="${HA_USER:-nixos}"
# ──────────────────────────────────────────────────────────────────────────
REACHABLE_1=false
REACHABLE_2=false
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
probe_node() {
local ip=$1
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${ip}" true 2>/dev/null && echo "ONLINE" || echo "OFFLINE"
}
section() { echo ""; echo "── $* ──"; }
echo "════════════════════════════════════════════════════"
echo " HA Cluster Health — $(date '+%Y-%m-%d %H:%M:%S')"
echo "════════════════════════════════════════════════════"
# ── Node reachability ──────────────────────────────────────────────────────
section "Nodes"
N1_STATUS=$(probe_node "$NODE1_IP")
N2_STATUS=$(probe_node "$NODE2_IP")
[[ "$N1_STATUS" == "ONLINE" ]] && REACHABLE_1=true
[[ "$N2_STATUS" == "ONLINE" ]] && REACHABLE_2=true
if ! $REACHABLE_1 && ! $REACHABLE_2; then
echo " ERROR: both nodes unreachable — cannot continue."
exit 1
fi
# ── Detect active node ─────────────────────────────────────────────────────
# crm_mon 2.x formats the Promoted line as " * Promoted: [ node ]" — the
# bullet * means ^\s*(Promoted|Masters): never matches. Filter out Unpromoted
# first, then match anywhere on the line.
ACTIVE_NODE=""
CRM_OUT=""
if $REACHABLE_1; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
elif $REACHABLE_2; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
ACTIVE_NODE=$(echo "${CRM_OUT:-}" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
STANDBY_NODE=""
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
STANDBY_NODE="$NODE2"
elif [[ "$ACTIVE_NODE" == "$NODE2" ]]; then
STANDBY_NODE="$NODE1"
fi
n1_tag=""; n2_tag=""
[[ "$ACTIVE_NODE" == "$NODE1" ]] && n1_tag=" [ACTIVE]" || n1_tag=" [STANDBY]"
[[ "$ACTIVE_NODE" == "$NODE2" ]] && n2_tag=" [ACTIVE]" || n2_tag=" [STANDBY]"
[[ -z "$ACTIVE_NODE" ]] && { n1_tag=""; n2_tag=""; }
printf " %-14s [%s]%s\n" "$NODE1" "$N1_STATUS" "$n1_tag"
printf " %-14s [%s]%s\n" "$NODE2" "$N2_STATUS" "$n2_tag"
if [[ -z "$ACTIVE_NODE" ]]; then
echo ""
echo " WARNING: could not determine active node from crm_mon."
echo " Pacemaker may still be settling, or both nodes may be in standby."
fi
# ── Quorum ─────────────────────────────────────────────────────────────────
section "Quorum"
if $REACHABLE_1; then
QUORUM=$(n1 "corosync-quorumtool -s 2>/dev/null" 2>/dev/null || echo "")
elif $REACHABLE_2; then
QUORUM=$(n2 "corosync-quorumtool -s 2>/dev/null" 2>/dev/null || echo "")
fi
if [[ -z "${QUORUM:-}" ]]; then
echo " corosync-quorumtool: unavailable"
else
QUORATE=$(echo "$QUORUM" | grep "Quorate:" | awk '{print $2}' || echo "?")
VOTES=$(echo "$QUORUM" | grep "Total votes:" | awk '{print $3}' || echo "?")
NEEDED=$(echo "$QUORUM" | grep "Quorum votes:" | awk '{print $3}' || echo "?")
echo " Quorate: $QUORATE Votes: $VOTES / Expected: $NEEDED"
fi
# ── DRBD ───────────────────────────────────────────────────────────────────
section "DRBD (ha-data)"
drbd_info_from() {
local node=$1 run=$2
local role dstate cs pct
role=$($run "drbdadm role ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
dstate=$($run "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
cs=$($run "grep -oE 'cs:[A-Za-z]+' /proc/drbd 2>/dev/null | head -1 | sed 's/cs://'" 2>/dev/null || echo "unknown")
pct=$($run "grep -oE \"sync'ed:[[:space:]]+[0-9.]+\" /proc/drbd 2>/dev/null | grep -oE '[0-9.]+$' | head -1" 2>/dev/null || echo "")
printf " %-14s role: %-22s dstate: %-25s cs: %s" \
"$node" "${role:-unknown}" "${dstate:-unknown}" "${cs:-unknown}"
[[ -n "$pct" ]] && printf " syncing: %s%%" "$pct"
echo ""
}
$REACHABLE_1 && drbd_info_from "$NODE1" n1 || echo " $NODE1 [OFFLINE]"
$REACHABLE_2 && drbd_info_from "$NODE2" n2 || echo " $NODE2 [OFFLINE]"
# ── Pacemaker (full crm_mon output) ───────────────────────────────────────
section "Pacemaker"
if [[ -n "${CRM_OUT:-}" ]]; then
echo "$CRM_OUT" | sed 's/^/ /'
else
echo " crm_mon returned no output — trying again without suppression:"
if $REACHABLE_1; then
n1 "crm_mon -1" || true
elif $REACHABLE_2; then
n2 "crm_mon -1" || true
fi
fi
# ── XFS mount ─────────────────────────────────────────────────────────────
section "XFS Mount ($XFS_MOUNT)"
check_mount() {
local node=$1 run=$2
local status
if $run "mountpoint -q '$XFS_MOUNT' 2>/dev/null" 2>/dev/null; then
local usage
usage=$($run "df -h '$XFS_MOUNT' 2>/dev/null | tail -1 | awk '{print \$3\"/\"\$2\" used (\"\$5\")\";}'" 2>/dev/null || echo "")
status="mounted"
[[ -n "$usage" ]] && status="mounted $usage"
else
status="not mounted"
fi
printf " %-14s %s\n" "$node" "$status"
}
$REACHABLE_1 && check_mount "$NODE1" n1 || echo " $NODE1 [OFFLINE]"
$REACHABLE_2 && check_mount "$NODE2" n2 || echo " $NODE2 [OFFLINE]"
# ── LAN VIP (NFS) — reachable from workstation ────────────────────────────────
section "LAN VIP ($VIP_LAN) — NFS"
if ping -c1 -W2 "$VIP_LAN" >/dev/null 2>&1; then
echo " Ping OK"
else
echo " Ping UNREACHABLE"
fi
if bash -c "echo >/dev/tcp/${VIP_LAN}/2049" 2>/dev/null; then
printf " %-10s port %-5s OK\n" "NFS" "2049"
else
printf " %-10s port %-5s UNREACHABLE\n" "NFS" "2049"
fi
# ── Storage VIP (NFS + iSCSI) — VLAN 20 internal bridge, tested via active node ─
section "Storage VIP ($VIP) — NFS + iSCSI (via ${ACTIVE_NODE:-unknown})"
run_active_raw() {
local active_ip=""
[[ "$ACTIVE_NODE" == "$NODE1" ]] && active_ip="$NODE1_IP"
[[ "$ACTIVE_NODE" == "$NODE2" ]] && active_ip="$NODE2_IP"
[[ -z "$active_ip" ]] && return 1
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${active_ip}" "$@" 2>/dev/null
}
if [[ -z "$ACTIVE_NODE" ]]; then
echo " Cannot determine active node — skipping"
else
if run_active_raw "ping -c1 -W2 '$VIP' >/dev/null 2>&1"; then
echo " Ping OK"
else
echo " Ping UNREACHABLE"
fi
if run_active_raw "bash -c 'echo >/dev/tcp/${VIP}/2049' 2>/dev/null"; then
printf " %-10s port %-5s OK\n" "NFS" "2049"
else
printf " %-10s port %-5s UNREACHABLE\n" "NFS" "2049"
fi
if run_active_raw "bash -c 'echo >/dev/tcp/${VIP}/3260' 2>/dev/null"; then
printf " %-10s port %-5s OK\n" "iSCSI" "3260"
else
printf " %-10s port %-5s UNREACHABLE\n" "iSCSI" "3260"
fi
fi
echo ""
echo "════════════════════════════════════════════════════"
if [[ -n "$ACTIVE_NODE" ]]; then
echo " Active: $ACTIVE_NODE Standby: $STANDBY_NODE"
else
echo " Active: unknown (Pacemaker not settled)"
fi
echo "════════════════════════════════════════════════════"
echo ""
+274
View File
@@ -0,0 +1,274 @@
#!/usr/bin/env bash
# resize-data-disk.sh — online resize of the HA cluster data disk
#
# Three-phase process (all online-safe, no downtime required):
# 1. Proxmox: grow scsi1 on both HA VMs (qm resize)
# 2. Guest: rescan block device on both nodes so the kernel sees the new size
# 3. DRBD + XFS: drbdadm resize, then xfs_growfs — active node only
#
# Usage:
# scripts/ha/resize-data-disk.sh --size +20G [--force] [--dry-run]
#
# --size +NNg amount to grow scsi1 by, e.g. +20G, +50G (required)
# XFS and DRBD cannot shrink; only positive deltas accepted
# --force skip the interactive confirmation prompt
# --dry-run show what would be done without changing anything
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=../env.sh
source "${SCRIPT_DIR}/../env.sh"
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}"
NODE2_IP="${NODE2_IP:-192.168.2.227}"
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"
DRBD_RESOURCE="${DRBD_RESOURCE:-ha-data}"
DATA_DISK_SLOT="${DATA_DISK_SLOT:-scsi1}" # Proxmox disk name (scsi1 = data disk)
HA_USER="${HA_USER:-nixos}"
PVE_HOST="${PVE_HOST:-${PVE1_HOST}}"
PVE_SSH_USER="${PVE_SSH_USER:-${PROXMOX_SSH_USER}}"
PVE_SUDO=""
[[ "$PVE_SSH_USER" != "root" ]] && PVE_SUDO="sudo"
# By-id symlink for the data disk; basename resolves to the raw block device.
# matches variables.nix's haServerDrbdDisk.
DATA_DISK_BYID="${DATA_DISK_BYID:-scsi-0QEMU_QEMU_HARDDISK_drive-${DATA_DISK_SLOT}}"
# ──────────────────────────────────────────────────────────────────────────
pve() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
"${PVE_SSH_USER}@${PVE_HOST}" "$@"; }
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
# ── Argument parsing ───────────────────────────────────────────────────────
SIZE=""
FORCE=false
DRY_RUN=false
while [[ $# -gt 0 ]]; do
case "$1" in
--size) shift; SIZE="${1:?--size requires a value (e.g. +20G)}" ;;
--force) FORCE=true ;;
--dry-run) DRY_RUN=true ;;
*) echo "Unknown argument: $1"
echo "Usage: $0 --size +NNg [--force] [--dry-run]"
exit 1 ;;
esac
shift
done
if [[ -z "$SIZE" ]]; then
echo "ERROR: --size is required (e.g. --size +20G)"
echo "Usage: $0 --size +NNg [--force] [--dry-run]"
exit 1
fi
# Only positive deltas — qm resize, DRBD, and XFS all refuse to shrink.
if [[ ! "$SIZE" =~ ^\+[0-9]+(G|M|T)$ ]]; then
echo "ERROR: --size must be a positive delta like +20G, +50G, +500M, +2T"
echo " (qm resize, drbdadm resize, and xfs_growfs can only grow, not shrink)"
exit 1
fi
DRY_PREFIX=""
$DRY_RUN && DRY_PREFIX="[dry-run] "
echo "════════════════════════════════════════════════════"
echo " HA Data Disk Resize — $(date '+%Y-%m-%d %H:%M:%S')"
echo " Size delta: $SIZE • Proxmox: $PVE_HOST"
$DRY_RUN && echo " MODE: dry-run — no changes will be made"
echo "════════════════════════════════════════════════════"
# ── Detect active node ─────────────────────────────────────────────────────
echo ""
echo "Detecting active node..."
CRM_OUT=""
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE1_IP}" true 2>/dev/null; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
fi
if [[ -z "$CRM_OUT" ]]; then
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE2_IP}" true 2>/dev/null; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
fi
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" (bullet *),
# so ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -z "$ACTIVE_NODE" ]]; then
echo "ERROR: could not determine active node from crm_mon."
echo " Is Pacemaker still settling? Try running scripts/ha/health.sh first."
exit 1
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
na() { n1 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
na() { n2 "$@"; }
fi
echo " Active node: $ACTIVE_NODE ($ACTIVE_IP)"
# ── Pre-check DRBD state ───────────────────────────────────────────────────
echo ""
echo "Pre-checks..."
DRBD_DSTATE=$(na "drbdadm dstate ${DRBD_RESOURCE} 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_DSTATE" | grep -q "UpToDate/UpToDate"; then
echo " WARNING: DRBD dstate is '$DRBD_DSTATE' (expected UpToDate/UpToDate)."
echo " Resizing with a partially-synced disk may cause issues."
if ! $FORCE; then
echo " Use --force to proceed anyway."
exit 1
fi
echo " --force specified — proceeding despite non-ideal DRBD state."
else
echo " DRBD dstate: $DRBD_DSTATE — OK"
fi
# ── Find VMIDs on Proxmox ─────────────────────────────────────────────────
echo ""
echo "Looking up VM IDs on ${PVE_HOST}..."
QM_LIST=$(pve "$PVE_SUDO qm list 2>/dev/null" || true)
VMID1=$(echo "$QM_LIST" | awk -v name="$NODE1" '$0 ~ name {print $1}' | head -1)
VMID2=$(echo "$QM_LIST" | awk -v name="$NODE2" '$0 ~ name {print $1}' | head -1)
if [[ -z "$VMID1" ]]; then
echo " ERROR: could not find VMID for $NODE1 on $PVE_HOST"
echo " qm list output:"
echo "$QM_LIST" | sed 's/^/ /'
exit 1
fi
if [[ -z "$VMID2" ]]; then
echo " ERROR: could not find VMID for $NODE2 on $PVE_HOST"
echo " qm list output:"
echo "$QM_LIST" | sed 's/^/ /'
exit 1
fi
echo " $NODE1: VMID $VMID1"
echo " $NODE2: VMID $VMID2"
# ── Confirm ────────────────────────────────────────────────────────────────
if ! $FORCE && ! $DRY_RUN; then
echo ""
echo " Plan:"
echo " Phase 1 — qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE} (on $PVE_HOST)"
echo " qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE} (on $PVE_HOST)"
echo " Phase 2 — block device rescan on $NODE1 and $NODE2"
echo " Phase 3 — drbdadm resize + xfs_growfs on $ACTIVE_NODE"
echo " No downtime required (all operations are online-safe)."
printf " Proceed? [y/N] "
read -r ANSWER
[[ "${ANSWER,,}" == "y" || "${ANSWER,,}" == "yes" ]] || { echo "Aborted."; exit 0; }
fi
# ═══════════════════════════════════════════════════════════════
# Phase 1 — Resize both VM data disks in Proxmox
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 1 — Proxmox disk resize (${DATA_DISK_SLOT} ${SIZE} on both VMs) ──"
echo " ${DRY_PREFIX}qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE} ($NODE1 on ${PVE_HOST})"
if ! $DRY_RUN; then
pve "$PVE_SUDO qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE}"
fi
echo " ${DRY_PREFIX}qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE} ($NODE2 on ${PVE_HOST})"
if ! $DRY_RUN; then
pve "$PVE_SUDO qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE}"
fi
echo " Phase 1 done."
# ═══════════════════════════════════════════════════════════════
# Phase 2 — Rescan block device on both guest nodes
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 2 — Block device rescan (both nodes) ──"
rescan_node() {
local node_name=$1 run_fn=$2
# Resolve block device name from the stable by-id symlink on the guest.
# Read-only lookup — safe to run even in dry-run so we show the real device.
local blk_dev=""
blk_dev=$($run_fn "bash -c 'basename \$(readlink -f /dev/disk/by-id/${DATA_DISK_BYID})'" 2>/dev/null || true)
if [[ -z "$blk_dev" ]]; then
echo " ERROR: /dev/disk/by-id/${DATA_DISK_BYID} not found on $node_name" >&2
echo " Check DATA_DISK_BYID or DATA_DISK_SLOT configuration." >&2
exit 1
fi
echo " ${DRY_PREFIX}Rescanning /dev/${blk_dev} on ${node_name}..."
if ! $DRY_RUN; then
$run_fn "bash -c 'echo 1 > /sys/block/${blk_dev}/device/rescan'" 2>/dev/null
local new_size
new_size=$($run_fn "lsblk -nd -o SIZE /dev/${blk_dev} 2>/dev/null" 2>/dev/null || echo "unknown")
echo " /dev/${blk_dev} on $node_name now reports: $new_size"
fi
}
rescan_node "$NODE1" n1
rescan_node "$NODE2" n2
echo " Phase 2 done."
# ═══════════════════════════════════════════════════════════════
# Phase 3 — Grow DRBD metadata, then XFS (active node only)
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 3 — DRBD resize + XFS grow (on active node: $ACTIVE_NODE) ──"
echo " ${DRY_PREFIX}drbdadm resize ${DRBD_RESOURCE}"
if ! $DRY_RUN; then
na "drbdadm resize ${DRBD_RESOURCE}"
fi
echo " ${DRY_PREFIX}xfs_growfs ${XFS_MOUNT}"
if ! $DRY_RUN; then
na "xfs_growfs ${XFS_MOUNT}"
fi
echo " Phase 3 done."
# ── Verify ────────────────────────────────────────────────────────────────
echo ""
echo "── Verify ──"
if ! $DRY_RUN; then
DF_OUT=$(na "df -h '${XFS_MOUNT}' 2>/dev/null" 2>/dev/null || echo "")
if [[ -n "$DF_OUT" ]]; then
echo " ${XFS_MOUNT}:"
echo "$DF_OUT" | sed 's/^/ /'
fi
DRBD_DSTATE_AFTER=$(na "drbdadm dstate ${DRBD_RESOURCE} 2>/dev/null" 2>/dev/null || echo "unknown")
echo " DRBD dstate: $DRBD_DSTATE_AFTER"
if ! echo "$DRBD_DSTATE_AFTER" | grep -q "UpToDate/UpToDate"; then
echo " NOTE: DRBD is resyncing — normal immediately after resize."
echo " Monitor: ssh nixos@${ACTIVE_IP} 'sudo watch -n3 cat /proc/drbd'"
fi
else
echo " [dry-run] would verify df -h ${XFS_MOUNT} and drbdadm dstate on $ACTIVE_NODE"
fi
echo ""
echo "════════════════════════════════════════════════════"
echo " Resize complete."
echo " Active node: $ACTIVE_NODE"
echo "════════════════════════════════════════════════════"
echo ""
+141 -130
View File
@@ -1,216 +1,227 @@
root-hashedPassword: ENC[AES256_GCM,data:Kp0nOZI7vDoLhJHiOJBwJn0rQZ5yhnwapGnAcA+qh8vlDETtFs/iQdetF/2ZxmANf62SviTNd+Ag0q5JIF1996x7onZGXqxgSMCuVzZLBdUlsO5IR0BslWWz47khYGTe4WkUg4NB1itBfQ==,iv:5Sra5vJ79V8hxQT3g9qJ+dOj2W2sumIhqpitqnHjJdk=,tag:3Igu0+8GeUZHqS3fKUVwog==,type:str] root-hashedPassword: ENC[AES256_GCM,data:Kp0nOZI7vDoLhJHiOJBwJn0rQZ5yhnwapGnAcA+qh8vlDETtFs/iQdetF/2ZxmANf62SviTNd+Ag0q5JIF1996x7onZGXqxgSMCuVzZLBdUlsO5IR0BslWWz47khYGTe4WkUg4NB1itBfQ==,iv:5Sra5vJ79V8hxQT3g9qJ+dOj2W2sumIhqpitqnHjJdk=,tag:3Igu0+8GeUZHqS3fKUVwog==,type:str]
nixos-hashedPassword: ENC[AES256_GCM,data:pT7tVRN6X4a+DNUgB7fIUUE3CbnetkjxmoSL1PxSU+ktsFU+fB0mEvJjA1uujsGH5Rcztg7YM815+M0Z67ILmHaXbza5DtFacrqhi4/b277xly0SHRX4yOvBwQh6mJG1jn/0O/wvUUIYdw==,iv:bp2nfhC8nFbk6o5iWDAugvbzu7J/a1xayFnBEtkhNpE=,tag:HqWgkIpSrSM/K9OK2WO+VQ==,type:str] nixos-hashedPassword: ENC[AES256_GCM,data:pT7tVRN6X4a+DNUgB7fIUUE3CbnetkjxmoSL1PxSU+ktsFU+fB0mEvJjA1uujsGH5Rcztg7YM815+M0Z67ILmHaXbza5DtFacrqhi4/b277xly0SHRX4yOvBwQh6mJG1jn/0O/wvUUIYdw==,iv:bp2nfhC8nFbk6o5iWDAugvbzu7J/a1xayFnBEtkhNpE=,tag:HqWgkIpSrSM/K9OK2WO+VQ==,type:str]
nix-github-token: ENC[AES256_GCM,data:k1vYz7SqVhzpWa6jTL6NUD8lKOCpHCgTm+HT4IcnbzbSTUZP/bJUYw==,iv:UqAULZnr/4+VcioUDfTwvOSuwM8K9JgGhiApvYQPyoc=,tag:1LKHXhWAO/AHPDIZFBb04A==,type:str] nix-github-token: ENC[AES256_GCM,data:k1vYz7SqVhzpWa6jTL6NUD8lKOCpHCgTm+HT4IcnbzbSTUZP/bJUYw==,iv:UqAULZnr/4+VcioUDfTwvOSuwM8K9JgGhiApvYQPyoc=,tag:1LKHXhWAO/AHPDIZFBb04A==,type:str]
beszel-token: ENC[AES256_GCM,data:OWmSRkZjb11y0Y8GdobqiE9GFwzdHOvvxCbYx69qUghGYARN,iv:i/JhGH0O7ThxPkL0SLAjfN0Fq8prm7tybI5kF2NRNpw=,tag:dBcqxOSHTnD4xngpOog55Q==,type:str]
nix-gitea-token: ENC[AES256_GCM,data:HRQ8ymx/D8pLcL/pYIhcSTv3tlBCKOU5nhXPp/W6rNI+t+yxfOoRPQ==,iv:0Av3lrxQew2bDFf67nX/UM+cD0/8CtbQ2fZSaKWHAzM=,tag:0K+atsB/YmYd/qCN3hMdBw==,type:str]
sops: sops:
age: age:
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrMGd3ZVNlNXdmOUZMUzdQ YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyR3lnUnZwUnlJTkFaeVdz
TW1acEs0NFA2Q01rM2dkc1h0NHkzQmhiWFZZCjMyK202VWdlaGhsZW04MnVwUVdO T3Q2eW52cWI2eTlhSFpHNDRHTEVNVGZJYm1NCldaMmlHaCt1K3lWRHhEZExHK3lo
alA0Q2FETThsYkhSS0hKdHBaS3VaY28KLS0tIG1Gdk8yalREOUtIZTUyY2p1UHlJ azZFck1URVg5ejBaRVdCWjdFMTc4dGsKLS0tIG9mNkxsZXI1N1hCRWk1NTZKcUUr
eG5iQnJsaTJBY3Y1dkw1c0VEaDQwdDQKfV04fLy32Lp2ZQ2VnvQ0h/Vsf+qdaJiv SlJoWGdWbXhqZEJHM3IzZmZSQ25QbzAKPzBIA/IJiZr5NpOhB6IPkUSDGQzPwpTU
DnLXGZ9hE5yzpKWkQIRgqYGBkF8PkH0YC4OIaVkA53wrtjqS4ZHR9Q== vgFLMze8OSEviaGXKLt/ZwTXHsr5As7V9yGvJKJHhS/hzuSLRjs8rQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjWFNRY2FiK3VkSm1RdHBn YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpYmZHNWVCL0ErNEF3OTlG
cWl1ZStLcTRFZWY5VVI5N0FhODZvR1ltM0FvCnBHZUtTUm9QeHNlbVBoZEx1V3Fa TTJDT1hZMXJNNUdLZ3ByUDNndVpLWVVGUm1RCkZSRmJmd3NyeHc0V0tpZEdYakF1
Nk9iMmJKVnhocEpERi9leE1ySUtNMFkKLS0tIDRRYkxnbU90S2RyMHdJNzRJNXBi WXhyWUJ0V1hTYzhrSUo1ZUNzR2J3OVUKLS0tIEthYXVvNVRibDVSQ2N0NWt3TmNP
QjRmZFhVakVic2tYODZHcWtJRmNQTDQK7G8eSJInt11P0DiL9uzNQ/ZHHLVNIYPe cTlteEtRYmxnUmdIUlBTREZzQXBBSVEKxQq10KDseuoVPe0cLBbk1+weuq0y6di+
bvlhuGkEuQ/+j5sVSKOfSI2Y7CvM7TpE3APyKBcLG3ajYg6F/Ev3SA== wJjaMShqVQBUI+MFWiQokBhPt8gZS7cs33LkWf3BegNALLDd6HXoOg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age19m0m7vdfg86yqy8l5mmle5jdd0unrn3f55t232w8h5ey42cqw34sfpt32n recipient: age19m0m7vdfg86yqy8l5mmle5jdd0unrn3f55t232w8h5ey42cqw34sfpt32n
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1S3dOcFdwV2NoMW1oMnY1 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArOE9oU210Vkk1Z1VKMS8x
VGdxeXJVR1lsbzNVZHlGb0NGOVo0SndiakVjCmg0QlNnZDV5RlFja2hCbVRXV1VF YTJGbFlSYmx6eWgvMXExZ243SndXWkZ0ekY4ClJVUTFSVkRERjY1UVZMZjlFeE1P
S1ZtbC9KU0U1ZW9zeVoyR3hxNW1XTFUKLS0tIFUzWTJhTzM4QnpWV3h1OFU0N3BK Vm40WHY4UjMwZWxVa05lblBMUW9iNEEKLS0tIGVlbDVxakR4MlBEaFlrM0tNOWRX
RnF1N2k0S0lIVitoNDJLUmZqdHRzZVkKUfNg24p8zxb3749v/A1BOKCNw75AUKpf elIvMnp5NmhnYVBYOFA0aUdtOE5DbEUKy+soKNLlRe0SC8kcnwrpKqvSrTGE114/
RUmFCw5DDWF2aNM0mZqcjjVmJ/FRKV2HXwwUGsHPKSOTnKfOUlPNKA== FaX2829gQWm0bYI0M4ixeTc5ME2O2Ct2tvYlzfZQPnAuub+jVx2k8g==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39 recipient: age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzazl5MUpUNEVNMTdLWXBp YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAzR0R1VEc2eTRIOWNxaVpa
c1hkaVhjcTZXTUNsWS96QVVWb0RVQlZ4VlNvCnE3TFRySU5jTFk2WjBONUQyQUhl RXVaNkxjaDVDTmxMVWJEU0dWNkxDeE56c2swCjdkZGVQM3V0SnhGMXFxSEFEeEpP
U1lNTDFTRmZhMUFyZmpVY2xpaUVxRW8KLS0tIE9MQ1M2U2ZXQmRCVll2UGRWL1RG OFNwTWQwYTY0S0trMy9FMjNjbWFxajAKLS0tIHhjUW1lZ1EycUh2Sm1yanhzS3dP
K2tnU0NOKzU5dkpiekF0Vk1VM0ZZZDgKK13aFypGAqrKWPOr3UwtXI1EoXf1+UzS WFlMcndzOThUVVVDZlJJeGFBT2JLTEUKHsJ6cwSPcO0IB1CQe2RqKeid8Q92BTNF
rBqcwnX6WPxSKUwWoins4Aojek4QhbhY4R5ei6rRS0KEQeryGxy8bg== RfURqE7Curj1yaFB45mzv2ThBgTKN5FE6y5BWgBnF6+szdMMXRw4uQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy recipient: age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZUpVT1p4TGZyNC9qMm5G YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlQkVPTHBYWkdMM3Jyd3NW
UzNHV08rWURjY2lqS3FTWFdoc0FYYTFjb0NBCk1mZ2JzaXk5RmExNE9xWGZ5K0pv bkNGa3VtYXNBanZPc1puSGdwUEN5ejIwRkFJCjlNNXFlTEV1eEZNVTFxWnNJZG5O
UEtqMkltV0dIWll5eVBUVVRNOUNDWUUKLS0tIGpKNVJudUM1UGNvaGl1UDBOeFA1 UG5hci9XVmMyeXF1ZHF2Rm5DUVdPcFEKLS0tIDBEZUtZNXR2R093OVNNYzRnbXlj
RjUyRlZ6a0Y4SXNsL21zSURVRk9KTFEKU1L6BQ6ZlYQQtqx3uF/uM5CQ1ercmvRT S3R2UzFoTmhZT3Y0K1d4STFjU2RRamsKU9LcaOLLjmcarmdir9Hnt/qaNvlxvSsE
TL3r2/Y07gE7CjRn3pR9z0co8KndGzxV6YR+ubyWptwBS8KQh5stkw== RdXIdOKuaQqJyJ1VEpuDCfuZgtIdkr7OG1360giXFUIEUDliM9OPiA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt recipient: age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsaWxxWS9xQXViY3VnUEx2 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBWUFJETUk2TGZTdVJqUTcz
V09FbTI0WGtNbW0yclhOSGZDbG5NUTNaTkFFCitkcjJ3OE9BSnN4bjFWcE9nYVBk MUZvUWhmWGR4SGVxOW8xcy9Pd1A2ZnhPcXpzCnhvL1lPTmZJclJ3OHVqbGtGZFMw
ZzMyVHlJQ2wwdU5JOXdCQm9oNkhNd2MKLS0tIE9tRzFYS05vSkUwWFRkaTdtc0k0 VysxdnY5ZzgwUEFiVUxpYStUb3NsTEUKLS0tIHRpS0dpb1ZNMXBiRUhFVVZiSHQ2
blVoMWV0QklBVkluT0Z4NHYyS1F0blUKO+Uc0of/V77ZUZOsxTzeH8/LmmAOQt+J UTRoTXFjNVFOeE84WXdsT25UQUprbE0K5U8S5xojEgUn8pAgY6X+Njllv/jqm/Qp
x/COHxnLCnZ4eWI6q1a0Qn5Br15OJYTxUI2QTV4goTnXBNUDo9wdpQ== tqeTbMw+w/afAhRxY80x/mTJdCAxUh4guLTKO6eojHYcFvwT6eZA4w==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs recipient: age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsLzFnZkdVdGQ2dHRwcnRt YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpY1FUZ3RKU080RVNPZFZ6
Ui8rc1NoQzNKSHVzTk0vYU1vMlRxcjBKZ2hNCmhLYXVGSis4RU9HNGVxZkpvUkd5 cDExYmhLdlU4K2xWU213MkZtUWY4elVxdlF3Ci9LK0cxV2hKbXZEMEd5dzVCWGVU
L0xvalhDYTQ4N21OcHRheTlkaUxvTkEKLS0tIFVkRGxtLzhQT0paV2U3ZnNScVdn SjErTmpYbDhmcDlSSENhNnRrN1QrcTgKLS0tIFRlbVlKOW51R3ZiZ2pLd0JXdks0
alZnaVppeGI3OUVscGpONkk3YTRXd3MK61na8x5qX7+dyMHasDz2dj7yeaUlX8me b0FsZ3VETmREVEhqTGI4Vm5KQnZaWjAKXq+8u2Qk84Vt+eDUxzE6sDk4DDm78P7H
N4/SIk1JDBhv9G7mdKLbKhSF1UJrSY7TJqJqx8/dqEc0uG3vptA1ew== KVnrZfhAmwP3X7dSuBhW+dK8in8D3jaqRK50d/eHUUWX0NIqniUMKg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx recipient: age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA4ZEtvOUhMU1FRWWpJQjF4 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZU2R4VVVkWTdqS1dxbDhv
NWphNXp5M0dLZXhkZndhT1Y3L09maytHazJVCm1MeEtMWXg1Zjg3bFVnZEorci9J K21KeWs2NVo4UTBvNEF6SDRYdE9WUldYUmpRClloR1lBRU9UYkpRSzYxY0NsR2c0
bkNZQU9Ta1dDTFFHaGFWQVBpK3pYRDQKLS0tIEhPVGliRDR3ZTF2aEl3ZnJEYWtR WHlXT0MycWJPdHhHaThWUHltdGZYQXcKLS0tIGJuaTEwSnpmMnFGVml4UUs1ZCtT
anh0SEpnVW8xdXNkZEZQSjcxU1BHMFEKVRJUA71fi1QawB2TnuTWMYhzQR18u4M2 U1V3U2NNN0l4L2JPRHFDOWpWWE9HaVkK/i5m6YFiAR6xtms/pbcDNhKaZreqIpjT
s1V4j4TwYyyKZFoNvt8kOUayjC499c5OBUufYs6G2ciC6gK2A9E0EQ== 8tvnqHz2HDSuCMAjAnZfluvuP1USHvjJQGZpBfreZ/XGhW0oqa7D7w==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e recipient: age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTV2IzSjBEMjUwZjB0dCtj YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKNklENW8yZ3VlMytlVHJs
aTdZVlEvYkVRbTNjUC9ZRVdNZVhPQTJEd1ZZClc3NDJiR1BVYkZkdVVxMVZGc0VN dUhDcDI0amdZWlhmN3UzbFJyR2x4RGVOdFNVCmovVzFlYnh6YWxYTnYxVzBNejVT
dWdSSXBFR2xxR0xrV2thRmUwSS96S0UKLS0tIHRRTXVlUi9UYnFRRlhsU21HZVY4 MGFhdjZzU2hnMW42d1RucENGeFdXa28KLS0tIGhIZy8rZ0pFM3o0cDBKTXQzaFNO
SDFYd0NwVEtVZXNsWUI1a1ZZU2xNRGMKuQUhOq2FRD+PGn5OkdODZItbxCzRKjne bG9XNzFGdVNyczRhWjRqQXlxNHFvL1UK8NMj76782tmIdJJ4qIzLicFytNhj6ZMk
E60UOYtHjanuGjJ1svuR9cYsLZz7lLOwItklecYaQYpMRZEwzzBGCQ== HIIOJGCrBnqgCtcKTiCrTGRhGbqGzhkId1oJkZkhMFRoU7kdSvwD8Q==
-----END AGE ENCRYPTED FILE-----
recipient: age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrSjJoUVUyd1JqRm1ZQzZx
SFJ2cWNUY0E4b3FndVA5Y0hIOEZnZUlVWlhBCm1vY1luOXZBelRUTmF5Y1NMeDBn
cE1BTDErc041UjJCWTBQbnk0Wk80dkEKLS0tIDVSZzd1UktvZGdyanFUMkVORUtl
eVB5TnJkMlp6dUpXSTlxRlplZ2NxUlEK0AYOxIbswjM0SUASDfmZ7PqcEU844fgI
ycFWVSEPodwUZ6UFoYXhHlJzHFcgpLvwUd1PMktLHe1qrZ7GOQJIMA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBPakhQcE92ZU0zYmk2QS9D YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAzbFU5YWREdFpsK3NScXNU
TG0xc1JPZXZCZ0tZOXA3MGNLVnBlZGVtRFRzCmIvMHhQKzFVWCtpMTQrQUhGVGJp NUFMMml2VEVOQU5pWkIrUjkzM2tZOG9vcTNRCmR6Vzc5OWRQaGdvU1Naam51dXhD
RU1jbldYckw3TXI2SlNpZVBIZHRsWWcKLS0tIGtJTUtJejFxem5jajFQUDFTQWU1 ME1yNFJObktNNTA2eXFsNExTL3JNeHMKLS0tIE13L0xYMCt5TE5ac3FEbkxVWmVv
VnlxYmVlNG04ay9ETi9FRmVYQXVoRkUK9oFNolI7jRjo9RUs1g4ghrx7aYV4U/ce V3NCSk9LUCtPZFZCbVNmSjR5QkFKTmMK5qFJXtZCKLjOCg1r+sVQpMKl75GNcrbI
ZTc2tFh57+7aKgrDi+2W3jwhfkjvBsThk//p5mLlqEEgw2lwlnhvPA== Dum/K/3HU03wv5reG51UDsQ1tMrsFDsaFh2fjR+LxLKSGlG7b3Rz4w==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th recipient: age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoVnh6dFAwTkY4cHJpaEs4 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEWFdCNGpPYU5pSXkzd3lG
QnZPeXZHK0tXYWZPNytmYXVsdGRWQVI3RlhRClFVb2I5OVZzZFNrRXFaa0JTUkRJ TGU1WkhuUEhmaDdybzFUbVVaVmVqU0FyQXhBCnRmRngrbXlsTVVMSDV6dkhBS1lD
OC9GQ1V5K0JhWlhkUjU1WStCa1lPV1kKLS0tIEhzdnBBZkRnK0NtV1FuTkVsNlgv MmNUd0NjTE5ybFEweFhkWXdGbDNwTmcKLS0tIGU5Sk5hM3Y2bUVsS3pTaDZiaTBi
QzVEcEVkQm5NL0Z5dUU1U0ZFaTJITnMKaWE9vlrOpQstr6FGP5ObdilsCYk4kYAj S21vU1ZVbjZ1SHE0WStab1VHQUNLRUUKBDW9hwI90Yn+B2mB7LUNTVxFGbwEFSw3
/phboR+Ym7QDTyUF9LZXJCU54YJp6vEWkRnlJFqC75UW/v/lgBhBMQ== LGGqgZdvnN5p6NMigBbJz+kSwOk1gQ/yo86HtRj8Ejllp7P7jRsfmw==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k recipient: age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkUHAzWk1KblJxVVZZV3FM YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaMWdPVkV4WnNHNmU1dTZZ
WndzRGRtbkw2azRVWnBuZmhKWElqRFk0RENzCk5DMHlMVWpwbXEwVUhkaFZUbkp2 ekVoVTJWY3laQ1ROM1NLUnlZY2FKTnJwNkFFCm8wVWtoNFpxMS91RkxzQWRhOWRs
QXNlZFV4SjBEdmR6UEw0N1JOUnhNKzAKLS0tIHo5RkNDUk1ESWRHQmV6bzkvSTlP V3k2STdvN1JRdjIrSHVFaFpSYmlmT00KLS0tIExnSmw2anNtdzVGMjRYdzdpcFRq
dk1GQ0Y3V0dTRlByb2xUOERVOTVwbVEKY4sAHyAhvGSYJzPuufWUIQD2xZcSt/nX alFrUTJVckpTVEVIcUxTK3pPbWNpUzAKSe3Z9V5u8+om2s+HqUcx6qXIaTmQUgOT
t2ZFXu891/QdEzyUXCIzdwAV+Y/LjvroIlCp5Hkbrk0s7N+ghqsB1A== zIrU8T0tLOHzdS1SAdh1Yb50yevN+P2+5TORdLjrtigpyzeYc2U/zQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7 recipient: age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBYVkwYXpYSjdmM1FpbDl3 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtWDI1Z2hVRG9ieHkyZnlP
QWpycXoyL1AyOEpZUmtpbjl3MFAwTkJoOWpVCmVDd0FBUWxaQmZCU2VmNkZGMk9o SVQ3VHNsSlZxMjdIMUsxQWdIclNQQk5OUWhzCnZSMzFZUGkvYlNSRTJiT1ZMU0h3
TUdLNGtac2N4REg2eVF1eVh0WnNaTE0KLS0tIDJvcFErSjRiWmhPMmpadjROOHdt L0RIWkt4R1F5bUlWNVlQQklKc0tGa3cKLS0tIEJpTXNzZkFUOWp1ZnFocjJEVExa
NXp6Y1JpdHFlSlRoa3JTaEt3emdnalUKjoFfZAiKMPF3noX+K0+vc3+p/XUHnhic dGlqRDRLd1BDYzYwSWZXaG1ueXhrSzAKdJ8yA+igZKyetNuQdoN2Woi2bl2I3Xj7
k888KdUwcZYl2/dAIc8UDSggbMnncJAJgoezoCHLkj97GNNAD7E+gQ== dDBVAn8bvxx5ZG1eyDXXcHSh76xUp5ZPRlLOFCvRK9oOLWd8FntBmw==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt recipient: age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqYnQyYzF5cktZNG5PcThy YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkb3RrRDNqTGZGYlR1eTls
SEszVEwrUkQ1VVRsM3pSTlRQaHVLN0VuSHhrCmN1Z3pwNlFsbDN2UGI0KzYyallM Y2FYQWxncmJBQ1ZiNWFpMTBMUmtlSnJzemhVCk5NTWZRcTJMWkIrTnVsbVAzcndB
SHJ5eklQeEIxSlhiYW5PUlpJcG5KNjQKLS0tIEk0QkpMdlBlRjVYMmJaMzJUbDNm MVp2NHJSazJjMVN5UXJHOWpnc0dLK1EKLS0tIGJlelUvbEZZZkFKd3BpdWx4VVNH
K25pZldwd3JoZi9vdURoa3Myb2RQNG8K6N6bO2YKooPfpKihgsYqilfz/yAYCLZD cHJSVmJ4ZWlVTlA3VXpWeE5DNkphcWsKTGpdWU/cE8vC/43lwmnwJDh4IPqHQoVV
XJ/THgT4URX2VNvSspvBtN8luOiJUVcchp5WtL2m9jARL5txEcDorA== yjnUdZnPDGyQHwwrydVgun2fdaAKH1zxAut8TZlH9pxD4ir0B69G+g==
-----END AGE ENCRYPTED FILE-----
recipient: age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvY0VpeW1nazVvRWJFaGNU
RGVzVjRmWWV4dnJzRHdsaW9ENVZYck5lWGtVClNtd3ppelowRjZpRFFSMC9EK09n
b2hqNnkzejdrTnhYNGNKblpteDNLRWcKLS0tIE02bWVjazRWNEVKbURITGlQODlR
cTJJVnBVdGIrdzBoSXExelNrVk1XcEUKc77o2EX7PCm/HjUo5GsUiQdm488WB2mg
wHd/qDbQhF1W75RrVTuIKgtEtrRjZqpmr8toe+aHJizPofcrToUfzw==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2 recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSMmZnd1pXRWh5NDVSN2xq YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBaZ0tVT09Sd1hXVHZPd0tD
MVhTN3N1OFpkcUh0a0tjYzJnWUlINGNVd3k0CmZkTnBac0l1dllxazdLY2l0Rzli YmZKaStnUXg3cERMTjRmOVh2RmxFbG1FZ0hJCjNMWWJmaUFqTURoL2NvRnJIdzNW
bE9sNTBVSkJNaWF1T3c0WktoOHl0NU0KLS0tIHEyeWZTUjdQeUN6U2t0d3JwNTBX OWFmSFAxWVRrdit6S2lpaFdERGNrOWcKLS0tIEJlVVFBSE5PK3k0dTRTRlZYMGZt
ZE1Za0tXb0gwc1FSakVYdU9OTHkyd28KkwmlzSYP8XofB0VGag+S18+S2TyQjLrM elcxVlltUnJVd256NHQ4TithUmFnSkkKkbFSRRktuO+tNxlfIEuJ27C7LWalHJMb
qaXtbBtLzJGNDhe9FhAKTPFcjTLWbohlG69vxcImyCyCns+QQ+gvug== aCnjE9301XMceGB8SstPtGi5SJRSHD4WrJleeJbKXYMronfvWnYDSg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2 recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2Q2RMWmZVOEwrOC9VcUxp YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEa3E2ek9BcHZRRUR1cUpK
VVE4R2NqYlZWZlhKSTBHRks4bDNoaUliMWlJCm5FbWdZS05GY0VLc0sxY2x1U21V UEFERzc1ZEtrYjc4bEdEeDRnSDNkWHV6elhFCjljdnJLa2syR0UzYTRveGhPYVk1
eExwK29GVVBqYlRPZ0l5RWVXRFhRNlEKLS0tIDl6dVZJQndwVStFVEJnRHRyMW1W cGNnWkNMNUVwbXFrOUZ1aE5UelFyMGsKLS0tIFNyQ29ieUpReW96OGhJaE4yU2Ni
NnFqc1F0SGJqT0xmREpaN21EdnlJK3MKRPE5rfFpVnH5wAOkuB5pNMlMd3omcpku Y3VWNUFYTU5LdHlmd0krK0NyQk9mMVEKfh7I/9+V9+0DLkzTf6n4sBKs+oZlMlTx
do2hFZwyI7t80jxF4+g3J7EolOx8AGjpc9Ba7Gj6IMDjye728q5N+g== ar203b95cR/jjsUekF0NoZjj6MW1IZSV7BkaDoizVwrAxh7WS0vwqQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c recipient: age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhSG94VE96TVlTNkEyclFE YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtSUcxeE5pM3NZVHlhcGNt
SmdCSkJpeWlVWDFIMDhwUEkxL1RUTDJ5UG5FCjhKRDB6VGtwTVozdUVzbUxGL3BW d1EwODM3Wm5BOGNybkNMdnM0SnJmTG5ScEhNCk5yZHhieVN4QnNPMkxzbUZ3SS9G
SnR3cmpSN2RxNnl4QmNvT2lkYmtoVFkKLS0tIHd2V2h2Wk5xOXlISzhjVzBsVkhz aG5WVENIbVBYRGI5ZzQ3QndUTG12NjgKLS0tIFF6ZFp5YWJ6UzlqMWRsb3pTS3l0
VVpRenVnSVpHUWJqV0JHNXNWWXJOdW8Kv7PJSTDbwFOAcl7pynALaJiTXU/87bSF ODRZZG9pRkNxL1Mxa1dwUG9vdkJZZTgKYrjkSkjNQCU2wMIuZMvJnd/regVplzMi
F3HQllYOwOoibGzBCe18H2N+VxyNxoQL9OWe0TvOIR6bgHFIIF0/Dg== 3kPGRBMWzO1t4CSFtf2PMqv5AHw754I+vVNs5CpTXBuxVUcQDcGJCA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn recipient: age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsMEZPUExRVHFFb3pSVHNO YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGNHY0Wm9oNk1BWFBiYTRm
cGJFN1ZzTDFVNEdneVpMZ253ekFJNjVtYkNnCnRGQjU2Q3dsRGRFV25LQ3pCbTJE OXBxNU9ETDNyTEhyMnBxZFRnZmFjVGVHN0RjCkdPNWpvRkVMTCtWdDhlR3Q0U3Nw
OTROaFBiT01xb200S1pUK0NYaTQ3R2sKLS0tIEdiQlZTbi9Vcm0zc2t6bHplZktF QUh6KzBVMU1ONXh2TTYrd0FaWlhCU28KLS0tIEhsaGxvcGFtaVF5T1NIc1NUOFBt
ekRySENXcjBuR2psdHZSSUJrR0xUdjgKvBsmnC+cbq5TUDFjXCyImIoPKvh8wsjE d2NGeW9YNVVxdUpYbFRoVnhUM2VTUGsK4adI9pgC2PipcAY4zXMRf9hPv5kilTvc
7Shk7Act8Jayrhx0lXBDRmfpHRrB4L16rDSmqO0DTE48VhT3TiFyug== BsUNz0Qr4YdRfVfrPlPBzMCPOTDofTp6qBv+Gc8FE4hvwxtbjIcJ3A==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68 recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxNDlya3RmYzNhU2p6RkVw YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBT25YN3BtMVZrVnhDazhO
cHJVWWY0Yi93cW1uZi9FWkVONmtpRWh5SzFzCmtWYTRIY3BkTWU5R2Jsa0ZJK3kz OXpzOEd0Wk5KUnZ4a2JjRGI3U0RsMExXN1FZClM3UFoyeUIzY3FVNnlkTU1NSCtn
SnZvZ3YwaGtoMVZ3V2laTlBBK3UyTmMKLS0tIFdpZWtqeGlacjlLbmFySHlSUUlj dXhNb3c4ZFlMTU5VTFc3blBUV0c5MkkKLS0tIHUzUzliL1NEUSt3cWx2d0tjRHcv
RmRqQWVHK0FUT3VDbFhLbXQ5WDhLeEEKcDkgV34lUFJRIHRoLB8F2IOvGAM93sM+ a1Nmbm1MZEptQjJuZ2ROWGUzN01zbDQKy3dPKE5oYwsTwE2vnhUi3auqJ/KBOBPX
AkmaM4+WRcGeYWQKMG2x6cYCUKFaT1lDXuWZ9kI8Fd7b9gTSnQMs6w== vSKOROUuS/uEfsdo8NoVxuZ2RaJHetrbHAYPHFHOMgpe/X72pCcQNQ==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz recipient: age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
- enc: | - enc: |
-----BEGIN AGE ENCRYPTED FILE----- -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjeFM4ajhHeVd4bzlqZmJ1 YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2Q24xSkJaVDVUZlBWcW11
MUxYMUpZVjdjR0NGTzVteVB6WFBQeFI0ZkZrCk55TEt0Zjdwbk51RnhYclNzam1H cWFOdjhXZEtuNkJtaUhQTEN3QkVUQnRRRjBZCnY4V1lCOTBNM3Mrc09naWNTQndn
cExKZXQxWFVLa1pDNFpkcGZzcnl6a0kKLS0tIFJ4ZEdJc3JVaEc5RU1aZk1uYm1l VXU2ZWt5Zi9CbjBFa3YrejFHM041VFEKLS0tIHd5L0hRN00wdmdsSUhkdFpTcUNS
d3RHS3hHSkRKRXFnN21FQmh0TlNtNmcKdc2G/1dhTJen6iT9kUWZM5OzCmDVprgx YUNiUGhVR2M2clFrOUkxNW5idkR2cnMKXO02RJh4ew22tf5GWH4lNdLlhf4bWyef
WN1Bl3JzYhLsNKn794887bVAICVqbXqkdpEZztNIS5n/Rw6geKsNvQ== +hW9R4TGCJOnIO1xSGpBX0wJsM3oiW4qy/1tgfQw2ejVih5qUr7JRg==
-----END AGE ENCRYPTED FILE-----
recipient: age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYa0gxNnBwNjNNTTgxKzgv
YkQwTjZxb25tQ053Ny9tSW4wNGVYYnlIaFZNCkVvOFNPOFkwQzhYZGxjb0FOZzJ5
ZjNXam1ZTWZrS1M4cGhhcHZaT1NjblkKLS0tIEdvL1dDTHpWcWF0S3ZqNkxrQW52
VlYwa29sZVloOS9qajJWQWFzY2FKRmsKy074SLdttogXsWycaFX8xso4ek7Cbjph
MMEhZd/svmnSiYM81nmeaze7qXEUcsZXuSmZCYATTBEGtx/Srll8aA==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0 recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
lastmodified: "2026-07-23T21:15:41Z" - enc: |
mac: ENC[AES256_GCM,data:qFhnPra6IE3wyKQ4WKweON0S0YtD5I0adGZVfA0m6BVilN6bX5oC/1j5NK2oHrsz920hSl0SOF8LrpqOrUyGjSRkPsN4kq8qr9bJcrX4URiktP0oRden5LLt6hf+ZRP7WmRXFqixPkPHJnZIoAvkNnTFce7cDq5NEAHkKUEKG7k=,iv:nyblUDGeu3TUfFivYylOn3C/HITj99qiPI2+mh8AGh4=,tag:FrtRzSCylC4wlIoqZdfx7w==,type:str] -----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1SnhhMEpwOS9vUWVuT0Iz
bWFCUzE4djFyUzN0alRvOG94bGJldUhlb1ZnCkxHd1pUU3B4b0ZhRklnczUyRzBR
djF1bS9scjBlQXJEbmtiUFgzTndKQVEKLS0tIDRpUnhWYTM2U3lLcWZyVFkwcVNL
NFpldXQraURraWZNdUNqZFNwVFY0WlEKhV8IZYbBXKGb0x+2E5pJkoMavH2ox4qp
dK/XRlBpEG0SCUkKZ5sDegzz/HyqbxRF26jC3IOL1uR8A5nsnwC2Cw==
-----END AGE ENCRYPTED FILE-----
recipient: age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEWmIvVG1GWEZmbnpFU252
bFhOajA2dnJXNlFGMmF0TDRuR1YrYmZqc0hnCk85WGFUMGZXdElWU2tvdFV3eE1v
czEyTHBkZXFnSi9MVWFQVHJabEpUb1UKLS0tIC9TNGZiQ0xYR1dQWHl6NTAxbzlD
YmNhQkFVZTZidWFtbURqYnY2eGMxZ2sKaUPhe5mQ4QSyxuQMqLNI1jqfkCUBnBWc
HR5ck3H80/H0pMwzP5uaBnN/hD5L4CT2FcZBgUygbyNz4G3VgbjaMA==
-----END AGE ENCRYPTED FILE-----
recipient: age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlU3BybGcyckNaT2dQR0ZO
dDVXa2NtNGFTS1RqZ0ViLzVJSk5PbWdYU3lVCkRZNTJ4MytYYVpON0JBQTh4SWRY
ay9xdkViektHNGRuSFNSeVoyMFdrNE0KLS0tIGJ1QW5RUjhWR2NlclR2dnh4SUQ5
SlJEWHhBTFI5Rm1rZVFjTGZlUWhZMGcKReNE3l+U35iutlQ07AZ+3fOFF4YdVbdx
bR/Sz3NqpqmZqBEmgjUjjjQI4h4Xturv4tT8/JUzEmVnEyDsqCbhHQ==
-----END AGE ENCRYPTED FILE-----
recipient: age16y0vfts5v2k20e2rj23qa5lc5gm96gm64uwsqsr0y688xl0ne46qcfr0sm
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTMjhleFpqRGRBMGIrVmU0
aXcrclh1TmtIV1JZSlNWRXRRc2RCYThyUVMwCmZOdXptWnhvajYxeERlOEJFVTU4
WTY0R2NZSXplMDNNSzZ6eCtEeTN1NjgKLS0tIFIvYmY3VHM4T1RTZzljR2RnV0ND
Z0ZhY2JpL0psb0RJcEJNN3FHUHJVQlEKJdoXKdvm6fGA+C21lsxY4Zq+VVPt/6k4
vH0IirsPd9Cg33+tBzrySwkF+GVNtYLtekup2L61pFQ6/Y/necfQ2A==
-----END AGE ENCRYPTED FILE-----
recipient: age1tm3zj5lp3elw2j832az4nj9xxmhqd66ag3cqcv3vskvmd7qdmqmsdpdcn0
lastmodified: "2026-07-30T07:49:07Z"
mac: ENC[AES256_GCM,data:D5Y+zyLvs3gz8BWS3+lrra/tc1TsDIk7FykxSPiz0Er1jUB5LOe4IKircytMpwGqdLcVhUKgXYdRA+C0KAXaI9KpQp5qKfRDvX2jX24jveVNFPgbvxA2LidYasVMzA7ayTU0I+oirPcEtj+5VH/ahSjDz9YtarTDeHMpXvBZMLU=,iv:KcIGEs+61yS8Ul734fqsrC95iA82wIinf72DE2bLk/A=,tag:RzL48FqVcMro2W6VAgIxzQ==,type:str]
unencrypted_suffix: _unencrypted unencrypted_suffix: _unencrypted
version: 3.13.2 version: 3.13.3
+22 -1
View File
@@ -1 +1,22 @@
STUB: run cluster-init.sh to generate, then: sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey {
"data": "ENC[AES256_GCM,data:JwjmYg0qzoer+8/jv7KOfK/BxmpObHOn04GNsUFQd6hSFBWbQlRbODfluL54hnWQ4gsS9MHgwWh8nrLciBhye05kRQCqADB7Crm0CZYubNb65WzM/devh14jmerc3MYIp1M3VmDisML3x5IqhJkFMSqKM/VaNAC3f0otcVR8TTgZw/fJyvPlfqpPTjikGmo5xg++Yk7Cy8A1Dj70kYk10+EQjQ78jf4k/agLoaS+YvMmKXFk4EljtHg8Fe5H6Rn9nfMSSKGTZeAH3Riix2e+fIH+nWEZiPHC0UlPjJrB3SeBqqGRSUUJshVKHFIdKI1Tq2Ea+GDWrAyIRI0qO8e7WQ==,iv:mr39NxbZuYUDHDw0e+fJDshbt9R1hLgVqkUxUVX/+l0=,tag:u31xZRPfvH5j3Sw5U8/nqg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtVW5ENXlkUW83elc3ZnhE\nOEZ4MWIvQWZ4aXhuUmZGYXZNQnhJemsvOWlVCjIwNmFjS3pJOWEwRnhBOGNRamlM\ncXZid1RNYVBXOFZpMlpnU3ZnVE1scnMKLS0tIGFhWFZFd2JPTFNrV1VHSDVKVHRK\nbmhoeFh6YlUySU1ZRVhHbjZnM1IxMFkKo7aHkz2pEeV64m+OEkBZ2V1e+PUzoChu\nUc/Mnh37dNXSSJtg2KnocHdyzDGi1yQbA72xKTxx6QjYJWrAC/jurg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBReWRLb09reXUxS0YyMUVV\nQlR2bEdYY08rK0RoR2pIelpLVEtYL0pqNUVNCklRaDNoSjBWT20xbk1UdmNqWHNt\nR3JoSlVwODVMUVBzMERUNktHMVViZm8KLS0tIHc0SXRIS1IreDF1VCthb0FBR0Na\ncGFRWTROUHF3clFtTmY1azEwNUpPNzQKsgCND/BZcMTgBTAcnHunQcT6LG1jNrtO\n+W7Yx7bFtFBajWnRYiNpUZPibQJlv5SE9os47WDH1gs86xftgV+uyA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGbllNTlhxNVVkd01GSUNi\nR0lxQWZoRXI5NWxoTExRUnBSb2FPR1BONnpzCkxCN3pNOXVrTW5GR2J0WFhZUE9r\ncm1QTThINDVVVlZEalJiS1RXcHJRS28KLS0tIHV6aGh5QjB0M1VPWDVMT1k3cTBh\nNTNGY25NV0tWSUl3UHlJSExFQVdpVDQK3ARj8xhFRYU6oqYxNQ5+Ryza86ALNUwX\n7yK/8ATnquYC8/ZIYUbgTPxzsocmXX9lVT0+ktIALqjtG8PgNTlOow==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r"
}
],
"lastmodified": "2026-07-28T16:04:51Z",
"mac": "ENC[AES256_GCM,data:9X0dkAEGJiug16LDy/8//QOkHITTzt4zwTKiIU13tRgSRXRpt/7b27+3eDauscXDMamg5wbYo1YF0+VwLl21Ip4icnnwrZBHd6HWleP30HTgef8rmo21SstshnclZz6RH8SEGVDO/vjrMDChaZ3A2WD+nbcomw47DpK3cG3kBIk=,iv:G7CScSD5Z1Z9WMEMydyeGK/RD4W43xA0PlcpmTCxLOc=,tag:DZmby/xkxpinoJztZZRWwA==,type:str]",
"version": "3.13.3"
}
}
+18
View File
@@ -0,0 +1,18 @@
{
"data": "ENC[AES256_GCM,data:81JCCVaOeEYNyqTT3vXkFDDV1oSAlOrElGmvN+1Jy+U+dF6EaCSeFYT0U2i2BvUu9VAYEiY6NRAXwWUgZXeQXsI65eziB6d/8NKTr7UWb0eQxvhHbthgzxoCfdCvofZH49DdGuCEl8kU6hppSeM+wnDIIYvEgWljO4JQJr4/qnQRegMqJ33564+ZvhiYqrbfojcBYpTjVBzh+4aitqZEAYoFx4xhFAtdBfmAA1W6FwDvZLrX09bbdFT5jOqnSSkPunvTHiyWtyAfLgaNzG656F1U+3eRDv4eZQYOaLXXo9H4BaBMC5jyJFV1rDgl5s3diCuQggSyDbWUnwSS3Prcixj3dkrF8h8heNk/LDV5lGBtGEV3BQsEAlCPl01PIe3O5McfsRaR8IqwD7wQQ8ULyA56fHHAYJR1xe9bNGgdLryhqLDBFWi06Ztidi1MQXFLnNFJxoIhvANIAmt//TBWimkHnXidkg==,iv:WNChOcSG+QiZOMUi8jpb2gmOCLonM3zK7vcZCrXt4Yg=,tag:zq6Zvf3xu4kofaCechUpcg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxMzVHK0tOazNoazU0RWZF\nT1h2N01NNktHRjVMQ0pEOWR4RDIxNVlQTWlZCkV2a2pQS0o0VFZWN1lyaklLWllp\nZXJvZEt6Zng1Q09xQTFkNnVtTDNNVDQKLS0tIFhnNW1sY09ZcWpVcGdmM1hrK0o3\nZ3VscjBXMVFEa0xtbVlZM0lMN1dHS2sKcYtCKhw6D1ax3Isf5Vk93cDteUEjx79j\n1fheqgOjytY3W9t8U2NOEpUP1hT7zWYswnaq+agm8nfEjf1fbwR9MQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA4K3VSL1NrSm9rMm5WNTk1\nVEpudHhJUkJUeWxieEtKSUdCL2c1YkNHWVdNCk9EeGNoWHdLOVNUR2dRY2YrMVlJ\nblliN08yQW1xdXN5VTRjbm9vZUxZaDgKLS0tICtLaldjemJHSkZzY2R1MHlySEsy\ncXJ6UGdZMi9hTnNjRHRTVW5UMlNMQkkKWhLTte9gTpppGCmA0lf/FEYu5b/ZaGYT\njn9j9qj3Lets0gNj9qBVaHUkCoJ2TAGORdkhnLEdY/Tm/J1w3XfjNQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age16y0vfts5v2k20e2rj23qa5lc5gm96gm64uwsqsr0y688xl0ne46qcfr0sm"
}
],
"lastmodified": "2026-07-30T09:34:38Z",
"mac": "ENC[AES256_GCM,data:oC5OPZyZ4kEh0A8Mmwoi5oZ3+fBNvEDN3A+o3n099RdPREn6mcH5QC+XD9BXvWVxp9BzQEwRBmExlyZBEoRuwrB6DQy5Fn8TneEjQoCvZrudpbXIB66/EvORk1utpUsbAov3wn9A8tXUDAUJaRsY93/Pe1mTc8KzIOGv/QTMR9w=,iv:+p87aL7dKsbyAfCWxc0pecVB36yUUa+ltfzQZc4oovk=,tag:tNRRC056bqMToaHz1UrqQQ==,type:str]",
"version": "3.13.3"
}
}
+18
View File
@@ -0,0 +1,18 @@
{
"data": "ENC[AES256_GCM,data:6Is5g/gqFdQ9aTE6dpe80g7kSgDbhRr3yIwffv6tT5CLqsQLA6i+V5XPAqqhH0zbpXnT7k5Rc2tA2880MDa0eVoI+x7qDnhadfLKVgErWZNwQ4Nz9ukovf+lDfoKXn9AWNFxbiBckpCoxNOms3zMv6kqf1gbEwV7eRIoiq/i2xypke4nOJSmRH22dqFUldfQOOUL1D0H6RFQpNK+5O/Okb7URxIzdUgmq6/9zIsZTOlX5j1V7kzpx8U4P+YPyoPoRSLTOxpK/5gh+Gx1BQbXVTf8z5lm42tJxuQAt5kXqu0Tu8aXZzUUcMKf1oGVbo4TwRaQN+9tFgkhegVMSS/l2don3X3vWPimBcYHTfXfvvCh3krH3hoSmej8es2jCG8AesObDEmCyMNu9hdqhGGdCWspakrXVEuefCJ2Tq8fJfOn5XQhh4AX1wAf04Us0FJO4NjByNrgW8Bo1344wocAStxBxuD3rw==,iv:kkk5muigMC6iTIiTHKwXFreCuJGl7CWmwp1W7ILmq7U=,tag:W49PPiOISdFjk54wroD8ZQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQN0NWRllWaEtPdTdlOHFz\nUXJBbG1IRXlPbnNzMEJhT0FuazJLYVZHWUdFCnJXdUtvV2RvUU05eDBaaVdxSlBw\nUVh1azM0T2dFZlpMMFhsU2tWbm4vSVUKLS0tIGNlUXMrbUlDYWpiRTQzV2svcXJH\nd1RIcmpGbkYvSlZwYkwxanBZajVTMUEKt4HCEEPjsSDKvp6XuSSrQjVXFQLQHwk0\n1Fi/HrkhhdkIO1f4DyTOOznvK5bc+Z4JKT5lrOkKQln92uAho2ECiw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBock0vOGVLMC9HZ0NYckxr\nZWRwK0NhbkVaV1FVVndDNmRoNWhGVXlUOEdjCk50NFVtdmdxbEFuZE92YU1IUDlN\nbVJ6QnN1TEk5c1VmWDQwNXJReFdCSXMKLS0tIEl6TW1iV1NvTk5OeUkyOTRtT0ZZ\nR3NsV1lwa00rdHdTRUkwbzdrc1VaVkkK2ioVnzacNrQD6cpNOomKz9WfRRq+B3oK\nnabokq5rEOoNsSFNip5TBeDjo/34kOTXZCKXpFYwmBSoHCrkdFHvYg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1tm3zj5lp3elw2j832az4nj9xxmhqd66ag3cqcv3vskvmd7qdmqmsdpdcn0"
}
],
"lastmodified": "2026-07-30T09:34:40Z",
"mac": "ENC[AES256_GCM,data:QOvDbiPiQBPYBGg3BzJNZzLBv//UccDfOnoOVqnpdskOpPw88uQOcpK0mcF/unW8o+B0AJe4wzr8/XhlhtlLSRi6buES2uZr8pjgCqrgFMCUX8WP+4rxiSq9DkZtmLO1XBtBtjJy37PgFr8bwUw1p8fNdhnOEJBYR75MDC8JG38=,iv:YCCfUyncprAkIbvxWwMGGZZywNHLTrr5bYjBBdQp2OI=,tag:mHyHw11ok4jm79iYn6L74Q==,type:str]",
"version": "3.13.3"
}
}
+19
View File
@@ -0,0 +1,19 @@
{
"data": "ENC[AES256_GCM,data:VpxnzrdX5FY2gfDcZpqg1f2wqL6mAawcrdE5A9A3fQOKr4BWXmCT/hjQEtt8qG+yVOTApdG+vF8SKDj1AKZ25KsZ253WS7W5C4j8oiT/xHo0CtvVani4fZqN60lB/9381h7aUZ+lcBMPaQcNn35zoQqYthuCGrC9PH1qQgD2VVHr0v8J6kceXKp0KTJldtDsQxky6ROC9Zyc44wwkrFPgPss4/yGaRWWqiojDvK5fK8238hmR8HyfD6Tf5KhgbqCsSGS2g6yt1muUhmegaZwtpi/KpoqBmf1XFaObuHGRPKvblq4Fa0x/sSjpPogBZyNuvRKIwRhla0FkKOH1bX4gC9bzOlgiUUfVcRJT6Nr9jocYuhIv9wvIT4TujGbgHyorYaDSQLFPJoJEUa3fgpGq+AC6p1DJJE3b/wqUSngNbdOtAPE/imaEuprOfnbS7cRx+Ti/Yu+ZYZuAlDtLkSVdXi1hQ4fLQ==,iv:rq/BzePXa/w/Gqewz8JfM/NdU+x4ShiH1OuQT7wRyCQ=,tag:IcO3ddoj4M57CkndHEmNkQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRdWFEUlRJMlZId2Yvc29F\nNnkwREkwQkllY25vZnRjV3Nob3BvNE1Zd21NCm56UVg5YUFGbEc2ak5MakxDbCs2\nTFVsRFhjVlRwMXdUbGpMKzQ1N2hBUUEKLS0tIEhmdm9hZEJqUEhRN0dsSk80bDY2\nY1JpajNOVFpEejhibjdKZlNHVjE1bGcKycyDfEBZ1WYc4EfAK2y5x/nKRqq7mnb4\nDlpR1Som4bSt7+B+OCZa48mC0Zv05HbP8PnoZdCe83swGNljvRONcQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0SGFVbXYycTE3amR3ZVhi\nVFJHVVhWNWVQQlJKZUJvZ2p4ditqdnYrWnd3ClZaaUVLUXFKaWlDdDQ4QW5SQUZY\nbG05c0RGNnZUenI1QTBlemVjT0szRncKLS0tIHBWTmNpd2FjWHc3MmR3cWt4SCtH\nWS8xZnVON1lGZVc4YzZFRmRBamdSVlUKdlQTlwiExTUVsiY05MXFG2/IQt1bZwKU\nlFDnuy6YCQzawBLQZuSUAp7WSGedupexlZKQKNUzPf+dis26XEYZoA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6"
}
],
"lastmodified": "2026-07-28T11:53:06Z",
"mac": "ENC[AES256_GCM,data:PMsPSffQkxRmqT6HwFezIoT4WKJ9WDlAcflWynsFLtzbqVTREN0YGOJgMwLd0yBR2sbGBWogG5xAuKjO16aiWUVpCFPRvP97um351679ZoXIlrOL+GlFn4NKwoLH9/eCp8BteB9mv5huhEEN631lqCr+jtYO6yS0ULk6QS4+7/c=,iv:oYWV4J1JUuGryvqntVXkj/HOzRDRr9E2C1RKNvr2bT4=,tag:e4p+I40y1LrgdlK9tLRicA==,type:str]",
"unencrypted_suffix": "_unencrypted",
"version": "3.13.3"
}
}
-6
View File
@@ -1,6 +0,0 @@
# STUB — not yet encrypted with sops.
# Bootstrap:
# bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
# sops updatekeys secrets/common.yaml (allows ha-server-1 to decrypt shared secrets)
# sops secrets/ha-server-1.yaml (create with: beszel-token)
beszel-token: REPLACE
+19
View File
@@ -0,0 +1,19 @@
{
"data": "ENC[AES256_GCM,data:Ea5AGwloFMyRgmQhJaDNTW19mpDvx0+dSc3ibENEXEniEIBV4Wv135mAwE6jQPbN8wF6f5ki3B5mDXHOMuhDQEqUXg6jcs4uEm2nwz7wAKYXEX4T8p0gUj7FPISiSQrw8O3jwzt4qF7AVcPsNhMtZLgrx9ssMtVmGct5do+E61Y7dFe0XbpJLynfzeiMKYvaBngTbHwQjv1mn9AMpX98apsp3tyzg8avLDR+WXRcMusds2mdov0jCoKuWPLG/srhMwjKOp1Z1Tu+aZCr9lHTT76mytpGIWGSbrP5l3UTp3oPLmy87Kwi/FJStZYdYikMnvmSo5qovePiYsLKVD74dS+d3oFGARyW4TILd9OQdXEjzdDRak5HcPv2/HhR638kvukVMdEdH0hpeJcqcV0h+BoLyIHe1e037iF1ZkCjuXE+A8LtlzRR0wqP202uzBIyg21ca4X5bRiHkkAOX4P3riaEcMgKfw==,iv:IXq4sHHsjnK6maVd3RRVZzhnxx1hOnxfsvYX8IKLn+4=,tag:jRZinAVLH8nGVf3gE4aSKA==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSByOTkrVVd0TXlBRzdkTktO\nalU3WlVOb2ltc1k5RG5sRE9EN2M2WUlnQVRnClFGZFYvSHdpVlhtNXExSXZMZnU3\ncnl6aHRzU3k2V3R3dU9RVDA3VWp3VkkKLS0tIHcwREM1dDREK0h0UFp0bU5JTW9Q\ncWdyanNMVjY0SnZUWUl3Y2R2SXNyc1UKI1FPIE66to19oQK2TwM5B4snGOVMLFfx\nl6JMEilDVSgy2RR/tSiovmo4NwOsv0hmmF1t34547rbZb+dB5KrMEQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnbWp2UWphSnFibnA4N0ZV\nRHlNUnBNYUNzMzNZakFVc3JXT1d0R3Yyakg0CkowMFlOcEhPcHE5dXd4ZUd3cDAx\nbjQ2T1NqejNRMnFvbTdNRGFGdW1pM2sKLS0tIDhGb1JBQ0RFQ3NxbmpkdXg2WVF6\nS25uZW1hMVo4Q1JGdGNQWWRyWE84b0EK0FmFZB6sN7uZynhzYU932x461zSZIUWU\n8oyDEKNoUuPqpWK8RktgjWlKHtB0oLXC+SLAOJEhtnhWF+GSqloGfw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r"
}
],
"lastmodified": "2026-07-28T11:53:32Z",
"mac": "ENC[AES256_GCM,data:EhWEEIamG62xHI6SqO4tzdK6gcEXxUU/UGcF25m+X11kUVXzHd6yLcfiM2DvqcwGLmwmOOf3SpFLABlQUzTka4QRzTMFIAJi7x1rzQV4J0YIGITHk9rgvy0V50TI0loONBb2du+vDt8IlTBNvF9WMbNCki+fHPAjIemyoRBdubo=,iv:wfsx0H4fD3L7WeGzz57SzZtajBr/xTRDljSpz22DD7Y=,tag:naDyGH0WTKkqgd954Ya8jA==,type:str]",
"unencrypted_suffix": "_unencrypted",
"version": "3.13.3"
}
}
-6
View File
@@ -1,6 +0,0 @@
# STUB — not yet encrypted with sops.
# Bootstrap:
# bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
# sops updatekeys secrets/common.yaml (allows ha-server-2 to decrypt shared secrets)
# sops secrets/ha-server-2.yaml (create with: beszel-token)
beszel-token: REPLACE
+3 -4
View File
@@ -1,4 +1,3 @@
beszel-token: ENC[AES256_GCM,data:meuzUP/6wCssJDVTgbC0XwiLZPMGyDl55HEIiON9xOXCD9k6,iv:TDqWcp+8Mxd8wN09r5otQRQXq3XTeQphaTWxvvuLTAs=,tag:cRPZQGlwB/dTguBAheWPQg==,type:str]
cache-priv-key: ENC[AES256_GCM,data:6vQKIf7eS0WNL2Eptoi4VWr18SRMZfN/H/aFUUtXdMYQY5LLyBp2EHRKqZcGFuh1nZhUdAxUztq/CVXx+QFxKW+ElHxCxUSp0QqI1fdSkBkKZb8hlit5SoX9JtLzZGg0HBNM3nJu,iv:0J+xmrPJhInHhFR/c41ACjuTfaIoMkQFSfbL2KkgFa8=,tag:f4s9Szs5oprVVRSyXaX48A==,type:str] cache-priv-key: ENC[AES256_GCM,data:6vQKIf7eS0WNL2Eptoi4VWr18SRMZfN/H/aFUUtXdMYQY5LLyBp2EHRKqZcGFuh1nZhUdAxUztq/CVXx+QFxKW+ElHxCxUSp0QqI1fdSkBkKZb8hlit5SoX9JtLzZGg0HBNM3nJu,iv:0J+xmrPJhInHhFR/c41ACjuTfaIoMkQFSfbL2KkgFa8=,tag:f4s9Szs5oprVVRSyXaX48A==,type:str]
sops: sops:
age: age:
@@ -38,7 +37,7 @@ sops:
IqA477+noQ/Rjrszb2hEvxID7keogcDUWMWQzQvdMc22+3mvAr9qIg== IqA477+noQ/Rjrszb2hEvxID7keogcDUWMWQzQvdMc22+3mvAr9qIg==
-----END AGE ENCRYPTED FILE----- -----END AGE ENCRYPTED FILE-----
recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68 recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
lastmodified: "2026-07-19T23:30:21Z" lastmodified: "2026-07-29T01:59:11Z"
mac: ENC[AES256_GCM,data:kLGE2xawQT7mx+sfw68hmGk5nCEGiEjZrqTEl9B1dtQmTrMwmoVr/1RISi4LfJrwxy31mDgff4lcIL4wIJuM373uk3X8j4RNyYQNTfKEkORT6r8NHeepNs267O77pKGd7OmcM4MT/BqOnB8ELS7Wlf2ect7CAlvUUVyc8icxgZE=,iv:EYLDsHYHZ1XOQXafOTqHHWpk/OBNq/R6IJnOBYV33E4=,tag:thxrCPC5oGvDjhK7Dz87YA==,type:str] mac: ENC[AES256_GCM,data:/nbcfause6G6F8IvMoyPZtkWS1XRLAivhwTFu6y5P0Mm0eCcO6M7/rgioN9dngKzPXrCUl3Dx/EvhrrWKe2/Saq9WEOFgvS2V05pTRbQgYjuugVzW2paPq1fgmoDYNjHz2yFYWAovbIFtjVxMR9tmcASMTe6r/FocvMXbFCJh7Y=,iv:n3KoaPPtDhWK8mxJNJRk7WPVUUNR59nZSA8JyysWNDc=,tag:tYhkNXsj2ohVjHkvxife2w==,type:str]
unencrypted_suffix: _unencrypted unencrypted_suffix: _unencrypted
version: 3.13.1 version: 3.13.3
-26
View File
@@ -1,26 +0,0 @@
{
"data": "ENC[AES256_GCM,data:aqlMnoVkGtH9z3fJRweeC0OYf7LGqJU2sWA9Q25dKK6NuNyJd4BvjPtpfeg/WhVtJsaOtcbwVm4WAhVK9FARE8g8j+vmq0f6BAU4s6mx0ZIhl+mP+/hIpt//LOdd+9YezelJxdpzUyZbdAngU99rsTluLRe2XmZ7Fquxd8yH/OHenSDY6dizp9+5jfEi8EU+EmuXvuWMPY59xnlnqYNPfSFxs43/pS402LzJoJ5H+cBPprddkUBVzy4cBQvMnrRFUSjnqp74ovZkfIWFqDWQ5YgSU2PjatBg18oulZ7wNRhQ6OLqj6gsu+xrMjNFwnp7rMlA3X//hIidxTkVcYITycXd8KzuMIaofUpnwoyT34fy6+H35/39iiEyG4LRTrOOKRDzXkY2rhJUxFSZ8GlhNhMd0RlkmLngVYrtjsswJ9meIwoAFLPYt7BC61PJf0TXdtk=,iv:mCA819J9LpAOj8QxFAkrHI9wFJIy8qVxv31D6IwWFnk=,tag:qqMtM0eyHbQEyl6ND/wf7g==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOZVpRYmNMQ1E4ZlFXMDVq\nbW1FcnYzWTBHQmdhSjlsZUtZK3dzNUZjY1FvCkV4VXU4MTcwNVZoZUpVVS9JeHE5\nbGk3UzE3YmpZQ1JISHV0RHJqSG9ZNXcKLS0tIHhBdnIvMjJkeWlVRE9Fb0FhWjNX\nUjBVWDltK2w0akkwOTJaTGNSWWNrRXMKqZRNnHiXvn1QBoSGdABp7vOqNlsEN6Xr\nDp3NByXow6PuRuWvQXHzd+WC+ADkwNaaiT6TUrbZcd/Pl8Ges9kcZg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2VFBxc2ZMUFp0YkJhVFhW\nc0ZNbnJUcGtkT1VLR2YrVGhRRFVTZFlyZXlJCmVnblNBNGxIdHh4Q0IweWR1b0dq\nRWFyOURuWkdkRE1RTnJRMEpXdk9HaUUKLS0tIFhKV3FoWW1zRk1pMlhuWWlhV2E5\nQzJTRHAyc0JtSjd2NHlJODZVbndaVDgKFA4565X/4FqNq/fZDZTg81/55hZi4c7b\nTti2AnyE3OcY/kurXJFHRinVMqURQf1fx9MxqUYRitiCz4qe5zFF+A==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwRWlKTFNocnBRN01pZ0hm\na3pyRUkxMWM3dE1iTkZZcGRYUFcyb1RDb2h3Ckt4blpGZHBjcnZ4SVE5UnF4cmY4\nWW0rZG5wUkZINVk5a2lmSzN2L3JJY1kKLS0tIExpWGlBRFZJVGpGbFhxRE5ZYjBo\nQVliTlFIZ1U5dE9xbDhHMUtxenpBSm8KY6sIFEfK8p+70IXsC4Jwb9Lm/pd9+V6K\n4JAzGrpA6mAuIwwSNnbdcA5j8FmBhCpK6nLBWmFhGm9Y+MRTaM7Jrw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoV0pIUXlyOFFTUHVTdllu\neG10VENOYWl5Wk0ycnpLUytTbnlrQ25pMkFRCnhyRU5xWW9vQXJJOHdFMGZqMkR1\nSktqT3lOdVVPRUN4YTlveGp0NXpqd0kKLS0tIGRaTktra0ZtSVpzSHBrY3VSSUph\nRVRZOSsxTTNmMmltMlJnVy9oT2VEWU0Krxf49B1BsrWn05fqg+cZ0k0PtfJJNfn0\nUL44RUWXWbK2igQHaIct9DfYe7DEonBJeROuxDYm8g7yNOv15S+P4Q==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp"
}
],
"lastmodified": "2026-07-28T01:44:26Z",
"mac": "ENC[AES256_GCM,data:eyOSn03dzHSgkshPzLVwc95382eEFaDQarHs9l83dtcsb1Ui9CjkKipl2DVSUb6bdMUH1qKYmXqJhwFnAZbFZFjT4VTKtutNtM+OkhVXfT+fJs+1+u7k+ZUYFL9HuxKA6AWpwX3eJ8vmJDJZaAayJbm4PRzOyJywvKeneQqdUS0=,iv:QRKltR4qzVofo/Elt2Us/lrHlD7BenX605X31x+Ng78=,tag:SZ97CcfgMJeu3yqNk8Y/cA==,type:str]",
"version": "3.13.2"
}
}
-43
View File
@@ -1,43 +0,0 @@
beszel-token: ENC[AES256_GCM,data:cbQOXhLzNk4g9d6hvm2DH7Q5ApTPCTzsW2txflDT2dD/UPIE,iv:V19MI1GEo5/0205Hrt7JImfkjduFiZ7f9aIkDVaI8mU=,tag:WCArgdrnIOudVe/Tw+oxRw==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZeDQyNFh3dm1YZ2FTMjdK
cDBWcDE2a1c1S0s1enhWRnVuMlVJc0gyTXk0CmxWNkRJMDhkeGpRTGltaitnVkZS
a3Q5TW4zYm5Ja2FETEhJcGF0N2ZKbmcKLS0tIDN0a1FqRGNOY3Y1UWxvUU8zWU1m
bE9DVzZESG1HTEhVWUdJOTF0bDhRVGcKP6OoyDAGLB9jQ69jpFyho5eaeK9XtZgN
RlSJpBm2Jo19h/crpH9AWXUAIG0BWueyr8mwBu12cQdFIU3IyZT6gg==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwRVFQcHpkQkNlSDhhM1Zp
bUZyOFduQXg1cDd6ZlNObFgzL2hmUmJhM1JFCnprMldPQXJNVW10dVRqQTdWcGlv
RnBYWWFsaVNrMkJpS0pkOGlQQzlJVVUKLS0tIFNsMEEwZTREZ1lwWFJGdE5YSVVU
ZEZ1bVpFMEQ5N0g0L2RacUpLMWQrVDQKxPzq6f960purgAmUJw6IZnZSnhkzNE8r
CSrFDowKTZI2KRdCtQ5fGhEoWO0ZPgVNxYV0KH7JBttylcpRLm6r5w==
-----END AGE ENCRYPTED FILE-----
recipient: age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0czVoT3ZyYTZhOEQ3RWtR
bVl1MW5vNERxUEpmNXV0MGVBRE9ySnBjT3pzCkM2aUVpZjg0SkNVTnRRMlhyMTN6
NlFrZDVKV09Yc0tuKzFzR0ZtQ2t6WkkKLS0tIG1mbUNFdHBycS9UOGc2cjNpeHVm
NUd1NThRQlZXeG1WbmR5Y3pTYXRKc3MKwSnE+0bGmxOAQUje6jHxuzIIyD6ZAwVz
b5AAYwbGRagKj6fimsHBUmi4ohyG1huIGGOU8HiUYpu4PGJgOscztg==
-----END AGE ENCRYPTED FILE-----
recipient: age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlVGRDMWNDZUR6c1VpUjRK
WDY3L2lNcWFFcm1UV3RPMjlqYnBGVEJLcFFzCjFxck4wdlp0Wmtzc1RKNS82MXpK
ZHBzOGhkc3ZuZUE2UmpUSTgycWdLSGMKLS0tIFUyYjczeUFWU2FyMlBTdzAxMTBE
VzhaVzlSL05nZzNmR0ZjNEFPTXYycHcKfiJ0KjdxtLWsXxsWKzAL+H3hYYjHrYO9
BjKknq1ZQJM0sB/Tid+GLqDwKi966MQK+AwHF5MqbsHW7eE5bO1nwg==
-----END AGE ENCRYPTED FILE-----
recipient: age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
lastmodified: "2026-07-19T02:30:40Z"
mac: ENC[AES256_GCM,data:rKHZjU/MH08ASTlu32HZO9uWmsBYuMCEC6M8gwVhzuWvmablnP05tS2z13XfaWaCEUXk6kmGJKuU0zu5+IKVZgamCF6DAMtxQb6bVCaLsoAm/GSqWQ5VI9eHqgnSSdN/o3ul/33Rf8iBQo4aw8FFAmDVuNz8bfAn0QefFTj0ByI=,iv:JD2gtqRinOY77etg6PUmZNovkYl1Q3F6ZvRi4x7RznQ=,tag:/5IMpWKRVt+l1luCTQE0BA==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.1
-43
View File
@@ -1,43 +0,0 @@
beszel-token: ENC[AES256_GCM,data:qg2eb7Eq/w7vIuiiDeK/s6h9nmHWuooyrd93GvdLD2hGOvka,iv:FWRBQTIY9PzV2V8J7z5ftEb5dvjN/S0hP0H9JV+Uuq8=,tag:WP7DHOu4elfqdLoW7vo9qg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZmozK3VjcENGb3l2MDNj
QVE3VVI3Z3R0MjduSlJGL3BFWkg5NHZmZms0CkNBTWZKZFJVNkJMZ0tpNXhXWW5q
WndvbmhuOVh5ZHhJMmJwenh0M2dEVlUKLS0tIGpXQ3dhRXVWUDZzdGg4RHN4V3p0
WDZnK1lMUHVIS09hRUEzVHRHQ0EwcWsKEd5NtstHSPH8okXcFLW7BId5xPGmgyC3
dAodIGkLBlJlaL1bZuRy6Vaac+mJZL5KJYD/o1kYSXRiHJfNpaulTg==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXWWhYOFU3VHZ3S1VuRTF2
QlZvMkJUNDZmT1RDWWZsMkxnZjJldjdrRG44CkU0NEJTR0NhQUpjMEY4RXBLSVBJ
bDNtZ1UrRDNUbzA0NDJ1cWJKdlBBMWMKLS0tIE9vTmEzTUxpODF4SG1MQ1czM1N5
dVgrMERMRXJwVmhtMldTZUlHNUJVTEkKzUf9cETbo0K9bCPkrlbUkgkhRIPvjCHR
GFVwgb2fcaOzZHcDwlLDLaP6HbRPtzJfVbLpQ1XU0N/xijSr9ENB6g==
-----END AGE ENCRYPTED FILE-----
recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlOVdWN0tHNWZmaWlWQ2hJ
MVFYd0pMNmJTdTlJQ3h1c0dzbDE3U2VqdzNzCitCcFdhbjRPYnl3Q214WnRxa2Jq
eTFnc2VBcXlGcTVYdWpWTFZiWXBMUDAKLS0tIC9YdVkrTU5EeXNlNzliLzBlYUR2
Qk1ITW9tMElzUHF4MVJyWS9uRGs1QncKj8OgxWYabf8MccWD6zYLxl+FvYPAvLCW
p/FJ8uMa9YWp0RjICvWIwyO0gK7oaxTh/JzfBjABngEZnZazNYP7Uw==
-----END AGE ENCRYPTED FILE-----
recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsbzFKckFQVDg0VEpJMFlv
WSs1ODVoYlZZK3YrSXlMU1hIQjdLTUhNdHhJCkRnbHY5Z2NGQVMvOHV4R2hTd0xu
LzdheXdZVVBsTjhVOU8zZGFKN0lPWjQKLS0tIFdaaG5SeElDN3YvMjFxdHVLWlNX
M1BQQmtJTUdsK2x5TEo5TmNrMEZidkEKhi4jNKhrkBAPhJoeYNg1d60LQTqmJt+G
Z/2X5uWzh12zF5keL8cNHH6sD0z0MOChtd4UF37yr+lb/LFJkia6LQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
lastmodified: "2026-07-26T01:12:11Z"
mac: ENC[AES256_GCM,data:0r68y6XmdiW7p/h9QmTR5h37TxKp+0wF44LP8QfG6cGP0tvaasuvv+V9EC3msnREYjwJhlnHjRd8hJ31HGVkOHbYWI4V5cF4ZnaiGgrZJhsGcJvOJSroAvQwev+vIlXZpAPgZz3Z35FDvOpgb71l5c0so0l4EJhc70JOh2FxhvQ=,iv:Or6l/v/E01+9wtZwyl+pCttiuReR6V0kmBzE9KhsfgQ=,tag:cAEUgj/9tKDToKFr00g5UQ==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
-25
View File
@@ -1,25 +0,0 @@
beszel-token: ENC[AES256_GCM,data:gjbT3uROiVKQOJaUeafTxjVknQO1Tvbyx/Pl2bTad7DezByX,iv:3ikf7OaT2omO8yd6G6UwYbaRBSzyvbn+NghxAe5bcgI=,tag:Zuc2EP8rUtdDhr5CzSW2Pw==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJL25EUUZack1FMzlnMmdk
Z1hnejZRNnMwVHpmWkFNdGcyeHVsNnRsSGtZCk9OdDhhcnR1WW9ZMEZ1OUVYbm1n
RmZRVy8wb1J3emJBK3Rrd1d4U1dYUDAKLS0tIGJaaElvSk1sOTBOM0lKck16OUtu
NnRZb3U0ZndmaHBZTm8zczhWdE1oaEUKkf6fLomAHoKPhuM4e9q96YmmH+h4VrEj
2x0rnwBwOoRzYWutB2MVtlsphAZmZ/PK0tEecT2MM0XXayVG/33qdg==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpeEJvaHl4akc1TFdEVGV2
czg3OVprU3p4ejNpTktXZEpneDgrOEhrZGlZCmJwT1dhSkZneHE5UmR0WTd5UENq
VHJEWG1EekJLY2pRZldtVGtxTHlGaGMKLS0tIEYrWHE0WTgyUlIwdktmNzNIS3FW
ZVRvT1dHa1Vzc2RSakVISzdMTlpnVGsKeT+edn4+LUkVtpRUNd/gKX3H1HG2bvNo
c8iI6qr/l6oxfP85OrKYFDU9IGvDMxSSdbixHtojPEb5OKVurV0WPQ==
-----END AGE ENCRYPTED FILE-----
recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
lastmodified: "2026-07-23T23:32:57Z"
mac: ENC[AES256_GCM,data:l9a/yNRoxY1hvSkLuR4N7deeKue/1JPlSvZvJfCSNbQ21p1qR433BbSDYvfW+kXQXS8GVcfgXSd9ywNzgVvkA5lR1++uYsZBLbYxJ+s3TKWs6/yECAZ0eM1KBA0BEm7cLSsHTOwd+2WspvmCYir++FDO9XRuS3guiMnQBglDf/E=,iv:eE6GVqexQNSiLYfmTTUdUx5AO//wyjSIsr96xAX1pcI=,tag:CNsoyDZYLUt5Seu7W5wJrw==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
+251 -153
View File
@@ -1,194 +1,294 @@
{ rec {
# Network / domains # ── Gitea / flake remote ──────────────────────────────────────────────────
lanDomain = "gitea.lan.ddnsgeek.com"; # Gitea/DDNS domain
homeDomain = "sweet.home"; # base LAN domain for service subdomains (pve., docker.) # External Gitea/DDNS domain — used only for the remote flake URL in
tailnetDomain = "tail13f623.ts.net"; # Tailscale MagicDNS suffix # Switch-nix / Test-nix aliases (modules/common/configuration.nix).
lanCidr = "192.168.2.0/24"; # LAN subnet giteaDomain = "gitea.lan.ddnsgeek.com";
lanGateway = "192.168.2.254"; # LAN default gateway (router)
lanPrefixLength = 24; # LAN subnet prefix length (/24 = 255.255.255.0) # Org/repo path within Gitea, combined with giteaDomain to form the
lxcLanInterface = "eth0"; # LAN NIC name in LXC containers (set by Proxmox --net0 name=eth0) # git+https:// URL used by Switch-nix / Test-nix.
vmLanInterface = "ens18"; # LAN NIC name in Proxmox VMs (virtio, first NIC) giteaRepoPath = "beatzaplenty/infrastructure";
vmStorageInterface = "ens19"; # storage NIC name in HA server VMs (virtio, second NIC on vmbr1)
# internal repo path to flake
giteaRepoFlakePath = "nixos";
# ── Network ───────────────────────────────────────────────────────────────
# Base LAN domain for service subdomains (pve., docker., nix-cache., …)
homeDomain = "sweet.home";
# Tailscale MagicDNS suffix for this tailnet
tailnetDomain = "tail13f623.ts.net";
lanCidr = "192.168.2.0/24";
lanGateway = "192.168.2.254";
lanPrefixLength = 24;
# NIC names inside guests — determined by the hypervisor/platform, not the OS.
lxcLanInterface = "eth0"; # LAN NIC in LXC containers (Proxmox --net0 name=eth0)
lxcStorageInterface = "eth1"; # storage-client NIC in LXC containers (vmbr2, --net1)
vmLanInterface = "ens18"; # LAN NIC in Proxmox VMs (virtio, first NIC)
vmStorageInterface = "ens19"; # cluster-internal NIC in HA VMs (vmbr1 — DRBD + Corosync)
vmStorageClientInterface = "ens20"; # storage-client NIC in HA VMs (vmbr2 — iSCSI/NFS VIP)
# ── Host IPs ──────────────────────────────────────────────────────────────
pxeServerIp = "192.168.2.223"; # pxe-boot LXC container LAN IP pxeServerIp = "192.168.2.223"; # pxe-boot LXC container LAN IP
nixCacheIp = "192.168.2.224"; # nix-cache LXC container LAN IP nixCacheIp = "192.168.2.224"; # nix-cache LXC container LAN IP
tailscaleRouterIp = "192.168.2.222"; # tailscale-router LXC container LAN IP tailscaleRouterIp = "192.168.2.222"; # tailscale-router LXC container LAN IP
torRelayIp = "192.168.2.221"; # tor-relay LXC container LAN IP torRelayIp = "192.168.2.221"; # tor-relay LXC container LAN IP
serverIp = "192.168.2.226"; # server (NFS/ZFS) Proxmox VM LAN IP
dockerIp = "192.168.2.225"; # docker Proxmox VM LAN IP dockerIp = "192.168.2.225"; # docker Proxmox VM LAN IP
pbsIp = "192.168.2.244"; # Proxmox Backup Server LAN IP (not NixOS-managed) pbsIp = "192.168.2.244"; # Proxmox Backup Server LAN IP (not NixOS-managed)
domainControllerIp = "192.168.2.253"; # FreeIPA domain controller — authoritative DNS for sweet.home (not NixOS-managed) domainControllerIp = "192.168.2.253"; # FreeIPA — authoritative DNS for sweet.home (not NixOS-managed)
ipaServer = "domain-controller.sweet.home"; # FreeIPA server hostname (used by security.ipa and Kerberos; must be a resolvable FQDN, not an IP)
# FreeIPA server FQDN used by security.ipa and Kerberos. Must be a
# resolvable name (not an IP); resolves to domainControllerIp.
ipaServer = "domain-controller.${homeDomain}";
# ── Cross-host references ─────────────────────────────────────────────────
# Cross-host references (LAN hostnames/users other hosts reach over the network)
nixCacheHost = "nix-cache"; # substituter/remote-builder hostname nixCacheHost = "nix-cache"; # substituter/remote-builder hostname
nfsServerHost = "server"; # NFS export source hostname
dockerHost = "docker"; # docker-compose stack host dockerHost = "docker"; # docker-compose stack host
# Raspberry Pi's own Tailscale hostname (not fronted by `server` — it # Raspberry Pi's own Tailscale hostname (not fronted by any server — it
# exports its own NFS share directly). Resolved as # exports its own NFS share directly). Resolved as
# "${raspberryPiHost}.${tailnetDomain}" in modules/raspi/mount-data.nix. # "${raspberryPiHost}.${tailnetDomain}" in modules/raspi/mount-data.nix.
raspberryPiHost = "raspberrypi"; raspberryPiHost = "raspberrypi";
remoteBuilderUser = "nixremote"; # remote builder SSH user remoteBuilderUser = "nixremote";
# nix-cache's own SSH host public key (not a secret — the private half # Tailscale's internal "Quad100" DNS resolver, reachable from any Tailscale
# never leaves the host). Wired into every client's # node via tailscale0. Used by modules/tailscale/ts-dns-forwarder.nix to
# programs.ssh.knownHosts by modules/nix-cache/remote-builder-client.nix # forward *.tailnetDomain queries on behalf of FreeIPA's conditional
# so distributed builds don't hit "Host key verification failed" on a # forwarder zone.
# fresh client that has never manually ssh'd to nix-cache before. Update tailscaleResolverIp = "100.100.100.100";
# this if nix-cache's host key is ever rotated or the host is rebuilt
# from scratch. # ── SSH keys ──────────────────────────────────────────────────────────────
# nix-cache's SSH host public key (not a secret — private half never leaves
# the host). Wired into every client's programs.ssh.knownHosts by
# modules/nix-cache/remote-builder-client.nix so distributed builds don't
# hit "Host key verification failed" on a fresh client. Update if nix-cache
# is ever rebuilt with a new host key.
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICuHUxGNH6ei3BZD+EfZs3l4X8uJNcjQiOsM/G4yo4O/ lxc-nix-cache"; nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICuHUxGNH6ei3BZD+EfZs3l4X8uJNcjQiOsM/G4yo4O/ lxc-nix-cache";
# Beszel hub's SSH public key — used by every agent to authenticate the
# hub's incoming connection. Update if the docker host is ever rebuilt and
# the hub generates a new keypair.
beszelHubKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
# Public keys authorized to SSH in as remoteBuilderUser on the nix-cache # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
# host (modules/nix-cache/server.nix) — one per client host that's allowed # host (modules/nix-cache/server.nix) — one per client host allowed to use
# to use it as a distributed builder. # it as a distributed builder.
remoteBuilderAuthorizedKeys = [ remoteBuilderAuthorizedKeys = [
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFDEA1S2ikpObREgbP5uVBWMxIOGbY8B+Wx7VTZK1m6t root@server" "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+ioWPhHixlgCB9KIQ0QTHTz6A+Oo2F3uKiINLip5rO root@docker"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPAYIT9ormlmxZ0SziyDQaUntnKI8HK9/s3Qac1ZKjP2 root@docker"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKKKzoEPl/ZW9KBRHBcp6/ThOngGpwMv5EhkTlgC4aDf root@nixos"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGtOWOCS+ImHc7NehguoyD7PbonGosKMZqc9+QR3v/h root@nixos"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHxXTQxFnArK5HXG7czeoybZebCGfxpUdusJkPn+BCSp root@server"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface" "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEj26SL/emsVjW2YhRucJVp2kTz8WgcEQgjBEBLRikk root@claude" "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII/rLceRhnDobVXQYiPceuhDHHvVjFQ1pc9A6un/eUlA root@server"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHj11bLPpRzH2oslnwFzEvY9cSgfEFtSZbLQaDm4nZMK root@pxe-boot"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH4/Sesm8NYpj73R0cbGhI0Ubvz73vIVWAnbEDTlBTdh root@tor-relay"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIBCFUUtePndW7pqtlawft1QCdHmBVs3O/c8EJO+RcXV root@tailscale-router"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBZ9WKKAlP9Z7GQdgaZ1Xgw9C+vja2lqEZO5rJFpVqYN root@ha-server-1"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEGNKlaaMckd8nLWNGz4B2QokXjnnIvM+rEUv+R6h0sp root@ha-server-2"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+XeMco7OxUpjrjZm54HogMs9QB5xlcKmElASRvrmlW root@nixos"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMseQwPpmaa6cgV5U8KhUsiVSYARG85zGa9rho0LJWks wayne@pve1"
]; ];
# Admin SSH public key, authorized on the primary user of every host and # Primary admin SSH public key, authorized on the primary user of every
# the installer image's nixos/root users. # host and the installer image's nixos/root users.
adminSshKey = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCq/Q5LvIXlZwO2kdeAN5nLGZ59nZB7JHYMEszHxmNtGMzv1lM31jiPNsr0z2EKVZhE7OOfa2IF9rhWYD7JUA9G0yzdZ4WTXFNGVVOJoOVH6vAF3XCxoVilOEwTc7h2Wiy+rzd0B28/3spffzQQWJhY6GRQVa8j+6xAGF60Fcvl1vLosYT9Bn2ZbK4TCWOwAn2jqXIieGpZdn/UNZbGOeKRiCvhktDfMAzuQzN/9jMu/oF4pkPn2X1UrsQdNlvp0Ci8md612MozIpncQJyAF1ADhunr3sMx0isUXiqD29R5DS4TftpekqLNLak+zcxFa8N7DcRNp3DcKfJvyTkwQrR4r+b7lFLYOLHLagSso9CzeW/paAS2q9I5SBm/2DtE1diLLg2jZikYcstsu/G5RgvbzbKqjiaMwTdXC3AMvDxQrs7U5pDRZFzoofG3cpODbTm+uy3m0kP70z0M1K45UbDG0p+itnTu9x40JbQEgefbx38AItNvAIx1A8HO4I1VX28= wayne@stream"; adminSshKey = "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCq/Q5LvIXlZwO2kdeAN5nLGZ59nZB7JHYMEszHxmNtGMzv1lM31jiPNsr0z2EKVZhE7OOfa2IF9rhWYD7JUA9G0yzdZ4WTXFNGVVOJoOVH6vAF3XCxoVilOEwTc7h2Wiy+rzd0B28/3spffzQQWJhY6GRQVa8j+6xAGF60Fcvl1vLosYT9Bn2ZbK4TCWOwAn2jqXIieGpZdn/UNZbGOeKRiCvhktDfMAzuQzN/9jMu/oF4pkPn2X1UrsQdNlvp0Ci8md612MozIpncQJyAF1ADhunr3sMx0isUXiqD29R5DS4TftpekqLNLak+zcxFa8N7DcRNp3DcKfJvyTkwQrR4r+b7lFLYOLHLagSso9CzeW/paAS2q9I5SBm/2DtE1diLLg2jZikYcstsu/G5RgvbzbKqjiaMwTdXC3AMvDxQrs7U5pDRZFzoofG3cpODbTm+uy3m0kP70z0M1K45UbDG0p+itnTu9x40JbQEgefbx38AItNvAIx1A8HO4I1VX28= wayne@stream";
# Additional SSH keys granted access alongside adminSshKey on every host
# (modules/common/configuration.nix) and on HA cluster root
# (modules/ha/cluster-config.nix). Single definition here prevents the
# two modules from drifting out of sync.
extraAdminSshKeys = [
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGygkCljN6uKpdJbHTOQtn8ZnH+wKXDLAwrDFbLrE/65 nixos@nixos"
];
# ── Wifi ──────────────────────────────────────────────────────────────────
# Prestaged wifi SSID for the gui host's NetworkManager profile # Prestaged wifi SSID for the gui host's NetworkManager profile
# (modules/networking/wifi.nix). The password is not here -- it's # (modules/networking/wifi.nix). Password is sops-encrypted in
# sops-encrypted in secrets/gui.yaml (wifi-password) instead, since this # secrets/gui.yaml (wifi-password) — not stored here.
# file isn't a secret store.
wifiSsid = "nbn-fttp-net-5G"; wifiSsid = "nbn-fttp-net-5G";
# Bare-metal gui host's two disks for a ZFS RAID0 (striped) root pool # ── Bare-metal GUI host ───────────────────────────────────────────────────
# (modules/disko/baremetal.nix). Only used transiently at disko-format
# time (partitioning); the resulting fileSystems/zpool import reference # Two disks for the ZFS RAID0 (striped) root pool on baremetal-gui
# by-partlabel/by-id paths afterward regardless, same as # (modules/disko/baremetal.nix). Only referenced at disko-format time;
# modules/disko/proxmox.nix's own plain "/dev/sda". # afterward the pool imports by-partlabel/by-id paths regardless.
guiRootDisk1 = "/dev/sda"; guiRootDisk1 = "/dev/sda";
guiRootDisk2 = "/dev/sdb"; guiRootDisk2 = "/dev/sdb";
# System # ── System / users ────────────────────────────────────────────────────────
timeZone = "Australia/Brisbane"; timeZone = "Australia/Brisbane";
# Main interactive user on every host. Every module that grants this user # Main interactive user on every host. Modules that grant this user a
# a group, a home directory, or tmpfiles ownership should reference # group, home directory, or tmpfiles ownership reference this so a rename
# vars.primaryUser rather than the literal "nixos", so renaming it is a # is a one-line change here.
# one-line change.
primaryUser = "nixos"; primaryUser = "nixos";
# Primary IPA/domain user. Home Manager is configured for this user on every # Primary IPA/domain user. Home Manager is configured for this user on
# IPA-enrolled host (see modules/ipa/client.nix) to manage the environment # every IPA-enrolled host (modules/ipa/client.nix).
# that IPA itself doesn't cover: dotfiles, user packages, session variables.
ipaUser = "wayne"; ipaUser = "wayne";
# HA file server cluster # GID of the IPA "docker-access" group (GID 50010 on the IPA server). The
# LAN IPs (vmbr0 / ens18) — client-facing: iSCSI initiators, NFS, management. # local "docker" group is pinned to this GID on every Docker host so IPA
# Storage IPs (vmbr1 / ens19) — isolated internal bridge, used for DRBD # group membership alone grants socket access — no per-host
# replication and Corosync heartbeat only; never leaves pve1. # users.groups.docker.members entries needed.
# haServerVip: floating virtual IP managed by Pacemaker's IPaddr2 resource; dockerAccessGid = 50010;
# NFS and iSCSI clients connect here regardless of which node is Active.
# ── HA file-server cluster ────────────────────────────────────────────────
#
# Three network segments, all internal to pve1:
# LAN VLAN 2 / vmbr0 / 192.168.2.x — management only
# Cluster VLAN 10 / vmbr1 / 192.168.10.x — DRBD replication + Corosync ring0
# Storage-client VLAN 20 / vmbr2 / 192.168.20.x — iSCSI + NFS client access
#
# The host octet is consistent across subnets: node1 = .228, node2 = .227,
# VIP = .229 everywhere.
#
# Protocol separation (firewall-enforced on HA nodes):
# NFS — both subnets; LAN VIP for pxe-boot/LAN clients, storage VIP for docker
# iSCSI — storage-client subnet only
haServer1Host = "ha-server-1"; haServer1Host = "ha-server-1";
haServer2Host = "ha-server-2"; haServer2Host = "ha-server-2";
haServer1Ip = "192.168.2.228"; # LAN IP, node 1
haServer2Ip = "192.168.2.227"; # LAN IP, node 2 haServer1Ip = "192.168.2.228"; # LAN IP, node 1 (vmbr0 / ens18)
haServerVip = "192.168.2.229"; # floating VIP (Pacemaker IPaddr2) haServer2Ip = "192.168.2.227"; # LAN IP, node 2 (vmbr0 / ens18)
haServer1StorageIp = "192.168.4.228"; # storage-net IP, node 1 (vmbr1 / ens19)
haServer2StorageIp = "192.168.4.227"; # storage-net IP, node 2 (vmbr1 / ens19) haServer1StorageIp = "192.168.10.228"; # cluster-net IP, node 1 (vmbr1 / ens19, VLAN 10)
haStorageCidr = "192.168.4.0/29"; # storage subnet — internal to pve1 only haServer2StorageIp = "192.168.10.227"; # cluster-net IP, node 2 (vmbr1 / ens19, VLAN 10)
haStoragePrefixLength = 29; # storage subnet prefix length (/29) haStorageCidr = "192.168.10.224/29"; # cluster subnet — VLAN 10, internal to pve1
haStoragePrefixLength = 29;
haServer1ClientIp = "192.168.20.228"; # storage-client IP, node 1 (vmbr2 / ens20, VLAN 20)
haServer2ClientIp = "192.168.20.227"; # storage-client IP, node 2 (vmbr2 / ens20, VLAN 20)
haServerVip = "192.168.20.229"; # storage-client floating VIP (Pacemaker vip-storage, VLAN 20)
haServerLanVip = "192.168.2.229"; # LAN floating VIP (Pacemaker vip-lan) — NFS for LAN clients
dockerStorageIp = "192.168.20.225"; # docker CT storage-client IP (vmbr2 / eth1, VLAN 20)
haClientCidr = "192.168.20.0/24"; # storage-client subnet — VLAN 20, internal to pve1
haClientPrefixLength = 24;
haStorageRoot = "/srv/ha-data"; # XFS-over-DRBD mount point on the Active node haStorageRoot = "/srv/ha-data"; # XFS-over-DRBD mount point on the Active node
# NFS VIP FQDNs — use these in fileSystems device strings so mounts
# survive a future VIP renumber via a DNS-only update, not a NixOS rebuild.
haStorageNfsFqdn = "nfs.storage.home"; # storage-client VIP (VLAN 20) — docker + future swarm
haLanNfsFqdn = "ha-vip-lan.${homeDomain}"; # LAN VIP (VLAN 2) — pxe-boot + other LAN clients
haIscsiIqn = "iqn.2026-01.home.sweet:ha-storage"; haIscsiIqn = "iqn.2026-01.home.sweet:ha-storage";
# Storage # DRBD backing disk — identified by SCSI controller path so it resolves to
storageRoot = "/tank"; # ZFS pool root on `server` # the correct block device regardless of OS-level naming (sda vs sdb can
# differ between VMs depending on disk-add order). drive-scsi1 is always
# the data disk; drive-scsi0 is the OS disk.
haServerDrbdDisk = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1";
# NFS datasets exported from `storageRoot` on `nfsServerHost` and mounted # ── Docker Swarm cluster ──────────────────────────────────────────────────
# by client hosts. `subpath` is relative to `storageRoot` — combined with #
# it to build both the export line in modules/build-types/server.nix and # Three network segments, all internal to pve1:
# the "<nfsServerHost>:<storageRoot>/<subpath>" device string each client # LAN vmbr0 192.168.2.0/24 — management; SSH + external service traffic
# mount uses in modules/docker/mount-data.nix. `mountpoint` is the # Storage-client vmbr2 192.168.20.0/24 — NFS from HA cluster VIP (shared with HA nodes)
# absolute local path clients mount it at, referenced by that same file's # Swarm cluster vmbr3 192.168.30.0/24 — Docker Swarm gossip + VXLAN overlay
# fileSystems attribute name plus every other place that needs to know #
# where the share lives locally (modules/build-types/docker.nix's # Host octet consistent across subnets: node1 = .230, node2 = .231.
# tmpfiles rules, modules/traefik/rotate-logs.nix's log path). Renaming a # IPs from the .230.239 expansion buffer documented in docs/ip-addressing.md.
# dataset or moving where it's mounted only needs changing it here — the #
# export and every client reference follow automatically. # Docker Swarm uses --advertise-addr and --data-path-addr on the swarm NIC
# (ens20/vmbr3) so all inter-node cluster traffic stays on the isolated
# internal bridge and never crosses the LAN.
#
# When expanding to a second Proxmox node, vmbr3 (VLAN 30) and vmbr1
# (VLAN 10) share the same inter-node trunk NIC via VLAN tagging — same
# physical wire, different VLAN IDs.
haDocker1Host = "ha-docker-1";
haDocker2Host = "ha-docker-2";
haDocker1Ip = "192.168.2.230"; # LAN management NIC (ens18, vmbr0)
haDocker2Ip = "192.168.2.231";
haDocker1StorageIp = "192.168.20.230"; # storage-client NIC (ens19, vmbr2)
haDocker2StorageIp = "192.168.20.231";
haDocker1SwarmIp = "192.168.30.230"; # swarm cluster NIC (ens20, vmbr3)
haDocker2SwarmIp = "192.168.30.231";
haDockerSwarmCidr = "192.168.30.0/24";
haDockerSwarmPrefixLength = 24;
# NIC names for ha-docker VMs. ens19/ens20 occupy the same guest bus
# positions as vmStorageInterface/vmStorageClientInterface on ha-server VMs
# but are attached to different bridges — storage (vmbr2) and swarm (vmbr3)
# respectively. Kept as named variables to avoid bare literals in modules.
haDockerStorageInterface = "ens19"; # vmbr2 — NFS client
haDockerSwarmInterface = "ens20"; # vmbr3 — Docker Swarm gossip + VXLAN
# ── Storage / NFS ─────────────────────────────────────────────────────────
# NFS share definitions — used by ha-server.nix (exports), docker/mount-data.nix,
# and pxe-boot/mount-pxe-images.nix (mounts). `subpath` is relative to
# haStorageRoot; `mountpoint` is the absolute local path on each client.
# Renaming a share only requires changing it here — exports and all client
# mounts follow automatically.
nfsShares = { nfsShares = {
options = "(rw,sync,no_subtree_check,no_root_squash)"; options = "(rw,sync,no_subtree_check,no_root_squash)";
dockerConfig = { dockerConfig = { subpath = "docker/config"; mountpoint = "/mnt/docker/config"; };
subpath = "docker/config"; dockerDatabases = { subpath = "docker/databases"; mountpoint = "/mnt/docker/databases"; };
mountpoint = "/mnt/docker/config"; dockerVolumes = { subpath = "docker/volumes"; mountpoint = "/mnt/docker/volumes"; };
}; nextcloudData = { subpath = "docker/nextcloud-data"; mountpoint = "/mnt/nextcloud-data"; };
dockerDatabases = { raspiVolumes = { subpath = "raspi/volumes"; mountpoint = "/mnt/raspi-backup"; };
subpath = "docker/databases"; proxmoxIsos = { subpath = "proxmox/iso"; mountpoint = "/mnt/iso"; };
mountpoint = "/mnt/docker/databases"; proxmoxLxcImages = { subpath = "proxmox/lxc"; mountpoint = "/mnt/lxc"; };
}; pxebootImages = { subpath = "pxe-boot/images"; mountpoint = "/mnt/pxe-images"; };
dockerVolumes = {
subpath = "docker/volumes";
mountpoint = "/mnt/docker/volumes";
};
nextcloudData = {
subpath = "docker/nextcloud-data";
mountpoint = "/mnt/nextcloud-data";
};
raspiVolumes = {
subpath = "raspi/volumes";
mountpoint = "/mnt/raspi-backup";
};
proxmoxIsos = {
subpath = "proxmox/iso";
mountpoint = "/mnt/iso";
};
proxmoxLxcImages = {
subpath = "proxmox/lxc";
mountpoint = "/mnt/lxc";
};
pxebootImages = {
subpath = "pxe-boot/images";
mountpoint = "/mnt/pxe-images";
};
}; };
# The Raspberry Pi's own NFS export — not under storageRoot/nfsServerHost, # The Raspberry Pi's own NFS export — not under haStorageRoot, served
# served directly by the Pi itself over Tailscale (see raspberryPiHost # directly by the Pi over Tailscale (see raspberryPiHost) and mounted by
# above) and mounted at raspiMountpoint by modules/raspi/mount-data.nix. # modules/raspi/mount-data.nix.
raspiNfsPath = "/home/raspi/raspi"; raspiNfsPath = "/home/raspi/raspi";
raspiMountpoint = "/mnt/raspi"; raspiMountpoint = "/mnt/raspi";
# ── Ports ─────────────────────────────────────────────────────────────────
#
# Every literal port referenced from modules/ or hosts/, grouped by the # Every literal port referenced from modules/ or hosts/, grouped by the
# service/host that opens or connects to it — kept as separate entries # service that opens or connects to it. Kept as separate entries even where
# even where two happen to share a number today (e.g. nixCacheHttp and # two share a number today (e.g. nixCacheHttp and pxeBootHttp are both 80)
# pxeBootHttp are both 80) so changing one service's port can never # so changing one service's port never silently changes another.
# silently change an unrelated one.
ports = { ports = {
# nix-cache's nginx reverse proxy in front of nix-serve # nix-cache's nginx reverse proxy in front of nix-serve
# (modules/nix-cache/server.nix). # (modules/nix-cache/server.nix)
nixCacheHttp = 80; nixCacheHttp = 80;
# pxe-boot's nginx asset server, also used to build pxeBaseUrl # pxe-boot's nginx asset server; also used to build pxeBaseUrl
# (modules/build-types/pxe-boot.nix). # (modules/build-types/pxe-boot.nix)
pxeBootHttp = 80; pxeBootHttp = 80;
# pxe-boot's atftpd TFTP server — UDP, not TCP # pxe-boot's atftpd TFTP server — UDP (modules/build-types/pxe-boot.nix)
# (modules/build-types/pxe-boot.nix).
pxeBootTftp = 69; pxeBootTftp = 69;
# `server`'s NFS exports: portmapper (rpcbind), NFS data, and the # DHCP proxy port opened by dnsmasq on the pxe-boot host
# mountd RPC service (used by showmount/NFSv3 mount protocol). # (modules/build-types/pxe-boot.nix)
# Mountd listens on a fixed port so the firewall can whitelist it dhcp = 67;
# explicitly rather than opening all of rpcbind's dynamic range.
# All three need both TCP and UDP (modules/build-types/server.nix and # DNS port opened on tailscale-router for FreeIPA's conditional forwarder
# modules/build-types/ha-server.nix). # (modules/tailscale/ts-dns-forwarder.nix)
dns = 53;
# NFS stack: portmapper (rpcbind), NFS data, and mountd RPC service.
# Mountd is pinned to a fixed port so the firewall can whitelist it
# without opening rpcbind's full dynamic range. All three need TCP + UDP
# (modules/build-types/ha-server.nix).
nfsRpcbind = 111; nfsRpcbind = 111;
nfsd = 2049; nfsd = 2049;
nfsMountd = 20048; nfsMountd = 20048;
# HA cluster ports opened on ha-server-1 and ha-server-2 # HA cluster ports (modules/ha/cluster-config.nix)
# (modules/build-types/ha-server.nix / modules/ha/cluster-config.nix).
haServerDrbd = 7789; # DRBD replication (TCP) haServerDrbd = 7789; # DRBD replication (TCP)
haServerIscsi = 3260; # iSCSI target (TCP) haServerIscsi = 3260; # iSCSI target (TCP)
haServerCorosync1 = 5404; # Corosync totem ring (UDP) haServerCorosync1 = 5404; # Corosync totem ring (UDP)
@@ -197,49 +297,47 @@
haServerPacemakerRemoted = 3121; # pacemaker-remoted (TCP) haServerPacemakerRemoted = 3121; # pacemaker-remoted (TCP)
haServerPcsd = 2224; # pcsd cluster daemon (TCP) haServerPcsd = 2224; # pcsd cluster daemon (TCP)
# Opened on the docker host's firewall for the Traefik-fronted # Docker host — Traefik HTTP/HTTPS listeners plus one additional exposed
# container stack (docker-compose config lives in the separate # service (modules/build-types/docker.nix)
# /home/debian/docker repo, not here): 80/443 are Traefik's own
# HTTP/HTTPS listeners; 8080 is an additional exposed service whose
# exact backend isn't declared in this repo (modules/build-types/docker.nix).
dockerHttp = 80; dockerHttp = 80;
dockerHttps = 443; dockerHttps = 443;
dockerExtra = 8080; dockerExtra = 8080;
# Beszel monitoring hub, reachable at # Docker Swarm inter-node ports (modules/build-types/ha-docker.nix).
# http://<dockerHost>.<homeDomain>:<beszelHub> from every agent # Firewalled to haDockerSwarmCidr only — vmbr3 is an isolated bridge
# (modules/beszel/enable-agent.nix, hosts/nixos/home.nix). # with no physical uplink, so these ports are unreachable from LAN.
dockerSwarmMgmt = 2377; # TCP — Raft consensus + cluster management
dockerSwarmDisc = 7946; # TCP+UDP — Serf gossip (container network discovery)
dockerSwarmVxlan = 4789; # UDP — VXLAN overlay data path
# Beszel monitoring hub on docker.sweet.home, reached by every agent
# (modules/beszel/enable-agent.nix, hosts/nixos/home.nix)
beszelHub = 8090; beszelHub = 8090;
# Proxmox VE and Proxmox Backup Server web UIs, opened as desktop # Proxmox VE and PBS web UIs — desktop shortcuts on the gui build type
# shortcuts on the gui build type (hosts/nixos/home.nix). # (hosts/nixos/home.nix, modules/build-types/gui.nix)
pveWeb = 8006; pveWeb = 8006;
pbsWeb = 8007; pbsWeb = 8007;
# Tor relay's ORPort — the port other Tor relays connect to for onion # Tor relay's ORPort (modules/tor/enable-relay.nix). Opened via
# routing traffic (modules/tor/enable-relay.nix). Tor's own conventional # services.tor.openFirewall rather than allowedTCPPorts directly, but
# default; opened via services.tor.openFirewall rather than # kept here so it's not a bare literal if ever referenced elsewhere.
# networking.firewall.allowedTCPPorts directly, but kept here anyway so
# it's not a bare literal duplicated between the relay's settings and
# anything else that ever needs to reference it.
torRelayOrPort = 9001; torRelayOrPort = 9001;
}; };
# ── Build / image settings ────────────────────────────────────────────────
# .raw disk image size for every proxmox-* host's standalone Disko image # .raw disk image size for every proxmox-* host's standalone Disko image
# build (modules/disko/proxmox.nix, config.system.build.diskoImagesScript # build (modules/disko/proxmox.nix — see docs/proxmox-images.md).
# — see docs/proxmox-images.md). Root fills whatever's left after the ESP
# and swap partitions within this total.
proxmoxImageSize = "50G"; proxmoxImageSize = "50G";
# nix-cache's Nix store garbage collection retention # nix-cache Nix store GC retention (modules/nix-cache/server.nix)
# (modules/nix-cache/server.nix).
nixCacheGcMaxAge = "30d"; nixCacheGcMaxAge = "30d";
# Traefik access log rotation, watched on the docker host at # Traefik access log rotation, watched on the docker host at
# nfsShares.dockerVolumes.mountpoint (modules/traefik/rotate-logs.nix). # nfsShares.dockerVolumes.mountpoint (modules/traefik/rotate-logs.nix)
traefikLogRotate = { traefikLogRotate = {
maxSize = "100M"; # rotate once a log file exceeds this size maxSize = "100M"; # rotate once a log file exceeds this size
keep = 20; # number of rotated logs to retain before deleting the oldest keep = 20; # number of rotated logs to retain
}; };
} }
@@ -1 +0,0 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPW/X9Mdrqs0wLR7XbEDTihk7TEkNZ3LcCeXoa2ITSDA linode-server
@@ -1,14 +0,0 @@
{
"data": "ENC[AES256_GCM,data:hzI2cJ81+Psv35MMZO06UHpC8F9FLVkfUzn0ReuR+WoUds8D5zhDFbQHeR6ByeHTuK8Hlj712okhoXh6Vm7l3WmhLJZkr8IGLsV9W+P2PYityjuOtBPrphrKUeSqxJfjQ31/EhZLOqw/508XUmQDNcs1/n2g0TtQ2UQSTAuO8r1OlRfRSPizvVdj7lu+Vqg3dDBRETSOJAYIh8XQXoQWl3M4dS7jOYUvgc4EsYOWrvPPPS+8xXzctBPeToasY0IyWtLzoeNajCs5EpH5u7S5K9S40/vFPa45Ic2IxEkb1HHClyN6lXWbnc5QlJ5HPrq+2YBU2y9dJHd9DQiwEbcEBIvR3/Lo9puVDppXx9kez7i4XR1UHU7WVu39nQZuP50BQQazj+eM0/HJapXfjoqtaaq0qAGN4onnmRd0vEmrZZl4WMXmx8m9HhxdNJoZ4C2Wb2skdHZok88tPzjd4r7xKzVhAbB+wqS4LML6FmCNk+m0USQXa25USXs0JDV7lgiTFQJPE/67JGa820WFicGs,iv:YmOeK2Ha3yBXumVO9strgLgqNmPOcnqwUDJv5QNR1WM=,tag:Z7kZvQLDOKx3uX4dRXJqeQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOSWl5ajRSVndlMHJJQTFp\nYzRRU3RQMEVzazB0Sm5FdWZmMzBRaUl5RUFvCjgwRUZWdUpvUjkzWHFVdThIZ1ZV\nM2RMNy9pUWF4VVBCVHBGWHgwakhKQ2cKLS0tIERIOTVVZm55QWkvL25SRGZkZmtI\nT2o3ZEJNQ2hmTDNoRUx2Z3UxcmlyekUKLOajmvRfLdCJL74PKSgBtIXDuAVd8NwM\nh4BtDs0hONOz82JaBqFw8Uz28hVFG/gcS80br1o2klqPd2gN62PCVw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:33Z",
"mac": "ENC[AES256_GCM,data:JJC+12gTCsDVvMWRtL5cj50kf1n6Xn2j1hNBnwvZXUA9Pdd96SXt61U/Q80h8GZ7Ycs/slsV7h5f3g6+8tV0PcdBM/vy0vPM5qX1zMySyMv2p+dkJb9MwQpPg2xAQ9jjYM9237p5n9nysgu74h4V7ccBqmBzp764bL9wx6hEzS4=,iv:U6u1GFvoxaqxmHv1zCht24nW0ZMJR2b4pZqCG3bGNzs=,tag:MocwywOBg/1rUuoSADuJVA==,type:str]",
"version": "3.13.2"
}
}
@@ -1 +0,0 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILVRdEddGd+AMNe4kXbmA9UXK8JRsPEuxPx1vhNT9ZG8 lxc-server
@@ -1,14 +0,0 @@
{
"data": "ENC[AES256_GCM,data:JTmbN++hNMddBdMKIdzqZ6hLD1So9hyAzwbce/QvqWBsa1At85cMRUx+P1Yi4Sq2aXCg1Tf2g8XC9DXsVfMusUGEHrHbJApwkia/xHNMcFOohscUkPltXnm/lRilm4hwJUv6ay6UyLvIqMQAzzC8YWebHkKo00rxRxQWQ77wjmUG4dJeujlK5wMH0zTXUkrf7tIdJ20Zzg1BrAU6s4FYLvRvcUSW2ROu+sF2Sxgy1qVNJfDrRnY4REh9013tTSB5IlgJzUSQLs1vHr7EpIfmA5MCCSkZjplGCKvxkcNHB3aaOkrxkGa18JeWoklRen31UPU9zMUhTjAZ6VbbIagxzaFGKVP27cOiHQPNpEIMXwaRzoprjBX2PJ/Bs6pxe6hBpmiKOb40XdtxFQx2rofPcbQwklTpM5SFwvK+/bgawy1m8O3NHMhcw1qvd+6KjUdqy++/ivFC2Nyy+VfUDyGu0JLK3X+YqU6JXXMcUWOq79pVQIL3q1ofbhVHQ8p17+V3wQQUB3kPo4EEcKikI7Z9,iv:TVYVDOiTsgXaIcuJdnd3djPWXCMDDGjpefAW0MR+7Es=,tag:jIYO1oatbvInp77d4fC8mQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5QlNzNEFqSkhKTVhWVisr\nS21zdVBZSnpITDdrb243c0pUdDR5WXl1bFVrCmkxaWR6U01sLzhLL0MwRGh1VG5D\nNEFiU1FtNXRBdGtPellFTXYraitCUTgKLS0tIFdwWGFnMWFCN1ZvK2VBMWR2VTJZ\nRlQ4dEpmczA4bEdvUkNhaC93OXhDTkEK0QBkaLV1mbTMlSnjmN4x+qljGipHM/DQ\nUlmBYyi3nEOrI36I/Mm8yoBZai/qWqdg6IG+sxDE49ZOLp9PhpAWww==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:44Z",
"mac": "ENC[AES256_GCM,data:/l6pa3LE7+kFYqH2pv2RIcYBycLrZfpb92Al3SIU2tMeFUUvh+C8q8P9CgmAaiQuQ8S2dfYIJVx67zc7cRqI/UL/dFvDSv0YaMTGQ8Wn4fIXSh01EL0f/QVaIfb+uuvyEsdjy2ScWTWcCf2ICnC/zaMmp+xP+MmR1DRBM8KPIkY=,iv:xsmaYZ6dyHLU3BVfT3jxbfWWeBvYKMT+D9MtRxF9jlo=,tag:VzOiutHQJGHqb5UCI+cxSA==,type:str]",
"version": "3.13.2"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIoCzxgfFjE3scsLzTitDcrmJ/Bh4MQ+3cEH8YOQ665z ha-docker-1
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:fd++6Dyfyq7JfeGflnUXkmAKWhTG9Z0tvMCmVreczaoVtPN9+t+atIIa9EB3I2PWbOEYoIKelO0fDuK3ZrY1mrp8K8Stb3vJdB8lz97dBB/Gn5BonRAMYszMpfp9+rgf8pZqc89KWt3xz7z9EFKjTE7Kf8n08K1pHyqVjZ9W+bH9Zxh4WEWGeaKAyH4nlmVLjZ7rKch9ErY28bMVBwiiXIDFeXy/TIkSk7pVODFeCqMcuXFFnPememm7vXme+wtepqNc6WfrKTQOuQFe9SDK/A36Cl/92hSqjooTLU9XXjQNrEf1xAOkZhpTpbvsF32O103ixtqDY72ZMmC/RExAvaTT+Jf+Utfnq0S8Gna+d2rHyzbbvtdUC68vqA6QhL9qJe3HHwyAPbcGJ2YONgTsKot9DR3zYEWVwTjydBL8Z3/JTw62A+sLve4WuXgq9yooYGF4s/SVX1+a0e1MkQ2ahn7a6rIEyj+YK2bX71G5Kw+KBXwHzFW6duJs4mN23Yu9N+xrU7RdAT0O3hWGeBgmLfp8ko4uWOxm2Kin,iv:+5am0O58f/8tqrSEiFld4Xu2GZNDsEobjFPDgNyv768=,tag:GLERbrGo1rRKUXZBnUXEvw==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxcDErcVphYUFWUnN0WlBa\nVU9jUlZkMzArZFlYbENRNmZFQzN5Q1ZNa2k4CmlYa21yYXIrZG1QUVVNSy9oWE40\nK2t1dCtrR0R3cVQ5cVcwakphUFRvQ2cKLS0tIFdVL1dwNmMxR0xJOHh2ZlUzQmNm\nV1NNMldMRGd4NDAxbU1nbGZ0bWJKTDgKnyzQCimHCBke8dM3kdkjVu4lDcfvkvOi\n+k4GJVCuTQ60rwWXPsB8kiPMDp/+8tNO09LG/fR3vcGRgX65KcjMSA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-30T09:22:17Z",
"mac": "ENC[AES256_GCM,data:MjPZqcKTTO3uoZk/u7fdfQhgpcHsjKggzu+/pJanXUgIDWtCiKfFbRsQmwH5nYiRyjHOJdX4xMcFRmaYEN6oz/hzDR5ty+0mSuVNIemBz8OpShwjGY2E+pAkdxDgWLsc/Q+CyywGscLslbgHSAKuKPdPlyzhuRyy1QTyk/ty7Xk=,iv:MSQsNuENPgde3YitmHTyDjTYR5ZuhzwRgcWb5DAPY6g=,tag:Yo231g12u6aMPUauIf65hg==,type:str]",
"version": "3.13.3"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPwfEcHZRbdI5GK9cn59SDrLruNOJJEjVfJXZve25nnE ha-docker-2
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:3sWc8kfLREvxWRf0dtZFgubyxrAnfgiOT8b7nGtAQyQVaoy3moi3TGhPVdeihwbEOaEQX+YHvLWIzfxtLQxTPDxjvu3v2+dAYrjobIDHJaon3Th9vLfF9dxl/NVQB0wy/uL26z1Q4bhYj2yOU2V9wIzJ4G/94OvnttZKUub2iKmiUji36KsZkq6FVL0e0fUcv134dbo65yTIjmVB6U4SSB4nCrVKNg7z5EeXBBZodulK4kJ5zsspgTHLZbI43cW/tXmnNoW2NTFZen7Q1rLhlB0ISkXSXp0Sn6rIQsSZpywdc24cZ0fenqER38UI4PhwUPXd34Mv/LxQ6D5TnblANDQn1rsfxjjFY2JHXWO7kydccS5Us0vBHWMHbgZbC50+D1VIvvrK9SoEnZ7LW5WRJc1QaCvAz5eVbaB1V1UQ+eJjZToBQY1yvCg3vyj2nlX0A+BAPdZZMZgRazUp5CZaspcvcszIjTK8TDwqMeefOmUfNkC2Hgz83ij2lu+Kr9/Z1BurvIMotPDFER7qqYFodZJR4HMQ40rS9egO,iv:qh4tRPY6wZfOHdk1bDzQme/QqBVfMSfh2gZVTy1BEN0=,tag:8UcO2ia1nlzZPHNKuMYrrw==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBK1h5RUtPdjQwWm9vWGJK\nVzZHbU1GU2dVN3NKSWxqOEhtTUZwbDkzbTA4CjREek1WbnZrVkdQZGU3djFVVjg4\nOUNkNzk3eXIvN2hDMVNsc3dZS3hJb28KLS0tIFgxdlhwUG9PdUlsRFdRdmJkOGhh\nVU1uWkpwSVhKL1VRTktoQmxEQVk5bjgKQHofvV/y5tPG0AJUoC1/oXBqAIlX6FB/\nAHKB7k3JxbgrzkR40uKuA9y6qmokGkVPoD3p7N5fd5G3aZY3fsYtjw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-30T09:22:26Z",
"mac": "ENC[AES256_GCM,data:bICdSPh5zGwWW1MI4rZZgKOZWM6X2h2OEL+uK3EQNGsGoRnW2JicBdIjqBkQsin5Ke4a0i0n/LudIB/G90+NkQYXu8/ixve3vN6VvfnoBb53BuDp/Hn7hJkvvn/1iXDmYsg1S75IM6xTm4RkY8nbLTuBltgiZvrN6mYMrxcXc9s=,iv:rGnu8V9P3vOrHA06GDq3Ru9l3+x93hfrVVZtN+hR9e4=,tag:rYZ1h95YoZzLg2ZaRxJmQA==,type:str]",
"version": "3.13.3"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN7IABG7x9ejHSENmHsL/BF9MQadGuBnHZa1eUEFMYnH ha-server-1
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:EhYMNPzlMDyby6g3hfvBKisKJnU+YAxZjf326tPdflQ2PcmPTzZ0GjHqJOPrOWyLU6IyvnsLMBV0JF5/UUi2c/H3p2wvoBgsxDMaaMW0YUK2IZFo+PZpWoYJSMrx40sH90W0qL0VBw17JLwPU6CjeUZrn5+sCN9EhQvncD8NoJ6QwZpWKhWLn9/7mxd/PF1uxjEQXzHYFeuUuVpQdKSUfV/BSJIT3pMM/VdH2203jTMtwUbK1/7UtGKtDWdfFbkEvJ6qsV7hnN3EYU16WtdyAiHEZvvZ3gqg6YK2DemojWG7z7VQkQGbGm40yA6/mYGW2dVK5ULzjxALl05G6lo1vzxODbMHhzsfJZVw6ocGx3r8xwaasiX8S3loq8WNUWJ9QgjbxuEX9uGZja6A3nPhcwVGGZExbUoU+syXUk2m+WfIQ0epplaPDyAxrYO5ZPWLdMYoPRxbDx94rx5fMxA3jst++IEL2kQMTieGQYJy3R8veQIe3Jb074zY+Y79EmRxfo8ruR6ZMqORtbZzJOBcd0OObjUVLUqB4rnC,iv:XkCfOKtmZNz+UTZdElLm8L0PpymkDzCM24VkAa9Y/Qo=,tag:hmr9ik0V7mxEKR0DcDB3/w==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqQ3J3VDVwUVBvUjVXSWNz\nU3YrRXZlT2NLRThCNzd3YWNHVTlHeS9xMHpzCi9HVW5kMXZrdjE4bXdML3NhRnI1\nL0NQQkNQUmg2M0NBYzh2cG1vdHA0clkKLS0tIDJZYUwzK0RGNzJWckJJekI0SFg1\nUXE3N05xWThtdjdCZkdJTFJ2YUx6cm8KVANudVL54WBNc9DK9s9h4WQRLMewUqgN\ntu4LdMOmi5oV3LX06lbxhBq79dmsV5uos/qszhJkVGhEZwD1RGBoZA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-28T06:42:29Z",
"mac": "ENC[AES256_GCM,data:Xn3/CWhQJHWtd3QHoqpdwLuJdZTp+oxx43k38j4jspQSBPdVDSG6Ifi7Sb2r1g9YTxpjUto8mv7FH+BqI0wSYUkLfjaxI51xs3dq6e6kl5NWOsvWwIOGmBuhfIwusHGgsfBguxk0J6Ev8Irw674CRFc37jozXTVdpOOzFhPTvFs=,iv:/7OFE9NYkw2kYEetzeZRXah0KXuD5OXmfjQMIeqI75Q=,tag:VamuInZqYGYRwlyU8TtL9g==,type:str]",
"version": "3.13.3"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINWuFwPKW2jwQEj6c//xb95DbtandPkJ7M0ceyTNd35G ha-server-2
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:GEebcQrPPrQxLC6mWXLLtytGkIy6Q2FbcQ/Z+ImGa3gahI/nigdNyR/avIz1DAUsOibpoMVlrk0GQUBMuxiEXulIwUEzeTh7sMI3XTZiugujOFHPyJoFeynlgeRNBbl2EVVjGmEmOsR/cZJGyKaet9aNTHTPDJUhodY5d+D/sE3SgpHoch1DI02/DRnqlnlOYiBoujk++BW71EDHLVyz3n9NsurTDYDK4L0Ch5cMgdlvwIuxGuPKtkhU9z6PtGxPbNlYlNP8U+3jG4XWEEY5hZHBGTVHn+bd5FViGY+sMiigO6yaOmddbUqzJoblKeI6F/rfrL3kXETLgVP7uVm3gw9KPEl5JaTdVqkG4hQ9G9/LvRlyEh619mCp9IRXxabm5SQ4NWFzah1ZwqIKp6GoAtqeWRgZNVqijoF+bnb0YmNFmjwpSrSMZvKsrZb5geyGmqah+NJii0xqVebnufB5p8lW9eytBSNgC/XLjj0olwHMw7CWGmnY6Q7lxF1tNYAD/K3dkbxQYFM3kBnmAdweC8WYc+hDjtk0HJul,iv:H4BonLmf3VoW2S9+IUE+SzPb0qiWHuNuGpVIUgUYScs=,tag:5+bq7vC81DMmOk4jn3u8lg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrQitsaDlqTmlYM2RvR1Ew\nb3NzZTJzR3F2bjM5NHhsa3dCUzZ4TW1pa3pBCkxIeEJtbWdaUFMwMkFyRHpCZ1Bl\nbmM5dGV3b3BDcDEwVlJ4UWV5RFNvRkUKLS0tIFRDd0U3dHBHb0Y3WDhNV3hkd2Ew\nclRDVEZHYjZUSzNVZkFjWGt3SGVQTlUKOULuXiYD9k2uVUmhuC15Kgezrd69rc9P\n7SocPa8kBliffP9IuxxW2S/hPbK8rqEx/sh/Km85ZIi4pG0AUQJ3fA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-28T06:45:10Z",
"mac": "ENC[AES256_GCM,data:ZnsqhgELqEdNzlTKJx2+xzcZRMNGTme/0a+PVvpbUv5IApimtif0zkTEC1LJsyLPut3MmO3QGprVlb8ihoRdm/IVvLaQonc36f6vN5cxrpQadru/RWEjLbnge1E8VkN/PVvHqRfLyIOPSq6wtGjAPaC/iHW1scBi50pWr0XO628=,iv:FlqAL1It6JuvoBhZotHWdSz17nckBf2rXnwovl6ZJBA=,tag:tNwcaL1sJakXtgejsB/Sbg==,type:str]",
"version": "3.13.3"
}
}
@@ -1 +0,0 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGQHSubxjvaIV9Xp5ABJSKsajCZBGyGmjsdaA2TWCFP8 proxmox-server
@@ -1,14 +0,0 @@
{
"data": "ENC[AES256_GCM,data:GdVHSekFtRtppf8xxMPsDE0cu8Hwq3ol0iJTfrwKwaAB0IW8YS8TFAeRnfOI5AE3P1JUaGDZ1Od4UXu4Jk2+uYCgwz074qc1WukzUsd8Q3G03ojdrTA13uH8YCUTRsm3bQfB+kmH5qefn9j88UJq7TYiTLe3U0DUU/B83lgyiFIygP8CKWDgFH3jWZOWPmyVa4Z1fMWVsiNhjO4hHduamN/I2FOY7U2TI5+zcYgGeunjnniLHcQdiO0iK9TOP5SckOVPZOy6daKzGcckEq03NV4y+miSucEUHlcQqMUyvsJwETIaCq28xR2MDTyDz42SL2HIodb7FyREtxd91W6gPN1j51yUZxt4WA9R8/WczDcUgmROK+HInPp+ktjWqv2ymXXV5xidxh7m+vrGTPC+YQqyFD4eIp3+h2S/vyzHimBa2rdEp9NNXTQlH3CfL2aLC95FOme+uKTDqWA2PHfB0rQ5uhHAuAgaX5vb/GTiQii5dq4k2O4MDf9u+fdPuOSCaGdlirWmSE45SvaXLJ3M2zQq7M/UTE+wSFyc,iv:EaIsYnGxf42LWQ3hzBU/HsBda/FHInhjZkwsPLcZMEY=,tag:16OuAs3Ram+xd4RfWEBXkw==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBETE4vOHhldGRFRjdYaXpi\nSmgzTFlqL3Qzak1CelRBR29jZzR1V0pOQWw0CkhSWDkvbkJWaDVHMzlKNDZUdzhH\nWGoyWXQ1KzJZQU1kQzFvaDk2V2lWQmsKLS0tIHlTY2tua2F1TXBhamsvcDZ0bmI1\nZVFwUGtQWW1HNTM3UUs4bHRZU0xKRDgKBG2iI9JP0lhU5VCWXrpN1b2rYEYk8sOZ\n9FUO14KKMg9QRfSa2iHOa84DByx2hxVRc9wLukBUpkKOAjGSxeGMeg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:58Z",
"mac": "ENC[AES256_GCM,data:jG7ohweKKndoadidquejYG/1w9iL+9Xb5/IsU/C9fn/Tq5RXEjrxO37COY8sAD7dvQf7iBNsly7upsjtHaMK8ybjQDaa6IQhoGBlfSOA2O968klaJZRQRiLPzCRet252KXzOtvDscrBvyYItvyqjnW5qBbw7lfT4y2J4OA2ieC0=,iv:mMo7Y6XEjQcchNOY6eaw5LOmjFKaQKx84o9bnGNXCKI=,tag:aHkPjs7uVfv8h0REHN8YmA==,type:str]",
"version": "3.13.2"
}
}