Compare commits

..
Author SHA1 Message Date
beatzaplenty 71381ad990 update default editor
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
2026-07-30 05:32:23 +10:00
beatzaplenty 857c81f64a updated beszel key
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
2026-07-29 21:33:48 +10:00
beatzaplenty 334ffbda09 updated beszel token
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
2026-07-29 21:27:16 +10:00
beatzaplentyandClaude Sonnet 4.6 620a344d78 feat(pxe-boot): switch NFS mount to HA cluster LAN VIP
Check NixOS configurations / eval-hosts (push) Successful in 10m31s
Updates mount-pxe-images.nix to mount pxe-boot/images from
ha-vip-lan.sweet.home (Pacemaker vip-lan, 192.168.2.229) instead of
server.sweet.home. Storage root changes from /tank to haStorageRoot
(/srv/ha-data). NFSv3+nolock options for LXC are unchanged.

Removes nfsServerHost and storageRoot from variables.nix — all NFS
clients now reference the HA cluster directly (docker via haStorageNfsFqdn
on VLAN 20, pxe-boot via ha-vip-lan on VLAN 2). Updates the nfsShares
comment to reflect the new single-source-of-truth role without the
dead server.nix references.

DNS records for server.sweet.home and server.storage.home were already
cleaned up automatically by IPA dynamic DNS when the VM was deleted.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 20:53:46 +10:00
beatzaplentyandClaude Sonnet 4.6 24e5c9fe0c feat: decommission server.sweet.home
Check NixOS configurations / eval-hosts (push) Failing after 3m46s
Removes all flake artifacts for the retired NFS/ZFS file server:
- flake.nix: drop linode-server, proxmox-server, lxc-server targets
- hosts/server/: delete host identity
- modules/build-types/server.nix: delete build type (ZFS pool + NFS exports)
- vars/per-machine/{linode,lxc,proxmox}-server/: delete clan vars
- secrets/server.keytab: delete IPA keytab
- .sops.yaml: remove server key anchors, strip from common.yaml recipients,
  delete server.yaml and server.keytab creation rules
- variables.nix: remove serverIp, serverStorageIp (nfsServerHost/storageRoot
  retained — pxe-boot/mount-pxe-images.nix still references them)
- docs: mark server decommissioned in ip-addressing.md

Docker NFS mounts already pointed at nfs.storage.home (previous commit).
VM destruction and DNS cleanup (server.sweet.home, server.storage.home
A/PTR records) to follow once the VM is confirmed stopped.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 20:46:19 +10:00
beatzaplentyandClaude Sonnet 4.6 4ed2db906a feat(docker): switch NFS mounts to HA cluster storage VIP
Check NixOS configurations / eval-hosts (push) Failing after 16m43s
Updates mount-data.nix to mount all docker shares from
nfs.storage.home (Pacemaker vip-storage, 192.168.20.229) over docker's
eth1/vmbr2 interface instead of server.sweet.home over the LAN.
Adds haStorageNfsFqdn variable to variables.nix for the storage.home
zone FQDN so mounts survive a future VIP renumber without a rebuild.
Storage root changes from /tank (server ZFS pool) to /srv/ha-data
(HA cluster XFS-over-DRBD). Share subpaths are unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 20:27:15 +10:00
beatzaplentyandClaude Sonnet 4.6 944af9597d docs(dns): document DNS zone layout for all three networks
Check NixOS configurations / eval-hosts (push) Failing after 14m52s
Adds a DNS Zones section to ip-addressing.md covering the three FreeIPA
zones (sweet.home / cluster.home / storage.home) with all A records and PTR
records as configured. Updates Stage 6 in network-cutover.md to reference
nfs.storage.home by DNS name rather than the raw VIP IP, and marks DNS as a
completed prerequisite for the HA storage cutover.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 20:12:25 +10:00
beatzaplentyandClaude Sonnet 4.6 4f0b07031d fix(ha): add NFS port check on storage VIP in health.sh
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Storage VIP now serves both NFS and iSCSI. Add port 2049 check
alongside the existing 3260 check in the storage VIP section.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 19:09:27 +10:00
beatzaplentyandClaude Sonnet 4.6 8c86144694 docs(ha): update docs to reflect implemented network segmentation
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
ip-addressing.md:
  - Storage-client network purpose updated: NFS primary, iSCSI available
  - Protocol separation box updated: NFS on both VIPs (each CIDR-restricted);
    iSCSI on VLAN 20 only but not in active use
  - docker role updated to NFS client (not iSCSI initiator)
  - vip-storage description updated to NFS + iSCSI endpoint

ha-network-audit.md:
  - Implementation status header added noting all audit items resolved
  - Records decisions that diverged from original recommendations
    (VLAN renumbering, dual VIP, NFS-over-iSCSI for docker/swarm)
  - Notes deferred items (STONITH, iSCSI ACLs)

network-cutover.md:
  - New Stage 6: HA storage cutover — docker NFS remount from
    server.sweet.home to vip-storage (192.168.20.229), final rsync
    steps, and server.sweet.home decommission procedure
  - Former Stage 6 (final cleanup) renumbered to Stage 7

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 19:04:42 +10:00
beatzaplentyandClaude Sonnet 4.6 3c28d48bc7 feat(ha): add NFS exports on storage-client network (VLAN 20)
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
vip-storage (192.168.20.229) now serves NFS as well as iSCSI, firewalled
to haClientCidr (192.168.20.0/24) only. This allows docker and future
swarm nodes to NFS-mount shared volumes from the storage network rather
than the LAN, keeping storage traffic off vmbr0.

LAN NFS (vip-lan 192.168.2.229) remains for pxe-boot and other LAN
clients. Protocol and subnet boundaries enforced by firewall on both nodes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 18:55:25 +10:00
beatzaplentyandClaude Sonnet 4.6 e9a4913069 fix(ha): split health.sh VIP check into LAN (NFS) and storage (iSCSI)
Check NixOS configurations / eval-hosts (push) Successful in 10m43s
Storage VIP (192.168.20.229) is on vmbr2 — an internal-only bridge with no
physical uplink, unreachable from the LAN. Test it via SSH to the active node
(which has VLAN 20 connectivity) instead of directly. LAN VIP (192.168.2.229)
continues to be tested directly from wherever the script runs.

Also adds VIP_LAN variable to config section.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 18:33:34 +10:00
beatzaplentyandClaude Sonnet 4.6 bf4836efac refactor(ha): renumber cluster→VLAN10, storage→VLAN20; iSCSI/NFS protocol separation
Check NixOS configurations / eval-hosts (push) Failing after 17m31s
Renumber internal HA bridges so VLAN ID matches IP third octet:
  vmbr1 (cluster): 192.168.4.x → 192.168.10.x (VLAN 10, /29)
  vmbr2 (storage): 192.168.5.x → 192.168.20.x (VLAN 20, /24)

Enforce protocol separation at the firewall on both HA nodes:
  - iSCSI (3260/tcp): storage-client subnet (192.168.20.0/24) only
  - NFS (111,2049,20048 tcp+udp): LAN subnet (192.168.2.0/24) only
  - Cluster subnet (192.168.10.224/29) accepted wholesale (DRBD+Corosync)
Moves from allowedTCPPorts/allowedUDPPorts to source-restricted extraCommands
rules on the nixos-fw chain.

NFS exports reduced to LAN-only (drop haClientCidr lines from mkNfsExports).
Storage-client network clients use iSCSI only; no NFS on VLAN 20.

All HA script VIP/storage IP defaults updated to match.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 18:19:07 +10:00
beatzaplentyandClaude Sonnet 4.6 5eea38d3ca fix(flake): fix /etc/flake-target on ha-server-{1,2} so Switch-nix alias works
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
mkTarget computed flakeTarget as "${platform}-${buildType}" which gives
"proxmox-ha-server" for both nodes, but the actual nixosConfigurations
keys are "proxmox-ha-server-1" / "proxmox-ha-server-2".  This caused
Switch-nix to fail immediately with an "unknown flake attribute" error.

Add an optional nameSuffix parameter to mkTarget and pass "-1" / "-2"
for the ha-server entries so /etc/flake-target matches the real
attribute name on each node.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
2026-07-29 16:25:51 +10:00
beatzaplentyandClaude Sonnet 4.6 6f38ad67e0 feat(ha): add dual VIP — retain 192.168.2.229 LAN alongside 192.168.5.229 storage
Check NixOS configurations / eval-hosts (push) Successful in 10m35s
Both VIPs fail over together in ha-group:
  vip-lan:     192.168.2.229/24 on ens18 (vmbr0) — LAN iSCSI + NFS access
  vip-storage: 192.168.5.229/24 on ens20 (vmbr2) — storage-client iSCSI + NFS

variables.nix: haServerLanVip = "192.168.2.229"
cluster-init.sh: both VIP primitives in ha-group XML; VIP_LAN var added
ip-addressing.md: both VIP entries documented

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 16:18:45 +10:00
beatzaplentyandClaude Sonnet 4.6 8d43b7039c feat(ha): add vmbr2 storage-client network and corosync ring1
Check NixOS configurations / eval-hosts (push) Successful in 10m39s
Infrastructure changes already applied to pve1:
- vmbr2 internal bridge created (192.168.5.0/24, no physical uplink)
- VM 200 (ha-server-1): net2 added → vmbr2 (ens20)
- VM 201 (ha-server-2): net2 added → vmbr2 (ens20)
- CT 105 (docker): net1 added → vmbr2 (eth1)
- VM 101 (server): net1 added → vmbr2 (ens19) — needs reboot to activate

NixOS config (deploy to ha nodes to complete; docker/server at cutover):
- ha-server-{1,2}/host.nix: ens20 with 192.168.5.{228,227}/24
- docker/host.nix: eth1 with 192.168.5.225/24
- server/host.nix: ens19 with 192.168.5.226/24
- cluster-config.nix: corosync ring1 on LAN IPs as backup heartbeat path
- cluster-config.nix: allow haClientCidr (192.168.5.0/24) in iptables
- ha-server.nix: NFS exports now allow both lanCidr and haClientCidr
- VIP moves from 192.168.2.229 (vmbr0/LAN) to 192.168.5.229 (vmbr2)
- iSCSI portal to be rebound from [::0] to 192.168.5.229 at cutover

variables.nix: haStorageCidr corrected to 192.168.4.224/29; new vars:
vmStorageClientInterface, lxcStorageInterface, haServer{1,2}ClientIp,
dockerStorageIp, serverStorageIp, haClientCidr/PrefixLength; haServerVip
updated to 192.168.5.229.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 16:02:53 +10:00
beatzaplentyandClaude Sonnet 4.6 8e58226d2b docs(ha): add network segmentation audit report
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Read-only discovery of pve1 bridges, HA cluster VMs (200/201), and
Docker CT (105) — current state, risk flags, recommended vmbr2 storage
bridge layout, gap list, and migration notes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 15:38:21 +10:00
beatzaplenty a5fb0404e2 Merge pull request 'fix(gc-hosts): filter discovered guests against flake to exclude non-NixOS hosts' (#107) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m19s
Reviewed-on: #107
2026-07-29 04:08:05 +00:00
beatzaplentyandClaude Sonnet 4.6 4ba9b141fe fix(gc-hosts): filter discovered guests against flake to exclude non-NixOS hosts
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m32s
pve1 runs non-NixOS guests (FreeIPA domain-controller, Proxmox Data Manager, etc.)
alongside NixOS ones. Validate each discovered hostname against the set of
hostnames defined in nixosConfigurations before adding it to the gc list.

Brings back the flake eval but uses it correctly: extract all hostname values
(not target-name keys) to build a filter set, then only include pve1 guests
whose name matches a flake-managed NixOS host.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 14:06:17 +10:00
beatzaplenty f3c2965f78 Merge pull request 'fix(gc-hosts): use pct/qm list output directly as hostnames' (#106) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
Reviewed-on: #106
2026-07-29 04:02:50 +00:00
beatzaplenty b9d3b51028 Merge pull request 'fix(ha): resolve data disk device via by-id symlink even in dry-run' (#105) from worktree-partitioned-swimming-pizza into main
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Reviewed-on: #105
2026-07-29 04:01:49 +00:00
beatzaplentyandClaude Sonnet 4.6 539bdf9833 fix(ha): resolve data disk device via by-id symlink even in dry-run
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m21s
The by-id lookup is read-only so it's safe to run in dry-run mode.
Previously it was gated behind `if ! $DRY_RUN`, which always triggered
the sdb fallback warning in dry-run — making it look like the device
path wasn't reliable when the symlink actually exists on both servers.

Now the lookup always runs and the script errors out with a clear message
if the by-id symlink is genuinely missing, instead of silently falling
back to a guessed /dev/sd* name.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:58:53 +10:00
beatzaplentyandClaude Sonnet 4.6 a5308a7ee5 fix(gc-hosts): use pct/qm list output directly as hostnames
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m32s
The flake hostname map approach was wrong: create-proxmox-resource.sh names
every guest after its NixOS hostname (pct create --hostname <nixos-hostname>,
qm create --name <nixos-hostname>), so pct/qm list already output the final
hostname directly. Looking those names up in a map keyed by flake target names
(lxc-docker, proxmox-server, ...) found nothing and the dynamic list was
always empty.

Drop the flake eval entirely; use the pct/qm list output as-is, filtering
only nix-cache and already-seen entries.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:58:45 +10:00
beatzaplenty dc6c37ed2e Merge pull request 'fix(ha): prefix qm commands with sudo for non-root Proxmox SSH user' (#104) from worktree-partitioned-swimming-pizza into main
Check NixOS configurations / eval-hosts (push) Failing after 4m22s
Reviewed-on: #104
2026-07-29 03:55:28 +00:00
beatzaplenty b5e61d62bd Merge pull request 'fix(gc-hosts): fix pve1 PATH and discovery sudo prompts' (#103) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Reviewed-on: #103
2026-07-29 03:53:50 +00:00
beatzaplentyandClaude Sonnet 4.6 ced1657407 fix(ha): prefix qm commands with sudo for non-root Proxmox SSH user
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m25s
qm lives at /usr/sbin/qm, which is not in the default PATH for
non-interactive SSH sessions as a non-root user.  Add PVE_SUDO (set to
"sudo" when PVE_SSH_USER != root, matching create-proxmox-resource.sh's
own sudo_prefix pattern) and prepend it to all three qm invocations in
the script.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:53:41 +10:00
beatzaplentyandClaude Sonnet 4.6 7b4794211d fix(gc-hosts): fix pve1 PATH and discovery sudo prompts
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m45s
Two bugs:

1. Dynamic discovery used BatchMode=yes for the pct/qm list SSH call, which
   silently suppressed any sudo password prompt and returned empty output.
   Fix: split SSH_OPTS into SSH_OPTS (BatchMode, for gc) and SSH_QUERY_OPTS
   (no BatchMode, for discovery) so sudo can prompt when needed.

2. pve1 gc used a non-login SSH session which doesn't source /etc/profile,
   so nix-collect-garbage was never on PATH. The Nix installer registers
   itself via /etc/profile.d/nix-daemon.sh which only runs in login shells.
   Fix: use "bash -l -c 'nix-collect-garbage -d'" instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:53:02 +10:00
beatzaplenty fdf41c659c Merge pull request 'Worktree crispy churning kernighan' (#102) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m50s
Reviewed-on: #102
2026-07-29 03:46:05 +00:00
beatzaplentyandClaude Sonnet 4.6 0ef8259225 fix(gc-hosts): source nix-daemon profile on pve1 before running gc
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
BatchMode SSH sessions don't source /etc/profile on non-NixOS hosts, so
nix-collect-garbage isn't on PATH for the wayne user. Source the nix-daemon
profile script explicitly, matching the pattern in scripts/lib/nix-bootstrap.sh.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:45:48 +10:00
beatzaplentyandClaude Sonnet 4.6 629c1457a9 refactor(gc-hosts): discover running pve1 guests dynamically each run
Replace the static host list with dynamic discovery: workstation (nixos)
and pve1 are hard-wired first and second; remaining hosts are discovered
on every run by SSHing to pve1, listing running VMs/containers via
pct/qm list, and resolving their NixOS hostnames from a single flake eval.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:44:26 +10:00
beatzaplenty 7ba603c005 Merge pull request 'feat(scripts): add gc-hosts.sh for parallel nix gc across all live hosts' (#101) from worktree-crispy-churning-kernighan into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #101
2026-07-29 03:36:09 +00:00
beatzaplentyandClaude Sonnet 4.6 decf3ddff9 feat(scripts): add gc-hosts.sh for parallel nix gc across all live hosts
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m29s
Runs nix-collect-garbage -d on all deployed NixOS hosts and pve1 in
parallel, skipping nix-cache to avoid evicting shared cache paths.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:35:24 +10:00
beatzaplenty 4490dfab7d Merge pull request 'feat(nix-cache): authorize pve1's wayne key as remote builder client' (#100) from worktree-pve1-nix-cache-builder into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #100
2026-07-29 03:16:53 +00:00
beatzaplentyandClaude Sonnet 4.6 fa163b613d feat(nix-cache): authorize pve1's wayne key as remote builder client
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
Generated /home/wayne/.ssh/id_ed25519 on pve1 (single-user Nix install
owned by wayne, no daemon — builds run as wayne not root). Adds the
public key to remoteBuilderAuthorizedKeys so nix-cache accepts it.

After deploying nix-cache, re-run configure-nix-cache-client.sh as
wayne on pve1 to wire in the builder line and known_hosts entry.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 13:16:30 +10:00
beatzaplenty 15bc5bd369 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Successful in 10m30s
2026-07-29 13:02:03 +10:00
beatzaplenty 5a4fbbf7ad enable beszel agent 2026-07-29 13:01:53 +10:00
beatzaplenty e720bbb018 Merge pull request 'fix(nfs): mask rpc-svcgssd on server and ha-server build types' (#99) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Failing after 9m46s
Reviewed-on: #99
2026-07-29 02:54:12 +00:00
beatzaplentyandClaude Sonnet 4.6 40cdf724b4 fix(nfs): mask rpc-svcgssd on server and ha-server build types
Check NixOS configurations / eval-hosts (pull_request) Failing after 9m43s
nfs-utils changed ConditionPathExists from !/etc/krb5.keytab to
/etc/krb5.keytab, so on IPA-joined hosts the service now starts instead
of skipping. Exports use standard auth (no sec=krb5) so the nfs/ Kerberos
principal is never provisioned and the service fails.

enable = false masks the unit so nfs-server's Wants= can't pull it in.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 12:51:50 +10:00
beatzaplenty aaa193645b Merge pull request 'refactor(beszel): move KEY and TOKEN fully into enable-agent.nix' (#98) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Successful in 10m45s
Reviewed-on: #98
2026-07-29 02:38:19 +00:00
beatzaplentyandClaude Sonnet 4.6 f22ff7db79 refactor(beszel): move KEY and TOKEN fully into enable-agent.nix
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m59s
KEY (hub's SSH public key) is now stored in variables.nix as beszelHubKey
and set by enable-agent.nix — no host file ever needs to touch it.
TOKEN was already universal; this removes the last per-host beszel config.

Importing enable-agent.nix in a build type is now the only step needed to
add a new host to beszel monitoring.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 12:37:54 +10:00
beatzaplenty 0da3c3070f updated flake.lock and added beszel agent environment
Check NixOS configurations / eval-hosts (push) Failing after 16m56s
2026-07-29 12:18:38 +10:00
beatzaplenty e1551feefd enabled beszel-agent
Check NixOS configurations / eval-hosts (push) Successful in 10m34s
2026-07-29 12:14:49 +10:00
beatzaplentyandClaude Sonnet 4.6 b4bc30cb2c chore(secrets): remove per-host beszel-token secrets
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
Five files contained only beszel-token and are deleted entirely.
nix-cache.yaml retains cache-priv-key; beszel-token removed from it.
The universal token in secrets/common.yaml replaces all of these.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 12:00:38 +10:00
beatzaplenty 5c8d55bd78 Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos
Check NixOS configurations / eval-hosts (push) Failing after 4m33s
2026-07-29 11:50:44 +10:00
beatzaplenty 999c9a3151 add beszel universal token 2026-07-29 11:50:01 +10:00
beatzaplenty 1e0fff1b26 Merge pull request 'refactor(beszel): switch to universal token via secrets/common.yaml' (#97) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Failing after 13m26s
Reviewed-on: #97
2026-07-29 01:47:15 +00:00
beatzaplentyandClaude Sonnet 4.6 24c6469f10 refactor(beszel): switch to universal token via secrets/common.yaml
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m40s
Replace per-host host-token.nix imports with a single beszel-token secret
in secrets/common.yaml, wired once in enable-agent.nix. Host files now only
need services.beszel.agent.environment.KEY — no imports block required.

Delete modules/beszel/host-token.nix (no longer referenced anywhere).

Action needed: run `sops secrets/common.yaml` and add `beszel-token: <value>`
from the beszel hub UI before deploying.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 10:58:15 +10:00
beatzaplenty c9458ac8a6 Merge pull request 'docs: add beszel agent setup guide' (#96) from worktree-docs-beszel-guide into main
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
Reviewed-on: #96
2026-07-29 00:41:22 +00:00
beatzaplentyandClaude Sonnet 4.6 99ba0ed52b docs: add beszel agent setup guide
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s
Documents how to wire beszel-agent into any flake build: the two-module
split (enable-agent.nix in build types, host-token.nix in host files),
sops secret creation, hub pairing flow, and optional EXTRA_FILESYSTEMS /
Docker monitoring knobs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 10:41:03 +10:00
beatzaplenty 1e4072029e update rmote-builder keys
Check NixOS configurations / eval-hosts (push) Successful in 10m38s
2026-07-29 10:39:53 +10:00
beatzaplenty 46649fc7e0 update remote-builder client keys
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
2026-07-29 10:29:40 +10:00
beatzaplenty 4cbcc3beb9 updated .gitignore and change default editor in home.nix
Check NixOS configurations / eval-hosts (push) Failing after 13m5s
2026-07-29 10:17:37 +10:00
beatzaplenty 35f696ccd5 Merge pull request 'feat(ha): add resize-data-disk.sh for online data disk expansion' (#95) from worktree-ha-resize-disk into main
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
Reviewed-on: #95
2026-07-29 00:01:49 +00:00
beatzaplentyandClaude Sonnet 4.6 e18b605706 feat(ha): add resize-data-disk.sh for online data disk expansion
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m27s
Three-phase online resize: Proxmox qm resize → guest block device
rescan → drbdadm resize + xfs_growfs. No downtime required.
Detects active node via crm_mon, pre-checks DRBD UpToDate/UpToDate,
auto-discovers VMIDs from qm list, and resolves the block device name
from the stable scsi-0QEMU... by-id symlink with a slot-index fallback.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 09:59:39 +10:00
beatzaplenty ee2451d87c remove stray pihole backup
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
2026-07-29 09:50:50 +10:00
beatzaplentyandClaude Sonnet 4.6 71a6c4738c fix(ha): fix crm_mon active-node detection in all three scripts
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
crm_mon 2.x formats the Promoted line as "    * Promoted: [ node ]" — the
asterisk bullet means the previous grep -E '^\s*(Promoted|Masters):' never
matched, so active-node detection silently returned empty in health.sh,
failover.sh, and acceptance-tests.sh.

Fix: pipe through grep -v Unpromoted first, then grep -E '(Promoted|Masters):'
without anchoring to start-of-line.

Also: remove the SSH_OPTS=-i ~/.ssh/... variable pattern in health.sh and
failover.sh; tilde is not expanded inside double-quoted strings, so $SSH_OPTS
was passing a literal ~ to SSH.  Inline the key path in each function
definition instead (same as acceptance-tests.sh already did).

Also: drop the 2>/dev/null embedded in the crm_mon argument string — the
outer 2>/dev/null on the n1/n2 call already suppresses SSH stderr; the
embedded one was harmless but noisy to reason about.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 02:36:22 +10:00
beatzaplentyandClaude Sonnet 4.6 793a2b5924 feat(ha): add failover.sh and health.sh operator scripts
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
failover.sh: graceful active→standby resource migration with pre-checks
(DRBD UpToDate/UpToDate, quorum), spinner wait for XFS mount on target,
rollback on timeout, --to/--force/--dry-run/--timeout flags.

health.sh: read-only status panel — node reachability, quorum, DRBD
role/dstate/cs per node, Pacemaker resources, failure history, XFS mount
usage, and service port checks (NFS 2049, iSCSI 3260) via VIP.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 02:23:47 +10:00
beatzaplenty 98b5429fb9 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 02:04:51 +10:00
beatzaplentyandClaude Sonnet 4.6 f658bdbabc fix(ha/cluster-init): enable Pacemaker maintenance-mode during DRBD sync
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
Even with both nodes in standby, Pacemaker's monitor operations keep running.
When the monitor sees DRBD is Primary on a standby node (that it didn't start),
it triggers a stop action — killing the initial sync after ~10 s.

Enable maintenance-mode after standby stops the existing resources but before
DRBD is manually brought up for the sync.  Maintenance-mode suspends all
start/stop/monitor actions so Pacemaker is completely hands-off during the
sync.  Disable it alongside crm_standby -v off once UpToDate/UpToDate is
confirmed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:49:15 +10:00
beatzaplentyandClaude Sonnet 4.6 d7e63cd80e fix(ha/cluster-init): fix DRBD metadata skip condition and StandAlone grace period
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
The metadata creation check used grep -q "UpToDate" which matches when the
resource is DOWN — drbdadm dstate reads metadata directly and returns just
"UpToDate" (no slash) for a down-but-previously-synced resource.  This caused
metadata creation to be silently skipped, leaving mismatched UUIDs from an
interrupted sync which cause DRBD to go WFConnection→StandAlone immediately.

Fix: require exact "UpToDate/UpToDate" (with slash, only possible when the
resource is UP and both disks are confirmed synced) before skipping.

Also give the StandAlone detection a 5-iteration (15 s) grace period before
dying, to avoid false-positive on the first few ticks while the peer is still
coming up.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:45:03 +10:00
beatzaplentyandClaude Sonnet 4.6 b88ea49880 fix(ha/cluster-init): keep Pacemaker in standby until DRBD sync completes
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
Clearing crm_standby before the initial sync finished caused Pacemaker's OCF
DRBD agent to race with the manual drbdadm up/primary calls.  The agent saw
DRBD in WFConnection or SyncSource and tore it down, driving the resource back
to StandAlone and killing the sync in seconds.

Move the crm_standby -v off calls to immediately after the sync-complete
break, so Pacemaker only resumes once DRBD is UpToDate/UpToDate and safe
to hand back.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:41:15 +10:00
beatzaplentyandClaude Sonnet 4.6 c7268ade8e fix(ha/cluster-init): detect StandAlone DRBD state in sync wait loop
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
If the DRBD peer connection drops during the initial sync wait (cs:StandAlone),
the loop would spin forever printing "waiting for sync progress" with no
indication of what's wrong. Now parses cs: from /proc/drbd each tick and
dies with a clear error if StandAlone is detected. Also shows the cs: field
in the non-syncing message so the actual connection state is always visible.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:29:04 +10:00
beatzaplentyandClaude Sonnet 4.6 367158548e fix(ha): wrap crm-fence-peer.sh to set PATH for kernel UMH callout
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
When the DRBD kernel module invokes fence-peer via the User Mode Helper
mechanism it provides a minimal PATH that omits /run/current-system/sw/bin.
crm-fence-peer.sh calls cibadmin, crm_mon etc.; without those in PATH a
pipeline breaks with SIGPIPE.  A signal-killed process has WEXITSTATUS()==0,
so the kernel sees exit code 0, logs "fence-peer helper broken, returned 0",
and retries forever — blocking Pacemaker failover completely.

Fix: use pkgs.writeShellScript to create thin PATH-fixing wrappers in the
Nix store.  The wrappers prepend /run/current-system/sw/bin before exec-ing
the real crm-fence-peer.sh / crm-unfence-peer.sh, giving them a working
Pacemaker toolchain regardless of what PATH the kernel provides.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 01:22:46 +10:00
beatzaplenty 585126beee secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 01:11:23 +10:00
beatzaplentyandClaude Sonnet 4.6 ceb491b18d fix(ha/cluster-init): drop TTY detection, always use \r for sync progress
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
The TTY check ([[ -t 1 ]]) evaluated false under sudo, causing each update
to print on a new line via log(). Since \r worked correctly in the original
code (user confirmed), drop the branching and always use printf \r.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:55:31 +10:00
beatzaplentyandClaude Sonnet 4.6 6d16eaff89 fix(ha/cluster-init): fix /proc/drbd sync progress parsing and TTY display
Check NixOS configurations / eval-hosts (push) Successful in 10m29s
The sync progress grep was silently failing because /proc/drbd formats the
percentage line as "sync'ed:  5.2%" (two spaces after colon), but the
pattern matched only one.  Use [[:space:]]+ throughout to tolerate any
whitespace.

Also make the display TTY-aware: interactive sessions get the spinning
\r in-place update; piped/logged sessions get a plain log line printed
once per unique percentage point (avoiding scroll spam).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:51:35 +10:00
beatzaplentyandClaude Sonnet 4.6 e085d4707c fix(ha): add DRBD sync progress indicator and pre-flight sync gate
Check NixOS configurations / eval-hosts (push) Successful in 10m22s
cluster-init.sh: replace the fixed 300-iteration sync wait with an
indefinite loop that prints an in-place progress line (%done, ETA, speed)
from /proc/drbd every 3 s. Clears the line with printf \r before logging
completion, so the output stays clean alongside the [cluster-init] log
lines.

acceptance-tests.sh: add a pre-flight check that hard-exits if
drbdadm dstate is not UpToDate/UpToDate, with a hint to the watch
command for monitoring progress. Tests cannot give accurate results
while the initial full-sync is in progress.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:46:57 +10:00
beatzaplenty c8d4440787 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:40:11 +10:00
beatzaplentyandClaude Sonnet 4.6 e7531b276e fix(ha): use crm_mon for Active node detection and add cleanup before VIP wait
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
acceptance-tests.sh: replace drbdadm-role-based Active node detection with
crm_mon-based detection. DRBD role briefly reads Secondary during Pacemaker
transitions, causing T2/T3/T4 to target the wrong node. crm_mon Promoted:/
Masters: lines are authoritative; wait up to 90 s for Pacemaker to settle
before falling back to NODE1.

cluster-init.sh: add crm_resource --cleanup before the VIP wait loop so
stale migration-threshold failure counters from previous cluster-init runs
are cleared before Pacemaker decides resource placement.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:24:47 +10:00
beatzaplenty 49a10d7cc5 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:16:51 +10:00
beatzaplentyandClaude Sonnet 4.6 ead4f55805 fix(ha): use stable by-id disk path for DRBD instead of /dev/sd*
Check NixOS configurations / eval-hosts (push) Failing after 9m44s
/dev/sda and /dev/sdb are assigned by the OS based on Proxmox disk-add
order, which is not consistent across VMs. Use the SCSI controller path
instead — drive-scsi1 is always the dedicated data disk on all HA nodes
regardless of which sda/sdb it gets assigned to.

variables.nix: replace per-node haServer{1,2}DrbdDisk with a single
haServerDrbdDisk using /dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1.

cluster-config.nix: revert to shared volume block (per-node block was
needed for /dev/sd* but the by-id path is identical on both nodes).

cluster-init.sh:
- single DRBD_DISK variable (matching haServerDrbdDisk)
- robust /etc/drbd.conf patch: NixOS manages this file as a symlink to
  a read-only Nix store path; cp --remove-destination breaks the symlink
  before sed -i so the edit actually takes effect
- scp helper script to NODE2 rather than bash -c over SSH to avoid
  quoting complexity

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-29 00:11:13 +10:00
beatzaplenty cac5ec45cc secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-29 00:02:48 +10:00
beatzaplentyandClaude Sonnet 4.6 ec90753a09 fix(ha): per-node DRBD disk assignment (NODE2 data disk is /dev/sda not /dev/sdb)
Check NixOS configurations / eval-hosts (push) Failing after 9m45s
Proxmox VM disk ordering differs between the two HA nodes:
  ha-server-1: sda=OS (50G), sdb=DRBD data (32G)
  ha-server-2: sda=DRBD data (32G), sdb=OS (50G)

The DRBD resource config was using a shared disk=/dev/sdb which targeted
the OS disk on ha-server-2, causing drbdmeta and drbdadm up to operate
on the mounted root filesystem (hence "Device or resource busy").

Changes:
- variables.nix: add haServer1DrbdDisk/haServer2DrbdDisk
- cluster-config.nix: move volume block inside per-host on{} sections
  so each node uses the correct backing disk
- cluster-init.sh: use NODE1_DRBD_DISK/NODE2_DRBD_DISK variables;
  add runtime check that patches /etc/drbd.d/*.res on the running nodes
  if the deployed config points to the wrong disk (workaround for VMs
  built before this fix; redeploy with --force-rebuild to make permanent)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:57:07 +10:00
beatzaplenty df1ddee735 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:48:23 +10:00
beatzaplentyandClaude Sonnet 4.6 8e8261be31 fix(ha/cluster-init): drop bash -c wrapper for NODE2 drbdmeta calls
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
drbdmeta is in /run/current-system/sw/bin which is already on sudo's
PATH on the HA nodes; the bash -c wrapper with PATH manipulation was
adding complexity without benefit and introduced quoting hazards that
likely caused write-dev-uuid to fail or run in the wrong context.

Use direct n2_ssh calls instead — the same pattern that works when
tested manually on NODE2.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:42:36 +10:00
beatzaplenty 5e9541741f secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:30:12 +10:00
beatzaplentyandClaude Sonnet 4.6 07543d7d56 fix(ha/cluster-init): add drbdmeta to PATH and remove openssl dependency
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
drbdmeta lives in the same Nix store dir as drbdadm but sudo doesn't
inherit the full PATH, so drbdmeta was not found (exit 127) even though
drbdadm was.  Resolve drbdmeta's directory from drbdadm's location and
prepend it to PATH.

Replace openssl rand for UUID generation with /proc/sys/kernel/random/uuid
— openssl is not guaranteed to be on PATH in a minimal NixOS root
environment, but /proc/sys/kernel/random/uuid is always present.

Apply the same PATH fix on NODE2 inline in the bash -c invocations that
call drbdmeta over SSH.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:24:01 +10:00
beatzaplentyandClaude Sonnet 4.6 a7c4a24fc3 fix(ha): use drbdmeta --force directly; fix T4 grep-c arithmetic bug
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
cluster-init.sh: replace 'drbdadm create-md --force' with direct drbdmeta
calls using --force on both steps.  drbdadm create-md --force passes --force
to the create-md sub-call but NOT to write-dev-uuid, which then fails when
/dev/sdb is busy and stdin is not a TTY ("stdin not a TTY, not waiting for
confirmation" → exit 20).  write-dev-uuid failing means DRBD has no UUID,
so the peer can't identify the device → connection fails → no sync → wrong
Active node detected by acceptance tests.

acceptance-tests.sh T4: grep -c returns exit code 1 when the count is 0
(no matches), triggering '|| echo "0"' and producing "0\n0" which breaks
[[ "$IQN_COUNT" -ge 1 ]] with "arithmetic syntax error".  Fixed by running
the pipe inside bash -c with '|| true' on the grep to suppress the
non-zero exit code.  Same fix applied to T5's IQN_ON_STANDBY check.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 23:09:06 +10:00
beatzaplenty a98955a8da secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:08:32 +10:00
beatzaplenty 7d60741e73 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 23:00:18 +10:00
beatzaplenty 427be2b287 secrets(ha): refresh sops host-key registrations for new VM instances 2026-07-28 22:55:02 +10:00
beatzaplentyandClaude Sonnet 4.6 684351b89b fix(ha): pre-seed SSH host key in disko image; fix DRBD init race
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
Root cause of recurring sops failures on new VM boots: disko builds raw
disk images, and create-proxmox-resource.sh only syncs the clan-var SSH
host key to the Proxmox node (for proxmox.nix to bake into the image) when
it actually builds — reusing a cached image skips sync_remote_host_keys, so
destroy+recreate reuses a stale image with the wrong or randomly-generated
key baked in.  On first boot the VM gets a different key than what .sops.yaml
was encrypted for, and sops fails permanently.

Fix 1 — deploy.sh Phase 3: always pass --force-rebuild so every VM creation
rebuilds the disko image fresh with the current clan-var key baked in via
NIXOS_HOST_KEYS_DIR (proxmox.nix already reads this under --impure).

Fix 2 — deploy.sh Phase 5.5: after VMs boot, scan their actual ed25519 host
keys and, if they drift from clan vars, update the clan var pub-key files,
rewrite the .sops.yaml age anchors, and re-encrypt all affected sops files.
Defence-in-depth: normally a no-op after Fix 1, but catches any residual
mismatch (e.g. --skip-create-vms reuse of an older image).

Fix 3 — cluster-init.sh: add crm_standby -v on for both nodes before DRBD
metadata init.  Without this, Pacemaker's OCF DRBD agent races: it sees
drbdadm down as a failure and immediately calls drbdadm up again, leaving
/dev/sdb busy when create-md / write-dev-uuid runs (drbdmeta exits 20 with
"stdin not a TTY, not waiting for confirmation").  Standby suppresses
resource scheduling during init; crm_standby -v off restores it after DRBD
is up on both nodes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 22:51:29 +10:00
beatzaplenty 327ff0b44d secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 22:27:29 +10:00
beatzaplenty 43b4cc6aa6 secrets(ha): regenerate IPA keytabs for ha-server-1 and ha-server-2
Check NixOS configurations / eval-hosts (push) Successful in 10m27s
2026-07-28 21:54:00 +10:00
beatzaplentyandClaude Sonnet 4.6 dd984019a1 fix(ha): prevent drbd.service from starting alongside Pacemaker
Check NixOS configurations / eval-hosts (push) Failing after 9m47s
drbd.service runs drbdadm up all at activation time, but Pacemaker's
OCF drbd agent manages the resource lifecycle (up/down/promote/demote).
When both run simultaneously the device is busy, causing drbdmeta
apply-al to abort with exit 20.

Set wantedBy = [] so the kernel module and config are still present
(via services.drbd.enable = true) but systemd doesn't auto-start the
service. Pacemaker's OCF agent calls drbdadm directly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 21:48:43 +10:00
beatzaplenty cb8a51b2fd updated sops keys
Check NixOS configurations / eval-hosts (push) Successful in 10m24s
2026-07-28 21:41:31 +10:00
beatzaplenty 18ab0ff254 secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 21:19:22 +10:00
beatzaplenty 5c5f22f84a +x on scripts and remove beszel from ha for now
Check NixOS configurations / eval-hosts (push) Failing after 5m35s
2026-07-28 20:48:51 +10:00
beatzaplenty 094eaa752b secrets(ha): encrypt corosync authkey generated by cluster-init 2026-07-28 20:35:12 +10:00
beatzaplentyandClaude Sonnet 4.6 fe9fc7364b fix(ha/cluster-init): explicitly start pacemaker after corosync restart
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
At first boot, pacemaker fails with 'Dependency failed' because ipa-activation
fails (IPA not enrolled yet) before corosync/authkey are ready, causing a
systemd boot-ordering race.  The service recovers fine when started manually.
Add an explicit 'systemctl start pacemaker' on both nodes immediately after
restarting corosync so cluster-init doesn't time out waiting for it.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:47:33 +10:00
beatzaplentyandClaude Sonnet 4.6 1ce4589830 fix(ha/deploy): remove _patch_targetctl — iscsi-target.nix now correct
Check NixOS configurations / eval-hosts (push) Successful in 10m23s
The _patch_targetctl runtime dropin was a workaround for the wrong targetctl
binary path in the original iscsi-target.nix (used pkgs.targetcli-fb, but
targetctl is in rtslib-fb's python3 env).  The module is now fixed to use
\${python3}/bin/targetctl, so new VM images have the correct ExecStart/ExecStop
baked in and the patch finds nothing — failing with exit 1 and killing the deploy.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:41:20 +10:00
beatzaplenty 7417cc1b0a Merge pull request 'feat(gui): add wayne's GUI Home Manager config matching the nixos user' (#94) from worktree-functional-mapping-russell into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #94
2026-07-28 09:29:28 +00:00
beatzaplentyandClaude Sonnet 4.6 cf3d8ea9a5 fix(sops): resolve merge conflicts in ha-server keytab files
Check NixOS configurations / eval-hosts (push) Successful in 10m26s
Both secrets/ha-server-{1,2}.keytab had unresolved conflict markers from
the stash/merge interaction.  Keep the upstream version (2 age recipients:
admin + the host's own key) so each host can decrypt its own keytab on boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:26:39 +10:00
beatzaplentyandClaude Sonnet 4.6 756e45743c fix(sops): resolve merge conflict in .sops.yaml keytab recipients
Check NixOS configurations / eval-hosts (push) Successful in 10m21s
The HA branch merge left conflict markers in the ha-server-{1,2}.keytab
path_regex entries.  Keep the upstream side (adds *proxmox-ha-server-{1,2}
as recipients) so each host can decrypt its own IPA keytab on boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:23:07 +10:00
57 changed files with 2215 additions and 899 deletions
+2
View File
@@ -23,3 +23,5 @@ host-keys/
# Temporary Milestone 1 audit checklist (remove-sensetive-info-refactor.md)
# - working notes only, never committed, deleted once every row is rotated.
secrets-inventory.md
.claude/worktrees/
.claude/settings.local.json
+2 -30
View File
@@ -7,23 +7,20 @@ keys:
- &linode-gui age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs
- &linode-minimal age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx
- &linode-nix-cache age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e
- &linode-server age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
- &linode-tailscale-router age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- &lxc-docker age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th
- &lxc-minimal age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k
- &lxc-nix-cache age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7
- &lxc-pxe-boot age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt
- &lxc-server age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
- &lxc-tailscale-router age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- &lxc-tor-relay age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
- &proxmox-docker age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c
- &proxmox-gui age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn
- &proxmox-nix-cache age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
- &proxmox-pxe-boot age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
- &proxmox-server age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
- &proxmox-tailscale-router age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
- &proxmox-ha-server-1 age1nxlnrevqs2msdatze562vz6ym4pgydt2zndye83lwqauy6flggtqrevyw5
- &proxmox-ha-server-2 age1scfc8p53q5aq2a87tcmsazmj8sfeft0s8kxg0et4nm3ucxyp3c3s6te82y
- &proxmox-ha-server-1 age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
- &proxmox-ha-server-2 age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
creation_rules:
# Shared across every currently-deployed host: root/nixos password hash,
@@ -41,20 +38,17 @@ creation_rules:
- *linode-gui
- *linode-minimal
- *linode-nix-cache
- *linode-server
- *linode-tailscale-router
- *lxc-docker
- *lxc-minimal
- *lxc-nix-cache
- *lxc-pxe-boot
- *lxc-server
- *lxc-tailscale-router
- *lxc-tor-relay
- *proxmox-docker
- *proxmox-gui
- *proxmox-nix-cache
- *proxmox-pxe-boot
- *proxmox-server
- *proxmox-tailscale-router
- *proxmox-ha-server-1
- *proxmox-ha-server-2
@@ -67,14 +61,6 @@ creation_rules:
- *lxc-nix-cache
- *proxmox-nix-cache
- path_regex: secrets/server\.yaml$
key_groups:
- age:
- *admin
- *linode-server
- *lxc-server
- *proxmox-server
- path_regex: secrets/tor-relay\.yaml$
key_groups:
- age:
@@ -174,14 +160,6 @@ creation_rules:
- *proxmox-gui
- *linode-gui
- path_regex: secrets/server\.keytab$
key_groups:
- age:
- *admin
- *linode-server
- *lxc-server
- *proxmox-server
- path_regex: secrets/docker\.keytab$
key_groups:
- age:
@@ -210,11 +188,8 @@ creation_rules:
key_groups:
- age:
- *admin
<<<<<<< Updated upstream
- *proxmox-ha-server-1
# proxmox-ha-server-1 added by sync-host-keys.sh
=======
>>>>>>> Stashed changes
# Host keytab for ha-server-2 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
@@ -222,8 +197,5 @@ creation_rules:
key_groups:
- age:
- *admin
<<<<<<< Updated upstream
- *proxmox-ha-server-2
# proxmox-ha-server-2 added by sync-host-keys.sh
=======
>>>>>>> Stashed changes
+7 -10
View File
@@ -21,9 +21,7 @@ machines when deployed.
`modules/installer/common.nix` (the auto-installer's own root/nixos login —
a deliberate, documented choice, see `docs/auto-installer.md`, not
accidental tech debt) and **SSH public keys** in `variables.nix`
(`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`) plus a couple of
per-host `KEY` values for beszel-agent auth (`hosts/server/host.nix`,
`hosts/nix-cache/host.nix`). Don't use the installer's hardcoded hash as a
(`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`, `vars.beszelHubKey`). Don't use the installer's hardcoded hash as a
template for a *real* host — every other host uses sops-nix
(`hashedPasswordFile`, see "Security Notes" in `README.md`). Flag any *new*
secret-like string you encounter instead of committing it.
@@ -375,9 +373,8 @@ removing a host.
- `hosts/<name>/host.nix` — per-machine identity **only**: hostname, hostId,
per-machine secrets, `system.stateVersion`. These files carry no `imports`
of their own beyond narrow parameterized helpers (see
`modules/beszel/host-token.nix` below) — all shared behavior comes from the
platform/build-type modules composed in `flake.nix`, not from the host file.
of their own — all shared behavior comes from the platform/build-type modules
composed in `flake.nix`, not from the host file.
- `modules/platforms/{linode,proxmox,lxc,baremetal}.nix` — platform-specific
config: boot method, guest tooling, and the hardware config, imported
directly by the platform module itself — **not** wired in from
@@ -434,10 +431,10 @@ removing a host.
substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the
full design (per-host local stores, no shared `/nix/store`, and how the
`nixremote` signing/SSH keys fit together).
- `modules/beszel/host-token.nix` — parameterized helper module
(`{ name, sopsFile }`) that wires a host's beszel-agent sops secret/template
and `environmentFile`; used by `hosts/server/host.nix` and
`hosts/nix-cache/host.nix` to avoid duplicating that boilerplate.
- `modules/beszel/enable-agent.nix` — enables beszel-agent, sets `HUB_URL`,
fixes the upstream `StateDirectory` bug, and wires the universal
`beszel-token` sops secret (from `secrets/common.yaml`) into the agent's
`environmentFile`; see `docs/beszel.md` for the full setup guide.
- `modules/tailscale/`, `modules/docker/`, `modules/networking/`,
`modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose,
single-host
+113
View File
@@ -0,0 +1,113 @@
# Beszel agent
[Beszel](https://github.com/henrygd/beszel) is the monitoring dashboard used
in this LAN. The hub runs as a Docker container on `docker.sweet.home` (port
`vars.ports.beszelHub`, 8090). Each monitored NixOS host runs a
`beszel-agent` that connects back to the hub.
---
## How it works
Everything is handled by a single module:
**`modules/beszel/enable-agent.nix`** — imported by a build type. It:
- Enables `beszel-agent`
- Sets `HUB_URL` to `docker.sweet.home:8090`
- Sets `KEY` from `vars.beszelHubKey` (`variables.nix`) — the hub's SSH
public key, shared by every agent. Update `beszelHubKey` if the docker
host is ever rebuilt and the hub generates a new keypair.
- Reads the universal `beszel-token` from `secrets/common.yaml` via sops
and passes it to the agent as `TOKEN` in an env file
- Fixes an upstream bug where the agent couldn't persist its hub-pairing
fingerprint across restarts (adds a real `StateDirectory`)
A host file needs no beszel configuration at all — just import the module
in the build type and add the system in the hub UI.
---
## Adding beszel to a new build type
Add `../beszel/enable-agent.nix` to the `imports` list in
`modules/build-types/<type>.nix`:
```nix
imports = [
../beszel/enable-agent.nix
# ... other imports
];
```
That's the only change required. The host file needs nothing.
---
## Adding a new system to the hub
1. Rebuild and deploy the host with its build type importing `enable-agent.nix`.
2. Open the beszel hub (`http://docker.sweet.home:8090`).
3. Go to **Systems → Add system**, enter the host's IP and the default port
(45876). The agent will connect and the system will appear as active.
---
## One-time setup: add the token to `secrets/common.yaml`
The universal token is stored once in the common secrets file, shared by all
agents. Only needed once, not per-host:
```sh
sops secrets/common.yaml
```
Add:
```yaml
beszel-token: <token from the beszel hub Settings → Keys>
```
`secrets/common.yaml` is already a sops recipient for every host via their
SSH host keys, so no additional sops recipient setup is needed.
---
## Optional: monitoring extra filesystems
To report disk usage for a mount beyond the root filesystem, add
`EXTRA_FILESYSTEMS` in the host file:
```nix
services.beszel.agent.environment = {
EXTRA_FILESYSTEMS = "/mnt/data"; # colon-separated for multiple paths
};
```
The `server` host uses this to expose its ZFS data pool:
```nix
services.beszel.agent.environment = {
EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
LOG_LEVEL = "debug";
};
```
---
## Optional: monitoring Docker containers
`enable-agent.nix` has a commented-out line for Docker monitoring:
```nix
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
```
Uncomment it if the host runs docker-socket-proxy and you want per-container
stats. Hosts without Docker should leave it commented out.
---
## If the hub key changes
If the docker host is ever rebuilt and beszel generates a new SSH keypair,
update `beszelHubKey` in `variables.nix` and rebuild all beszel-enabled hosts.
The new key is visible in the beszel hub under **Settings → Keys**.
+261
View File
@@ -0,0 +1,261 @@
# Storage/Cluster Network Segmentation Audit — pve1.sweet.home
**Date:** 2026-07-29
**Scope:** Read-only discovery of pve1.sweet.home host networking, HA cluster VMs (200/201), and Docker CT (105). No changes made.
> **Implementation status — 2026-07-29:** All recommendations from this audit have been
> implemented in the same session. See `docs/ip-addressing.md` for the current state.
> Key decisions that diverged from the original recommendations:
> - VLAN IDs renumbered: cluster → VLAN 10 (192.168.10.x), storage-client → VLAN 20 (192.168.20.x)
> - Two Pacemaker VIPs: `vip-lan` (192.168.2.229, NFS for LAN) and `vip-storage` (192.168.20.229, NFS + iSCSI for VLAN 20)
> - NFS served on **both** VIPs (each firewalled to its own subnet); iSCSI available on VLAN 20 but NFS is preferred for docker to support future Docker Swarm multi-host access
> - `corosync.conf` ring1 added using LAN IPs (RF-1 resolved)
> - `vmbr2` created and NICs added to HA VMs and docker CT (RF-6/RF-7 resolved)
> - iSCSI portal remains on `[::0]`; firewall enforces VLAN 20 restriction (RF-4 mitigated)
> - STONITH still disabled (RF-3 deferred — accepted risk during development phase)
> - iSCSI ACLs not configured (RF-5 deferred — iSCSI not in active use)
---
## 1. Current State Summary
### pve1.sweet.home Host — Physical NICs
| Interface | Speed/Duplex | Notes |
|-----------|-------------|-------|
| `nic0` | 2500 Mb/s / Full (2.5GbE) | Only active physical NIC; sole bridge port for vmbr0 |
| `nic1` | (not connected / no data) | Present in config, not UP |
| `wlp4s0` | DOWN | WiFi, unused |
No bonding configured. Every guest's traffic ultimately funnels through the single 2.5GbE `nic0`.
### Proxmox Bridges
| Bridge | Physical NIC | Host IP | Subnet | VLAN-aware | Purpose (current) |
|--------|-------------|---------|--------|------------|-------------------|
| `vmbr0` | `nic0` (2.5GbE) | 192.168.2.245/24 | 192.168.2.0/24 | No | General LAN, management, **iSCSI/NFS VIP** |
| `vmbr1` | **none** (internal-only) | — | 192.168.4.224/29 | No | Corosync heartbeat + DRBD replication |
`vmbr1` has `bridge-ports none` in `/etc/network/interfaces.d/vmbr1.conf` — it is a purely software bridge with zero physical uplink. All traffic on it stays inside the hypervisor's memory.
### Guest NIC Assignments
| Guest | VMID | Role | NIC | Bridge | IP | Traffic type |
|-------|------|------|-----|--------|----|--------------|
| nix-cache | CT 102 | Build cache | eth0 | vmbr0 | DHCP/LAN | LAN |
| pxe-boot | CT 103 | PXE/TFTP | eth0 | vmbr0 | DHCP/LAN | LAN |
| tor-relay | CT 104 | Tor | eth0 | vmbr0 | DHCP/LAN | LAN |
| **docker** | **CT 105** | **Docker host** | **eth0** | **vmbr0** | **192.168.2.225/24** | **LAN only** |
| pdm | CT 106 | Proxmox mgmt | eth0 | vmbr0 | 192.168.2.220/24 | LAN |
| server | VM 101 | General server | net0 | vmbr0 | DHCP/LAN | LAN |
| tailscale-router | VM 107 | Tailscale exit | net0 | vmbr0 | DHCP/LAN | LAN |
| domain-controller | VM 108 | FreeIPA | net0 | vmbr0 | 192.168.2.253/24 | LAN |
| **ha-server-1** | **VM 200** | **HA primary** | **net0** | **vmbr0** | **192.168.2.228/24 + VIP 192.168.2.229/24** | **LAN + VIP** |
| **ha-server-1** | **VM 200** | **HA primary** | **net1** | **vmbr1** | **192.168.4.228/29** | **Corosync + DRBD** |
| **ha-server-2** | **VM 201** | **HA secondary** | **net0** | **vmbr0** | **192.168.2.227/24** | **LAN** |
| **ha-server-2** | **VM 201** | **HA secondary** | **net1** | **vmbr1** | **192.168.4.227/29** | **Corosync + DRBD** |
### Corosync/Pacemaker State
- **Transport:** knet/UDP
- **Rings:** 1 only — ring0 on `192.168.4.228` / `192.168.4.227` (vmbr1/ens19)
- **Cluster status:** Both nodes online, DC = ha-server-1, quorum achieved
- **STONITH:** `stonith-enabled: false`
- **no-quorum-policy:** `ignore`
- **Resources (all active on ha-server-1):**
- `ms-drbd0` — promotable DRBD clone (Primary: ha-server-1, Secondary: ha-server-2)
- `xfs-data` — XFS on `/dev/drbd0``/srv/ha-data`
- `iscsi-target` — targetctl service
- `nfs-server` — nfs-server service
- `vip` — IPaddr2 at **192.168.2.229/24** (no `nic=` parameter specified; floats to ens18/vmbr0 automatically based on subnet match)
- **Resource ordering:** ha-group starts only after DRBD is promoted; collocated with Promoted DRBD clone.
### DRBD State
- **Resource:** `ha-data` (DRBD 8.4.11 kernel module, config in `/etc/drbd.conf`)
- **Protocol:** C (synchronous)
- **Replication endpoints:**
- ha-server-1: `192.168.4.228:7789` (ens19 / vmbr1)
- ha-server-2: `192.168.4.227:7789` (ens19 / vmbr1)
- **State at audit time:** Initial sync in progress — ~20% complete, ~40 MB/s, ~34 min remaining (100 GB disk)
- **Fencing config:** `fencing resource-only` + fence-peer/unfence-peer handlers
### iSCSI Target
- **IQN:** `iqn.2026-01.home.sweet:ha-storage`
- **Portal:** `[::0]:3260` — confirmed listening on all interfaces (`ss -tnlp` shows `*:3260 *:*`)
- **LUN 0:** fileio backstore — `/srv/ha-data/iscsi-lun.img` (10 GiB, write-thru)
- **ACLs:** **None** (`no-gen-acls`, `no-auth`)
- **VIP (intended portal):** 192.168.2.229 — on vmbr0/LAN, no storage-NIC-specific binding
### NFS Exports
Served from the same `ha-group` as iSCSI (starts/stops together):
| Export path | Client subnet |
|------------|---------------|
| `/srv/ha-data/docker/{config,databases,volumes,nextcloud-data}` | 192.168.2.0/24 |
| `/srv/ha-data/proxmox/{iso,lxc}` | 192.168.2.0/24 |
| `/srv/ha-data/pxe-boot/images` | 192.168.2.0/24 |
| `/srv/ha-data/raspi/volumes` | 192.168.2.0/24 |
All NFS exports restrict to 192.168.2.0/24 and are served via the VIP at 192.168.2.229. `rw,sync,no_subtree_check,no_root_squash`.
### Docker Host Current State
- **Single NIC:** eth0 on vmbr0, 192.168.2.225/24, gateway 192.168.2.254
- **Route to storage network (192.168.4.x):** none — no NIC and no route
- **iSCSI sessions:** none
- **iSCSI nodes discovered:** none
- **Docker networks:** several active compose-project networks (core_traefik, core_nextcloud, core_passbolt, core_gramps, core_docker-socket-proxy, plus CI isolation networks)
---
## 2. Risk Flags
### RF-1: Corosync has only one ring (no heartbeat path redundancy)
`corosync.conf` defines only `ring0_addr` for each node, using 192.168.4.x on vmbr1. No `ring1_addr` / second knet link is configured. On a single Proxmox host, vmbr1 is a software bridge (no physical NIC), so physical link failure is not the concern — but a kernel network stack hiccup, a `pveproxy` restart dropping bridge state, or vmbr1 getting disrupted during heavy DRBD sync all leave corosync with zero fallback path. Missed heartbeats on a two-node cluster with `no-quorum-policy: ignore` do not cause a clean shutdown; they cause a false failover or split-brain.
Adding the LAN addresses (192.168.2.228 / 192.168.2.227 via ens18/vmbr0) as a second knet link would provide a backup path with no infrastructure changes needed.
### RF-2: Corosync heartbeat and DRBD replication share vmbr1 — no isolation between them
Both corosync (knet/UDP, ~1 kB heartbeat packets every ~100 ms) and DRBD replication (protocol C, synchronous, syncing at ~40 MB/s on a 100 GB initial fill at audit time) traverse the same `vmbr1` virtual bridge and terminate on the same ens19 NIC pair inside each HA VM. Under heavy DRBD write load, the guest-kernel scheduler's NIC transmit queue processes both flows together. While corosync's heartbeat is tiny, the absence of QoS/priority marking on vmbr1 means a DRBD burst can delay a heartbeat enough to trigger a ring fault warning. This is a latent risk that grows under high-write workloads.
### RF-3: STONITH disabled — split-brain protection relies solely on DRBD's resource-only fencing
`stonith-enabled: false` in the CIB. With `no-quorum-policy: ignore`, both nodes will continue running if corosync loses communication. DRBD's `fencing resource-only` does call `fence-peer` before allowing a Primary promotion, which provides some protection, but there is no hard external power fence to guarantee the other node actually stops. In a real split-brain (both nodes believe they are Primary), data corruption on the shared XFS filesystem is possible. **This is the highest-severity risk in the current setup.**
Getting STONITH to work on Proxmox-hosted VMs requires either a `fence_pve` agent (Proxmox API fencing) or `fence_virtd` (QEMU guest agent fencing). Neither is configured.
### RF-4: iSCSI portal bound to `[::0]:3260` — listens on every interface, not just the VIP
The targetcli portal is `[::0]:3260` (confirmed: `*:3260 *:*` in ss). This means the target is reachable on:
- 192.168.2.229 (VIP — correct, failover-safe)
- 192.168.2.228 (ha-server-1 LAN IP — does **not** move during failover; an initiator session connecting here would break on failover)
- 192.168.4.228 (storage NIC — not reachable by the Docker host today, but unintentionally exposed)
Binding the portal explicitly to the VIP IP instead of wildcard eliminates the non-VIP reachability risks.
### RF-5: iSCSI has zero ACLs and no authentication
`targetcli ls` shows `acls: 0`, `no-gen-acls`, `no-auth`. Any host that can reach port 3260 on any of the above IPs can log into the LUN with no credentials. The Docker host is not yet configured as an initiator — but neither is it blocked.
### RF-6: Docker host has no path to the storage network — iSCSI would traverse vmbr0/nic0
CT 105 (docker, 192.168.2.225) has one NIC, on vmbr0. To reach the VIP at 192.168.2.229, iSCSI traffic would travel:
```
docker (eth0/vmbr0) → nic0 (2.5GbE) → vmbr0 → tap200i0 (VM 200 net0/ens18)
```
All of the following share this same path over vmbr0 → nic0:
- Docker container traffic (outbound and inter-container)
- CI/CD runner traffic (Gitea Actions jobs visible in `docker network ls`)
- NFS mounts from pxe-boot, proxmox host itself, and other LAN clients
- iSCSI block traffic (protocol-sensitive to latency and retransmit)
A Nextcloud upload or a CI `nix build` job can saturate nic0 and starve the iSCSI session, causing command timeouts and filesystem errors on the Docker host.
### RF-7: VIP is on the LAN interface with no storage-specific binding
The Pacemaker `vip` resource specifies `ip=192.168.2.229, cidr_netmask=24` with no `nic=` parameter. Pacemaker's IPaddr2 agent selects the interface by longest-prefix match, landing it on ens18 (vmbr0/LAN). There is no way to keep this VIP from competing with general LAN traffic on nic0 without moving the VIP to a separate subnet on a different virtual bridge.
---
## 3. Recommended Target Layout
### Design constraints
- Single Proxmox host: all traffic ultimately shares nic0's bandwidth. The goal is QoS partitioning via separate bridges and subnets, not true physical isolation.
- Future physical split: bridge/VLAN IDs chosen here should map cleanly to physical uplink VLAN tags when the HA nodes move to separate hardware.
### Proposed bridge layout
| Bridge | Physical port | VLAN tag (future) | Subnet | Purpose |
|--------|-------------|-------------------|--------|---------|
| `vmbr0` | nic0 | untagged / VLAN 1 | 192.168.2.0/24 | **LAN/management only** — no storage traffic |
| `vmbr1` | (none / VLAN 10 on future trunk) | VLAN 10 | 192.168.4.224/29 | **Corosync heartbeat + DRBD replication** (current, keep) |
| `vmbr2` *(new)* | (none / VLAN 20 on future trunk) | VLAN 20 | 192.168.5.0/24 | **Storage: iSCSI + NFS client access** |
This is the minimum-disruption path: vmbr1 stays as-is (no DRBD reconfiguration needed), and the new vmbr2 gives the Docker host a direct path to the storage VIP without crossing vmbr0.
If stricter isolation is later desired, DRBD can be migrated from vmbr1 to vmbr2 in a separate maintenance window (see §5), leaving vmbr1 as corosync-only.
### Per-guest NIC assignments in target layout
| Guest | VMID | NIC | Bridge | Proposed IP | Purpose |
|-------|------|-----|--------|-------------|---------|
| ha-server-1 | VM 200 | net0 | vmbr0 | 192.168.2.228/24 | LAN/management (keep) |
| ha-server-1 | VM 200 | net1 | vmbr1 | 192.168.4.228/29 | Corosync + DRBD (keep) |
| ha-server-1 | VM 200 | **net2 (new)** | **vmbr2** | **192.168.5.1/24** | iSCSI + NFS storage client |
| ha-server-2 | VM 201 | net0 | vmbr0 | 192.168.2.227/24 | LAN/management (keep) |
| ha-server-2 | VM 201 | net1 | vmbr1 | 192.168.4.227/29 | Corosync + DRBD (keep) |
| ha-server-2 | VM 201 | **net2 (new)** | **vmbr2** | **192.168.5.2/24** | iSCSI + NFS storage client |
| docker | CT 105 | net0 | vmbr0 | 192.168.2.225/24 | LAN/management (keep) |
| docker | CT 105 | **net1 (new)** | **vmbr2** | **192.168.5.10/24** | iSCSI + NFS |
**VIP target:** `192.168.5.100/24` on vmbr2. The Pacemaker `vip` resource changes from `ip=192.168.2.229` to `ip=192.168.5.100, nic=<ens20>` (whichever name the new NIC gets inside the HA VMs). The existing `192.168.2.229` LAN VIP can optionally be retained as a separate static alias on ens18 for management-plane access, but should not be the iSCSI portal target.
**iSCSI portal:** Bind to `192.168.5.100:3260` instead of `[::0]:3260`. In targetcli: remove the wildcard portal, add `portals/ create 192.168.5.100`.
**NFS exports:** NFS is a file-level protocol and is fine being accessed over a routed path. After the VIP moves, non-docker LAN clients (proxmox host, pxe-boot, raspi) can reach NFS either via a static route to 192.168.5.0/24 or by keeping a secondary static alias at 192.168.2.229 on ens18 dedicated to NFS. Either approach works — NFS handles reconnect gracefully in ways iSCSI block I/O cannot.
**Corosync second ring (independent, low-disruption improvement):**
Add a second knet link using the LAN addresses as a backup heartbeat path. Edit `corosync.conf` on both nodes:
```
node { ring0_addr: 192.168.4.228; ring1_addr: 192.168.2.228; name: ha-server-1; nodeid: 1; }
node { ring0_addr: 192.168.4.227; ring1_addr: 192.168.2.227; name: ha-server-2; nodeid: 2; }
```
Requires a corosync service restart (brief cluster pause, ~5 seconds), no interface or bridge changes.
**Future physical-host split:**
When ha-server-1 and ha-server-2 move to separate physical machines, vmbr1 and vmbr2 become VLAN-tagged sub-interfaces on a physical trunk (e.g. VLAN 10 → cluster, VLAN 20 → storage). The bridge/subnet/IP layout above is designed so the tag numbers can be layered onto the existing addresses without renumbering.
---
## 4. Gap List
| Gap | Action needed |
|-----|--------------|
| `vmbr2` does not exist on pve1 | Create internal bridge: `/etc/network/interfaces.d/vmbr2.conf` with `bridge-ports none`, `inet manual` |
| VM 200 and VM 201 have no net2 | `qm set 200 --net2 virtio,bridge=vmbr2` / `qm set 201 --net2 virtio,bridge=vmbr2` (hot-plug, no reboot needed) |
| CT 105 has no net1 | `pct set 105 --net1 name=eth1,bridge=vmbr2,ip=192.168.5.10/24` |
| HA VMs have no OS config for the new NIC | NixOS `networking.interfaces.<ens20>` with `ipv4.addresses = [{address="192.168.5.1"; prefixLength=24;}]` per host (name may differ — check `ip link` after hotplug) |
| VIP needs to move to 192.168.5.100 on vmbr2 | `pcs resource update vip ip=192.168.5.100 cidr_netmask=24 nic=<ens20>` |
| iSCSI portal bound to `[::0]` | `targetcli /iscsi/iqn.2026-01.home.sweet:ha-storage/tpg1/portals delete ::0 3260` then `create 192.168.5.100`; save and restart via `pcs resource restart iscsi-target` |
| iSCSI ACLs empty | Get Docker initiator IQN via `iscsiadm -m iface` on CT 105, then add via targetcli `acls/ create <iqn>` |
| Docker host has no iSCSI initiator config | `iscsiadm -m discoverydb -t sendtargets -p 192.168.5.100 -D` then `iscsiadm -m node -l` once ACLs are set |
| Corosync single ring | Add `ring1_addr` entries in `corosync.conf` using LAN IPs; restart corosync cluster-wide (one node at a time) |
| STONITH not configured | Evaluate `fence_pve` (Proxmox API agent); document accepted risk if deferred |
| DRBD still on vmbr1 (optional, separate window) | Stop ms-drbd0 via pcs, edit `/etc/drbd.conf` on both nodes (change `192.168.4.x``192.168.5.x`), restart DRBD, re-enable via pcs |
---
## 5. Migration Notes
### Non-disruptive (no service impact)
- **Create vmbr2 on pve1:** Bridge definition edit only; no effect on existing bridges or guests.
- **Hot-add net2 to VMs 200/201:** Proxmox allows adding a NIC without reboot (`qm set 200 --net2 ...`). The NIC appears inside the VM immediately via QEMU hotplug but will be unconfigured (down) inside NixOS until the NixOS config is deployed — no impact on running services.
- **Add net1 to CT 105:** LXC NIC hotplug works similarly; CT does not need to restart.
- **Add corosync ring1:** Requires `systemctl restart corosync` on both nodes (one at a time). Pacemaker briefly sees corosync go offline and recover; with two nodes and `wait_for_all: 0`, this typically completes in under 5 seconds and resources stay running.
### Disruptive — requires maintenance window
- **Move VIP from 192.168.2.229 to 192.168.5.100:** `pcs resource update vip ip=192.168.5.100` causes Pacemaker to immediately stop the old VIP and start the new one. Any NFS mounts referencing 192.168.2.229 will stall until remounted at the new address (or a static alias is added at 192.168.2.229 on ens18). No iSCSI sessions exist yet, so no iSCSI disruption.
- **Change iSCSI portal from `[::0]` to VIP-specific:** Requires `pcs resource restart iscsi-target` after the targetcli portal change — brief target unavailability. Any initiator sessions (once configured) will need to re-login.
- **Migrate DRBD replication from 192.168.4.x to 192.168.5.x** (optional — only needed to give corosync sole ownership of vmbr1):
1. `pcs resource disable ms-drbd0` — demotes DRBD Primary, stops ha-group (unmounts XFS, stops iSCSI + NFS + VIP)
2. `drbdadm down ha-data` on both nodes
3. Edit `/etc/drbd.conf` on both nodes (change `address` lines)
4. `drbdadm up ha-data` on both nodes
5. `pcs resource enable ms-drbd0` — Pacemaker re-promotes, mounts, starts services
DRBD does **not** require a full resync when only the address changes — the disk data and metadata are unchanged; only the TCP connection endpoint changes. However, the initial sync was in progress at audit time (~20% at ~40 MB/s). Recommend waiting for that sync to complete before scheduling this migration.
+115 -18
View File
@@ -2,16 +2,88 @@
## Subnets
| Subnet | CIDR | Purpose | Routed? |
|---|---|---|---|
| LAN | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
| Storage | `192.168.4.0/29` | HA file server DRBD replication | No — internal `vmbr1` only, no uplink |
| Subnet | VLAN | CIDR | Purpose | Routed? |
|---|---|---|---|---|
| LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
| Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
| Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink |
The storage subnet never leaves pve1. `vmbr1` is a Proxmox Linux bridge with no physical port
attached; traffic between the two HA file server VMs stays in-kernel.
The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
The host octet is consistent across subnets for any host that has multiple interfaces — e.g.
ha-node1 is always `.228` (LAN: `192.168.2.228`, storage: `192.168.4.228`).
VLAN IDs match the third octet of each subnet (VLAN 2 → 192.168.**2**.x, VLAN 10 → 192.168.**10**.x,
VLAN 20 → 192.168.**20**.x). The host octet is consistent across all subnets — e.g. ha-node1
is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20.228` (storage client).
**Protocol separation** (enforced by firewall on HA nodes):
- NFS (ports 111, 2049, 20048): both subnets, each restricted to its own CIDR
- VLAN 2 only → `vip-lan` (192.168.2.229) — pxe-boot and other LAN clients
- VLAN 20 only → `vip-storage` (192.168.20.229) — docker, future swarm nodes
- iSCSI (port 3260): VLAN 20 only — available but not in active use; NFS is preferred
for multi-host access (shared volumes across a Docker Swarm require a shared filesystem,
not per-host block devices)
---
## DNS Zones
FreeIPA (domain-controller.sweet.home) is authoritative for all zones. Three
zones correspond to the three subnets — one per VLAN. All zones are internal
only; no external delegation.
### sweet.home — VLAN 2 (192.168.2.x)
General LAN zone. All infrastructure hostnames live here.
| Hostname | A record | Notes |
|---|---|---|
| `domain-controller.sweet.home` | `192.168.2.253` | FreeIPA / KDC / DNS |
| `ha-vip-lan.sweet.home` | `192.168.2.229` | Pacemaker `vip-lan` — NFS for LAN clients |
| `ha-server-1.sweet.home` | `192.168.2.228` | HA node 1 management NIC |
| `ha-server-2.sweet.home` | `192.168.2.227` | HA node 2 management NIC |
| `server.sweet.home` | `192.168.2.226` | Current ZFS/NFS server (retiring) |
| `docker.sweet.home` | `192.168.2.225` | Docker/Traefik host |
| `nix-cache.sweet.home` | `192.168.2.224` | Nix binary cache + remote builder |
| `pxe-boot.sweet.home` | `192.168.2.223` | PXE / TFTP / HTTP netboot |
| `tailscale-router.sweet.home` | `192.168.2.222` | Tailscale exit node |
| `tor-relay.sweet.home` | `192.168.2.221` | Tor relay |
| `pdm.sweet.home` | `192.168.2.220` | Proxmox Deploy Manager |
| `nixos.sweet.home` | `192.168.2.39` | Bare-metal workstation (DHCP) |
| `pve1.sweet.home` | `192.168.2.245` | Proxmox VE hypervisor |
| `pbs.sweet.home` | `192.168.2.244` | Proxmox Backup Server |
PTR records exist for all static hosts. The workstation (`nixos.sweet.home`) is
DHCP-assigned; its PTR is omitted.
### cluster.home — VLAN 10 (192.168.10.x)
Internal only — Corosync ring0 heartbeat and DRBD replication between HA nodes.
No VIP exists on this subnet (DRBD/Corosync endpoints are static per-node IPs).
| Hostname | A record | Notes |
|---|---|---|
| `ha-server-1.cluster.home` | `192.168.10.228` | HA node 1 cluster NIC (ens19 / vmbr1) |
| `ha-server-2.cluster.home` | `192.168.10.227` | HA node 2 cluster NIC (ens19 / vmbr1) |
PTR records exist for both. DNS here is for debugging convenience — DRBD and
Corosync use the IPs from the NixOS config directly, not DNS.
### storage.home — VLAN 20 (192.168.20.x)
Internal only — NFS (and iSCSI) client access to the HA storage VIP. NFS clients
mount from **`nfs.storage.home`** (the Pacemaker floating VIP) so mounts survive
failover transparently without reconfiguration.
| Hostname | A record | Notes |
|---|---|---|
| `nfs.storage.home` | `192.168.20.229` | Pacemaker `vip-storage` — NFS + iSCSI VIP |
| `ha-server-1.storage.home` | `192.168.20.228` | HA node 1 storage-client NIC (ens20 / vmbr2) |
| `ha-server-2.storage.home` | `192.168.20.227` | HA node 2 storage-client NIC (ens20 / vmbr2) |
| `docker.storage.home` | `192.168.20.225` | Docker host storage-client NIC (eth1 / vmbr2) |
| `server.storage.home` | `192.168.20.226` | server VM storage-client NIC (decommissioned — remove DNS record after VM is destroyed) |
PTR records exist for all five. Remove `server.storage.home`, `server.sweet.home`,
and their PTRs from FreeIPA DNS once the server VM is destroyed.
---
@@ -56,10 +128,10 @@ All VMs and LXC containers run on pve1.
| IP | Hostname | Role | Status |
|---|---|---|---|
| `192.168.2.229` | ha-vip | HA file server iSCSI floating VIP (Pacemaker) | Future |
| `192.168.2.228` | ha-node1 | HA file server node 1 (DRBD + XFS + iSCSI) | Future |
| `192.168.2.227` | ha-node2 | HA file server node 2 (DRBD + XFS + iSCSI) | Future |
| `192.168.2.226` | server | Current NFS/ZFS file server — retires when HA is live | Retiring |
| `192.168.2.229` | ha-vip-lan | HA file server LAN floating VIP (Pacemaker `vip-lan`) — LAN iSCSI + NFS | Active |
| `192.168.2.228` | ha-node1 | HA file server node 1 — management NIC | Active |
| `192.168.2.227` | ha-node2 | HA file server node 2 — management NIC | Active |
| `192.168.2.226` | server | Former NFS/ZFS file server — decommissioned | Removed from flake |
| `192.168.2.225` | docker | Docker / Traefik stack | Active |
| `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
| `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active |
@@ -78,16 +150,40 @@ MAC address churn.
---
## Storage network — 192.168.4.0/29
## Cluster network — VLAN 10 — 192.168.10.224/29
Internal to pve1 only. Proxmox bridge `vmbr1`, no physical NIC attached.
| IP | Hostname | Interface role |
|---|---|---|
| `192.168.4.228` | ha-node1 | DRBD replication NIC |
| `192.168.4.227` | ha-node2 | DRBD replication NIC |
| `192.168.10.228` | ha-node1 | DRBD replication + Corosync ring0 (primary heartbeat) |
| `192.168.10.227` | ha-node2 | DRBD replication + Corosync ring0 (primary heartbeat) |
| — | no gateway | Isolated — not routed to LAN or internet |
Corosync ring1 (backup heartbeat only) uses the LAN IPs (`192.168.2.228` / `192.168.2.227`)
over `vmbr0` — no additional bridge needed, and DRBD traffic never crosses ring1.
---
## Storage-client network — VLAN 20 — 192.168.20.0/24
Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached.
| IP | Hostname | Interface / role |
|---|---|---|
| `192.168.20.229` | ha-vip-storage | Pacemaker floating VIP — NFS + iSCSI endpoint |
| `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
| `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
| `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — decommissioned |
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client |
| — | no gateway | Isolated — not routed to LAN or internet |
NFS clients mount from `192.168.20.229` (surviving failover transparently via the VIP).
Firewall on each HA node restricts NFS and iSCSI ports to `192.168.20.0/24` — LAN hosts
cannot reach either service on this VIP. The `vip-storage` endpoint is not reachable
from the workstation directly (internal bridge only); health checks proxy through the
active HA node.
---
## Migration reference
@@ -101,9 +197,10 @@ Current → target IP for every host being renumbered.
| pve1 | `192.168.2.250` | `192.168.2.245` | `/etc/network/interfaces` on Proxmox host |
| pbs | `192.168.2.108` | `192.168.2.244` | static config on PBS host |
| nixos workstation | `192.168.2.119` | `192.168.2.243` | `networking.interfaces` / NetworkManager on guest |
| ha-node1 | — | `192.168.2.228` | future |
| ha-node2 | — | `192.168.2.227` | future |
| ha-vip | — | `192.168.2.229` | future (Pacemaker resource) |
| ha-node1 | — | `192.168.2.228` (LAN), `192.168.10.228` (cluster/VLAN 10), `192.168.20.228` (storage/VLAN 20) | active |
| ha-node2 | — | `192.168.2.227` (LAN), `192.168.10.227` (cluster/VLAN 10), `192.168.20.227` (storage/VLAN 20) | active |
| ha-vip-lan | — | `192.168.2.229` (vmbr0 / Pacemaker `vip-lan`) — NFS endpoint for LAN clients | active |
| ha-vip-storage | — | `192.168.20.229` (vmbr2 / Pacemaker `vip-storage`) — iSCSI endpoint for VLAN 20 clients | active |
| server | `192.168.2.252` | `192.168.2.226` | static config on guest |
| docker | `192.168.2.249` | `192.168.2.225` | static config on guest |
| nix-cache | `192.168.2.120` | `192.168.2.224` | static config on guest |
+83 -1
View File
@@ -336,7 +336,89 @@ temporarily use either.
---
## Stage 6 — Final cleanup
## Stage 6 — HA storage cutover (docker NFS remount)
> **Prerequisites:**
> - HA cluster fully deployed and `vip-storage` (`nfs.storage.home` → 192.168.20.229) serving NFS ✓
> - DNS configured: `storage.home` zone populated, `nfs.storage.home` resolves to 192.168.20.229 ✓
> - docker CT has eth1 on vmbr2 (`docker.storage.home` → 192.168.20.225) ✓
> - Final rsync from server.sweet.home to `/srv/ha-data` complete before step 6b
docker.sweet.home currently NFS-mounts its persistent volumes from `server.sweet.home`
(`192.168.2.226:/tank/docker/...`). This stage moves those mounts to the HA cluster's
storage VIP so server can be decommissioned.
### 6a. Final rsync from server to HA cluster
Run from server.sweet.home (or over SSH from the workstation) to sync any data written
since the initial rsync:
```bash
# Confirm active HA node and mount point
ssh wayne@192.168.2.228 'sudo findmnt /srv/ha-data' # check which node is active
# rsync each dataset (adjust source paths to match /tank layout on server)
sudo rsync -av --delete /tank/docker/config/ wayne@<active-node-ip>:/srv/ha-data/docker/config/
sudo rsync -av --delete /tank/docker/databases/ wayne@<active-node-ip>:/srv/ha-data/docker/databases/
sudo rsync -av --delete /tank/docker/volumes/ wayne@<active-node-ip>:/srv/ha-data/docker/volumes/
sudo rsync -av --delete /tank/docker/nextcloud-data/ wayne@<active-node-ip>:/srv/ha-data/docker/nextcloud-data/
```
### 6b. Update docker NixOS config to mount from vip-storage
In `hosts/docker/host.nix` (or wherever the NFS mount fileSystems are declared), change
the NFS server from `server.sweet.home` / `192.168.2.226` to `nfs.storage.home`:
```nix
# Before:
fileSystems."/mnt/docker/config" = {
device = "server:/tank/docker/config"; # or 192.168.2.226:...
...
};
# After:
fileSystems."/mnt/docker/config" = {
device = "nfs.storage.home:/srv/ha-data/docker/config";
...
};
```
Using the DNS name (`nfs.storage.home`) rather than the VIP IP means the mount
config survives a future VIP renumber without touching the NixOS config.
Repeat for all four docker shares (`config`, `databases`, `volumes`, `nextcloud-data`).
Then rebuild docker:
```bash
# On the workstation — or via Switch-nix on docker itself
sudo nixos-rebuild switch --no-write-lock-file --refresh \
--flake "git+https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos.git#lxc-docker"
```
### 6c. Verify mounts and container health
```bash
ssh wayne@192.168.2.225 'findmnt | grep 192.168.20' # mounts should show vip-storage
ssh wayne@192.168.2.225 'docker ps' # all containers running
```
Spot-check Nextcloud, Traefik, and any database containers for connectivity.
### 6d. Decommission server.sweet.home
Once docker is confirmed healthy on the HA NFS mounts:
```bash
# Stop server VM on pve1
ssh wayne@192.168.2.245 'sudo qm stop 101'
# (Optional) Archive the ZFS pool snapshot before destroying
# Then after a settling period:
ssh wayne@192.168.2.245 'sudo qm destroy 101 --destroy-unreferenced-disks 1'
```
---
## Stage 7 — Final cleanup
Once all hosts are at their new IPs and verified:
Generated
+3 -3
View File
@@ -259,11 +259,11 @@
},
"nixpkgs_2": {
"locked": {
"lastModified": 1785104993,
"narHash": "sha256-eKbrvPoAOFutbYMdbB3r5EQVmFxKv24iKqHPPUXA0gM=",
"lastModified": 1785133411,
"narHash": "sha256-Yjv0WEg39KRYS0rBdTbu6Fc/or/ihAKk13W9sQ6VWd0=",
"owner": "NixOS",
"repo": "nixpkgs",
"rev": "8623c4c20aa4ca2f5fb81510d2944066c3fb0d96",
"rev": "2f5a153c270b70cb0f8c11f46d96d6d3bc39f4e3",
"type": "github"
},
"original": {
+4 -7
View File
@@ -45,9 +45,9 @@
# (hostName, hostId, per-machine secrets). Every build type except
# nix-cache itself consumes the nix-cache substituter and remote
# builder.
mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix }:
mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix, nameSuffix ? "" }:
let
flakeTarget = "${platform}-${buildType}";
flakeTarget = "${platform}-${buildType}${nameSuffix}";
in
nixpkgs.lib.nixosSystem {
inherit system;
@@ -109,9 +109,6 @@
proxmox-nix-cache = mkTarget { platform = "proxmox"; buildType = "nix-cache"; hostPath = ./hosts/nix-cache/host.nix; };
lxc-nix-cache = mkTarget { platform = "lxc"; buildType = "nix-cache"; hostPath = ./hosts/nix-cache/host.nix; };
linode-server = mkTarget { platform = "linode"; buildType = "server"; hostPath = ./hosts/server/host.nix; };
proxmox-server = mkTarget { platform = "proxmox"; buildType = "server"; hostPath = ./hosts/server/host.nix; };
lxc-server = mkTarget { platform = "lxc"; buildType = "server"; hostPath = ./hosts/server/host.nix; };
linode-docker = mkTarget { platform = "linode"; buildType = "docker"; hostPath = ./hosts/docker/host.nix; };
proxmox-docker = mkTarget { platform = "proxmox"; buildType = "docker"; hostPath = ./hosts/docker/host.nix; };
@@ -131,8 +128,8 @@
lxc-tor-relay = mkTarget { platform = "lxc"; buildType = "tor-relay"; hostPath = ./hosts/tor-relay/host.nix; };
proxmox-ha-server-1 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-1/host.nix; };
proxmox-ha-server-2 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-2/host.nix; };
proxmox-ha-server-1 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-1/host.nix; nameSuffix = "-1"; };
proxmox-ha-server-2 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-2/host.nix; nameSuffix = "-2"; };
};
# Auto-install environments (migrated from the former nix-auto-installer
+4 -4
View File
@@ -5,10 +5,10 @@
hostName = "docker";
hostId = "007f0200";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.dockerIp;
prefixLength = vars.lanPrefixLength;
}];
interfaces = {
${vars.vmLanInterface}.ipv4.addresses = [{ address = vars.dockerIp; prefixLength = vars.lanPrefixLength; }];
${vars.lxcStorageInterface}.ipv4.addresses = [{ address = vars.dockerStorageIp; prefixLength = vars.haClientPrefixLength; }];
};
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
+5 -18
View File
@@ -1,30 +1,17 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "ha-server-1";
sopsFile = ../../secrets/ha-server-1.yaml;
})
];
networking = {
hostName = vars.haServer1Host;
hostId = "3a4b5c6d";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.haServer1Ip;
prefixLength = vars.lanPrefixLength;
}];
interfaces.${vars.vmStorageInterface}.ipv4.addresses = [{
address = vars.haServer1StorageIp;
prefixLength = vars.haStoragePrefixLength;
}];
interfaces = {
${vars.vmLanInterface}.ipv4.addresses = [{ address = vars.haServer1Ip; prefixLength = vars.lanPrefixLength; }];
${vars.vmStorageInterface}.ipv4.addresses = [{ address = vars.haServer1StorageIp; prefixLength = vars.haStoragePrefixLength; }];
${vars.vmStorageClientInterface}.ipv4.addresses = [{ address = vars.haServer1ClientIp; prefixLength = vars.haClientPrefixLength; }];
};
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# Set KEY after pairing this host with the beszel hub; the token is sops-managed.
services.beszel.agent.environment.KEY = "";
system.stateVersion = "26.05";
}
+5 -18
View File
@@ -1,30 +1,17 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "ha-server-2";
sopsFile = ../../secrets/ha-server-2.yaml;
})
];
networking = {
hostName = vars.haServer2Host;
hostId = "7e8f9a0b";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.haServer2Ip;
prefixLength = vars.lanPrefixLength;
}];
interfaces.${vars.vmStorageInterface}.ipv4.addresses = [{
address = vars.haServer2StorageIp;
prefixLength = vars.haStoragePrefixLength;
}];
interfaces = {
${vars.vmLanInterface}.ipv4.addresses = [{ address = vars.haServer2Ip; prefixLength = vars.lanPrefixLength; }];
${vars.vmStorageInterface}.ipv4.addresses = [{ address = vars.haServer2StorageIp; prefixLength = vars.haStoragePrefixLength; }];
${vars.vmStorageClientInterface}.ipv4.addresses = [{ address = vars.haServer2ClientIp; prefixLength = vars.haClientPrefixLength; }];
};
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
# Set KEY after pairing this host with the beszel hub; the token is sops-managed.
services.beszel.agent.environment.KEY = "";
system.stateVersion = "26.05";
}
-11
View File
@@ -1,13 +1,6 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "nix-cache";
sopsFile = ../../secrets/nix-cache.yaml;
})
];
networking = {
hostName = vars.nixCacheHost;
useDHCP = false;
@@ -19,10 +12,6 @@
nameservers = [ vars.domainControllerIp ];
};
services.beszel.agent.environment = {
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
};
# Preserved from the pre-refactor `nix-cache` target — stateVersion must
# never be bumped on an already-installed machine.
system.stateVersion = "25.05";
+1 -1
View File
@@ -26,7 +26,7 @@
# Optional: set environment vars
sessionVariables = {
EDITOR = "vim";
EDITOR = "nano";
SOPS_AGE_KEY_FILE = "${config.home.homeDirectory}/.config/sops/age/keys.txt";
};
+3 -1
View File
@@ -11,7 +11,9 @@
defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
services.beszel.agent.environment = {
# KEY = "";
};
# Preserved from the pre-refactor `pxe-boot` target — stateVersion must
# never be bumped on an already-installed machine.
system.stateVersion = "25.05";
-32
View File
@@ -1,32 +0,0 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "server";
sopsFile = ../../secrets/server.yaml;
})
];
networking = {
hostName = vars.nfsServerHost;
hostId = "6689f93e";
useDHCP = false;
interfaces.${vars.vmLanInterface}.ipv4.addresses = [{
address = vars.serverIp;
prefixLength = vars.lanPrefixLength;
}];
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
nameservers = [ vars.domainControllerIp ];
};
services.beszel.agent.environment = {
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
LOG_LEVEL = "debug";
};
# Preserved from the pre-refactor `server` target — stateVersion must never
# be bumped on an already-installed machine.
system.stateVersion = "25.05";
}
-11
View File
@@ -1,13 +1,6 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "tailscale-router";
sopsFile = ../../secrets/tailscale-router.yaml;
})
];
networking = {
hostName = "tailscale-router";
useDHCP = false;
@@ -19,10 +12,6 @@
nameservers = [ vars.domainControllerIp ];
};
services.beszel.agent.environment = {
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
};
# No networking.hostId: only ZFS-touching hosts (server, docker) need one
# for pool-import safety, and this host does neither.
-11
View File
@@ -1,13 +1,6 @@
{ vars, ... }:
{
imports = [
(import ../../modules/beszel/host-token.nix {
name = "tor-relay";
sopsFile = ../../secrets/tor-relay.yaml;
})
];
networking = {
hostName = "tor-relay";
useDHCP = false;
@@ -22,10 +15,6 @@
# No networking.hostId: only ZFS-touching hosts need one for pool-import
# safety, and this host does neither.
services.beszel.agent.environment = {
KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
};
# A genuinely new host (not a pre-refactor carry-over), so it tracks the
# flake's current nixpkgs release rather than being pinned to an older one.
system.stateVersion = "26.05";
+18 -5
View File
@@ -1,10 +1,23 @@
{ vars, ... }:
{ config, vars, ... }:
{
services.beszel.agent.enable = true;
services.beszel.agent.environment = {
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
# Universal token shared by all beszel agents. Add to secrets/common.yaml:
# sops secrets/common.yaml
# beszel-token: <value from the beszel hub UI>
sops.secrets."beszel-token" = { };
sops.templates."beszel.env".content = ''
TOKEN=${config.sops.placeholder."beszel-token"}
'';
services.beszel.agent = {
enable = true;
environmentFile = config.sops.templates."beszel.env".path;
environment = {
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
KEY = vars.beszelHubKey;
};
};
# The upstream module runs beszel-agent under DynamicUser with
-11
View File
@@ -1,11 +0,0 @@
{ name, sopsFile }:
{ config, ... }:
{
sops.secrets."beszel-token".sopsFile = sopsFile;
sops.templates."${name}-beszel.env".content = ''
TOKEN=${config.sops.placeholder."beszel-token"}
'';
services.beszel.agent.environmentFile = config.sops.templates."${name}-beszel.env".path;
}
+1 -1
View File
@@ -97,7 +97,7 @@
sops
];
sessionVariables = {
EDITOR = "vim";
EDITOR = "nano";
SOPS_AGE_KEY_FILE = "/home/${vars.ipaUser}/.config/sops/age/keys.txt";
};
file = {
+10 -4
View File
@@ -17,12 +17,15 @@
{ lib, pkgs, vars, ... }:
let
# Generates /etc/exports lines for all nfsShares data entries. Shared
# pattern with modules/build-types/server.nix — both export the same
# set of shares, differing only in the storage root they serve from.
# Generates /etc/exports lines for all nfsShares data entries.
# LAN (VLAN 2): NFS via vip-lan (192.168.2.229) for pxe-boot and other LAN clients.
# Storage-client (VLAN 20): NFS via vip-storage (192.168.20.229) for docker and
# future swarm nodes; firewall restricts these ports to haClientCidr only.
mkNfsExports = storageRoot:
lib.concatMapStrings
(share: " ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n")
(share:
" ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n" +
" ${storageRoot}/${share.subpath} ${vars.haClientCidr}${vars.nfsShares.options}\n")
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares));
in
{
@@ -45,4 +48,7 @@ in
# Pacemaker controls nfs-server — prevent systemd from starting it at boot
# on both nodes (only the Active node should be serving NFS).
systemd.services.nfs-server.wantedBy = lib.mkForce [ ];
# Same reason as server.nix: exports use standard auth, not Kerberos.
systemd.services.rpc-svcgssd.enable = false;
}
+1
View File
@@ -314,6 +314,7 @@ in
imports = [
../pxe-boot/stage-installer-artifacts.nix
../pxe-boot/mount-pxe-images.nix
../beszel/enable-agent.nix
];
environment.systemPackages = with pkgs; [
-115
View File
@@ -1,115 +0,0 @@
{ vars, lib, pkgs, ... }:
let
poolName = lib.removePrefix "/" vars.storageRoot;
# For each NFS share subpath, generate every ancestor path so ZFS datasets
# are created parent-first. e.g. "docker/config" → ["docker" "docker/config"]
ancestors = path:
let parts = lib.splitString "/" path;
in lib.imap1 (i: _: lib.concatStringsSep "/" (lib.take i parts)) parts;
poolDatasets = lib.unique (
lib.concatMap (share: ancestors share.subpath)
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares))
);
# Generates /etc/exports lines for all nfsShares data entries (every
# attrset value — excludes the bare `options` string). Both server and
# ha-server export the same share set from different storage roots, so
# this helper is the single source of truth for the export line format.
mkNfsExports = storageRoot:
lib.concatMapStrings
(share: " ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n")
(lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares));
in
{
imports = [
../beszel/enable-agent.nix
../services/zfs/enable-service.nix
];
boot.zfs.extraPools = [ poolName ];
# On a fresh image deploy the data disk (scsi1) starts blank — no pool
# exists yet, so zfs-import-tank.service would spin for 60 s and fail.
# This service runs first: if the pool is already present it exits instantly;
# otherwise it creates it (with all required datasets) so the standard
# import service finds it ready on the very first boot.
systemd.services."zfs-init-${poolName}" = {
description = "Initialize '${poolName}' ZFS pool on first boot if not present";
wantedBy = [ "zfs-import-${poolName}.service" ];
before = [ "zfs-import-${poolName}.service" ];
after = [ "systemd-udev-settle.service" ];
unitConfig.DefaultDependencies = false;
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
};
path = [ pkgs.zfs_unstable ];
script = ''
# Already imported nothing to do.
if zpool list "${poolName}" >/dev/null 2>&1; then
exit 0
fi
# Locate the data disk first used for both the fallback import
# attempt and, only if the disk is genuinely blank, pool creation.
DATA_DISK=""
for candidate in /dev/disk/by-id/scsi-*drive-scsi1; do
[[ "$candidate" == *-part* ]] && continue
[ -b "$candidate" ] && DATA_DISK="$candidate" && break
done
if [ -z "$DATA_DISK" ]; then
echo "zfs-init-${poolName}: no data disk found (expected /dev/disk/by-id/scsi-*drive-scsi1)" >&2
exit 1
fi
# Try importing via the by-id symlink directory first (normal path),
# then fall back to scanning the disk directly. The two-step exists
# because of a udev race: systemd-udev-settle.service can clear before
# /dev/disk/by-id/ entries are fully populated, causing the first
# import to fail even when the pool is intact on the disk.
if zpool import -d /dev/disk/by-id -N "${poolName}" 2>/dev/null; then
exit 0
fi
if zpool import -d "$DATA_DISK" -N "${poolName}" 2>/dev/null; then
exit 0
fi
# Both import attempts failed. Before creating a new pool, verify the
# disk is genuinely blank if ZFS label metadata is present the import
# failed for some other reason and we must not clobber existing data.
if zdb -l "$DATA_DISK" 2>/dev/null | grep -q "name: '${poolName}'"; then
echo "zfs-init-${poolName}: $DATA_DISK has ZFS pool '${poolName}' metadata but import failed refusing to overwrite existing data. Run 'zpool import -d $DATA_DISK ${poolName}' manually to investigate." >&2
exit 1
fi
# Disk is genuinely blank: create the pool. -f is intentionally
# omitted so that if we somehow reach this point with an existing pool
# on the disk, zpool refuses rather than silently destroying data.
echo "zfs-init-${poolName}: creating pool on $DATA_DISK"
zpool create "${poolName}" "$DATA_DISK"
${lib.concatMapStrings (ds: ''
zfs create "${poolName}/${ds}"
'') poolDatasets}
'';
};
systemd.services.nfs-server = {
after = [ "zfs-mount.service" ];
requires = [ "zfs-mount.service" ];
};
services.nfs.server = {
enable = true;
exports = mkNfsExports vars.storageRoot;
};
# mountd (20048) is needed for showmount/NFSv3 mount protocol — without it
# clients can reach portmapper (111) and get the mountd port back, then
# time out trying to connect to it. All three ports need TCP and UDP.
networking.firewall.allowedTCPPorts = [ vars.ports.nfsRpcbind vars.ports.nfsd vars.ports.nfsMountd ];
networking.firewall.allowedUDPPorts = [ vars.ports.nfsRpcbind vars.ports.nfsd vars.ports.nfsMountd ];
}
+13 -18
View File
@@ -10,24 +10,19 @@ let
# non-blocking behavior, so they don't need `nofail` too).
automountOpts = if config.boot.isContainer then [ "nofail" ] else [ "x-systemd.automount" ];
# A bare hostname here never resolves reliably: systemd-resolved only
# ever tries LLMNR for single-label names (never DNS, regardless of any
# configured search domain), and a *global* search domain (the first fix
# attempted here) backfires worse -- confirmed live on lxc-docker, adding
# `networking.search` made systemd-resolved prioritize its domain-matched
# but server-less global scope over eth0's correctly-configured one for
# every "*.sweet.home" query, silently sending them to public fallback
# DNS instead. `resolvectl query --interface=eth0 server.sweet.home`
# resolved fine throughout, proving the LAN DNS server was never the
# problem -- only the ambient, unqualified device string was. Using the
# FQDN directly sidesteps all of that, matching the pattern
# ../raspi/mount-data.nix already uses for the same reason.
nfsServer = "${vars.nfsServerHost}.${vars.homeDomain}";
# FQDN in the storage.home zone — resolves to the Pacemaker vip-storage
# (192.168.20.229) on docker's eth1/vmbr2 interface. Using the DNS name
# rather than the raw IP means a future VIP renumber only requires a DNS
# update, not a NixOS rebuild. The storage.home zone is served by the same
# FreeIPA nameserver (domainControllerIp) that docker already uses, so
# resolution reaches it over eth0 without any extra routing.
nfsServer = vars.haStorageNfsFqdn;
storageRoot = vars.haStorageRoot;
in
{
fileSystems = {
${vars.nfsShares.dockerConfig.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath}";
device = "${nfsServer}:${storageRoot}/${vars.nfsShares.dockerConfig.subpath}";
fsType = "nfs";
options = [
@@ -38,7 +33,7 @@ in
};
${vars.nfsShares.dockerDatabases.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath}";
device = "${nfsServer}:${storageRoot}/${vars.nfsShares.dockerDatabases.subpath}";
fsType = "nfs";
options = [
@@ -49,7 +44,7 @@ in
};
${vars.nfsShares.dockerVolumes.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
device = "${nfsServer}:${storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
fsType = "nfs";
options = [
@@ -60,7 +55,7 @@ in
};
${vars.nfsShares.nextcloudData.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath}";
device = "${nfsServer}:${storageRoot}/${vars.nfsShares.nextcloudData.subpath}";
fsType = "nfs";
options = [
@@ -71,7 +66,7 @@ in
};
${vars.nfsShares.raspiVolumes.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath}";
device = "${nfsServer}:${storageRoot}/${vars.nfsShares.raspiVolumes.subpath}";
fsType = "nfs";
options = [
+55 -16
View File
@@ -21,7 +21,26 @@
# STONITH device during the testing phase. Switch to resource-and-stonith
# once the fence_pve_ssh STONITH resource is active (see
# scripts/ha/cluster-enable-stonith.sh).
{ lib, vars, ... }:
#
# PATH wrapper: when the DRBD kernel module invokes the fence-peer handler
# via the UMH (User Mode Helper) mechanism it provides a minimal PATH that
# omits /run/current-system/sw/bin. crm-fence-peer.sh calls cibadmin,
# crm_mon etc.; if those aren't found a pipeline in the script breaks with
# SIGPIPE. A process killed by signal has WEXITSTATUS() == 0, so the kernel
# sees exit code 0 and logs "fence-peer helper broken, returned 0", looping
# forever. The writeShellScript wrappers below prepend the NixOS sw path
# before exec-ing the real handler, giving it a working Pacemaker toolchain.
{ lib, pkgs, vars, ... }:
let
fencePeerWrapper = pkgs.writeShellScript "drbd-fence-peer" ''
export PATH="/run/current-system/sw/bin:/run/current-system/sw/sbin:$PATH"
exec /run/current-system/sw/lib/drbd/crm-fence-peer.sh "$@"
'';
unfencePeerWrapper = pkgs.writeShellScript "drbd-unfence-peer" ''
export PATH="/run/current-system/sw/bin:/run/current-system/sw/sbin:$PATH"
exec /run/current-system/sw/lib/drbd/crm-unfence-peer.sh "$@"
'';
in
{
# Root SSH access — same key set as nixos user so all admin keys can reach root.
users.users.root.openssh.authorizedKeys.keys = [
@@ -37,6 +56,12 @@
# DRBD lock-file directory (drbd-utils checks for it; missing → harmless but noisy warnings).
systemd.tmpfiles.rules = [ "d /var/lib/drbd 0750 root root -" ];
# Prevent drbd.service from auto-starting at boot / nixos-rebuild switch.
# Pacemaker's OCF drbd agent calls drbdadm up/down directly when managing
# the resource. If drbd.service also runs drbdadm up all while DRBD is
# already Primary under Pacemaker, apply-al fails with "device busy" (exit 20).
systemd.services.drbd.wantedBy = lib.mkForce [ ];
services.drbd = {
enable = true;
config = ''
@@ -56,15 +81,15 @@
fencing resource-only;
}
handlers {
fence-peer "/run/current-system/sw/lib/drbd/crm-fence-peer.sh";
unfence-peer "/run/current-system/sw/lib/drbd/crm-unfence-peer.sh";
fence-peer "${fencePeerWrapper}";
unfence-peer "${unfencePeerWrapper}";
}
}
resource ha-data {
volume 0 {
device /dev/drbd0;
disk /dev/sdb;
disk ${vars.haServerDrbdDisk};
meta-disk internal;
}
@@ -97,33 +122,47 @@
services.corosync = {
clusterName = "ha-cluster";
nodelist = [
{ nodeid = 1; name = vars.haServer1Host; ring_addrs = [ vars.haServer1StorageIp ]; }
{ nodeid = 2; name = vars.haServer2Host; ring_addrs = [ vars.haServer2StorageIp ]; }
# ring0: cluster-internal vmbr1 (primary heartbeat + DRBD path)
# ring1: LAN vmbr0 (backup heartbeat only — never carries DRBD)
{ nodeid = 1; name = vars.haServer1Host; ring_addrs = [ vars.haServer1StorageIp vars.haServer1Ip ]; }
{ nodeid = 2; name = vars.haServer2Host; ring_addrs = [ vars.haServer2StorageIp vars.haServer2Ip ]; }
];
};
networking.firewall = {
allowedTCPPorts = [
vars.ports.haServerIscsi
vars.ports.haServerPacemakerRemoted
vars.ports.haServerPcsd
vars.ports.haServerDrbd
vars.ports.nfsRpcbind
vars.ports.nfsd
vars.ports.nfsMountd
];
allowedUDPPorts = [
vars.ports.haServerCorosync1
vars.ports.haServerCorosync2
vars.ports.haServerCorosyncCrypto
vars.ports.nfsRpcbind
vars.ports.nfsd
vars.ports.nfsMountd
];
# Protocol separation: iSCSI (VLAN 20 / storage clients only),
# NFS (VLAN 2 / LAN only). Cluster-internal subnets accepted wholesale
# since they are isolated bridges with no external uplink.
extraCommands = ''
iptables -A INPUT -s ${vars.haServer1Ip}/32 -j ACCEPT
iptables -A INPUT -s ${vars.haServer2Ip}/32 -j ACCEPT
iptables -A INPUT -s ${vars.haStorageCidr} -j ACCEPT
iptables -A nixos-fw -s ${vars.haServer1Ip}/32 -j nixos-fw-accept
iptables -A nixos-fw -s ${vars.haServer2Ip}/32 -j nixos-fw-accept
iptables -A nixos-fw -s ${vars.haStorageCidr} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.haClientCidr} --dport ${toString vars.ports.haServerIscsi} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsRpcbind} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsRpcbind} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsd} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsd} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsMountd} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.lanCidr} --dport ${toString vars.ports.nfsMountd} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsRpcbind} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsRpcbind} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsd} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsd} -j nixos-fw-accept
iptables -A nixos-fw -p tcp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsMountd} -j nixos-fw-accept
iptables -A nixos-fw -p udp -s ${vars.haClientCidr} --dport ${toString vars.ports.nfsMountd} -j nixos-fw-accept
'';
};
}
+5 -5
View File
@@ -1,14 +1,14 @@
{ config, lib, vars, ... }:
let
# Use the same FQDN approach as docker/mount-data.nix — a bare hostname is
# unreliable: systemd-resolved only tries LLMNR for single-label names, and
# a global search domain causes it to skip the interface-scoped LAN DNS.
nfsServer = "${vars.nfsServerHost}.${vars.homeDomain}";
# FQDN of the LAN NFS VIP (Pacemaker vip-lan, 192.168.2.229). Using the
# FQDN rather than a raw IP or bare hostname avoids systemd-resolved LLMNR
# quirks and survives a future VIP renumber via a DNS-only update.
nfsServer = "ha-vip-lan.${vars.homeDomain}";
in
{
fileSystems.${vars.nfsShares.pxebootImages.mountpoint} = {
device = "${nfsServer}:${vars.storageRoot}/${vars.nfsShares.pxebootImages.subpath}";
device = "${nfsServer}:${vars.haStorageRoot}/${vars.nfsShares.pxebootImages.subpath}";
fsType = "nfs";
options = [
"_netdev"
Binary file not shown.
+237
View File
@@ -0,0 +1,237 @@
#!/usr/bin/env bash
# gc-hosts.sh — Run nix-collect-garbage -d on all live NixOS hosts.
#
# The host list is rebuilt on every run:
# 1. This workstation (nixos) — always first
# 2. pve1 — always second (non-NixOS Proxmox node with Nix installed)
# 3. Every NixOS guest currently running on pve1 (discovered via pct/qm list)
#
# nix-cache is excluded: gc-ing the shared binary cache evicts store paths
# that other hosts depend on for substitution.
#
# NixOS hosts: tries "sudo -n nix-collect-garbage -d" first (works when
# wheelNeedsPassword = false, e.g. the HA cluster). Falls back to user-level
# "nix-collect-garbage -d" if sudo needs a password — still collects
# unreferenced store paths and old nixos-user profile generations, but leaves
# old system generations in place.
# pve1: runs "bash -l -c nix-collect-garbage -d" as the login user so
# /etc/profile is sourced and the Nix daemon's PATH is set up automatically.
#
# Usage (from repo root):
# bash scripts/gc-hosts.sh [--dry-run]
set -euo pipefail
cd "$(dirname "$0")/.."
source scripts/env.sh 2>/dev/null || true
source scripts/lib/nix-eval.sh 2>/dev/null || true
# ── config ────────────────────────────────────────────────────────────────────
: "${MAX_JOBS:=8}"
: "${NIXOS_USER:=nixos}"
: "${PVE1_SSH_USER:=${PROXMOX_SSH_USER:-wayne}}"
# GC connections use BatchMode — no interactive prompts, just succeed or fail.
SSH_OPTS=(-o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=10)
# Discovery connections do NOT use BatchMode so that sudo can prompt if needed
# (pct/qm list require root access on Proxmox).
SSH_QUERY_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=10)
DRY_RUN=0
for arg in "$@"; do
case "$arg" in
--dry-run) DRY_RUN=1 ;;
*) echo "Unknown option: $arg" >&2; exit 1 ;;
esac
done
# ── build the host list ───────────────────────────────────────────────────────
# ORDERED_HOSTS: names in display/execution order.
# HOST_TARGET[name]: SSH target string (user@host).
# HOST_TYPE[name]: "nixos" (try sudo gc, fallback user) | "nix" (login-shell gc).
declare -a ORDERED_HOSTS=()
declare -A HOST_TARGET=()
declare -A HOST_TYPE=()
declare -A _SEEN_HOSTNAMES=() # dedup tracker
_add_host() {
local name="$1" target="$2" type="$3"
if [[ -n "${_SEEN_HOSTNAMES[$name]+_}" ]]; then return; fi
_SEEN_HOSTNAMES[$name]=1
ORDERED_HOSTS+=("$name")
HOST_TARGET[$name]="$target"
HOST_TYPE[$name]="$type"
}
# 1. Workstation (hard-wired first)
_add_host "nixos" "${NIXOS_USER}@nixos" "nixos"
# 2. pve1 (hard-wired second; non-NixOS, no system generations)
_add_host "pve1" "${PVE1_SSH_USER}@${PVE1_HOST}" "nix"
# 3. Dynamically discover running NixOS guests on pve1
#
# create-proxmox-resource.sh names every guest after its NixOS hostname:
# pct create ... --hostname <nixos-hostname> (LXC)
# qm create ... --name <nixos-hostname> (VM)
# So pct/qm list output already contains the NixOS hostname directly.
# We validate against the flake to filter out non-NixOS guests on pve1
# (e.g. FreeIPA, Proxmox Backup Server) that share the same Proxmox node.
echo "Discovering running guests on ${PVE1_HOST}..."
# Eval the flake once to get the set of hostnames that are actually NixOS.
# Values are NixOS hostnames (e.g. "docker"); keys are flake targets ("lxc-docker").
nixos_hostnames=""
nixos_hostnames="$(
nix eval --json "${NIX_EVAL_FLAGS[@]}" .#nixosConfigurations \
--apply 'cfgs: builtins.attrValues (builtins.mapAttrs (_: cfg: cfg.config.networking.hostName) cfgs)' \
2>/dev/null | jq -r '.[]' | sort -u
)" || { echo " warning: flake eval failed — non-NixOS guests will not be filtered" >&2; }
# SSH_QUERY_OPTS (no BatchMode) so sudo can prompt if needed for pct/qm.
if ssh "${SSH_QUERY_OPTS[@]}" "${PVE1_SSH_USER}@${PVE1_HOST}" "true" 2>/dev/null; then
running_guests="$(
ssh "${SSH_QUERY_OPTS[@]}" "${PVE1_SSH_USER}@${PVE1_HOST}" bash -s <<'DISCOVER'
sudo pct list 2>/dev/null | awk 'NR>1 && $2=="running" { print $NF }'
sudo qm list 2>/dev/null | awk 'NR>1 && $3=="running" { print $2 }'
DISCOVER
)" || running_guests=""
while IFS= read -r hostname; do
[[ -z "$hostname" ]] && continue
# Exclude nix-cache.
case "$hostname" in *nix-cache*) continue ;; esac
# Skip if not a flake-managed NixOS host (filters non-NixOS pve1 guests).
if [[ -n "$nixos_hostnames" ]] && ! grep -qxF "$hostname" <<< "$nixos_hostnames"; then
continue
fi
# Skip if already in the list (e.g. a proxmox-gui guest whose hostname is nixos).
if [[ -n "${_SEEN_HOSTNAMES[$hostname]+_}" ]]; then continue; fi
echo " + $hostname"
_add_host "$hostname" "${NIXOS_USER}@${hostname}" "nixos"
done <<< "$(echo "$running_guests" | sort -u)"
else
echo " warning: ${PVE1_HOST} unreachable — skipping dynamic host discovery" >&2
fi
echo ""
echo "Hosts: ${ORDERED_HOSTS[*]}"
echo ""
# ── dry-run ───────────────────────────────────────────────────────────────────
if [[ "$DRY_RUN" -eq 1 ]]; then
echo "[dry-run] commands that would run:"
for host in "${ORDERED_HOSTS[@]}"; do
target="${HOST_TARGET[$host]}"
type="${HOST_TYPE[$host]}"
if [[ "$type" == "nixos" ]]; then
echo " ssh ${SSH_OPTS[*]} $target 'sudo -n nix-collect-garbage -d'"
echo " # fallback: ssh ... $target 'nix-collect-garbage -d'"
else
echo " ssh ${SSH_OPTS[*]} $target 'bash -l -c nix-collect-garbage -d'"
fi
done
exit 0
fi
# ── gc worker ─────────────────────────────────────────────────────────────────
gc_one() {
local host="$1" target="${HOST_TARGET[$1]}" type="${HOST_TYPE[$1]}" logfile="$2"
if ! ssh "${SSH_OPTS[@]}" "$target" "true" 2>>"$logfile"; then
echo "unreachable"; return
fi
if [[ "$type" == "nixos" ]]; then
if ssh "${SSH_OPTS[@]}" "$target" "sudo -n nix-collect-garbage -d" \
>>"$logfile" 2>>"$logfile"; then
echo "ok(sudo)"; return
fi
echo "[sudo needs password — falling back to user-level gc]" >>"$logfile"
if ssh "${SSH_OPTS[@]}" "$target" "nix-collect-garbage -d" \
>>"$logfile" 2>>"$logfile"; then
echo "ok(user)"; return
fi
else
# Non-NixOS node: use a login shell so /etc/profile is sourced and the
# Nix daemon's bin dir is on PATH (set up by /etc/profile.d/nix-daemon.sh
# which the Nix installer adds to /etc/profile).
if ssh "${SSH_OPTS[@]}" "$target" "bash -l -c 'nix-collect-garbage -d'" \
>>"$logfile" 2>>"$logfile"; then
echo "ok"; return
fi
fi
echo "failed:$?"
}
# ── parallel execution ────────────────────────────────────────────────────────
echo "Running gc on ${#ORDERED_HOSTS[@]} hosts (up to ${MAX_JOBS} parallel)..."
echo ""
TMPDIR_GC="$(mktemp -d)"
trap 'rm -rf "$TMPDIR_GC"' EXIT
declare -A LOGS=()
job_count=0
for host in "${ORDERED_HOSTS[@]}"; do
logfile="${TMPDIR_GC}/${host}.log"
resultfile="${TMPDIR_GC}/${host}.result"
LOGS[$host]="$logfile"
: > "$logfile"
( result="$(gc_one "$host" "$logfile")"; echo "$result" > "$resultfile" ) &
(( job_count++ )) || true
if [[ "$job_count" -ge "$MAX_JOBS" ]]; then
wait -n 2>/dev/null || wait
(( job_count-- )) || true
fi
done
wait
# ── summary ───────────────────────────────────────────────────────────────────
echo "Results:"
echo "──────────────────────────────"
ok_hosts=()
warn_hosts=()
fail_hosts=()
for host in "${ORDERED_HOSTS[@]}"; do
result="$(cat "${TMPDIR_GC}/${host}.result" 2>/dev/null || echo "failed:missing")"
case "$result" in
ok|"ok(sudo)"|"ok(user)")
printf " %-22s %s\n" "$host" "$result"
ok_hosts+=("$host") ;;
unreachable)
printf " %-22s UNREACHABLE\n" "$host"
warn_hosts+=("$host") ;;
*)
printf " %-22s FAILED (%s)\n" "$host" "$result"
fail_hosts+=("$host") ;;
esac
done
echo ""
echo " ${#ok_hosts[@]} succeeded, ${#warn_hosts[@]} unreachable, ${#fail_hosts[@]} failed"
for host in "${warn_hosts[@]+"${warn_hosts[@]}"}" "${fail_hosts[@]+"${fail_hosts[@]}"}"; do
logfile="${LOGS[$host]}"
if [[ -s "$logfile" ]]; then
echo ""
echo "── $host ──"
cat "$logfile"
fi
done
echo ""
[[ "${#fail_hosts[@]}" -eq 0 ]]
Regular → Executable
+47 -10
View File
@@ -12,7 +12,7 @@ NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.2.229}" # vars.haServerVip
VIP="${VIP:-192.168.20.229}" # vars.haServerVip
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
ISCSI_IQN="${ISCSI_IQN:-iqn.2026-01.home.sweet:ha-storage}" # vars.haIscsiIqn
# ──────────────────────────────────────────────────────────────────────────
@@ -34,18 +34,55 @@ echo "════════════════════════
echo " HA Cluster Acceptance Tests — $(date '+%Y-%m-%d %H:%M:%S')"
echo "════════════════════════════════════════════════════"
# ── Detect Active/Standby nodes ────────────────────────────────────────────
# Pacemaker can promote either node; determine which is currently Active
# (DRBD Primary / holds ha-group resources) before running tests.
# ── Pre-flight: DRBD sync must be complete ────────────────────────────────
# Tests that check disk state, XFS mount, and iSCSI will fail or give false
# results while the initial full sync is in progress. Block until done.
echo ""
echo "Detecting Active/Standby nodes..."
if n1 "drbdadm role ha-data" 2>/dev/null | grep -q "^Primary"; then
ACTIVE_NODE="$NODE1"; ACTIVE_IP="$NODE1_IP"
echo "Pre-flight: verifying DRBD sync is complete..."
DRBD_PREFLIGHT=$(n1 "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_PREFLIGHT" | grep -q "^UpToDate/UpToDate$"; then
echo ""
echo " ERROR: DRBD initial sync not complete."
echo " Current dstate on $NODE1: $DRBD_PREFLIGHT"
echo ""
echo " Monitor progress:"
echo " ssh nixos@$NODE1_IP 'sudo watch -n3 cat /proc/drbd'"
echo ""
echo " Re-run this script once dstate shows UpToDate/UpToDate."
exit 1
fi
echo " dstate: $DRBD_PREFLIGHT — ready."
# ── Detect Active/Standby nodes ────────────────────────────────────────────
# Use crm_mon to detect which node holds the Promoted (Primary) DRBD resource.
# Pacemaker is authoritative; DRBD role can briefly read as Secondary while
# Pacemaker is mid-transition, giving a false Active/Standby swap.
# Wait up to 90 s for Pacemaker to settle before giving up.
echo ""
echo "Detecting Active/Standby nodes (waiting for Pacemaker to settle)..."
ACTIVE_NODE=""
for i in $(seq 1 30); do
# crm_mon -1 output contains "Promoted: [ <node> ]" for the DRBD master.
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || n2 "crm_mon -1" 2>/dev/null || true)
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" — the * bullet
# means ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | grep -E '(Promoted|Masters):' | grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
[[ -n "$ACTIVE_NODE" ]] && break
sleep 3
done
if [[ -z "$ACTIVE_NODE" ]]; then
echo " WARNING: could not determine Active node from crm_mon after 90 s — defaulting to $NODE1"
ACTIVE_NODE="$NODE1"
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
STANDBY_NODE="$NODE2"; STANDBY_IP="$NODE2_IP"
na() { n1 "$@"; }
ns() { n2 "$@"; }
else
ACTIVE_NODE="$NODE2"; ACTIVE_IP="$NODE2_IP"
ACTIVE_IP="$NODE2_IP"
STANDBY_NODE="$NODE1"; STANDBY_IP="$NODE1_IP"
na() { n2 "$@"; }
ns() { n1 "$@"; }
@@ -97,7 +134,7 @@ fi
# ── T4: iSCSI target visible on Active node ───────────────────────────────
echo ""
echo "[T4] iSCSI target"
IQN_COUNT=$(na "ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn" || echo "0")
IQN_COUNT=$(na "bash -c 'ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn || true'" 2>/dev/null || echo "0")
if [[ "$IQN_COUNT" -ge 1 ]]; then
pass "iSCSI IQN active on $ACTIVE_NODE ($IQN_COUNT target(s))"
else
@@ -130,7 +167,7 @@ done
if $MOVED; then
pass "XFS mounted on $STANDBY_NODE after failover"
IQN_ON_STANDBY=$(ns "ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn" || echo "0")
IQN_ON_STANDBY=$(ns "bash -c 'ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn || true'" 2>/dev/null || echo "0")
[[ "$IQN_ON_STANDBY" -ge 1 ]] \
&& pass "iSCSI target active on $STANDBY_NODE after failover" \
|| fail "iSCSI target NOT active on $STANDBY_NODE after failover"
View File
+176 -27
View File
@@ -26,12 +26,18 @@ NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.2.229}" # vars.haServerVip
VIP="${VIP:-192.168.20.229}" # vars.haServerVip (storage-client, vmbr2, VLAN 20)
VIP_LAN="${VIP_LAN:-192.168.2.229}" # vars.haServerLanVip (LAN, vmbr0)
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
ISCSI_IQN="${ISCSI_IQN:-iqn.2026-01.home.sweet:ha-storage}" # vars.haIscsiIqn
ISCSI_LUN_FILE="${XFS_MOUNT}/iscsi-lun.img"
ISCSI_LUN_SIZE="10G"
DRBD_DEVICE="/dev/drbd0"
# DRBD backing disk — by-id path that resolves correctly on both nodes
# regardless of whether the OS-level name is sda or sdb (Proxmox VM disk
# ordering is not guaranteed). Matches haServerDrbdDisk in variables.nix.
# Override DRBD_DISK if your hardware uses a different controller/slot path.
DRBD_DISK="${DRBD_DISK:-/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1}"
VMID_NODE1="${VMID_NODE1:-}" # set by deploy.sh; needed for STONITH
VMID_NODE2="${VMID_NODE2:-}"
PVE_HOST="${PVE_HOST:-pve1.sweet.home}"
@@ -72,6 +78,18 @@ if ! command -v mkfs.xfs &>/dev/null; then
|| die "mkfs.xfs not found — add xfsprogs to ha-server.nix environment.systemPackages and rebuild"
fi
# drbdmeta lives alongside drbdadm but may not be in PATH when run via sudo.
if ! command -v drbdmeta &>/dev/null; then
_drbd_bin=$(dirname "$(command -v drbdadm)" 2>/dev/null || true)
[[ -n "$_drbd_bin" ]] && export PATH="$_drbd_bin:$PATH" \
|| die "drbdmeta not found — is drbd-utils in ha-server environment.systemPackages?"
fi
# Portable 16-hex-char UUID generator (no openssl required).
_rand_uuid() {
cat /proc/sys/kernel/random/uuid 2>/dev/null | tr -d '-' | cut -c1-16 | tr '[:lower:]' '[:upper:]'
}
# Inter-node SSH/SCP helpers — abstract over root-to-root vs nixos+sudo.
_SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=10"
[[ -n "$HA_KEY" ]] && _SSH_OPTS="-i $HA_KEY $_SSH_OPTS"
@@ -103,11 +121,16 @@ n2_ssh "mkdir -p /etc/corosync"
n2_scp "$AUTHKEY" "$AUTHKEY"
n2_ssh "chmod 0400 '${AUTHKEY}'"
log "Restarting corosync on both nodes..."
log "Restarting corosync and pacemaker on both nodes..."
systemctl restart corosync
n2_ssh "systemctl restart corosync"
sleep 3
log "Starting pacemaker on both nodes (may have failed at boot before authkey was placed)..."
systemctl start pacemaker 2>/dev/null || systemctl restart pacemaker 2>/dev/null || true
n2_ssh "systemctl start pacemaker 2>/dev/null || systemctl restart pacemaker 2>/dev/null || true"
sleep 2
# ── 1. Corosync quorum ────────────────────────────────────────────────────
log "Waiting for corosync quorum..."
for i in $(seq 1 30); do
@@ -130,14 +153,91 @@ for i in $(seq 1 30); do
done
# ── 2. DRBD initialisation ────────────────────────────────────────────────
# Put both nodes in Pacemaker standby first so it stops managed resources
# cleanly, then enable maintenance-mode so Pacemaker's monitor operations are
# suspended. Without maintenance-mode, Pacemaker keeps monitoring: when it
# sees DRBD Primary on a standby node (that it didn't start), it triggers a
# stop action — killing the initial sync after ~10 s. Maintenance-mode
# disables all start/stop/monitor actions for the duration of the sync; it is
# cleared after UpToDate/UpToDate is confirmed.
log "Setting both nodes to Pacemaker standby for DRBD metadata init..."
crm_standby -N "$NODE1" -v on 2>/dev/null || true
crm_standby -N "$NODE2" -v on 2>/dev/null || true
# Wait for Pacemaker to actually stop DRBD (if it was managing it).
log "Waiting for DRBD to stop under Pacemaker control..."
for i in $(seq 1 30); do
n1_role=$(drbdadm role ha-data 2>/dev/null || echo "Unconfigured")
n2_role=$(n2_ssh "drbdadm role ha-data 2>/dev/null" 2>/dev/null || echo "Unconfigured")
if [[ "$n1_role" == "Unconfigured" ]] && [[ "$n2_role" == "Unconfigured" ]]; then
log "DRBD stopped on both nodes"
break
fi
[[ $i -eq 30 ]] && warn "DRBD still active after 60s standby — forcing down anyway"
sleep 2
done
log "Enabling Pacemaker maintenance-mode (suspends monitor/start/stop during sync)..."
crm_attribute -t crm_config -n maintenance-mode -v true 2>/dev/null || true
log "Detaching DRBD on $NODE1 (belt-and-suspenders after standby)..."
drbdadm down ha-data 2>/dev/null || true
log "Detaching DRBD on $NODE2..."
n2_ssh "drbdadm down ha-data 2>/dev/null || true"
sleep 2
# Ensure /etc/drbd.conf on both nodes points to DRBD_DISK (the stable by-id
# path). VMs built before this fix may have /dev/sda or /dev/sdb hardcoded.
# NixOS makes /etc/drbd.conf a symlink into the read-only Nix store, so
# sed -i on the symlink target would fail — we break the symlink first with
# cp --remove-destination, creating a regular writable copy.
# Rebuild+redeploy (--force-rebuild) to make this permanent.
_PATCH_DRBD=$(mktemp)
cat > "$_PATCH_DRBD" << 'PATCHEOF'
#!/bin/bash
WANT="$1"
conf=/etc/drbd.conf
if [[ -L "$conf" ]]; then
cp --remove-destination "$(readlink -f "$conf")" "$conf"
fi
cur=$(drbdadm sh-ll-dev ha-data 2>/dev/null | head -1 || true)
if [[ -n "$cur" && "$cur" != "$WANT" ]]; then
echo "[cluster-init] WARNING: patching $conf: $cur → $WANT (rebuild to make permanent)"
sed -i "s,${cur},${WANT},g" "$conf"
fi
PATCHEOF
chmod +x "$_PATCH_DRBD"
bash "$_PATCH_DRBD" "$DRBD_DISK"
n2_scp "$_PATCH_DRBD" "/tmp/patch-drbd-disk.sh"
n2_ssh "bash /tmp/patch-drbd-disk.sh ${DRBD_DISK}"
n2_ssh "rm -f /tmp/patch-drbd-disk.sh"
rm -f "$_PATCH_DRBD"
log "Initialising DRBD metadata on $NODE1..."
if ! drbdadm dstate ha-data 2>/dev/null | grep -q "UpToDate\|Inconsistent\|Diskless"; then
drbdadm create-md ha-data --force
# Use drbdmeta --force directly for BOTH create-md and write-dev-uuid.
# drbdadm create-md --force passes --force to drbdmeta create-md but NOT to
# the write-dev-uuid sub-call it makes internally, so write-dev-uuid fails when
# the backing disk is still busy and stdin is not a TTY:
# "stdin not a TTY, not waiting for confirmation" → exit 20.
# Calling drbdmeta --force directly bypasses the exclusive-open confirmation on
# both steps without needing a TTY, regardless of whether the device is busy.
# Skip metadata creation only if DRBD is UP and fully synced (UpToDate/UpToDate).
# When the resource is down, drbdadm dstate reads metadata and returns just
# "UpToDate" (no slash) — that must not be treated as "already synced".
# Mismatched UUIDs from an interrupted sync cause instant WFConnection→StandAlone,
# so we always recreate metadata unless the sync is genuinely complete.
if [[ "$(drbdadm dstate ha-data 2>/dev/null)" != "UpToDate/UpToDate" ]]; then
UUID1=$(_rand_uuid)
drbdmeta --force 0 v08 "${DRBD_DISK}" internal create-md
drbdmeta --force 0 v08 "${DRBD_DISK}" internal write-dev-uuid "$UUID1"
fi
log "Initialising DRBD metadata on $NODE2..."
# Use grep -E for ERE alternation inside the remote bash -c string (avoids \| quoting issues).
n2_ssh "bash -c 'drbdadm dstate ha-data 2>/dev/null | grep -qE \"UpToDate|Inconsistent|Diskless\" || drbdadm create-md ha-data --force'"
if [[ "$(n2_ssh "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null)" != "UpToDate/UpToDate" ]]; then
UUID2=$(n2_ssh "cat /proc/sys/kernel/random/uuid 2>/dev/null | tr -d '-' | cut -c1-16 | tr '[:lower:]' '[:upper:]'")
n2_ssh "drbdmeta --force 0 v08 ${DRBD_DISK} internal create-md"
n2_ssh "drbdmeta --force 0 v08 ${DRBD_DISK} internal write-dev-uuid ${UUID2}"
fi
log "Bringing up DRBD on both nodes..."
drbdadm up ha-data 2>/dev/null || true
@@ -145,18 +245,49 @@ n2_ssh "drbdadm up ha-data" 2>/dev/null || true
log "Forcing $NODE1 to DRBD Primary for initial sync..."
drbdadm primary ha-data --force
# NOTE: Pacemaker standby is intentionally kept ON until after the sync
# completes. Clearing it here races with the OCF DRBD agent: Pacemaker
# sees DRBD in WFConnection/SyncSource and may call drbdadm-down thinking
# something went wrong, killing the sync. Standby is cleared below, after
# UpToDate/UpToDate is confirmed.
log "Waiting for DRBD to finish initial sync (this may take several minutes)..."
for i in $(seq 1 300); do
state=$(drbdadm dstate ha-data 2>/dev/null || echo "unknown")
if echo "$state" | grep -q "UpToDate/UpToDate"; then
log "DRBD sync complete: $state"
log "Waiting for DRBD initial sync to complete (32 GB may take 1020 min)..."
log " (monitor with: watch -n3 cat /proc/drbd)"
_sync_chars=('|' '/' '-' $'\\')
_sync_iter=0
while true; do
_dstate=$(drbdadm dstate ha-data 2>/dev/null || echo "unknown")
if echo "$_dstate" | grep -q "UpToDate/UpToDate"; then
printf "\r%-80s\r" ""
log "DRBD initial sync complete (dstate: $_dstate)"
break
fi
[[ $i -eq 300 ]] && warn "DRBD not UpToDate after 300 s — continuing anyway (check drbdadm status)"
sleep 1
# Parse connection state from /proc/drbd (cs:SyncSource, cs:Connected, cs:StandAlone …)
_cs=$(grep -oE 'cs:[A-Za-z]+' /proc/drbd 2>/dev/null | head -1 | sed 's/cs://' || echo "unknown")
# /proc/drbd uses variable whitespace: "sync'ed: 5.2%" (two spaces).
_pct=$(grep -oE "sync'ed:[[:space:]]+[0-9.]+" /proc/drbd 2>/dev/null | grep -oE "[0-9.]+" | head -1 || echo "")
_eta=$(grep -oE "finish:[[:space:]]+[0-9:]+" /proc/drbd 2>/dev/null | grep -oE "[0-9:]+$" | head -1 || echo "")
_spd=$(grep -oE "speed:[[:space:]]+[0-9,]+" /proc/drbd 2>/dev/null | grep -oE "[0-9,]+$" | head -1 || echo "")
_sync_iter=$(( _sync_iter + 1 ))
_sc="${_sync_chars[$_sync_iter % 4]}"
if [[ "$_cs" == "StandAlone" && $_sync_iter -gt 5 ]]; then
printf "\r%-80s\r" ""
die "DRBD is StandAlone after 15 s — peer connection lost (dstate: $_dstate). " \
"Check corosync/network and re-run cluster-init."
elif [[ -n "$_pct" ]]; then
printf "\r [%s] syncing: %s%% done — ETA %s @ %s K/s " \
"$_sc" "$_pct" "${_eta:-??:??:??}" "${_spd:-?}"
else
printf "\r [%s] cs:%s dstate:%s — waiting for sync to start " "$_sc" "$_cs" "$_dstate"
fi
sleep 3
done
log "Disabling Pacemaker maintenance-mode and clearing standby — handing DRBD back to Pacemaker..."
crm_attribute -t crm_config -n maintenance-mode -v false 2>/dev/null || true
crm_standby -N "$NODE1" -v off 2>/dev/null || true
crm_standby -N "$NODE2" -v off 2>/dev/null || true
# ── 3. XFS filesystem ─────────────────────────────────────────────────────
log "Creating XFS on ${DRBD_DEVICE}..."
if ! xfs_info "${DRBD_DEVICE}" &>/dev/null; then
@@ -218,7 +349,7 @@ log "Unmounting ${XFS_MOUNT} — Pacemaker manages it..."
umount "${XFS_MOUNT}" || { sync; umount -l "${XFS_MOUNT}"; }
log "Demoting DRBD to Secondary — Pacemaker manages primary role..."
[[ "$(drbdadm role ha-data 2>/dev/null)" == "Primary/Secondary" ]] && drbdadm secondary ha-data || true
drbdadm role ha-data 2>/dev/null | grep -q "^Primary" && drbdadm secondary ha-data || true
# ── 7. Pacemaker resources ────────────────────────────────────────────────
log "Configuring Pacemaker cluster properties..."
@@ -286,15 +417,28 @@ cibadmin --replace --scope resources --xml-text '<resources>
<op id="nfs-monitor" name="monitor" interval="30s" timeout="40s"/>
</operations>
</primitive>
<primitive id="vip" class="ocf" type="IPaddr2" provider="heartbeat">
<instance_attributes id="vip-attrs">
<nvpair id="vip-ip" name="ip" value="192.168.2.229"/>
<nvpair id="vip-cidr" name="cidr_netmask" value="24"/>
<primitive id="vip-storage" class="ocf" type="IPaddr2" provider="heartbeat">
<instance_attributes id="vip-storage-attrs">
<nvpair id="vip-storage-ip" name="ip" value="192.168.20.229"/>
<nvpair id="vip-storage-cidr" name="cidr_netmask" value="24"/>
<nvpair id="vip-storage-nic" name="nic" value="ens20"/>
</instance_attributes>
<operations>
<op id="vip-start" name="start" interval="0" timeout="20s"/>
<op id="vip-stop" name="stop" interval="0" timeout="20s"/>
<op id="vip-monitor" name="monitor" interval="10s" timeout="20s"/>
<op id="vip-storage-start" name="start" interval="0" timeout="20s"/>
<op id="vip-storage-stop" name="stop" interval="0" timeout="20s"/>
<op id="vip-storage-monitor" name="monitor" interval="10s" timeout="20s"/>
</operations>
</primitive>
<primitive id="vip-lan" class="ocf" type="IPaddr2" provider="heartbeat">
<instance_attributes id="vip-lan-attrs">
<nvpair id="vip-lan-ip" name="ip" value="192.168.2.229"/>
<nvpair id="vip-lan-cidr" name="cidr_netmask" value="24"/>
<nvpair id="vip-lan-nic" name="nic" value="ens18"/>
</instance_attributes>
<operations>
<op id="vip-lan-start" name="start" interval="0" timeout="20s"/>
<op id="vip-lan-stop" name="stop" interval="0" timeout="20s"/>
<op id="vip-lan-monitor" name="monitor" interval="10s" timeout="20s"/>
</operations>
</primitive>
</group>
@@ -306,13 +450,16 @@ cibadmin --replace --scope constraints --xml-text '<constraints>
<rsc_colocation id="coloc-group-with-drbd" score="INFINITY" rsc="ha-group" with-rsc="ms-drbd0" with-rsc-role="Promoted"/>
</constraints>'
log "Clearing stale Pacemaker failure history..."
crm_resource --cleanup 2>/dev/null || true
log "Waiting for resources to start..."
for i in $(seq 1 60); do
if crm_resource -r vip --locate 2>/dev/null | grep -q "running on"; then
log "VIP is up: $(crm_resource -r vip --locate)"
if crm_resource -r vip-storage --locate 2>/dev/null | grep -q "running on"; then
log "VIPs are up: $(crm_resource -r vip-storage --locate)"
break
fi
[[ $i -eq 60 ]] && { warn "VIP not up after 120 s — check: crm_mon -1"; break; }
[[ $i -eq 60 ]] && { warn "VIPs not up after 120 s — check: crm_mon -1"; break; }
sleep 2
done
@@ -320,9 +467,11 @@ log ""
log "═══════════════════════════════════════════════════════════════"
log " HA cluster initialised."
log ""
log " crm_mon -1 — cluster status"
log " iscsiadm -m discovery -t st -p ${VIP} — verify iSCSI target"
log " showmount -e ${VIP} — verify NFS exports"
log " crm_mon -1 — cluster status"
log " iscsiadm -m discovery -t st -p ${VIP} — verify iSCSI (storage net)"
log " iscsiadm -m discovery -t st -p ${VIP_LAN} — verify iSCSI (LAN)"
log " showmount -e ${VIP} — verify NFS exports (storage net)"
log " showmount -e ${VIP_LAN} — verify NFS exports (LAN)"
log ""
log " To enable STONITH (after deploying fence SSH key):"
log " 1. Fill in VMID_NODE1 / VMID_NODE2 in cluster-enable-stonith.sh"
+100 -63
View File
@@ -32,6 +32,7 @@
# --skip-create-vms Skip VM creation (VMs already exist)
# --skip-add-hardware Skip net1/scsi1 attachment (already attached)
# --skip-boot-wait Skip boot/SSH wait (VMs already running)
# --skip-refresh-sops-keys Skip scanning running VMs for fresh SSH host keys
# --skip-cluster-init Skip cluster formation (cluster already configured)
# --skip-tests Skip acceptance tests
# --force-rebuild Pass --force-rebuild to create-proxmox-resource.sh
@@ -70,6 +71,7 @@ SKIP_SYNC_KEYS=false
SKIP_CREATE_VMS=false
SKIP_ADD_HARDWARE=false
SKIP_BOOT_WAIT=false
SKIP_REFRESH_SOPS_KEYS=false
SKIP_CLUSTER_INIT=false
SKIP_TESTS=false
FORCE_REBUILD=false
@@ -82,9 +84,9 @@ NODE1_HOST="ha-server-1"
NODE2_HOST="ha-server-2"
NODE1_IP="192.168.2.228"
NODE2_IP="192.168.2.227"
STORAGE_IP1="192.168.4.228"
STORAGE_IP2="192.168.4.227"
STORAGE_CIDR="192.168.4.0/29"
STORAGE_IP1="192.168.10.228"
STORAGE_IP2="192.168.10.227"
STORAGE_CIDR="192.168.10.224/29"
SSH_USER="${PROXMOX_SSH_USER:-wayne}"
# ── Argument parsing ──────────────────────────────────────────────────────────
@@ -108,8 +110,9 @@ while [[ $# -gt 0 ]]; do
--skip-sync-keys) SKIP_SYNC_KEYS=true; shift ;;
--skip-create-vms) SKIP_CREATE_VMS=true; shift ;;
--skip-add-hardware) SKIP_ADD_HARDWARE=true; shift ;;
--skip-boot-wait) SKIP_BOOT_WAIT=true; shift ;;
--skip-cluster-init) SKIP_CLUSTER_INIT=true; shift ;;
--skip-boot-wait) SKIP_BOOT_WAIT=true; shift ;;
--skip-refresh-sops-keys) SKIP_REFRESH_SOPS_KEYS=true; shift ;;
--skip-cluster-init) SKIP_CLUSTER_INIT=true; shift ;;
--skip-tests) SKIP_TESTS=true; shift ;;
--force-rebuild) FORCE_REBUILD=true; shift ;;
--destroy) DESTROY=true; shift ;;
@@ -275,14 +278,16 @@ fi
if ! $SKIP_CREATE_VMS; then
log "Phase 3: Building and creating VMs on ${NODE}"
REBUILD_FLAG=""
$FORCE_REBUILD && REBUILD_FLAG="--force-rebuild"
CREATE="${REPO_ROOT}/scripts/proxmox/create-proxmox-resource.sh"
for spec in "${VMID1}:ha-server-1:proxmox-ha-server-1" "${VMID2}:ha-server-2:proxmox-ha-server-2"; do
IFS=: read -r vmid host_name flake_target <<< "$spec"
log "Creating ${flake_target} (VMID ${vmid}) on ${NODE}..."
# Always --force-rebuild: create-proxmox-resource.sh only calls
# sync_remote_host_keys (which populates host-keys/ for proxmox.nix to
# bake the clan-var SSH key into the disko image) when it actually builds.
# Reusing a cached image skips that step, so destroy+recreate would reuse
# an image with a stale/random key baked in → sops fails on first boot.
run bash "$CREATE" \
--type vm \
--host "$host_name" \
@@ -291,7 +296,7 @@ if ! $SKIP_CREATE_VMS; then
--storage "$STORAGE" \
--memory "$MEMORY_MB" \
--cores "$CORES" \
${REBUILD_FLAG}
--force-rebuild
done
fi
@@ -325,6 +330,90 @@ if ! $SKIP_BOOT_WAIT; then
sleep 10
fi
# ── Phase 5.5: Refresh sops host-key registrations ───────────────────────────
#
# Disko builds raw disk images; each new VM boots with a freshly-generated SSH
# host key rather than the one pre-seeded in clan vars. This phase scans the
# actual running VMs, and if their ed25519 host keys differ from what clan vars
# record: updates the clan var pub-key files, rewrites the .sops.yaml age-key
# anchors, and re-encrypts all affected sops files so the nodes can decrypt
# secrets on the next nixos-rebuild. Safe no-op when keys haven't changed.
if ! $SKIP_REFRESH_SOPS_KEYS; then
if $DRY_RUN; then
logn "[dry-run] Would scan VM host keys and refresh .sops.yaml / secrets if needed"
else
log "Phase 5.5: Refreshing sops host-key registrations (disko key drift fix)"
SOPS_UPDATED=false
for spec in \
"${NODE1_IP}:proxmox-ha-server-1:${NODE1_HOST}" \
"${NODE2_IP}:proxmox-ha-server-2:${NODE2_HOST}"; do
IFS=: read -r node_ip flake_target host_name <<< "$spec"
CLAN_PUB="${REPO_ROOT}/vars/per-machine/${flake_target}/openssh/ssh_host_ed25519_key.pub/value"
logn "Scanning ed25519 host key from ${host_name} (${node_ip})..."
RAW=$(ssh-keyscan -t ed25519 "${node_ip}" 2>/dev/null | grep -v "^#") || true
if [[ -z "$RAW" ]]; then
logn "WARNING: no ed25519 key returned by ssh-keyscan for ${node_ip} — skipping"
continue
fi
# ssh-keyscan returns: <ip> ssh-ed25519 <b64key>
SCANNED_TYPE=$(awk '{print $2}' <<< "$RAW")
SCANNED_KEY=$(awk '{print $3}' <<< "$RAW")
SCANNED_PUBKEY="${SCANNED_TYPE} ${SCANNED_KEY} ${host_name}"
CURRENT=$(tr -d '\n' < "$CLAN_PUB" 2>/dev/null || true)
if [[ "$SCANNED_PUBKEY" == "$CURRENT" ]]; then
logn "${host_name}: clan var matches running key — no update needed"
continue
fi
logn "${host_name}: key drift detected — updating clan var"
logn " old: ${CURRENT}"
logn " new: ${SCANNED_PUBKEY}"
echo "$SCANNED_PUBKEY" > "$CLAN_PUB"
SOPS_UPDATED=true
# Rewrite the .sops.yaml anchor for this host with the new age key.
ANCHOR="${flake_target}" # e.g. proxmox-ha-server-1
NEW_AGE=$(echo "$SCANNED_PUBKEY" | \
nix run --quiet --no-warn-dirty nixpkgs#ssh-to-age 2>/dev/null)
if [[ -z "$NEW_AGE" ]]; then
err "ssh-to-age produced no output for ${host_name} — check nixpkgs#ssh-to-age"
fi
logn " new age key: ${NEW_AGE}"
sed -i "/&${ANCHOR} /s| age[a-z0-9]*$| ${NEW_AGE}|" "${REPO_ROOT}/.sops.yaml"
done
if $SOPS_UPDATED; then
logn "Running sops updatekeys on affected secrets..."
SOPS="nix run --quiet --no-warn-dirty nixpkgs#sops --"
(cd "${REPO_ROOT}" && \
$SOPS updatekeys -y secrets/common.yaml && \
$SOPS updatekeys -y secrets/ha-server-1.yaml && \
$SOPS updatekeys -y secrets/ha-server-2.yaml && \
$SOPS updatekeys -y secrets/ha-server-1.keytab && \
$SOPS updatekeys -y secrets/ha-server-2.keytab)
# Note: ha-corosync-authkey is re-generated and re-encrypted by cluster-init below.
logn "Committing refreshed host keys and re-encrypted secrets..."
(cd "${REPO_ROOT}" && \
git add \
vars/per-machine/proxmox-ha-server-1/openssh/ssh_host_ed25519_key.pub/value \
vars/per-machine/proxmox-ha-server-2/openssh/ssh_host_ed25519_key.pub/value \
.sops.yaml \
secrets/common.yaml \
secrets/ha-server-1.yaml \
secrets/ha-server-2.yaml \
secrets/ha-server-1.keytab \
secrets/ha-server-2.keytab && \
git commit -m "secrets(ha): refresh sops host-key registrations for new VM instances" || true)
logn "Sops keys refreshed and committed."
fi
fi
fi
# ── Phase 6: Cluster init ─────────────────────────────────────────────────────
if ! $SKIP_CLUSTER_INIT; then
@@ -355,58 +444,6 @@ if ! $SKIP_CLUSTER_INIT; then
"sudo mkdir -p /root/.ssh && sudo cp /tmp/cluster-init-key /root/.ssh/cluster-init-key && \
sudo chmod 600 /root/.ssh/cluster-init-key && rm -f /tmp/cluster-init-key"
# Fix targetctl.service on both nodes: the iscsi-target.nix module bakes
# pkgs.targetcli-fb for the targetctl binary, but targetctl is actually in
# rtslib-fb (a different store path). Apply a runtime dropin that corrects
# both ExecStart and ExecStop before Pacemaker ever touches the service.
# The fixed iscsi-target.nix module will make this redundant on next rebuild.
logn "Patching targetctl.service on both nodes..."
_patch_targetctl() {
local ip="$1"
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${ip}" sudo bash << 'PATCH'
set -euo pipefail
TC=$(find /nix/store -maxdepth 4 -path '*/python3*env/bin/targetctl' 2>/dev/null | head -1)
PY=$(find /nix/store -maxdepth 4 -path '*/python3*env/bin/python3' -name 'python3' 2>/dev/null | \
while IFS= read -r p; do "$p" -c "import rtslib_fb" 2>/dev/null && echo "$p" && break; done | head -1)
[[ -n "$TC" && -n "$PY" ]] || { echo "targetctl or python3+rtslib_fb not found"; exit 1; }
# Write stop script that saves LIO config then tears down kernel state
"$PY" - "$TC" "$PY" << 'PYEOF'
import sys, os, stat
tc, py = sys.argv[1], sys.argv[2]
script = f"""#!{py}
import subprocess, sys, rtslib_fb
root = rtslib_fb.RTSRoot()
targets = list(root.targets)
if targets:
r = subprocess.run(["{tc}", "save", "/etc/target/saveconfig.json"], capture_output=True)
print(f"saved {{len(targets)}} target(s); rc={{r.returncode}}")
else:
print("no active LIO targets")
for t in targets:
try:
for tpg in list(t.tpgs): tpg.enable = False
t.delete()
except Exception as e: print(f"warn: {{e}}", file=sys.stderr)
for so in list(root.storage_objects):
try: so.delete()
except Exception as e: print(f"warn: {{e}}", file=sys.stderr)
print("LIO kernel target cleared")
"""
path = "/run/ha-targetctl-stop.py"
with open(path, "w") as f: f.write(script)
os.chmod(path, 0o755)
print(f"wrote {path}")
PYEOF
mkdir -p /run/systemd/system/targetctl.service.d
printf '[Service]\nExecStart=\nExecStart=%s restore /etc/target/saveconfig.json\nExecStop=\nExecStop=/run/ha-targetctl-stop.py\n' \
"$TC" > /run/systemd/system/targetctl.service.d/fix-exec.conf
systemctl daemon-reload
echo "patched on $(hostname)"
PATCH
}
_patch_targetctl "${NODE1_IP}"
_patch_targetctl "${NODE2_IP}"
logn "Uploading cluster-init.sh to ${NODE1_HOST}..."
scp -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"$CLUSTER_INIT" "${HA_USER}@${NODE1_IP}:/tmp/cluster-init.sh"
@@ -415,7 +452,7 @@ PATCH
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo env NODE1=${NODE1_HOST} NODE2=${NODE2_HOST} \
NODE1_IP=${NODE1_IP} NODE2_IP=${NODE2_IP} \
VIP=192.168.2.229 XFS_MOUNT=/srv/ha-data \
VIP=192.168.20.229 XFS_MOUNT=/srv/ha-data \
ISCSI_IQN=iqn.2026-01.home.sweet:ha-storage \
VMID_NODE1=${VMID1} VMID_NODE2=${VMID2} \
HA_USER=${HA_USER} HA_KEY=/root/.ssh/cluster-init-key \
@@ -454,7 +491,7 @@ if ! $SKIP_TESTS; then
else
NODE1="$NODE1_HOST" NODE2="$NODE2_HOST" \
NODE1_IP="$NODE1_IP" NODE2_IP="$NODE2_IP" \
VIP="192.168.2.229" \
VIP="192.168.20.229" \
bash "${REPO_ROOT}/scripts/ha/acceptance-tests.sh"
fi
fi
+256
View File
@@ -0,0 +1,256 @@
#!/usr/bin/env bash
# failover.sh — graceful HA cluster failover
#
# Detects which node is active and moves all resources to the other node by
# putting the active node into Pacemaker standby. Waits for the XFS mount to
# appear on the target before returning.
#
# Usage:
# scripts/ha/failover.sh [--to node1|node2] [--force] [--timeout <s>] [--dry-run]
#
# --to node1|node2 target node (default: the node that is NOT currently active)
# --force skip the interactive confirmation prompt
# --timeout <s> seconds to wait for resources to move (default: 120)
# --dry-run show what would be done without changing anything
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}"
NODE2_IP="${NODE2_IP:-192.168.2.227}"
VIP="${VIP:-192.168.20.229}"
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"
HA_USER="${HA_USER:-nixos}"
# ──────────────────────────────────────────────────────────────────────────
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
# ── Argument parsing ───────────────────────────────────────────────────────
TARGET_NODE=""
FORCE=false
DRY_RUN=false
TIMEOUT=120
while [[ $# -gt 0 ]]; do
case "$1" in
--to)
shift
case "${1:-}" in
node1|ha-server-1) TARGET_NODE="$NODE1" ;;
node2|ha-server-2) TARGET_NODE="$NODE2" ;;
*) echo "ERROR: --to must be node1, node2, ha-server-1, or ha-server-2"; exit 1 ;;
esac
;;
--force) FORCE=true ;;
--dry-run) DRY_RUN=true ;;
--timeout) shift; TIMEOUT="${1:?--timeout requires a value}" ;;
*) echo "Unknown argument: $1"; echo "Usage: $0 [--to node1|node2] [--force] [--timeout <s>] [--dry-run]"; exit 1 ;;
esac
shift
done
DRY_PREFIX=""
$DRY_RUN && DRY_PREFIX="[dry-run] "
echo "════════════════════════════════════════════════════"
echo " HA Cluster Failover — $(date '+%Y-%m-%d %H:%M:%S')"
$DRY_RUN && echo " MODE: dry-run — no changes will be made"
echo "════════════════════════════════════════════════════"
# ── Detect active node ─────────────────────────────────────────────────────
echo ""
echo "Detecting active node..."
CRM_OUT=""
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" true 2>/dev/null; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
fi
if [[ -z "$CRM_OUT" ]]; then
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" true 2>/dev/null; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
fi
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" — the * bullet
# means ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -z "$ACTIVE_NODE" ]]; then
echo ""
echo "ERROR: could not determine active node from crm_mon."
echo " Is Pacemaker still settling? Try running scripts/ha/health.sh first."
echo " If Pacemaker is down on both nodes, manual recovery is required."
exit 1
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
STANDBY_NODE="$NODE2"
STANDBY_IP="$NODE2_IP"
na() { n1 "$@"; }
ns() { n2 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
STANDBY_NODE="$NODE1"
STANDBY_IP="$NODE1_IP"
na() { n2 "$@"; }
ns() { n1 "$@"; }
fi
echo " Active: $ACTIVE_NODE ($ACTIVE_IP)"
echo " Standby: $STANDBY_NODE ($STANDBY_IP)"
# ── Validate target ────────────────────────────────────────────────────────
if [[ -n "$TARGET_NODE" ]]; then
if [[ "$TARGET_NODE" == "$ACTIVE_NODE" ]]; then
echo ""
echo "ERROR: $TARGET_NODE is already the active node — nothing to do."
exit 1
fi
echo " Target: $TARGET_NODE (as requested)"
else
echo " Target: $STANDBY_NODE (auto — the other node)"
fi
# ── Pre-checks ─────────────────────────────────────────────────────────────
echo ""
echo "Pre-checks..."
DRBD_DSTATE=$(na "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_DSTATE" | grep -q "UpToDate/UpToDate"; then
echo ""
echo " WARNING: DRBD dstate is '$DRBD_DSTATE' (not UpToDate/UpToDate)."
echo " Failing over with a partially-synced disk risks split-brain."
if ! $FORCE; then
echo " Use --force to proceed anyway (not recommended)."
exit 1
fi
echo " --force specified — proceeding despite non-ideal DRBD state."
else
echo " DRBD dstate: $DRBD_DSTATE — OK"
fi
QUORUM_OK=$(na "corosync-quorumtool -s 2>/dev/null | grep -c 'Quorate:.*Yes'" 2>/dev/null || echo "0")
if [[ "$QUORUM_OK" -lt 1 ]]; then
echo " ERROR: cluster does not have quorum — failover would be unsafe."
exit 1
fi
echo " Quorum: OK"
# ── Confirm ────────────────────────────────────────────────────────────────
if ! $FORCE && ! $DRY_RUN; then
echo ""
echo " This will move all resources from $ACTIVE_NODE$STANDBY_NODE."
echo " VIP and services will be unreachable for ~1030 seconds."
printf " Proceed? [y/N] "
read -r ANSWER
[[ "${ANSWER,,}" == "y" || "${ANSWER,,}" == "yes" ]] || { echo "Aborted."; exit 0; }
fi
# ── Capture active node's crm_node name ───────────────────────────────────
# crm_node -n returns the node name as registered in Pacemaker (may differ
# from hostname if Pacemaker was configured with explicit node names).
ACTIVE_CRMD_NAME=$(na "crm_node -n 2>/dev/null" 2>/dev/null || echo "$ACTIVE_NODE")
# ── Perform failover ───────────────────────────────────────────────────────
echo ""
echo "${DRY_PREFIX}Putting $ACTIVE_NODE into standby (resources will migrate to $STANDBY_NODE)..."
if ! $DRY_RUN; then
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v on" 2>/dev/null || true
fi
# ── Wait for resources to move ─────────────────────────────────────────────
echo "${DRY_PREFIX}Waiting up to ${TIMEOUT}s for XFS to mount on $STANDBY_NODE..."
MOVED=false
SPIN_CHARS=('|' '/' '-' '\')
SPIN_I=0
if $DRY_RUN; then
echo " [dry-run] would wait for mountpoint $XFS_MOUNT on $STANDBY_NODE"
MOVED=true
else
for i in $(seq 1 "$TIMEOUT"); do
if ns "mountpoint -q '${XFS_MOUNT}' 2>/dev/null" 2>/dev/null; then
printf "\r%-80s\r" ""
echo " Resources moved in ${i}s"
MOVED=true
break
fi
SPIN_I=$(( SPIN_I + 1 ))
SC="${SPIN_CHARS[$((SPIN_I % 4))]}"
printf "\r [%s] waiting... (%ds) " "$SC" "$i"
sleep 1
done
fi
if ! $MOVED; then
echo ""
echo "ERROR: XFS did not mount on $STANDBY_NODE within ${TIMEOUT}s."
echo ""
echo " Current resource state:"
na "crm_mon -1 2>/dev/null" 2>/dev/null | grep -E 'Started|Stopped|Promoted|Unpromoted|FAILED' | sed 's/^/ /' || true
echo ""
echo " Clearing standby to restore $ACTIVE_NODE (undo the failover attempt)..."
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
na "crm_resource --cleanup" 2>/dev/null || true
exit 1
fi
# ── Clear failure history ──────────────────────────────────────────────────
echo "${DRY_PREFIX}Clearing Pacemaker failure history..."
if ! $DRY_RUN; then
ns "crm_resource --cleanup 2>/dev/null" 2>/dev/null || true
fi
# ── Re-enable original active node as standby ─────────────────────────────
echo "${DRY_PREFIX}Re-enabling $ACTIVE_NODE (now standby — will not claim resources)..."
if ! $DRY_RUN; then
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
fi
# ── Wait briefly for DRBD resync to begin ─────────────────────────────────
if ! $DRY_RUN; then
sleep 5
fi
# ── Final state ────────────────────────────────────────────────────────────
echo ""
echo "Failover complete. Final state:"
echo ""
CRM_OUT_AFTER=""
if ! $DRY_RUN; then
CRM_OUT_AFTER=$(ns "crm_mon -1" 2>/dev/null || na "crm_mon -1" 2>/dev/null || true)
else
CRM_OUT_AFTER="$CRM_OUT"
fi
NEW_ACTIVE=$(echo "$CRM_OUT_AFTER" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -n "$NEW_ACTIVE" ]]; then
if [[ "$NEW_ACTIVE" == "$ACTIVE_NODE" ]]; then
echo " WARNING: $ACTIVE_NODE is still showing as active in crm_mon."
echo " Pacemaker may still be settling — check again in a few seconds."
else
echo " Active: $NEW_ACTIVE"
echo " Standby: $ACTIVE_NODE"
fi
fi
echo ""
RESOURCES_AFTER=$(echo "$CRM_OUT_AFTER" | awk '/Full List of Resources/,0' | tail -n +2 || true)
[[ -z "$RESOURCES_AFTER" ]] && RESOURCES_AFTER=$(echo "$CRM_OUT_AFTER" | \
grep -E 'Started|Stopped|Promoted|Unpromoted|FAILED|Master|Slave' || true)
[[ -n "$RESOURCES_AFTER" ]] && echo "$RESOURCES_AFTER" | sed 's/^/ /'
echo ""
echo " (DRBD resync of $ACTIVE_NODE may take a moment; monitor with:"
echo " ssh nixos@${ACTIVE_IP} 'sudo watch -n3 cat /proc/drbd')"
echo ""
echo "════════════════════════════════════════════════════"
Regular → Executable
View File
+206
View File
@@ -0,0 +1,206 @@
#!/usr/bin/env bash
# health.sh — HA cluster health snapshot (read-only, non-destructive)
#
# Prints a compact status panel across both nodes: SSH reachability, quorum,
# DRBD state, Pacemaker resources, and service ports via the VIP.
# Run from any host with SSH access to the HA nodes.
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.20.229}" # vars.haServerVip (storage-client, VLAN 20 — internal only)
VIP_LAN="${VIP_LAN:-192.168.2.229}" # vars.haServerLanVip (LAN, VLAN 2 — reachable from workstation)
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
HA_USER="${HA_USER:-nixos}"
# ──────────────────────────────────────────────────────────────────────────
REACHABLE_1=false
REACHABLE_2=false
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
probe_node() {
local ip=$1
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${ip}" true 2>/dev/null && echo "ONLINE" || echo "OFFLINE"
}
section() { echo ""; echo "── $* ──"; }
echo "════════════════════════════════════════════════════"
echo " HA Cluster Health — $(date '+%Y-%m-%d %H:%M:%S')"
echo "════════════════════════════════════════════════════"
# ── Node reachability ──────────────────────────────────────────────────────
section "Nodes"
N1_STATUS=$(probe_node "$NODE1_IP")
N2_STATUS=$(probe_node "$NODE2_IP")
[[ "$N1_STATUS" == "ONLINE" ]] && REACHABLE_1=true
[[ "$N2_STATUS" == "ONLINE" ]] && REACHABLE_2=true
if ! $REACHABLE_1 && ! $REACHABLE_2; then
echo " ERROR: both nodes unreachable — cannot continue."
exit 1
fi
# ── Detect active node ─────────────────────────────────────────────────────
# crm_mon 2.x formats the Promoted line as " * Promoted: [ node ]" — the
# bullet * means ^\s*(Promoted|Masters): never matches. Filter out Unpromoted
# first, then match anywhere on the line.
ACTIVE_NODE=""
CRM_OUT=""
if $REACHABLE_1; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
elif $REACHABLE_2; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
ACTIVE_NODE=$(echo "${CRM_OUT:-}" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
STANDBY_NODE=""
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
STANDBY_NODE="$NODE2"
elif [[ "$ACTIVE_NODE" == "$NODE2" ]]; then
STANDBY_NODE="$NODE1"
fi
n1_tag=""; n2_tag=""
[[ "$ACTIVE_NODE" == "$NODE1" ]] && n1_tag=" [ACTIVE]" || n1_tag=" [STANDBY]"
[[ "$ACTIVE_NODE" == "$NODE2" ]] && n2_tag=" [ACTIVE]" || n2_tag=" [STANDBY]"
[[ -z "$ACTIVE_NODE" ]] && { n1_tag=""; n2_tag=""; }
printf " %-14s [%s]%s\n" "$NODE1" "$N1_STATUS" "$n1_tag"
printf " %-14s [%s]%s\n" "$NODE2" "$N2_STATUS" "$n2_tag"
if [[ -z "$ACTIVE_NODE" ]]; then
echo ""
echo " WARNING: could not determine active node from crm_mon."
echo " Pacemaker may still be settling, or both nodes may be in standby."
fi
# ── Quorum ─────────────────────────────────────────────────────────────────
section "Quorum"
if $REACHABLE_1; then
QUORUM=$(n1 "corosync-quorumtool -s 2>/dev/null" 2>/dev/null || echo "")
elif $REACHABLE_2; then
QUORUM=$(n2 "corosync-quorumtool -s 2>/dev/null" 2>/dev/null || echo "")
fi
if [[ -z "${QUORUM:-}" ]]; then
echo " corosync-quorumtool: unavailable"
else
QUORATE=$(echo "$QUORUM" | grep "Quorate:" | awk '{print $2}' || echo "?")
VOTES=$(echo "$QUORUM" | grep "Total votes:" | awk '{print $3}' || echo "?")
NEEDED=$(echo "$QUORUM" | grep "Quorum votes:" | awk '{print $3}' || echo "?")
echo " Quorate: $QUORATE Votes: $VOTES / Expected: $NEEDED"
fi
# ── DRBD ───────────────────────────────────────────────────────────────────
section "DRBD (ha-data)"
drbd_info_from() {
local node=$1 run=$2
local role dstate cs pct
role=$($run "drbdadm role ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
dstate=$($run "drbdadm dstate ha-data 2>/dev/null" 2>/dev/null || echo "unknown")
cs=$($run "grep -oE 'cs:[A-Za-z]+' /proc/drbd 2>/dev/null | head -1 | sed 's/cs://'" 2>/dev/null || echo "unknown")
pct=$($run "grep -oE \"sync'ed:[[:space:]]+[0-9.]+\" /proc/drbd 2>/dev/null | grep -oE '[0-9.]+$' | head -1" 2>/dev/null || echo "")
printf " %-14s role: %-22s dstate: %-25s cs: %s" \
"$node" "${role:-unknown}" "${dstate:-unknown}" "${cs:-unknown}"
[[ -n "$pct" ]] && printf " syncing: %s%%" "$pct"
echo ""
}
$REACHABLE_1 && drbd_info_from "$NODE1" n1 || echo " $NODE1 [OFFLINE]"
$REACHABLE_2 && drbd_info_from "$NODE2" n2 || echo " $NODE2 [OFFLINE]"
# ── Pacemaker (full crm_mon output) ───────────────────────────────────────
section "Pacemaker"
if [[ -n "${CRM_OUT:-}" ]]; then
echo "$CRM_OUT" | sed 's/^/ /'
else
echo " crm_mon returned no output — trying again without suppression:"
if $REACHABLE_1; then
n1 "crm_mon -1" || true
elif $REACHABLE_2; then
n2 "crm_mon -1" || true
fi
fi
# ── XFS mount ─────────────────────────────────────────────────────────────
section "XFS Mount ($XFS_MOUNT)"
check_mount() {
local node=$1 run=$2
local status
if $run "mountpoint -q '$XFS_MOUNT' 2>/dev/null" 2>/dev/null; then
local usage
usage=$($run "df -h '$XFS_MOUNT' 2>/dev/null | tail -1 | awk '{print \$3\"/\"\$2\" used (\"\$5\")\";}'" 2>/dev/null || echo "")
status="mounted"
[[ -n "$usage" ]] && status="mounted $usage"
else
status="not mounted"
fi
printf " %-14s %s\n" "$node" "$status"
}
$REACHABLE_1 && check_mount "$NODE1" n1 || echo " $NODE1 [OFFLINE]"
$REACHABLE_2 && check_mount "$NODE2" n2 || echo " $NODE2 [OFFLINE]"
# ── LAN VIP (NFS) — reachable from workstation ────────────────────────────────
section "LAN VIP ($VIP_LAN) — NFS"
if ping -c1 -W2 "$VIP_LAN" >/dev/null 2>&1; then
echo " Ping OK"
else
echo " Ping UNREACHABLE"
fi
if bash -c "echo >/dev/tcp/${VIP_LAN}/2049" 2>/dev/null; then
printf " %-10s port %-5s OK\n" "NFS" "2049"
else
printf " %-10s port %-5s UNREACHABLE\n" "NFS" "2049"
fi
# ── Storage VIP (NFS + iSCSI) — VLAN 20 internal bridge, tested via active node ─
section "Storage VIP ($VIP) — NFS + iSCSI (via ${ACTIVE_NODE:-unknown})"
run_active_raw() {
local active_ip=""
[[ "$ACTIVE_NODE" == "$NODE1" ]] && active_ip="$NODE1_IP"
[[ "$ACTIVE_NODE" == "$NODE2" ]] && active_ip="$NODE2_IP"
[[ -z "$active_ip" ]] && return 1
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${active_ip}" "$@" 2>/dev/null
}
if [[ -z "$ACTIVE_NODE" ]]; then
echo " Cannot determine active node — skipping"
else
if run_active_raw "ping -c1 -W2 '$VIP' >/dev/null 2>&1"; then
echo " Ping OK"
else
echo " Ping UNREACHABLE"
fi
if run_active_raw "bash -c 'echo >/dev/tcp/${VIP}/2049' 2>/dev/null"; then
printf " %-10s port %-5s OK\n" "NFS" "2049"
else
printf " %-10s port %-5s UNREACHABLE\n" "NFS" "2049"
fi
if run_active_raw "bash -c 'echo >/dev/tcp/${VIP}/3260' 2>/dev/null"; then
printf " %-10s port %-5s OK\n" "iSCSI" "3260"
else
printf " %-10s port %-5s UNREACHABLE\n" "iSCSI" "3260"
fi
fi
echo ""
echo "════════════════════════════════════════════════════"
if [[ -n "$ACTIVE_NODE" ]]; then
echo " Active: $ACTIVE_NODE Standby: $STANDBY_NODE"
else
echo " Active: unknown (Pacemaker not settled)"
fi
echo "════════════════════════════════════════════════════"
echo ""
+274
View File
@@ -0,0 +1,274 @@
#!/usr/bin/env bash
# resize-data-disk.sh — online resize of the HA cluster data disk
#
# Three-phase process (all online-safe, no downtime required):
# 1. Proxmox: grow scsi1 on both HA VMs (qm resize)
# 2. Guest: rescan block device on both nodes so the kernel sees the new size
# 3. DRBD + XFS: drbdadm resize, then xfs_growfs — active node only
#
# Usage:
# scripts/ha/resize-data-disk.sh --size +20G [--force] [--dry-run]
#
# --size +NNg amount to grow scsi1 by, e.g. +20G, +50G (required)
# XFS and DRBD cannot shrink; only positive deltas accepted
# --force skip the interactive confirmation prompt
# --dry-run show what would be done without changing anything
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=../env.sh
source "${SCRIPT_DIR}/../env.sh"
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}"
NODE2_IP="${NODE2_IP:-192.168.2.227}"
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}"
DRBD_RESOURCE="${DRBD_RESOURCE:-ha-data}"
DATA_DISK_SLOT="${DATA_DISK_SLOT:-scsi1}" # Proxmox disk name (scsi1 = data disk)
HA_USER="${HA_USER:-nixos}"
PVE_HOST="${PVE_HOST:-${PVE1_HOST}}"
PVE_SSH_USER="${PVE_SSH_USER:-${PROXMOX_SSH_USER}}"
PVE_SUDO=""
[[ "$PVE_SSH_USER" != "root" ]] && PVE_SUDO="sudo"
# By-id symlink for the data disk; basename resolves to the raw block device.
# matches variables.nix's haServerDrbdDisk.
DATA_DISK_BYID="${DATA_DISK_BYID:-scsi-0QEMU_QEMU_HARDDISK_drive-${DATA_DISK_SLOT}}"
# ──────────────────────────────────────────────────────────────────────────
pve() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=10 \
"${PVE_SSH_USER}@${PVE_HOST}" "$@"; }
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
# ── Argument parsing ───────────────────────────────────────────────────────
SIZE=""
FORCE=false
DRY_RUN=false
while [[ $# -gt 0 ]]; do
case "$1" in
--size) shift; SIZE="${1:?--size requires a value (e.g. +20G)}" ;;
--force) FORCE=true ;;
--dry-run) DRY_RUN=true ;;
*) echo "Unknown argument: $1"
echo "Usage: $0 --size +NNg [--force] [--dry-run]"
exit 1 ;;
esac
shift
done
if [[ -z "$SIZE" ]]; then
echo "ERROR: --size is required (e.g. --size +20G)"
echo "Usage: $0 --size +NNg [--force] [--dry-run]"
exit 1
fi
# Only positive deltas — qm resize, DRBD, and XFS all refuse to shrink.
if [[ ! "$SIZE" =~ ^\+[0-9]+(G|M|T)$ ]]; then
echo "ERROR: --size must be a positive delta like +20G, +50G, +500M, +2T"
echo " (qm resize, drbdadm resize, and xfs_growfs can only grow, not shrink)"
exit 1
fi
DRY_PREFIX=""
$DRY_RUN && DRY_PREFIX="[dry-run] "
echo "════════════════════════════════════════════════════"
echo " HA Data Disk Resize — $(date '+%Y-%m-%d %H:%M:%S')"
echo " Size delta: $SIZE • Proxmox: $PVE_HOST"
$DRY_RUN && echo " MODE: dry-run — no changes will be made"
echo "════════════════════════════════════════════════════"
# ── Detect active node ─────────────────────────────────────────────────────
echo ""
echo "Detecting active node..."
CRM_OUT=""
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE1_IP}" true 2>/dev/null; then
CRM_OUT=$(n1 "crm_mon -1" 2>/dev/null || true)
fi
if [[ -z "$CRM_OUT" ]]; then
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
"${HA_USER}@${NODE2_IP}" true 2>/dev/null; then
CRM_OUT=$(n2 "crm_mon -1" 2>/dev/null || true)
fi
fi
# crm_mon 2.x formats Promoted lines as " * Promoted: [ node ]" (bullet *),
# so ^\s*(Promoted|Masters): never matches; filter Unpromoted first instead.
ACTIVE_NODE=$(echo "$CRM_OUT" | grep -v 'Unpromoted\|Unmanaged' | \
grep -E '(Promoted|Masters):' | \
grep -oE '\b(ha-server-[0-9]+)\b' | head -1 || true)
if [[ -z "$ACTIVE_NODE" ]]; then
echo "ERROR: could not determine active node from crm_mon."
echo " Is Pacemaker still settling? Try running scripts/ha/health.sh first."
exit 1
fi
if [[ "$ACTIVE_NODE" == "$NODE1" ]]; then
ACTIVE_IP="$NODE1_IP"
na() { n1 "$@"; }
else
ACTIVE_IP="$NODE2_IP"
na() { n2 "$@"; }
fi
echo " Active node: $ACTIVE_NODE ($ACTIVE_IP)"
# ── Pre-check DRBD state ───────────────────────────────────────────────────
echo ""
echo "Pre-checks..."
DRBD_DSTATE=$(na "drbdadm dstate ${DRBD_RESOURCE} 2>/dev/null" 2>/dev/null || echo "unknown")
if ! echo "$DRBD_DSTATE" | grep -q "UpToDate/UpToDate"; then
echo " WARNING: DRBD dstate is '$DRBD_DSTATE' (expected UpToDate/UpToDate)."
echo " Resizing with a partially-synced disk may cause issues."
if ! $FORCE; then
echo " Use --force to proceed anyway."
exit 1
fi
echo " --force specified — proceeding despite non-ideal DRBD state."
else
echo " DRBD dstate: $DRBD_DSTATE — OK"
fi
# ── Find VMIDs on Proxmox ─────────────────────────────────────────────────
echo ""
echo "Looking up VM IDs on ${PVE_HOST}..."
QM_LIST=$(pve "$PVE_SUDO qm list 2>/dev/null" || true)
VMID1=$(echo "$QM_LIST" | awk -v name="$NODE1" '$0 ~ name {print $1}' | head -1)
VMID2=$(echo "$QM_LIST" | awk -v name="$NODE2" '$0 ~ name {print $1}' | head -1)
if [[ -z "$VMID1" ]]; then
echo " ERROR: could not find VMID for $NODE1 on $PVE_HOST"
echo " qm list output:"
echo "$QM_LIST" | sed 's/^/ /'
exit 1
fi
if [[ -z "$VMID2" ]]; then
echo " ERROR: could not find VMID for $NODE2 on $PVE_HOST"
echo " qm list output:"
echo "$QM_LIST" | sed 's/^/ /'
exit 1
fi
echo " $NODE1: VMID $VMID1"
echo " $NODE2: VMID $VMID2"
# ── Confirm ────────────────────────────────────────────────────────────────
if ! $FORCE && ! $DRY_RUN; then
echo ""
echo " Plan:"
echo " Phase 1 — qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE} (on $PVE_HOST)"
echo " qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE} (on $PVE_HOST)"
echo " Phase 2 — block device rescan on $NODE1 and $NODE2"
echo " Phase 3 — drbdadm resize + xfs_growfs on $ACTIVE_NODE"
echo " No downtime required (all operations are online-safe)."
printf " Proceed? [y/N] "
read -r ANSWER
[[ "${ANSWER,,}" == "y" || "${ANSWER,,}" == "yes" ]] || { echo "Aborted."; exit 0; }
fi
# ═══════════════════════════════════════════════════════════════
# Phase 1 — Resize both VM data disks in Proxmox
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 1 — Proxmox disk resize (${DATA_DISK_SLOT} ${SIZE} on both VMs) ──"
echo " ${DRY_PREFIX}qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE} ($NODE1 on ${PVE_HOST})"
if ! $DRY_RUN; then
pve "$PVE_SUDO qm resize $VMID1 ${DATA_DISK_SLOT} ${SIZE}"
fi
echo " ${DRY_PREFIX}qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE} ($NODE2 on ${PVE_HOST})"
if ! $DRY_RUN; then
pve "$PVE_SUDO qm resize $VMID2 ${DATA_DISK_SLOT} ${SIZE}"
fi
echo " Phase 1 done."
# ═══════════════════════════════════════════════════════════════
# Phase 2 — Rescan block device on both guest nodes
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 2 — Block device rescan (both nodes) ──"
rescan_node() {
local node_name=$1 run_fn=$2
# Resolve block device name from the stable by-id symlink on the guest.
# Read-only lookup — safe to run even in dry-run so we show the real device.
local blk_dev=""
blk_dev=$($run_fn "bash -c 'basename \$(readlink -f /dev/disk/by-id/${DATA_DISK_BYID})'" 2>/dev/null || true)
if [[ -z "$blk_dev" ]]; then
echo " ERROR: /dev/disk/by-id/${DATA_DISK_BYID} not found on $node_name" >&2
echo " Check DATA_DISK_BYID or DATA_DISK_SLOT configuration." >&2
exit 1
fi
echo " ${DRY_PREFIX}Rescanning /dev/${blk_dev} on ${node_name}..."
if ! $DRY_RUN; then
$run_fn "bash -c 'echo 1 > /sys/block/${blk_dev}/device/rescan'" 2>/dev/null
local new_size
new_size=$($run_fn "lsblk -nd -o SIZE /dev/${blk_dev} 2>/dev/null" 2>/dev/null || echo "unknown")
echo " /dev/${blk_dev} on $node_name now reports: $new_size"
fi
}
rescan_node "$NODE1" n1
rescan_node "$NODE2" n2
echo " Phase 2 done."
# ═══════════════════════════════════════════════════════════════
# Phase 3 — Grow DRBD metadata, then XFS (active node only)
# ═══════════════════════════════════════════════════════════════
echo ""
echo "── Phase 3 — DRBD resize + XFS grow (on active node: $ACTIVE_NODE) ──"
echo " ${DRY_PREFIX}drbdadm resize ${DRBD_RESOURCE}"
if ! $DRY_RUN; then
na "drbdadm resize ${DRBD_RESOURCE}"
fi
echo " ${DRY_PREFIX}xfs_growfs ${XFS_MOUNT}"
if ! $DRY_RUN; then
na "xfs_growfs ${XFS_MOUNT}"
fi
echo " Phase 3 done."
# ── Verify ────────────────────────────────────────────────────────────────
echo ""
echo "── Verify ──"
if ! $DRY_RUN; then
DF_OUT=$(na "df -h '${XFS_MOUNT}' 2>/dev/null" 2>/dev/null || echo "")
if [[ -n "$DF_OUT" ]]; then
echo " ${XFS_MOUNT}:"
echo "$DF_OUT" | sed 's/^/ /'
fi
DRBD_DSTATE_AFTER=$(na "drbdadm dstate ${DRBD_RESOURCE} 2>/dev/null" 2>/dev/null || echo "unknown")
echo " DRBD dstate: $DRBD_DSTATE_AFTER"
if ! echo "$DRBD_DSTATE_AFTER" | grep -q "UpToDate/UpToDate"; then
echo " NOTE: DRBD is resyncing — normal immediately after resize."
echo " Monitor: ssh nixos@${ACTIVE_IP} 'sudo watch -n3 cat /proc/drbd'"
fi
else
echo " [dry-run] would verify df -h ${XFS_MOUNT} and drbdadm dstate on $ACTIVE_NODE"
fi
echo ""
echo "════════════════════════════════════════════════════"
echo " Resize complete."
echo " Active node: $ACTIVE_NODE"
echo "════════════════════════════════════════════════════"
echo ""
+131 -130
View File
@@ -1,234 +1,235 @@
root-hashedPassword: ENC[AES256_GCM,data:Kp0nOZI7vDoLhJHiOJBwJn0rQZ5yhnwapGnAcA+qh8vlDETtFs/iQdetF/2ZxmANf62SviTNd+Ag0q5JIF1996x7onZGXqxgSMCuVzZLBdUlsO5IR0BslWWz47khYGTe4WkUg4NB1itBfQ==,iv:5Sra5vJ79V8hxQT3g9qJ+dOj2W2sumIhqpitqnHjJdk=,tag:3Igu0+8GeUZHqS3fKUVwog==,type:str]
nixos-hashedPassword: ENC[AES256_GCM,data:pT7tVRN6X4a+DNUgB7fIUUE3CbnetkjxmoSL1PxSU+ktsFU+fB0mEvJjA1uujsGH5Rcztg7YM815+M0Z67ILmHaXbza5DtFacrqhi4/b277xly0SHRX4yOvBwQh6mJG1jn/0O/wvUUIYdw==,iv:bp2nfhC8nFbk6o5iWDAugvbzu7J/a1xayFnBEtkhNpE=,tag:HqWgkIpSrSM/K9OK2WO+VQ==,type:str]
nix-github-token: ENC[AES256_GCM,data:k1vYz7SqVhzpWa6jTL6NUD8lKOCpHCgTm+HT4IcnbzbSTUZP/bJUYw==,iv:UqAULZnr/4+VcioUDfTwvOSuwM8K9JgGhiApvYQPyoc=,tag:1LKHXhWAO/AHPDIZFBb04A==,type:str]
beszel-token: ENC[AES256_GCM,data:OWmSRkZjb11y0Y8GdobqiE9GFwzdHOvvxCbYx69qUghGYARN,iv:i/JhGH0O7ThxPkL0SLAjfN0Fq8prm7tybI5kF2NRNpw=,tag:dBcqxOSHTnD4xngpOog55Q==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJR3NKa2kxZGRHd3RWdWRR
WDkreDcxZzE1RVZ5NFFUN1pLUktzVHZSUVd3CmJkMzVsRVVraGZWVEZLWkQydVJI
NGF5Ui9ranRZU1lqS2RKZzRmWGJVYmMKLS0tIGdIRlRMOFBDMnNnT3hZdjcvUVA3
cUxXa0tjWWk3K0ZTNUx2OU9MRHpobzQKMZLQH+z8o27s1bAXyJI8HD8jHnU5JaZV
WLHwwptGYz7pYyCkWc25IkA0nR3KR4lfHT2o5eGn2BBivpI5o/QQMw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiU2IxZ056SUs3SVZiUTlQ
NXZEYmg3VzJzeWJPRnZXcUtOMGQ1MElBU0dVCm5xSURZZ0sxTjVxQnhpVVNEU092
bDBHbFAzZ3hzejc5NmM2U0FQTkRNSTQKLS0tIC9OOURPRlcvQXIxMVZXR1JjUWRX
bE5ja3FoWXFIZ2ZLTklqWHhMRHBTRWcK2Z4sEcdyvx1rdBIGrbNw7bYvEPXdobvF
h0aNihn817DvtVr8eeucj22iYCS40etElQNcO5i68t5wab4/d3rUbQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpOVVzNDZvcE1Kb1Zua2pF
cjllRWZkTDZUb0JtVW84Z01OOHF0Q3JsZW1RCmswV0lxUW02NEZzY3lWeHNXQ1Ez
K0VILzFhemZjTTFLbEpVZVdJY21HTTAKLS0tIHVqZ2E5aU1NVk10bEJ5UXkvMVFK
dXdSWEx2OXBSZG9aM0VGNVQzTjVRUkUKYgZjGG7I0ea9I+gG4Ah1VSkiONNAHdDQ
5zG9LnKQZ5faY4tWsX7JHFWDX2Y83mXbHV6jMCutAfQG2zt/5psTMw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrN0lNNGtYbE8rRTB0U2hL
eUNOZUNFRGk1RzE3R1VjV0Q3ckszcDNHWGxJCkhFN25JbDVSaWJIZ09uNzczbkM1
ZWZ1Y2wrTHZ6VzlnVWRoT2JOdDZHM0UKLS0tIHQ2ZE9BaFRCeDNqbldRQkpLclI4
dXZqOXNaR2Q2QWJtRlZUa0hLaTh2YlkKeCh3+yIt0sjh2Ig8m/YQifhMDHnTpuZE
zHBE74SUyWFmqLgtd1b8Rx5gn9vBcS/3SMv3yKWispg4DZ3bCOeNFA==
-----END AGE ENCRYPTED FILE-----
recipient: age19m0m7vdfg86yqy8l5mmle5jdd0unrn3f55t232w8h5ey42cqw34sfpt32n
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxS3VuaHlaVGJObVNPb3Rh
UEdBMk9meDQvWi85NVRHR3dXSlhkcFZVa1RBClZDR1MxOUdwVjhzSjFaWVFWelVX
cUgzMWNrUzl3L1R6elc1ZTRGamxDU2MKLS0tIDFwUUJFQ0hIUWNTVmFwdGt0czZi
V3VGSndCT0tQb2Nzd2w2TWxBbEFBeUEK8kQQVqISb3h0snOtqM0w/mhWEpjIxlhk
N84QzgxhD40tvutfO+57HsfOIqjKi4Yhdmbd8+iW9GzGnrbfrDuVGw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5c2luYmpYcVVxRHROQlpk
cTM4cm9uNEtVS21pK2pDRVRySmxYS3NwdGg0CnQvS3g4Q2tSdlpVeWZxbzArcTM1
YWRMcDAxcWgySFRqRC9XSElsTVorVEEKLS0tIFd6YjdrY2ZXQVBaZ01wMTIvV2sv
ZEk0TG0ybENlTlhpZnovMXh0OGd5V1kKEbjrmKP6Su16KRDQ4PMqi0tHRS6+PfuT
0v5DEyi8EDtIYSMzzdF7Jk2C0h5XuRux45Z6E6JhKW5c788dGQHz/A==
-----END AGE ENCRYPTED FILE-----
recipient: age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoYXBpZlpjT2oxMURTcW9t
NTdNWkMyT0d0SVBCSFBqVC9vNkxYUUk5Wnd3CmlyNUhIa0FmR0lwSFEwK2QzRjlO
ajNKMitzZGlDaXpRS1pNNDhhaFdPYjgKLS0tIFVtSjNXSWdlQTVwOEUzcjdEVExu
aklINzU2QUFsZmxPaUI5aFR2Z3V0bXMK/rDuNYW2r022tHjuk4KqIEOxqvjQsBIg
D2lG4ih5h3idm++gnhCSlP/9tKSaVHoq+BpunsmFObrrWzZRhKI6jw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB1eW5kd1FrSGJaSDkwTzdY
SnJhR2hqRkNTTGtSTWsvYmZpSlVWV0xvYkRZCmtqR1FQOTRQR3dGYU8zNFpMdkUr
SWN2KzhXaE1rYWZYNHJsOVVvRDY3R00KLS0tIG1aUldSWHlHeHkrcjF2TENvazR3
YXJLUXV3VVNSWEdseFlZQUJPQUF6OFkKOPKH52muUha+MV2leWDiqpuTaIj0ZKzV
keEdyqkIck6KKfIjphNKTW5ujiuKjGQbRliquBknfCRGMnRlMNfmkA==
-----END AGE ENCRYPTED FILE-----
recipient: age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArbkhEaEhoUk1SUHNjVm5C
Q2svcmpselJ1eU5qMVJKV2JWV1J0eUpSWlV3CnhIWDF2NXFMM0VMbDQyeHdkNDJV
aForNXJ0UFd0aXFGZzZhekNTdUd2U3cKLS0tIFljclNTa2I5RHJTUU1yMm5XakVo
amlHVmZzSDZRTndFSEpjQlVOUTVhRVEKz4Tx0PceUCxw8MsREB2HRPDyC/lIskhs
/HWmseMlnsoEMu8E1OIO0TpWY3qowHrdFFr4njpxMJUTXusGvRK8Qw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmTDExNFpCZHZmWnpwYis4
bFJGZXV5UksyS01MT0d3cXE2Wno2Zzh6OFFnCm9hT3MyZ0hTbnkzdXk3NmN1TUVF
OGtCZnlJRmZJaWJWaE1pMGcrTzJzZm8KLS0tIFE1L3ZxeUl6WVA2UXN2b0lIcTBl
YmtXZnZaMml6ejY1Y3lRUFFtTEVMS0UKZbcuunA9GRqctCfC0Fh2foKt7n3l8KkE
FNjjxUzL7OQjNG9M1aAtRMMO5KXL7YKXRUJWOPcpiLeLUNvDLeLXvA==
-----END AGE ENCRYPTED FILE-----
recipient: age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXbm1jWW1DZDVNdnVZOE5E
bTRLTzRhckVMNjV5VkMrcllLZk9NeVRiT0NrCkh3RDRicXFTQTNqSmw4VEQ0QlNt
cGJNRlFTSjU5YWFXWmM1MjBjVGZSeVkKLS0tIHFPOERUTGVOVSs5U0VMalAwUXcx
QTYvK2RScDErVENFdUhpYUVsejBYR0UKCNBN3n4q2X1YLturxpjDv4IlnLXoPtOM
DqXMNEgGubagOBhfVFrOjpFxonP7JThSfzEJT/pmL+S7bCvgyosYXA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBQYlp5U0FvVkdIKzBGZVRl
UEtOOG5kVHVLQ0ptVTVpdG9Lb1Y3bzlOODNzCmNYN0Y5ZGNzTGFGYmxJcWFxaExn
NkkrT2RLSlZTdmFiNks3RVZXQlBEYnMKLS0tIDdCVlBxUG8vYU05d3c2MXhjQjhM
TTVJUGxlWUxWalBRYzFKQUQzd1R6dHcKPz1ciNICph7qnqSH5WYmOjUCNHsIAmTX
lRnbFCZdD/ZIFRK8OYAyCGfQ42UYSTf16kH2jDdTFUy7Fk2+O7lNIA==
-----END AGE ENCRYPTED FILE-----
recipient: age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXdDRCMlVyaTdPWFQ5VVhh
UWMzUTBkZVYrSDRQWUpWLytDeHVmcVh0L0g0CjVIZE1jZHA0N3RtSmZKVThrcmt5
N3VrenFGTS9LUDdZRHBiNDhORE4wa2sKLS0tIGdieDQxQ0lWZnVMZUZzSExKWEJU
azl2Q01YdHZacDRSaElpZ0h3SG9xcGsK4s30qRC2eXbuKqPUHfRUJrE8FCMdz5EQ
25UhdrspVBadt0G92hHV06Uwm/KKnG4Mi5crLKIMI+HAF+5Uxrh/xA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBiemVqWW9LdlV0RjRuRHpF
U2MwN3RDeHU5SWhuQ2l4YStRVVE2eU9XSUc4ClNGMkNXQVdkWFkwcys0aFVhZnZv
Y3J6RmxwakdaZVo5VWZhb0lVTUdTYzQKLS0tIEQ2NW0yb2lGQ0NLZm1PSWNFVkQ1
a0Y1NlYwZ0dnbHhsQVB5eS9kZVdqbmsKgFLUpZMqGywO9Ext5WRwco2bpenVNrzL
XrGCqhn/7TW9DrJlPnFe8r4fl4DFP6GWByn7axDl58Qy1OzeBZwnNg==
-----END AGE ENCRYPTED FILE-----
recipient: age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwL2xrRnJyOWxRT01YbHI0
OG8vU2xQaEJySkZkQnp6ck90dmkzQlV1RER3CmtGc3duTGRxZXhlbDlHd2hGUDdH
QmxYTjBVbkVWRVlIcEhpaml2MlVzc0kKLS0tIFRWTnZGQm94VDhJK0J6SS9oa2FB
aGpJUGFydUhvQlpFaURWSm9VS1AzM28KWvU5knwB/ViUrSxbP0zPR4iUE4LXxYi3
qraWrbv3jXZwu4Sgv0H+/k6St/Xo5RU+nIqSOCKEy/kpACwk4BhMXg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLRzlzWEcwTUx6M1JNYWgr
ZWxnVXhFWkhQVjBrSndGTU9oY2MrOTQxTGhFCjhUdERPc3hzQlkzMjBEa3AvaWVX
T0RJWXFFZHdmUVBObGhpQktCQ2NPbWcKLS0tIDN2YkE4bEVtbjJlcndWVmlrQ3ZT
dlVvRmt4RklOdUJyZ25ia0ZoYjU4ZHcK5fL1yvRcty6+EMzfSkwwJM3qvrr6h46d
d7Zs+EElXrsGDQvL0fevB/zDcICAtudYnVv4nP3Jf8GQnL19zWlwbg==
-----END AGE ENCRYPTED FILE-----
recipient: age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3K2tIY25TZCtReTQ2Zkc5
d2w2aGZqYXNBSmZ2VVc5WWtZNDU4YlJpNm5VCkpGUkNOdGVSeXU1WG1YV2NQYjhu
c2NKd0czSm5FUWxmNTdHVGVVWGlUMWcKLS0tIDJmRy9vbGJDNGxMVDM3b25TRVVI
blpZTkZsTzNrdmFwUGQzR2RIOVJINlkKwa06bE6TmYtXPx0daHadbwL9/u1iYuGm
n62kEwTKdyfncODl895qqjWkiA3JnbwQxzsaVDtEGfauzUph9DPY2g==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhZElRNXNLK0tVUjQ0ektQ
Tzc5NDJReldiYVFFZnRLS3l5dWw3UVQ0WWhJCm1DYVloZGpYeVJtNUNRcEJvVUZE
Q0xMcmR0WWdVVTBaWko0dGI5V1hqaUkKLS0tIFBBeXdNY1pFai9QT2t3NWFLV2sw
cWpRU2RISSs1UXh3Wkl1dWI3VEk1bXMKP6v3Lm0XZkp3Gn3iJkB0K0Y0E5BeilqR
plmPh1rKbT/wkefFwpa8e2Wwmhx80YK+bjn633MH7mLQDf6AiK83Xw==
-----END AGE ENCRYPTED FILE-----
recipient: age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1ZlVxYkZ4UFJmRVBHeExi
OWZ0akNwc0dMenlHQ2dUSDlERGxSNzZtdFJJCnRYK2cwNHJRK0xtY0NnYlh1a1pG
aTk0WURSYVdUY1ZTSUhBNVpZZUFyaUUKLS0tIHZKTFZmS2ZsNlZ6UjZUWmhNK0Q4
RC9TREh3ZWZRTzc0WmFkQm5qVlZJcFkKUngIUXCVV1CKpUKfAlal1KjoJwk83mq5
1MvU9mjx8Oq/suUlW4axFfAO99mUc9yCwb8TCMUZDhQp802v5FjBcw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMVk8yT1FtMVoxYjF2YVBs
UTFVbmhzblBCRno2V3YvaERRN0xWcHd2NVE4CmVyK3p1SWtodk5iWjdkQjJRc0dM
Vm8zZHczYlFFSG91QThmUGNaWHBoN0UKLS0tIDZMMXoxbUM0TUVaNFUzRExLc2VM
OVFrNGZNL256cjg0Q3ZBQW9xUFcxSUEKv4HqaTcTzNYRJCmNEeUQLFfEoFXyOBEC
yZrg8D+ROBn98484bxNeoyOjQdcVn7OYiCv1y6utnyeA9sdMUB0ong==
-----END AGE ENCRYPTED FILE-----
recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRM0hrc0I3UCtBOXJWUm1H
MW5QWHAwV3lETFp2TGtPWVZHZ2s0ZHJmQ2w4CnBlQmlTVmEvRTlLMjVXMkRJeWIw
QUR3aXpPUndocCt4aWUybndxK3RUN00KLS0tIDJrNkY0d1Jqby9pa0tXWVVuTER5
YTlVNklTakFlaGQxdjJqRks5cUZoK3MKkV199oB7lPBzNYd30nSY0J6Cd09ViBA8
uZWlmMGKlveBQ98qOt/vDTouRDxedASz0ZGg7+jxUXkUKbujkCYYXw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1Rkl0UXduS3k3WVU1RzdG
N2IybE1TZ1kwVmM4Y2lvUS9DVG0zSkdpZGhRCnFkZnJVS2FBZm9Xa0V4SjRwaGdR
TWpaaTh4VEk4cUFrb2pURERzbWtSR0UKLS0tIEsybHdvUWg3UlUvL0lhdHIrUE0y
R0pMdnI5QlpIcWVWdGRYQ3JCaElLWTgK5cTfvehj4BPU22tYiR5piZdouUdeh5Gr
x/xSbFWqYVauDB6rpzZvhGMI4KCJh2RmDdt7FrlCb/YCB3QoSgvl8w==
-----END AGE ENCRYPTED FILE-----
recipient: age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwVmlDOHh6dUFaR1YxOHJM
N09LdWxscStGSTYvMm1QaWRkZVVod1JTTlJnCkp0Z1pSa3lVU2JqUnU0ZnAzWWEv
bHc3S2ZjK3E5Z1BRL2pYWklvSFoxSnMKLS0tIGxTd29MK0dRWFFnaGVwa2NVQXM2
TjRQdHRZNDFTTWNWR2FvZXhQMVZORncKdAcSOB50bNQsaTGtAqnvjYyFpcjdMGzO
NvwI5ZquoT/B37Xsg4NQ7++/ZkemzkuX5VYoXLVAQ9WJwQxewoMFKg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwL0NNbGlGS3lTZjEydGJm
MWZRK0Z5YktuNVNudk84MndLUUdXR0RlRFF3ClB4MWRUTFBWYUdwbityeUU2SlR4
azZMb012YXQzWHUxZnZzeVp3MmdzcU0KLS0tIFJ2SUNyMTBEY1NWejdVQ3hyUytQ
VnlQWHRDYWZFaU1ON2RQTXBwNFNGYUEKwnRUWCdsiStia1jLHJ6UNP/kgEt4Aw8U
pA8PKKu6TZVPI8pw7svtI/uvJuW+8ARuNcyb1FdYmX+sOHnOEqHyjg==
-----END AGE ENCRYPTED FILE-----
recipient: age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3V0M5bnBzamdGemU4Y0g0
TEdZdzQvUVpYSkozZlRsMWM4MStmN2lxckYwCms0ZTF3cTB5OTRzWWE0a1FXWkhG
SGptVCtiOHptV3V0aVNQUUJ4NkE0dnMKLS0tIEEvK0lwa1pnc1YwWXhkYmh3cy94
S2E0SUk4RDdSWktQTjZoTkNUallLTVEKNqTmhQo74Q00ZhrSrD+/JB3TwxL9OQdC
XSONu6XXzPzfhllNIwe3BnmRVJM/1ru8rvllk98Cvle0k4K5+xBJ/g==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBxM0FyaFlTR2FzMkhGTzZx
d1d3dWNGOG1ESllETEVxRGJlYWkrTUNIdkYwCmI4andLMUUvNVU3R0N2WCsvWWhV
UmlkcHdMSFA3bFp0ZG5OckE3YmJrejAKLS0tIDhEVEJYOFc5VWVHcGcvMm5yanBl
eGxaNEFxdEN4NWh3cnlGeElBUTJmUzAKvCSJF+LGf1JqEdkzT/Wk6hEzferseyWO
vAYR8Z+u6GDwQi7JdFz9iZKD21LAs0IMzd7cZqXKIXVedyXuaIhPtg==
-----END AGE ENCRYPTED FILE-----
recipient: age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKYnFtM29UUzVRYWpTN0w2
MDVmQk83THEwR3liN0hmemxBZlF3eDVZV0RRCjRFcjZKQk8yaXViWm0vTUdrSlM3
UDNnemoyMG0wbnpzK0FRYXNiRTRnaW8KLS0tIHd1QVYvWWl2QlcxVGZzbFBLbjlu
djV4dVpnQzF6LzFiWTREaTYrMHAreUkK82l9Njt3B7HJBLtdLjNSnQ4TDTvBgG9/
iUEn3fd9OmvfSGcbp4hhwB7q0UJqb7AP47ctwDOEg/FYeF7eSBvVgg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGZ2M0cG5nMU5nYVliRzlx
TVpDcHpwclhXNkxuc2VKanBvcnRxRW04UXh3CmNJeEl3K0FrWWVHQWRwK3Q4Tzk2
MEZQbFV1QURLZEhCdEhaMjBRUzllaVUKLS0tIFhSUTJ4SXJwRHhVbkZFc2Rwb3Fh
QzAxTnpWSHZaWlF3NUlFMlBHUkRUa2sK3Kim0mp2adSvruI6Necd3UpZHow0EFmT
JFOIpIK+3Pc06gsiqrDtDAx2Qtc10Ja4ILTutPlp4va6zzmzrPABiw==
-----END AGE ENCRYPTED FILE-----
recipient: age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoZnVZekVieVVRT2ViWUpN
bFk3ckJUU1RHRUJrbkNkTUZkbVEzRGd1VWtRCk03MFEweENBOWtOS3gvM3ZpNTF2
WG1UWTM3OFp3RERNNC9iV05aOTZnWmMKLS0tIFZqZ0NhZVhxODMyWENhVlI3SUR0
b2doaGp1VHBOUVFacG51Rk02d2o1MWMKnNYktA42DgSWWGuy0XTOtUN9ry49WOBd
NaGq27ke16XhQ0ZC+r2a1g7YAFaL+qWJxpaH+0ojhUjti3x9O3pWqw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwTUZ1RzJKcVF5bnJGamlU
bWJvcU1sTTJuME94YjhMZWRWRWR0aGJ3UUNvCjh6U1BkaEg0TDlCeTNyUlNTRWV6
Nnd0NjE3NmplSWFjVWpSRy9qWjBUbnMKLS0tIGdMZUFzbGdMejZoMlFBVUJXSE13
TW4rUHFjZk5oVkw0UGtpaG5mOHRRbHcKFX4ZyNQ0TUeqwXEaz1qmAXQRIOctDkvz
TIwxJseAjS3qVxCslnxzgaqmCh8DzCfjrRxLUS/HVTnwOaE1YAh0vg==
-----END AGE ENCRYPTED FILE-----
recipient: age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRTkQ0cm9MSTMzZjF5T0Vh
TWlOdVJzZ1JvQWdLTzZsYWZ6djJhTjk2dnpFCmxUZkhrOW5OTWtrS2FLOUdsWm9F
WVB2VzhTdGwxNzZIZ2lSRmRielAzZkUKLS0tIEd1RlIwQXhocmJkVWtvVGMvTlVB
WWtyVjUrWjYrWDFmc3VtdzJUaEFPeGMKvWONIbN6B9Ims3f0l/lfUTU116TFPXew
K6kAvyySbh9Z9JK8Uyp31WLSbqVy56eGr2Su8Hj6rAGIoJdo0v5NoQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTWStVRDhlblZDMDJ3ejIr
MkwyeHRwcnpWQVBmOXY0a3NCU1JnTjkvVGtVClVvVUpLTVhUNGNNLzY4U0o2SDRD
bGZteUU1Uzlzc0NjTW56OVNCcnVpUUEKLS0tIC8wN1V2Nkc0cVU3OHNYa1NBVlhq
YTV5QUpsQWppNHRpT1dMb2JDQnpoR3MKCoBngnNg5qpktg5j4CcuTUKiooOVI8IN
04plE+1lra8q5ZK8q4r1/K8hBM23poyfq+XushO1GZ17dPtWs9wITg==
-----END AGE ENCRYPTED FILE-----
recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlSWtxbHJsQllITit4NVZv
b2Z3YzJZTHFOTGU3VzkwWEFZUjdWTC96UjNVCi83SkYyaFdoTVN3eFhLQnA0bDVa
OExZRFl1M28yV0VTUC9hZUJJR3dYTWMKLS0tIGNuc00rZDI2dHFqWEc2R1ZVb1E5
Vm45YUpvWlMvMXUxdDB0NHpncWpIMjgKcMIZK2ww/VPuuBXlL8klSTm8ySME2njC
vbhCzg9VIqIV3q2i8WEbWFmo7uyiyF89Z52xf/3uUc/TUEqjVVDnZQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBUU2ZSazhZNWFZVXk0UXdx
Mjh3R1NLMThlaHUvWVFMZEQ4Z0NYOW83dWgwCkVnd2dhdzQrZkxaL1NRbDJodERQ
SkM5S0kxcVBtY3laNWp5OXpXZHhqZ0UKLS0tIHVOT2VUZU5DZjJjVW1kcU5qS3FD
YXNhRkpaREMzRDB5aWgrdVNvUk1qcWsKQnPqE+5xSBGeDMmLfKB0tEBt2eylBOxT
oi26mC+EoO5T2pCIpbD+DODAAlWKLattqCYHneUO9wRk9DnhuEXRBg==
-----END AGE ENCRYPTED FILE-----
recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZRkRvN2dBZ1ErZmNhSktm
LzNLdzJLR3RhR2VqS0xCdlVSVUJtRnpLTkVNCjFmZlp3U05Vb2FwbDBBdmhjVUJt
Uk01dDVzS1lWeEZueDlmcUw4MlFFTHMKLS0tIGJ1bENFSFZPa0tNNFZsblFOM05x
THAyY3draFB5eHpydk94Um1qU1BZb0UKUVszMlSqUPsEj3vVseGksI+SEEjEblNo
zK5fwkYH0aDubsotkVcJjoXc2ZArHyygIMImLNpRBJYIScDGzzTBRA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZUlFlenlOVDk1NmxQVGUx
UHhVK3ZtMHdzaXdtWmE1TXhOekpsTmUwSlJ3CmdncjMxOUw0alBQZkhZZjA0YzJZ
bFNVOHUySXRuYW8wVjdHMDN4ZThuQmsKLS0tIGdvUUdHcXlpWlZkL0xBcVpMN1E1
NGZCQURYQVY5ZzdDODNNY0o3SG04eDAKiYVuKsJLaSC7/bQRB51tX282/yqpAjWT
XNnqPQBQ7H3xuK0tDVdCAUMruOSlqAe9YRZwmiou1irpg9vZYwgTPA==
-----END AGE ENCRYPTED FILE-----
recipient: age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBic1JIWi9xKzVkVUtqdHBx
OVU1djJDbDcxYTFCSEQ0MTVRaS8zYm5QWFhRCjVNNFE0RUtPTlE5U3pRTlh2eVNy
OUU0cnpLNlluemdkVVNjWGxhV1NTRUEKLS0tIDNkWmhoSTNkWExWek4zdjduaFBB
UEJjYnZwYVN3L3p5Vk9sRGVQM3dld0kKsFjQKYOClBjWBaU3kCEP5yYGBphUfgOO
E1epPU6UI3asa2AC/svfFBZI3u/7EpxDH9jaSSpPec4QTwkheiVVrw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnUUlPS3ZPS1lSVmxwV0lq
WWE2QXltb1BzU3p3cFRuMzVieGJta05nWno4Cnk1RG1CUXZjM3dNQzlJK1ZGaHpZ
b3drMTB2L3hidkpoTXdMQ05vOUtTb28KLS0tIHhyUllpQnlzL093MCs4YWx4VlVn
S3Q3QnpVNEx6ek54RGx5bmlqUk5hbVEKHVUvfk/xKtlOk2o73LodweIivSB4Oii/
6dwnE8wa+Qx/OU0vDJVelKOUE6JiLwckX/0u2YIUOybkx49n+tYs/Q==
-----END AGE ENCRYPTED FILE-----
recipient: age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEb0VlcGQrWnZHWlM4YjhG
NlN6c2I3UnpSdkhYTXg4NlM5aG82K3YzUGwwCkdsM0JLUm03RTk0cm5SSitzR1ls
bVBLZ3kzaElDaTRHR1Q3bjdXYm9BMk0KLS0tIEpHNE9MTDJuMDEzV002YTVpT2J5
Rm9nem5FSnA3M1NzNi91NnBtRi9XMmMKaGB7uE6HCE2cYHfI1VscO8meeq6O5JPi
7bTansMUKCPKhDTBYlSnxxjlDE3DRB8ZdxW7LiG+8iVI7HUbsjTejQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3MHlrTTdxYit0MTkxa293
eFFIcmRLMUpkZjd3eWdsOXYySEljNDB2NUdNCnAyanVkS0dmenZSa2dpSWZTSkZw
Tm5LMkRoWEk3L2ExVktnZDZMM09hZ0EKLS0tIGVGcHFuMjh4OTUxdHNJQSsrR1pq
WU1jVG5UN2hMbW9iQ3JmbHlwc2VveFUK4JC32glesv/9nMWBR0lFeX7MPgIt9sIo
9HsAkOMTxPTI0NZW8q79+G5hq8qG+QvQhwO6wQtwz/KEHPXntKDX+Q==
-----END AGE ENCRYPTED FILE-----
recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlSnYrVVFObDRmUE5UN0xY
K0ZIMzNtUWM5RE8xMmlhTm5pcHNEemVwYVZNCnVLWGRObkxPcWdaOGVMcExqRlRw
VngrNGJFWXBHZGh3YXVBQmtSZXBsTzgKLS0tIFovVG43V0FBbmRJYW1qRGJCRUZp
RENmSmFuMmhFV0xHUHB2N2xWbHVvSWcKJXaXCfz3Zh4SRgTJkMuPabOq3laRorIo
F4R4bJDkao5L5QWDH1BWSi97kTTLz3QgmsEGPHm/SpODM7Llx7EPMQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRbEZGNmtlUjJVZ2dlVGZC
MGVEQXV1SFRpMjFqWks1Y2F6c29TY25oL1MwCjVFRk90Q2dJRXhUMmIrRVdHVFRD
TWgvQS91VWdFWERJWEYrL3VnRXFraUkKLS0tIExUU01VRWlENlpBSGt3V1Izc1dS
RFBhSTVKTGV2KzFidDQ3b3lmUHZ3d0UKHWvWO/Om6ahwBkONreGeENqdFG9UI6JB
fRA3gRHy98EorEiia1him6lGY+ewYkTtVNo6iEt0abncW5x5xMQywQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyR3VNVisxbnoyQXZGaTJV
Y0lMemtYK1F1QjAyWFVqSm9VRzhhbHRCRFRJCnZINVpiS0ZCRXUrVEJhcVlSZk9H
TnhaeDU4b0loQjdOazJVbkpEbDZ4K3cKLS0tIGFKRjd0Rkh3RTk2WG9uQUVpWU1K
c3IxVmU0Tnc5dnBBWUZhYUgxV0NaaE0K+00bh1AiHdTL3gsA98fvFI2/IDsnWiiK
3tOBK4UB5OE5PRupqVXlpEK3ZqKXju/MQbu0/KdaVwHxlL+WXdjAGA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2dFFpeDU2Y2dtQU12cytZ
WUJjKzZ5S2RiZmtvYVJScStYVHI5d2FvcmpzCm9COGZmRllyaElqb2NsczhKRjhO
ZGhwZCtoaFcrWmNZbGVHMUNLMFQ3Ym8KLS0tIHJQR1JOblZlY0cydnhNUnBCT1Fh
aHJiZVpMU3Y3NXd0ajc0MGMxSGp4MmcKky1oaS8yFydszcvZE2pAL90ZUKoN9MnJ
1TdqatDSMtmo3PWkdFeHbWxPjTizQGIcKEcsv6GmO/5KCAJDCHDNMg==
-----END AGE ENCRYPTED FILE-----
recipient: age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjSHFHbmU0WGg2a1poMzRF
Ry85ZWZCYVdVWkVoNnRWWTZXWWNqQURtendJCndHcmtIaFNPRGMrblJndElMKy9Q
NHlaeDVOS2k1OSt1bWs0RlU3a0dJTFEKLS0tIFRrMGJWYThKbVQ5SzdjLzlQbk9a
b0xlZGFvTVJGRjZHQk5XM3VObFRnaUkKMavcISNlQh+5yHpA1M5JIkQEF2qasnXH
Pd/JWKhnvk3Lyd45ZBJEqFV2wOknJF7v4Z0jdbo4WiUaLh5shqOp2A==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrd3QwWE4yaVlIakdiYkNX
bFVmR01yNjdWa0d6V29tcWh4M0hWRmQwR3pBCkFRV2gzSVdPQmQ5dXZYc0F1dW1r
OFZNTkRwSzRFR0tlUXE2UzFxeU1Ld0kKLS0tIEZ6RlZkMFFiZytPZ05DSEI1Ni9X
WEtpUlMycVhkMERsWDRGQ2lrNHdQcDAKRqVAOzQY4ZM1uH7LyZSqX1T6+1PwNhBY
5M0RxSuI43E8YKC9axcppvKFJMQI1GAGvgfQxg2J0CEpnmPSVsO5DA==
-----END AGE ENCRYPTED FILE-----
recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3ckNVU2lUQS94RkRGeTFo
aWNJRTdpcWxUOWZEOHF5ODIrRkZ0MkpnRndBClREbDNNUW9RZWFSRHdON3pOT3VZ
bUlFZnFGTnJpSzhFVjFLMVhKZTNDWUkKLS0tIG44Ui9YL2RPYzJLck1UVVBiRTRJ
MWxURWJlVEFBQ3RSZzZFYWJvaW1FVW8KX8/o/4LP+Wp/qOfF4wt7cTt0O+kAZbWR
Xu+dthboRmZBV21zzrEvzKGRFBj2T3EMyGDqqPcfX/vuhbVEJJgZyg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvb0xNMloyRW5tTUUvZGph
YWxIc0ZhSWI5RHNmaEIvdjZJNHFDM2xBMkNzCldlM2RUeFZCNE9tMVp0MFlWM2sr
Sm44MjkrWlNkY0U0eUlyNlE2ODl5SFkKLS0tIEdISElQa2VBZlduajhsV0ZaU3dZ
ci9COWE4bldUMS9lSVB1eEk4TG1sVjQKYu9zby36DP+41dTV8sRdtDil3Az40pf0
vIhqhCTxQymnJHEwEukiEzM/w3Z85R/W/MbOKyzEZmxzG0a5O5svhg==
-----END AGE ENCRYPTED FILE-----
recipient: age1nxlnrevqs2msdatze562vz6ym4pgydt2zndye83lwqauy6flggtqrevyw5
recipient: age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYck0zNDJaMmhLZUxxVUxR
OGJqdTBHMFpzb2tQV3NsUVhPc1JTeGxCWWkwCmtWeWhqRXJFNVRoVVBTcjJtWlM2
UmQ1OXh3SnJVQWQ2NnloVjRCdUZnaDQKLS0tIGZITFdwcjJFMU5iRHhvRllKaTA0
dW1xN2ZhUmZ3VlNzWFprYWNOVG90aVUKH8Gd6gkbDIZydnb1mL0tpujmucLhbPbm
sUm8KZpNUfWpP+t7SK5M91HFaAZiqGJOARb31oszwqLGcNDliV40MQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZlFtOHI0NnlMeWdjZEsr
UnQ0UFJXY1pHTm4yLyszY0dBaldtUkIxdzJvCkNQbk9mN0wzK3BtN2pEa3JSWWFk
RXNoemFFbDZ1SlUxOEZMUm9mY1RaZW8KLS0tIEk3bCtyaUVGM3MwNUwxS3QyTjA4
YVpTTFlOdzljbmI2cnZUNG1nd1NrVFkKxARm3bpBL685Wky4Y06eA18VvUZRkAl3
BBYPJMn3lfDgiQPvfXINfhJI6O+bWyjt0WnfodCGFM6EJLARXiTaqA==
-----END AGE ENCRYPTED FILE-----
recipient: age1scfc8p53q5aq2a87tcmsazmj8sfeft0s8kxg0et4nm3ucxyp3c3s6te82y
lastmodified: "2026-07-23T21:15:41Z"
mac: ENC[AES256_GCM,data:qFhnPra6IE3wyKQ4WKweON0S0YtD5I0adGZVfA0m6BVilN6bX5oC/1j5NK2oHrsz920hSl0SOF8LrpqOrUyGjSRkPsN4kq8qr9bJcrX4URiktP0oRden5LLt6hf+ZRP7WmRXFqixPkPHJnZIoAvkNnTFce7cDq5NEAHkKUEKG7k=,iv:nyblUDGeu3TUfFivYylOn3C/HITj99qiPI2+mh8AGh4=,tag:FrtRzSCylC4wlIoqZdfx7w==,type:str]
recipient: age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
lastmodified: "2026-07-29T11:33:40Z"
mac: ENC[AES256_GCM,data:hXns2uTM8+bX9J1EWxKcA7v1oYayBMfyB57pMgayEQHV8PvdIWi7rYez+4OCuMBZzBNVAZWn2AfzYxC2VJJnKrajrvwGZ/neJ1qvd2sfY9e9TskZv+c6nqnRuXl0OGi+kP+6EVSmZPiSFUODZd+xtAku8dIMGO8LZeZoGYjCBfs=,iv:EFZZpw/QKiuEtShBhB2bP6qq69T6jlrJk81FpqSt5m8=,tag:VCqctIBaFK2DQqoj1sxv7g==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
version: 3.13.3
+8 -8
View File
@@ -1,22 +1,22 @@
{
"data": "ENC[AES256_GCM,data:olqZm/3yxtVnlDeYzAWnXjUP9pgkruFEpeS6qONHSfdrj0lJOch44zn1pbG6Jg5QYcksny0rHki9PJEZSbRIyrZ2mSKfTJing3n9M5ODm9xPMbNYn8cpVB4AzZmCSAH7mN4p/98+5hJ1qzopGwCgKTcOOJ9DMUl9AStXORY=,iv:R2K3OehIfyKZXRFH3Y9KYnvEp6CjHPT4Ki2P1YrC6+I=,tag:1FM/E0n9teARAZ1HZM4b3A==,type:str]",
"data": "ENC[AES256_GCM,data:JwjmYg0qzoer+8/jv7KOfK/BxmpObHOn04GNsUFQd6hSFBWbQlRbODfluL54hnWQ4gsS9MHgwWh8nrLciBhye05kRQCqADB7Crm0CZYubNb65WzM/devh14jmerc3MYIp1M3VmDisML3x5IqhJkFMSqKM/VaNAC3f0otcVR8TTgZw/fJyvPlfqpPTjikGmo5xg++Yk7Cy8A1Dj70kYk10+EQjQ78jf4k/agLoaS+YvMmKXFk4EljtHg8Fe5H6Rn9nfMSSKGTZeAH3Riix2e+fIH+nWEZiPHC0UlPjJrB3SeBqqGRSUUJshVKHFIdKI1Tq2Ea+GDWrAyIRI0qO8e7WQ==,iv:mr39NxbZuYUDHDw0e+fJDshbt9R1hLgVqkUxUVX/+l0=,tag:u31xZRPfvH5j3Sw5U8/nqg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3RFRoRW1rRklxbXYybUtu\nL1lEOXJBMDNCN1paU1k5N0hQRTJYQ1dQcTBVCk9OYnloOFZad2toOHo0UXFYSzNR\nZDUwV00vKzVnckMwSE9MMnE4RUozZTgKLS0tIFByL3Z3SzJFeEVNZjhuRDNTMzFo\nNU5tZ2tQU1ZORE5qNm1JUEJrTGdzUHMKypYeJz6BeUUY4aPKazB1nxncOA3DGkal\nfemLr9uwCJw+D3xfXzrIKrI0w3OH7bu2LmWqrNDSz3Bwa5VDuHoLqQ==\n-----END AGE ENCRYPTED FILE-----\n",
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBtVW5ENXlkUW83elc3ZnhE\nOEZ4MWIvQWZ4aXhuUmZGYXZNQnhJemsvOWlVCjIwNmFjS3pJOWEwRnhBOGNRamlM\ncXZid1RNYVBXOFZpMlpnU3ZnVE1scnMKLS0tIGFhWFZFd2JPTFNrV1VHSDVKVHRK\nbmhoeFh6YlUySU1ZRVhHbjZnM1IxMFkKo7aHkz2pEeV64m+OEkBZ2V1e+PUzoChu\nUc/Mnh37dNXSSJtg2KnocHdyzDGi1yQbA72xKTxx6QjYJWrAC/jurg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqTzhtOHM1Q1lWRXdadHB6\nZ1VrMXFsQWZEbDBMMnQvYnZSN090VnpyQnd3CkdJaHBsbzQ4SERmU0dBS2RZMUpi\nZVYrZmtSNXhiaDNTZkNMVEZud3ZzT3cKLS0tIFhETER5eWJRK05ValFhenRnSGg1\nc1prdWVTVmMyWWt2UkNXWVRLcWRHcm8KjVn1faGmsWiFzcNg4PnxZQfeQONFKz/i\nyJGJc7w2KSZ3La+44jfitMziJQ3AFRUlAhGDX25kWZJ7PtjmFvYwuQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1nxlnrevqs2msdatze562vz6ym4pgydt2zndye83lwqauy6flggtqrevyw5"
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBReWRLb09reXUxS0YyMUVV\nQlR2bEdYY08rK0RoR2pIelpLVEtYL0pqNUVNCklRaDNoSjBWT20xbk1UdmNqWHNt\nR3JoSlVwODVMUVBzMERUNktHMVViZm8KLS0tIHc0SXRIS1IreDF1VCthb0FBR0Na\ncGFRWTROUHF3clFtTmY1azEwNUpPNzQKsgCND/BZcMTgBTAcnHunQcT6LG1jNrtO\n+W7Yx7bFtFBajWnRYiNpUZPibQJlv5SE9os47WDH1gs86xftgV+uyA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkVXFQazlvUEx2NzVUNlF1\nN3FQNTdFL1FtSDhCdmNQclRMdG1vRVhvcEhzClF4Ykh1ZS9JekFzOHkvOVphejVy\nd2s0a0IrZ1d4V3diOGNvMlBHUVlwTmMKLS0tIEFhTE13Q2F1di9ESXdJVDhJUVl4\na1kzNDBLcHhlRGg2Y3hSSmdEc2k5MlEKpU2sIv5Pq0LaKkvqA/fRkiB9PcvsSjvq\no4iEFzGNDi63QzXqftWnzengEy/6nWGNA31fzScQftESesGlKfYYlA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1scfc8p53q5aq2a87tcmsazmj8sfeft0s8kxg0et4nm3ucxyp3c3s6te82y"
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBGbllNTlhxNVVkd01GSUNi\nR0lxQWZoRXI5NWxoTExRUnBSb2FPR1BONnpzCkxCN3pNOXVrTW5GR2J0WFhZUE9r\ncm1QTThINDVVVlZEalJiS1RXcHJRS28KLS0tIHV6aGh5QjB0M1VPWDVMT1k3cTBh\nNTNGY25NV0tWSUl3UHlJSExFQVdpVDQK3ARj8xhFRYU6oqYxNQ5+Ryza86ALNUwX\n7yK/8ATnquYC8/ZIYUbgTPxzsocmXX9lVT0+ktIALqjtG8PgNTlOow==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r"
}
],
"lastmodified": "2026-07-28T06:46:00Z",
"mac": "ENC[AES256_GCM,data:xJHyh4JuY+qnKljS3mtfN4Cy0pGPUszhU4ud2L8JxTFGUHTXhZzwQ3s1vM88efLTwM7yDK7oBMBUH2xligAzglDnqkE1T0l7R9k+hyldm4nYqOK6Y8GyPt/aald6oLYGCx3J7Ap0Oqq/crU+s4NiY9enf+R+08jliFyrPBHuRSY=,iv:FGtURIfSrf1tRGqpPfn40GqybhNYFBcvR7n1YkcAMGY=,tag:ir0TU7YuxpPuew+nyC9/eg==,type:str]",
"lastmodified": "2026-07-28T16:04:51Z",
"mac": "ENC[AES256_GCM,data:9X0dkAEGJiug16LDy/8//QOkHITTzt4zwTKiIU13tRgSRXRpt/7b27+3eDauscXDMamg5wbYo1YF0+VwLl21Ip4icnnwrZBHd6HWleP30HTgef8rmo21SstshnclZz6RH8SEGVDO/vjrMDChaZ3A2WD+nbcomw47DpK3cG3kBIk=,iv:G7CScSD5Z1Z9WMEMydyeGK/RD4W43xA0PlcpmTCxLOc=,tag:DZmby/xkxpinoJztZZRWwA==,type:str]",
"version": "3.13.3"
}
}
+7 -19
View File
@@ -1,31 +1,19 @@
{
<<<<<<< Updated upstream
"data": "ENC[AES256_GCM,data:4Plrw4SFb7Qx7ofyHwXkHHiCaY+54QT+XLnqoMullnxoTEwf4aZGmjZKx+J5eNnXO/+aPcheg50pe6vXggIgA02a736l3xkIU8mmRdqDSXwROwy7DWBfpcsT5bcL5umGpJ3tnJUAoA8TWVvxoj14J2k=,iv:IwoLgVeWJ+1E1yjeVLjMGEcEDNK0MD1caaP0hcbYJhs=,tag:KPT4mFN2MitDcWAggBV7Mg==,type:str]",
"data": "ENC[AES256_GCM,data:VpxnzrdX5FY2gfDcZpqg1f2wqL6mAawcrdE5A9A3fQOKr4BWXmCT/hjQEtt8qG+yVOTApdG+vF8SKDj1AKZ25KsZ253WS7W5C4j8oiT/xHo0CtvVani4fZqN60lB/9381h7aUZ+lcBMPaQcNn35zoQqYthuCGrC9PH1qQgD2VVHr0v8J6kceXKp0KTJldtDsQxky6ROC9Zyc44wwkrFPgPss4/yGaRWWqiojDvK5fK8238hmR8HyfD6Tf5KhgbqCsSGS2g6yt1muUhmegaZwtpi/KpoqBmf1XFaObuHGRPKvblq4Fa0x/sSjpPogBZyNuvRKIwRhla0FkKOH1bX4gC9bzOlgiUUfVcRJT6Nr9jocYuhIv9wvIT4TujGbgHyorYaDSQLFPJoJEUa3fgpGq+AC6p1DJJE3b/wqUSngNbdOtAPE/imaEuprOfnbS7cRx+Ti/Yu+ZYZuAlDtLkSVdXi1hQ4fLQ==,iv:rq/BzePXa/w/Gqewz8JfM/NdU+x4ShiH1OuQT7wRyCQ=,tag:IcO3ddoj4M57CkndHEmNkQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIaDJyeGJKeDhVNU9xYzhk\nb2Z5K0VTbnM2OExVWWZPaVBoYkNrOVVwbjJBCnBmQlRsdjdPZHZydU14UDl2Nm5G\nUkJWd1N0SXljTW9Kb2VMVXpETGZlS1kKLS0tIEZaWkdrWkVqMENQUmpQOERSY0hQ\ncHU5anBjeUQ0U2ZFYmFaa3ozSmpGWXcKU8QpWAk7Zkc3N/ZmWn5xLsOSTAtMxU5W\nIQOvdDNzIn7Ecoj4zQ91T9puAhaO6PZ1wWjo+rAN0TyWjM+TAwTuFw==\n-----END AGE ENCRYPTED FILE-----\n",
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRdWFEUlRJMlZId2Yvc29F\nNnkwREkwQkllY25vZnRjV3Nob3BvNE1Zd21NCm56UVg5YUFGbEc2ak5MakxDbCs2\nTFVsRFhjVlRwMXdUbGpMKzQ1N2hBUUEKLS0tIEhmdm9hZEJqUEhRN0dsSk80bDY2\nY1JpajNOVFpEejhibjdKZlNHVjE1bGcKycyDfEBZ1WYc4EfAK2y5x/nKRqq7mnb4\nDlpR1Som4bSt7+B+OCZa48mC0Zv05HbP8PnoZdCe83swGNljvRONcQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMbGg2WW1nOUw0RzhaNXNa\nbk1iWmRFWng5SlgwY1V1ZXZIcEp1SWlKYnlBCmJ1Q3Z1KzJoY2ZZWi9UckFOTWZC\nUGxZeUlJZDdRci9ZbzhsdTM4cXk0SjgKLS0tIEhWUy91bTZkTyt2ZFRIb2VKL1Rn\nMm0wN2RjUUJ3dHAxQU1nd0VRTGlQbUEKsb6AFoMZw+FxZW1A1rf4YUlNaODUgbS3\npZfGZIocS2y9uMTaXWQ56Y078zDD/F40wLYsuknm0ECVfNysSVacVw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1nxlnrevqs2msdatze562vz6ym4pgydt2zndye83lwqauy6flggtqrevyw5"
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0SGFVbXYycTE3amR3ZVhi\nVFJHVVhWNWVQQlJKZUJvZ2p4ditqdnYrWnd3ClZaaUVLUXFKaWlDdDQ4QW5SQUZY\nbG05c0RGNnZUenI1QTBlemVjT0szRncKLS0tIHBWTmNpd2FjWHc3MmR3cWt4SCtH\nWS8xZnVON1lGZVc4YzZFRmRBamdSVlUKdlQTlwiExTUVsiY05MXFG2/IQt1bZwKU\nlFDnuy6YCQzawBLQZuSUAp7WSGedupexlZKQKNUzPf+dis26XEYZoA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6"
}
],
"lastmodified": "2026-07-28T06:44:56Z",
"mac": "ENC[AES256_GCM,data:N95kOsHFH2nKVhr79EB/zILMpJ0rgh+QZ1weYbQJgvxRYjLMl+AK/ALjtx7Mrj9ufOnIWnndrUpD7X2cod2oPrlAPT3c8P0IY8LaJTNDx6TUljKHFe9Nvm/JcnI9i5skioY0X8sRC1oosL0loVTmVb9yaf5xsf+wursIsj7pI3A=,iv:kQLx4FLeiKZIevOOOPS3NAu2NUUhnRw/jpws3FZ6zcM=,tag:rCp0h+wujci4N5VFuzCuLA==,type:str]",
=======
"data": "ENC[AES256_GCM,data:LbqG6wmya62HfWjTiQad1+k0XwR/iwgoxvvBaCfbY8NsjXYdh8rsl20r9OUFkp4G0DaEjfoNXburz5Jled7OEWReXymt9oYqD4V8JM06x3xwdY2Zotxit13FjcgDqclv7tX3eewgge4F7K+qxXSQaoCLt4VgDswAWX+mz7GNUSD8WnRt/n57VSLKaG/mce53S/hH77sMStosRhy47R8zlKrnmp5T33r1Xz7Y9cOZozxid3QsDp0MGZjYEYjOYkmTLYno62DeTxiGQ1sABW+RpGlZuSyDmk840pXqTk8oikSsfjkEWZH62CYelNzyhVXFe9K/NscX5C8U//AkpHHK/9Od8C4Wts66Q9K3dLIQbsJvuVyx30xEL1MF6FaHlvROkzsbgYN1wJMHw+B0H+gLI6H6UYojg3VIrBQxGHMzajX5UEQviLmaxjHKVsfC3+fFQCXBQkP4fOjoA/WKPjTAT13wz2bYVg==,iv:OuO1VFB4koP1kPRmT6MWIv9gouUGB9LLnhkL+FwRUM4=,tag:1dv+YprSKLTGN8tKM5A4Yg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB6S25jcEwwYkEyTkhjSlgr\naWljSXNVQ0ZHOVkxQXgwZDYvTi9YOStsNUJJClB3Qk1hTUpvZVVJOTBpN05vcmdi\nbENuMlhsLzFuUTdoTXhYK2JIZ3IwVmsKLS0tIFZLbmx4R1IrS0pQVmVCdDZPZjBy\na0JIL0MvOXhhTloxZVpBQlY3TWVZemMK2XMLHNm6wDrX9VqIOPErQWIUwXLhhNJN\nlWEQvkZWjDY9PQ06CYpF/B/VE1ukJFtuslzNc3a2yuXMx7BtAhiX7A==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-28T07:50:04Z",
"mac": "ENC[AES256_GCM,data:M90tmlmYarfx6UcS3rFedpyMBtefZ802ecouExpGczsWTLtAqDtfdsendiXDgq0aV9D4cCA7KecstFvKAFIICIXF9FHCxHPtHMUWW3RsSjH0N1IIu5oeERSfzR+EIYwE9/bE+bx3dVXRMBdnuGgMXKPt7UatBNBAw43xAjl3FkA=,iv:JfVKxfpk7QYtK7XrdzpHouO8z0nRGgVlxyzB/0JwVRM=,tag:pUHX4zsX4clkQU+TeTJa1g==,type:str]",
>>>>>>> Stashed changes
"lastmodified": "2026-07-28T11:53:06Z",
"mac": "ENC[AES256_GCM,data:PMsPSffQkxRmqT6HwFezIoT4WKJ9WDlAcflWynsFLtzbqVTREN0YGOJgMwLd0yBR2sbGBWogG5xAuKjO16aiWUVpCFPRvP97um351679ZoXIlrOL+GlFn4NKwoLH9/eCp8BteB9mv5huhEEN631lqCr+jtYO6yS0ULk6QS4+7/c=,iv:oYWV4J1JUuGryvqntVXkj/HOzRDRr9E2C1RKNvr2bT4=,tag:e4p+I40y1LrgdlK9tLRicA==,type:str]",
"unencrypted_suffix": "_unencrypted",
"version": "3.13.3"
}
}
-25
View File
@@ -1,25 +0,0 @@
beszel-token: ENC[AES256_GCM,data:tT2a1I6AhVgXWfc=,iv:1iXoOMAakHXpI2lYiipxcVSzXODUdpqYmfrpNKZwljg=,tag:Vuxxz7pyIsZYbkg2x6GvgQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAzeEVTTVZ1ZHdVNWNNUW53
amN2d1NhQ2lydVNJUXdyaG52ditNTHNJZmdZCnljakFpaW92a0xEbjNZSEhXdDcr
ZUN1MmN2dlA0T2lYTVNibjB6UExqU1UKLS0tIGliYmVicTBrWmE3dFZWVzhwUm9r
T2dUUUdUQUVzTWRvbzlWSExKRUorTGMKktkw3uuydGeChy+9pxtysHLvssZ5LKoK
8i+CNr76/nyKCRns6tHj5lHYefY/B8IpdmlQ4fXQKSrna/dDjkdWTA==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0dmd5ZkdDcEM1akJna3Yv
RnZFYVJNZ1IvY2R0QnZ6WC9PQnFmVkdWNHpFClVJWnNaRFV1UVBQTWlXb1NEZEVp
Z0ZiUC8yTXVyMFdmRnB5OWdwQVVTSUEKLS0tIFFxLzdyNlZ2WlFPdEJFTTl0Mnhh
TXVMdVIvKzRTeGRIRXBqTUNxM0Z1cmsKMLHPN6n2i18LHkgjinq033qZsrt7BdFG
NuQA4EbCqg5uiHPCSGU6/EVk2XmfROrJz5eyMWIwrwCbt53jsUNTFw==
-----END AGE ENCRYPTED FILE-----
recipient: age1nxlnrevqs2msdatze562vz6ym4pgydt2zndye83lwqauy6flggtqrevyw5
lastmodified: "2026-07-28T06:44:32Z"
mac: ENC[AES256_GCM,data:rtDzbONqaNpbEKo1btGiHArAJVCVsStHhgvM8MTgkRMUNGU8p9W4tiuQLmobDpFh81kj/mDnkx4cCpScuiMhQojz9H74l7P9o9b1Wxy0fSiCtGoQhaq6JoIVJoKtEZoHfMStdIio5EBW9DBB5abvjPLTaC0afgHo5pNn3A9qr6g=,iv:SqtJhel/5qW0V7IV6V8dCRWqPnsK6QCzbC15/BKeq1k=,tag:vF7RJJccm0q7BSyNGYnplQ==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.3
+7 -19
View File
@@ -1,31 +1,19 @@
{
<<<<<<< Updated upstream
"data": "ENC[AES256_GCM,data:Gi6HP0EuWQcTESaFOEq7wgAvuvlyevuFIKAnShDXJIUBg7xNNlGJ3PmBfx+YyFvChCqOqThAS8TxStogMVQqyWxm0vtCoVshkHG+9Q9xHV72AiuTcmJdsZ0DiSf9jwJYLZ80x1VI9a++Mqg0yLs5few=,iv:H9YYUQ4nOhQTBaonA/H7F7KroMRyU7kGul/bL/AnQ2Y=,tag:7ESjzSEH+XTqvvtxpinKJg==,type:str]",
"data": "ENC[AES256_GCM,data:Ea5AGwloFMyRgmQhJaDNTW19mpDvx0+dSc3ibENEXEniEIBV4Wv135mAwE6jQPbN8wF6f5ki3B5mDXHOMuhDQEqUXg6jcs4uEm2nwz7wAKYXEX4T8p0gUj7FPISiSQrw8O3jwzt4qF7AVcPsNhMtZLgrx9ssMtVmGct5do+E61Y7dFe0XbpJLynfzeiMKYvaBngTbHwQjv1mn9AMpX98apsp3tyzg8avLDR+WXRcMusds2mdov0jCoKuWPLG/srhMwjKOp1Z1Tu+aZCr9lHTT76mytpGIWGSbrP5l3UTp3oPLmy87Kwi/FJStZYdYikMnvmSo5qovePiYsLKVD74dS+d3oFGARyW4TILd9OQdXEjzdDRak5HcPv2/HhR638kvukVMdEdH0hpeJcqcV0h+BoLyIHe1e037iF1ZkCjuXE+A8LtlzRR0wqP202uzBIyg21ca4X5bRiHkkAOX4P3riaEcMgKfw==,iv:IXq4sHHsjnK6maVd3RRVZzhnxx1hOnxfsvYX8IKLn+4=,tag:jRZinAVLH8nGVf3gE4aSKA==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1QVJsNHBYaUt6WWFDRjFX\nYVkrMHRtd1pTVFJsano1TG9hVyt4NGZyUEY0Ck1RU2U3MTQ1YTVkaXVhNlM5clg5\nWDZKZ1duTHZrM3FxWldnVXB1alJBeGsKLS0tIC9wUmZWcG1yLzBIL05qQ3lVV0xp\nS2lhWWVaTU9NWjd0K2hEaTcwNWE5YjgKmy3SHvYur/2sn6PjVLZFYzHnVMTFe66r\nm7MsWeUXU6zs4o4RWPPNBrpsVXIGZJx5Gbl+lp9Zubn61Wt/KVqyfw==\n-----END AGE ENCRYPTED FILE-----\n",
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSByOTkrVVd0TXlBRzdkTktO\nalU3WlVOb2ltc1k5RG5sRE9EN2M2WUlnQVRnClFGZFYvSHdpVlhtNXExSXZMZnU3\ncnl6aHRzU3k2V3R3dU9RVDA3VWp3VkkKLS0tIHcwREM1dDREK0h0UFp0bU5JTW9Q\ncWdyanNMVjY0SnZUWUl3Y2R2SXNyc1UKI1FPIE66to19oQK2TwM5B4snGOVMLFfx\nl6JMEilDVSgy2RR/tSiovmo4NwOsv0hmmF1t34547rbZb+dB5KrMEQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRNG5SQ0FvWWVvaGhOTmI1\naTE2anlXbTJDcHVBdWlxamlBam9jQlVyL2kwCmVJMkx2NnU4dno2T2dQOGUrRFRp\nQnl3TkJ2R0ZOS3Q2K2g3VDZQZ2VWS2cKLS0tIHgvdjJ3WURmcHVSQytHUGlwSnpm\nc1k2UHZORWt5TlJvaVp0YlpzWDdob00KKJ4bSM2CQD675b0ceEv9tz+wBtCIx/wz\n7Nr2CJoY68cB8KjM1eMzioZI1Rlt49tTQuD4XSs2/9Ej/dDDbP8p6A==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1scfc8p53q5aq2a87tcmsazmj8sfeft0s8kxg0et4nm3ucxyp3c3s6te82y"
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnbWp2UWphSnFibnA4N0ZV\nRHlNUnBNYUNzMzNZakFVc3JXT1d0R3Yyakg0CkowMFlOcEhPcHE5dXd4ZUd3cDAx\nbjQ2T1NqejNRMnFvbTdNRGFGdW1pM2sKLS0tIDhGb1JBQ0RFQ3NxbmpkdXg2WVF6\nS25uZW1hMVo4Q1JGdGNQWWRyWE84b0EK0FmFZB6sN7uZynhzYU932x461zSZIUWU\n8oyDEKNoUuPqpWK8RktgjWlKHtB0oLXC+SLAOJEhtnhWF+GSqloGfw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r"
}
],
"lastmodified": "2026-07-28T06:45:59Z",
"mac": "ENC[AES256_GCM,data:13BJaaM1Zrcg8/SZToXdlEM8J4GtFZnkoVt6vBWoeLU58wsGi58jdEAqMzWpnT3Vf2Sb8OjNUB1cbBVv9uMitbffFR27y06diKkAICnVc4shS0czoO/6u9O0k9gXicU04jUr/FKLL+sSiNi2ypCFSZT1C3WY44sTB15RNYQR7XQ=,iv:/4owaf5HYndcojBy5rZv6LZp/ipe3UuXy34dq3XI4Qc=,tag:i3Qk8jdl5hUmLScLCirm9w==,type:str]",
=======
"data": "ENC[AES256_GCM,data:YGR1qsYnOOJyaRx49FmR8Qs5pTZtxUu3izJxl6y+Zp9EJ2ERnJKBhQ7d7wxWb/Nd/SjHh4abAVPzX5ALoUkqo3avl2j9pJnAiBGtlQKwzwZrTwf+OBmM3GCvv0Myx3TPvo0vH9QuZnwdDQRI3uWgHeZ+KMRODMpKqzxiJNtwI0mHhz/eaIK8lIWM8ssrPc/g7Q5pb8K+Jz+duLX5M9COd5ohIoJWwGCuuWnCA4yrKoioizTHuJMDlFUip4kkdJoXUXJL7TL38BZU12TUM0wkpScxojel94xezw7tyUh9/ak9IDA2n5+LrUVwXYcJUT4BN19Pxy0zQnTWAWA/9p0tUa4m1dnYzkYOM6YPFpGuV/310ZHbjMp65gw9nSojBgpnvflQZqpNn5TEzc2ox63Y8uMLiEjf6Qcb/yA32Oy/lfEZwPFDEZel2MAefb9kYKC4EEkPUxywqS80Yzg8UhI3H/HwSMUe+Q==,iv:Bbn8oDmFHUY4/apwGd7cZyKnVeZCrTT5/IVgciyL7mQ=,tag:e4H6PO5KfRysRltjkpDY3w==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA5N3F6N1paaUFkek44SUdW\ncDIvVVNPN0R6M1VVS1BHMUJaL3ZHUU44U2tJCjI3cHZqMmU5M1hvamtDVnpBdGhV\nRG9mdnROaXVTNEN2d0tmTUxkUjRhTVEKLS0tIENqeHRjZ3NLd1BpSHRHQzArQmlw\naVVxYXlXUXFoWXpGOGxzeTBsRFhXRUUKkGZzx82vm2kkw6nM5pnOkDXwLxuh0Cq0\nW/rdAwHwQtskgsL2A7mXd46bDdtPLiT+6ajLsCighJ9EZgaP2TfUBw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-28T07:50:16Z",
"mac": "ENC[AES256_GCM,data:jRj0folhDTavBa+J4VivfHFpkqJBihdodMwu6EZ9G1hmJP782AzSWoVkeoPK7UhBzipki8aDzYS0uWpRxvSeFFAnkPmif0z2yRL4Z0MahXuEb2RHnexbqOjfHT2k4u1UKzEJ/s0IlXby61HXXG1uoB1PXeDauDgGVnrvIo/Bsjg=,iv:DJ2zXvDW2FKb+F7XxMCHtWA6ICkyfWbNJOB4WshEdis=,tag:M8GqOQq343OksrFK+sKkdw==,type:str]",
>>>>>>> Stashed changes
"lastmodified": "2026-07-28T11:53:32Z",
"mac": "ENC[AES256_GCM,data:EhWEEIamG62xHI6SqO4tzdK6gcEXxUU/UGcF25m+X11kUVXzHd6yLcfiM2DvqcwGLmwmOOf3SpFLABlQUzTka4QRzTMFIAJi7x1rzQV4J0YIGITHk9rgvy0V50TI0loONBb2du+vDt8IlTBNvF9WMbNCki+fHPAjIemyoRBdubo=,iv:wfsx0H4fD3L7WeGzz57SzZtajBr/xTRDljSpz22DD7Y=,tag:naDyGH0WTKkqgd954Ya8jA==,type:str]",
"unencrypted_suffix": "_unencrypted",
"version": "3.13.3"
}
}
-25
View File
@@ -1,25 +0,0 @@
beszel-token: ENC[AES256_GCM,data:0wooMaPiytDTGug=,iv:/WATBCEakrIvujt4oAeprw18W1n/WoK2zjpQdlVJl4Y=,tag:lhJa9hJgFqNFSB1Ogj5CtQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhdmNmSFlmei9KbDRWU3lj
TVJoSW9FRE9sWUdBRkNIMTB3aVNERUp0U0c4CnZUUzIvYm1xL1JNMGFKcWlBK2xT
SCt4TTdlU0ZralArcUVFZEh1YTRQaWcKLS0tIEtKbnBZVlkrU2U3VzlycS9odXVi
VjhXeWtZdDhpTG12cUZwVDhKeWZPMmsKOtXLVB8Z0faO05BPCVQXBn8gbMOZeHxC
sHmt3vMWO0jrxM0WLql59VWAFsczNwb1fw89NdZMfd/XoVq2EUV8dA==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBucCtBR3N3Qm9JbTlLdTZT
eTlxWEsxUWZsUnMrWi9zSDdRVWZWejBUWFNVCnpyVEVLbTllVHg1a2RXRkVRZ05z
ZWlsSTBDWFF3bzkzaTFMVDQrVW1yczgKLS0tIFhKMll2OVdzWTcyeWdoaU85Vmla
WWEwRFgxNDF5czA3Q01LZ0NjY2toaTQKKawEiHUWss0NPuJng6UimekEOlxipiyC
ielJbLxma8WPLeLwBukFOBOTyNFxgsVBvwmBJ2k1iMIg1lCjhLD7RA==
-----END AGE ENCRYPTED FILE-----
recipient: age1scfc8p53q5aq2a87tcmsazmj8sfeft0s8kxg0et4nm3ucxyp3c3s6te82y
lastmodified: "2026-07-28T06:45:50Z"
mac: ENC[AES256_GCM,data:f50JYTzMJiEjB9+fNIs734CXrkN1IRgJKcoJ0GHoI5p75cfLfPxuzUifty9kQ/4DbM7LbWa5w+9occoV/yNOa86BKpPtU0LmlzXf904/SgOCUKhyHqhyO5mp8w71GoyLxx3ld8CbqfBu8t5l4gwbQamftsmQRxnUdU/Oa/CzdZQ=,iv:EewXBjeihw89h8ypH5u/G4YfoKZzkHqRWgSSIXU3/a0=,tag:lEWMvAkebMexDGGpE+wqkg==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.3
+3 -4
View File
@@ -1,4 +1,3 @@
beszel-token: ENC[AES256_GCM,data:meuzUP/6wCssJDVTgbC0XwiLZPMGyDl55HEIiON9xOXCD9k6,iv:TDqWcp+8Mxd8wN09r5otQRQXq3XTeQphaTWxvvuLTAs=,tag:cRPZQGlwB/dTguBAheWPQg==,type:str]
cache-priv-key: ENC[AES256_GCM,data:6vQKIf7eS0WNL2Eptoi4VWr18SRMZfN/H/aFUUtXdMYQY5LLyBp2EHRKqZcGFuh1nZhUdAxUztq/CVXx+QFxKW+ElHxCxUSp0QqI1fdSkBkKZb8hlit5SoX9JtLzZGg0HBNM3nJu,iv:0J+xmrPJhInHhFR/c41ACjuTfaIoMkQFSfbL2KkgFa8=,tag:f4s9Szs5oprVVRSyXaX48A==,type:str]
sops:
age:
@@ -38,7 +37,7 @@ sops:
IqA477+noQ/Rjrszb2hEvxID7keogcDUWMWQzQvdMc22+3mvAr9qIg==
-----END AGE ENCRYPTED FILE-----
recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
lastmodified: "2026-07-19T23:30:21Z"
mac: ENC[AES256_GCM,data:kLGE2xawQT7mx+sfw68hmGk5nCEGiEjZrqTEl9B1dtQmTrMwmoVr/1RISi4LfJrwxy31mDgff4lcIL4wIJuM373uk3X8j4RNyYQNTfKEkORT6r8NHeepNs267O77pKGd7OmcM4MT/BqOnB8ELS7Wlf2ect7CAlvUUVyc8icxgZE=,iv:EYLDsHYHZ1XOQXafOTqHHWpk/OBNq/R6IJnOBYV33E4=,tag:thxrCPC5oGvDjhK7Dz87YA==,type:str]
lastmodified: "2026-07-29T01:59:11Z"
mac: ENC[AES256_GCM,data:/nbcfause6G6F8IvMoyPZtkWS1XRLAivhwTFu6y5P0Mm0eCcO6M7/rgioN9dngKzPXrCUl3Dx/EvhrrWKe2/Saq9WEOFgvS2V05pTRbQgYjuugVzW2paPq1fgmoDYNjHz2yFYWAovbIFtjVxMR9tmcASMTe6r/FocvMXbFCJh7Y=,iv:n3KoaPPtDhWK8mxJNJRk7WPVUUNR59nZSA8JyysWNDc=,tag:tYhkNXsj2ohVjHkvxife2w==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.1
version: 3.13.3
-26
View File
@@ -1,26 +0,0 @@
{
"data": "ENC[AES256_GCM,data:aqlMnoVkGtH9z3fJRweeC0OYf7LGqJU2sWA9Q25dKK6NuNyJd4BvjPtpfeg/WhVtJsaOtcbwVm4WAhVK9FARE8g8j+vmq0f6BAU4s6mx0ZIhl+mP+/hIpt//LOdd+9YezelJxdpzUyZbdAngU99rsTluLRe2XmZ7Fquxd8yH/OHenSDY6dizp9+5jfEi8EU+EmuXvuWMPY59xnlnqYNPfSFxs43/pS402LzJoJ5H+cBPprddkUBVzy4cBQvMnrRFUSjnqp74ovZkfIWFqDWQ5YgSU2PjatBg18oulZ7wNRhQ6OLqj6gsu+xrMjNFwnp7rMlA3X//hIidxTkVcYITycXd8KzuMIaofUpnwoyT34fy6+H35/39iiEyG4LRTrOOKRDzXkY2rhJUxFSZ8GlhNhMd0RlkmLngVYrtjsswJ9meIwoAFLPYt7BC61PJf0TXdtk=,iv:mCA819J9LpAOj8QxFAkrHI9wFJIy8qVxv31D6IwWFnk=,tag:qqMtM0eyHbQEyl6ND/wf7g==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOZVpRYmNMQ1E4ZlFXMDVq\nbW1FcnYzWTBHQmdhSjlsZUtZK3dzNUZjY1FvCkV4VXU4MTcwNVZoZUpVVS9JeHE5\nbGk3UzE3YmpZQ1JISHV0RHJqSG9ZNXcKLS0tIHhBdnIvMjJkeWlVRE9Fb0FhWjNX\nUjBVWDltK2w0akkwOTJaTGNSWWNrRXMKqZRNnHiXvn1QBoSGdABp7vOqNlsEN6Xr\nDp3NByXow6PuRuWvQXHzd+WC+ADkwNaaiT6TUrbZcd/Pl8Ges9kcZg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA2VFBxc2ZMUFp0YkJhVFhW\nc0ZNbnJUcGtkT1VLR2YrVGhRRFVTZFlyZXlJCmVnblNBNGxIdHh4Q0IweWR1b0dq\nRWFyOURuWkdkRE1RTnJRMEpXdk9HaUUKLS0tIFhKV3FoWW1zRk1pMlhuWWlhV2E5\nQzJTRHAyc0JtSjd2NHlJODZVbndaVDgKFA4565X/4FqNq/fZDZTg81/55hZi4c7b\nTti2AnyE3OcY/kurXJFHRinVMqURQf1fx9MxqUYRitiCz4qe5zFF+A==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwRWlKTFNocnBRN01pZ0hm\na3pyRUkxMWM3dE1iTkZZcGRYUFcyb1RDb2h3Ckt4blpGZHBjcnZ4SVE5UnF4cmY4\nWW0rZG5wUkZINVk5a2lmSzN2L3JJY1kKLS0tIExpWGlBRFZJVGpGbFhxRE5ZYjBo\nQVliTlFIZ1U5dE9xbDhHMUtxenpBSm8KY6sIFEfK8p+70IXsC4Jwb9Lm/pd9+V6K\n4JAzGrpA6mAuIwwSNnbdcA5j8FmBhCpK6nLBWmFhGm9Y+MRTaM7Jrw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoV0pIUXlyOFFTUHVTdllu\neG10VENOYWl5Wk0ycnpLUytTbnlrQ25pMkFRCnhyRU5xWW9vQXJJOHdFMGZqMkR1\nSktqT3lOdVVPRUN4YTlveGp0NXpqd0kKLS0tIGRaTktra0ZtSVpzSHBrY3VSSUph\nRVRZOSsxTTNmMmltMlJnVy9oT2VEWU0Krxf49B1BsrWn05fqg+cZ0k0PtfJJNfn0\nUL44RUWXWbK2igQHaIct9DfYe7DEonBJeROuxDYm8g7yNOv15S+P4Q==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp"
}
],
"lastmodified": "2026-07-28T01:44:26Z",
"mac": "ENC[AES256_GCM,data:eyOSn03dzHSgkshPzLVwc95382eEFaDQarHs9l83dtcsb1Ui9CjkKipl2DVSUb6bdMUH1qKYmXqJhwFnAZbFZFjT4VTKtutNtM+OkhVXfT+fJs+1+u7k+ZUYFL9HuxKA6AWpwX3eJ8vmJDJZaAayJbm4PRzOyJywvKeneQqdUS0=,iv:QRKltR4qzVofo/Elt2Us/lrHlD7BenX605X31x+Ng78=,tag:SZ97CcfgMJeu3yqNk8Y/cA==,type:str]",
"version": "3.13.2"
}
}
-43
View File
@@ -1,43 +0,0 @@
beszel-token: ENC[AES256_GCM,data:cbQOXhLzNk4g9d6hvm2DH7Q5ApTPCTzsW2txflDT2dD/UPIE,iv:V19MI1GEo5/0205Hrt7JImfkjduFiZ7f9aIkDVaI8mU=,tag:WCArgdrnIOudVe/Tw+oxRw==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZeDQyNFh3dm1YZ2FTMjdK
cDBWcDE2a1c1S0s1enhWRnVuMlVJc0gyTXk0CmxWNkRJMDhkeGpRTGltaitnVkZS
a3Q5TW4zYm5Ja2FETEhJcGF0N2ZKbmcKLS0tIDN0a1FqRGNOY3Y1UWxvUU8zWU1m
bE9DVzZESG1HTEhVWUdJOTF0bDhRVGcKP6OoyDAGLB9jQ69jpFyho5eaeK9XtZgN
RlSJpBm2Jo19h/crpH9AWXUAIG0BWueyr8mwBu12cQdFIU3IyZT6gg==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwRVFQcHpkQkNlSDhhM1Zp
bUZyOFduQXg1cDd6ZlNObFgzL2hmUmJhM1JFCnprMldPQXJNVW10dVRqQTdWcGlv
RnBYWWFsaVNrMkJpS0pkOGlQQzlJVVUKLS0tIFNsMEEwZTREZ1lwWFJGdE5YSVVU
ZEZ1bVpFMEQ5N0g0L2RacUpLMWQrVDQKxPzq6f960purgAmUJw6IZnZSnhkzNE8r
CSrFDowKTZI2KRdCtQ5fGhEoWO0ZPgVNxYV0KH7JBttylcpRLm6r5w==
-----END AGE ENCRYPTED FILE-----
recipient: age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0czVoT3ZyYTZhOEQ3RWtR
bVl1MW5vNERxUEpmNXV0MGVBRE9ySnBjT3pzCkM2aUVpZjg0SkNVTnRRMlhyMTN6
NlFrZDVKV09Yc0tuKzFzR0ZtQ2t6WkkKLS0tIG1mbUNFdHBycS9UOGc2cjNpeHVm
NUd1NThRQlZXeG1WbmR5Y3pTYXRKc3MKwSnE+0bGmxOAQUje6jHxuzIIyD6ZAwVz
b5AAYwbGRagKj6fimsHBUmi4ohyG1huIGGOU8HiUYpu4PGJgOscztg==
-----END AGE ENCRYPTED FILE-----
recipient: age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlVGRDMWNDZUR6c1VpUjRK
WDY3L2lNcWFFcm1UV3RPMjlqYnBGVEJLcFFzCjFxck4wdlp0Wmtzc1RKNS82MXpK
ZHBzOGhkc3ZuZUE2UmpUSTgycWdLSGMKLS0tIFUyYjczeUFWU2FyMlBTdzAxMTBE
VzhaVzlSL05nZzNmR0ZjNEFPTXYycHcKfiJ0KjdxtLWsXxsWKzAL+H3hYYjHrYO9
BjKknq1ZQJM0sB/Tid+GLqDwKi966MQK+AwHF5MqbsHW7eE5bO1nwg==
-----END AGE ENCRYPTED FILE-----
recipient: age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
lastmodified: "2026-07-19T02:30:40Z"
mac: ENC[AES256_GCM,data:rKHZjU/MH08ASTlu32HZO9uWmsBYuMCEC6M8gwVhzuWvmablnP05tS2z13XfaWaCEUXk6kmGJKuU0zu5+IKVZgamCF6DAMtxQb6bVCaLsoAm/GSqWQ5VI9eHqgnSSdN/o3ul/33Rf8iBQo4aw8FFAmDVuNz8bfAn0QefFTj0ByI=,iv:JD2gtqRinOY77etg6PUmZNovkYl1Q3F6ZvRi4x7RznQ=,tag:/5IMpWKRVt+l1luCTQE0BA==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.1
-43
View File
@@ -1,43 +0,0 @@
beszel-token: ENC[AES256_GCM,data:qg2eb7Eq/w7vIuiiDeK/s6h9nmHWuooyrd93GvdLD2hGOvka,iv:FWRBQTIY9PzV2V8J7z5ftEb5dvjN/S0hP0H9JV+Uuq8=,tag:WP7DHOu4elfqdLoW7vo9qg==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZmozK3VjcENGb3l2MDNj
QVE3VVI3Z3R0MjduSlJGL3BFWkg5NHZmZms0CkNBTWZKZFJVNkJMZ0tpNXhXWW5q
WndvbmhuOVh5ZHhJMmJwenh0M2dEVlUKLS0tIGpXQ3dhRXVWUDZzdGg4RHN4V3p0
WDZnK1lMUHVIS09hRUEzVHRHQ0EwcWsKEd5NtstHSPH8okXcFLW7BId5xPGmgyC3
dAodIGkLBlJlaL1bZuRy6Vaac+mJZL5KJYD/o1kYSXRiHJfNpaulTg==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXWWhYOFU3VHZ3S1VuRTF2
QlZvMkJUNDZmT1RDWWZsMkxnZjJldjdrRG44CkU0NEJTR0NhQUpjMEY4RXBLSVBJ
bDNtZ1UrRDNUbzA0NDJ1cWJKdlBBMWMKLS0tIE9vTmEzTUxpODF4SG1MQ1czM1N5
dVgrMERMRXJwVmhtMldTZUlHNUJVTEkKzUf9cETbo0K9bCPkrlbUkgkhRIPvjCHR
GFVwgb2fcaOzZHcDwlLDLaP6HbRPtzJfVbLpQ1XU0N/xijSr9ENB6g==
-----END AGE ENCRYPTED FILE-----
recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlOVdWN0tHNWZmaWlWQ2hJ
MVFYd0pMNmJTdTlJQ3h1c0dzbDE3U2VqdzNzCitCcFdhbjRPYnl3Q214WnRxa2Jq
eTFnc2VBcXlGcTVYdWpWTFZiWXBMUDAKLS0tIC9YdVkrTU5EeXNlNzliLzBlYUR2
Qk1ITW9tMElzUHF4MVJyWS9uRGs1QncKj8OgxWYabf8MccWD6zYLxl+FvYPAvLCW
p/FJ8uMa9YWp0RjICvWIwyO0gK7oaxTh/JzfBjABngEZnZazNYP7Uw==
-----END AGE ENCRYPTED FILE-----
recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsbzFKckFQVDg0VEpJMFlv
WSs1ODVoYlZZK3YrSXlMU1hIQjdLTUhNdHhJCkRnbHY5Z2NGQVMvOHV4R2hTd0xu
LzdheXdZVVBsTjhVOU8zZGFKN0lPWjQKLS0tIFdaaG5SeElDN3YvMjFxdHVLWlNX
M1BQQmtJTUdsK2x5TEo5TmNrMEZidkEKhi4jNKhrkBAPhJoeYNg1d60LQTqmJt+G
Z/2X5uWzh12zF5keL8cNHH6sD0z0MOChtd4UF37yr+lb/LFJkia6LQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
lastmodified: "2026-07-26T01:12:11Z"
mac: ENC[AES256_GCM,data:0r68y6XmdiW7p/h9QmTR5h37TxKp+0wF44LP8QfG6cGP0tvaasuvv+V9EC3msnREYjwJhlnHjRd8hJ31HGVkOHbYWI4V5cF4ZnaiGgrZJhsGcJvOJSroAvQwev+vIlXZpAPgZz3Z35FDvOpgb71l5c0so0l4EJhc70JOh2FxhvQ=,iv:Or6l/v/E01+9wtZwyl+pCttiuReR6V0kmBzE9KhsfgQ=,tag:cAEUgj/9tKDToKFr00g5UQ==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
-25
View File
@@ -1,25 +0,0 @@
beszel-token: ENC[AES256_GCM,data:gjbT3uROiVKQOJaUeafTxjVknQO1Tvbyx/Pl2bTad7DezByX,iv:3ikf7OaT2omO8yd6G6UwYbaRBSzyvbn+NghxAe5bcgI=,tag:Zuc2EP8rUtdDhr5CzSW2Pw==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJL25EUUZack1FMzlnMmdk
Z1hnejZRNnMwVHpmWkFNdGcyeHVsNnRsSGtZCk9OdDhhcnR1WW9ZMEZ1OUVYbm1n
RmZRVy8wb1J3emJBK3Rrd1d4U1dYUDAKLS0tIGJaaElvSk1sOTBOM0lKck16OUtu
NnRZb3U0ZndmaHBZTm8zczhWdE1oaEUKkf6fLomAHoKPhuM4e9q96YmmH+h4VrEj
2x0rnwBwOoRzYWutB2MVtlsphAZmZ/PK0tEecT2MM0XXayVG/33qdg==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpeEJvaHl4akc1TFdEVGV2
czg3OVprU3p4ejNpTktXZEpneDgrOEhrZGlZCmJwT1dhSkZneHE5UmR0WTd5UENq
VHJEWG1EekJLY2pRZldtVGtxTHlGaGMKLS0tIEYrWHE0WTgyUlIwdktmNzNIS3FW
ZVRvT1dHa1Vzc2RSakVISzdMTlpnVGsKeT+edn4+LUkVtpRUNd/gKX3H1HG2bvNo
c8iI6qr/l6oxfP85OrKYFDU9IGvDMxSSdbixHtojPEb5OKVurV0WPQ==
-----END AGE ENCRYPTED FILE-----
recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
lastmodified: "2026-07-23T23:32:57Z"
mac: ENC[AES256_GCM,data:l9a/yNRoxY1hvSkLuR4N7deeKue/1JPlSvZvJfCSNbQ21p1qR433BbSDYvfW+kXQXS8GVcfgXSd9ywNzgVvkA5lR1++uYsZBLbYxJ+s3TKWs6/yECAZ0eM1KBA0BEm7cLSsHTOwd+2WspvmCYir++FDO9XRuS3guiMnQBglDf/E=,iv:eE6GVqexQNSiLYfmTTUdUx5AO//wyjSIsr96xAX1pcI=,tag:CNsoyDZYLUt5Seu7W5wJrw==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.2
+50 -34
View File
@@ -7,13 +7,14 @@
lanGateway = "192.168.2.254"; # LAN default gateway (router)
lanPrefixLength = 24; # LAN subnet prefix length (/24 = 255.255.255.0)
lxcLanInterface = "eth0"; # LAN NIC name in LXC containers (set by Proxmox --net0 name=eth0)
lxcStorageInterface = "eth1"; # storage-client NIC name in LXC containers (vmbr2, --net1)
vmLanInterface = "ens18"; # LAN NIC name in Proxmox VMs (virtio, first NIC)
vmStorageInterface = "ens19"; # storage NIC name in HA server VMs (virtio, second NIC on vmbr1)
vmStorageInterface = "ens19"; # cluster-internal NIC in HA VMs (vmbr1 — DRBD + Corosync only)
vmStorageClientInterface = "ens20"; # storage-client NIC in HA VMs (vmbr2 — iSCSI/NFS VIP)
pxeServerIp = "192.168.2.223"; # pxe-boot LXC container LAN IP
nixCacheIp = "192.168.2.224"; # nix-cache LXC container LAN IP
tailscaleRouterIp = "192.168.2.222"; # tailscale-router LXC container LAN IP
torRelayIp = "192.168.2.221"; # tor-relay LXC container LAN IP
serverIp = "192.168.2.226"; # server (NFS/ZFS) Proxmox VM LAN IP
dockerIp = "192.168.2.225"; # docker Proxmox VM LAN IP
pbsIp = "192.168.2.244"; # Proxmox Backup Server LAN IP (not NixOS-managed)
domainControllerIp = "192.168.2.253"; # FreeIPA domain controller — authoritative DNS for sweet.home (not NixOS-managed)
@@ -21,7 +22,6 @@
# Cross-host references (LAN hostnames/users other hosts reach over the network)
nixCacheHost = "nix-cache"; # substituter/remote-builder hostname
nfsServerHost = "server"; # NFS export source hostname
dockerHost = "docker"; # docker-compose stack host
# Raspberry Pi's own Tailscale hostname (not fronted by `server` — it
@@ -40,17 +40,25 @@
# from scratch.
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICuHUxGNH6ei3BZD+EfZs3l4X8uJNcjQiOsM/G4yo4O/ lxc-nix-cache";
# Beszel hub's SSH public key — used by every agent to authenticate the
# hub's incoming connection. Update if the docker host is ever rebuilt and
# the hub generates a new keypair.
beszelHubKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
# Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
# host (modules/nix-cache/server.nix) — one per client host that's allowed
# to use it as a distributed builder.
remoteBuilderAuthorizedKeys = [
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFDEA1S2ikpObREgbP5uVBWMxIOGbY8B+Wx7VTZK1m6t root@server"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPAYIT9ormlmxZ0SziyDQaUntnKI8HK9/s3Qac1ZKjP2 root@docker"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKKKzoEPl/ZW9KBRHBcp6/ThOngGpwMv5EhkTlgC4aDf root@nixos"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGtOWOCS+ImHc7NehguoyD7PbonGosKMZqc9+QR3v/h root@nixos"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHxXTQxFnArK5HXG7czeoybZebCGfxpUdusJkPn+BCSp root@server"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+ioWPhHixlgCB9KIQ0QTHTz6A+Oo2F3uKiINLip5rO root@docker"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEj26SL/emsVjW2YhRucJVp2kTz8WgcEQgjBEBLRikk root@claude"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII/rLceRhnDobVXQYiPceuhDHHvVjFQ1pc9A6un/eUlA root@server"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHj11bLPpRzH2oslnwFzEvY9cSgfEFtSZbLQaDm4nZMK root@pxe-boot"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH4/Sesm8NYpj73R0cbGhI0Ubvz73vIVWAnbEDTlBTdh root@tor-relay"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIBCFUUtePndW7pqtlawft1QCdHmBVs3O/c8EJO+RcXV root@tailscale-router"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBZ9WKKAlP9Z7GQdgaZ1Xgw9C+vja2lqEZO5rJFpVqYN root@ha-server-1"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEGNKlaaMckd8nLWNGz4B2QokXjnnIvM+rEUv+R6h0sp root@ha-server-2"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+XeMco7OxUpjrjZm54HogMs9QB5xlcKmElASRvrmlW root@nixos"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIMseQwPpmaa6cgV5U8KhUsiVSYARG85zGa9rho0LJWks wayne@pve1"
];
# Admin SSH public key, authorized on the primary user of every host and
@@ -92,37 +100,45 @@
dockerAccessGid = 50010;
# HA file server cluster
# LAN IPs (vmbr0 / ens18) — client-facing: iSCSI initiators, NFS, management.
# Storage IPs (vmbr1 / ens19) — isolated internal bridge, used for DRBD
# replication and Corosync heartbeat only; never leaves pve1.
# haServerVip: floating virtual IP managed by Pacemaker's IPaddr2 resource;
# NFS and iSCSI clients connect here regardless of which node is Active.
# LAN IPs (vmbr0 / ens18) — management only after storage migration.
# Cluster IPs (vmbr1 / ens19) — VLAN 10 (192.168.10.x), isolated internal bridge,
# DRBD replication and Corosync heartbeat only; never leaves pve1.
# Storage-client IPs (vmbr2 / ens20) — VLAN 20 (192.168.20.x), isolated internal
# bridge for iSCSI; docker and server VMs connect here instead of crossing vmbr0.
# haServerVip: floating virtual IP on vmbr2, managed by Pacemaker IPaddr2;
# iSCSI clients connect here regardless of which node is Active.
# Protocol separation: iSCSI on storage-client subnet (VLAN 20) only;
# NFS on LAN subnet (VLAN 2) only. Enforced by firewall on the HA nodes.
haServer1Host = "ha-server-1";
haServer2Host = "ha-server-2";
haServer1Ip = "192.168.2.228"; # LAN IP, node 1
haServer2Ip = "192.168.2.227"; # LAN IP, node 2
haServerVip = "192.168.2.229"; # floating VIP (Pacemaker IPaddr2)
haServer1StorageIp = "192.168.4.228"; # storage-net IP, node 1 (vmbr1 / ens19)
haServer2StorageIp = "192.168.4.227"; # storage-net IP, node 2 (vmbr1 / ens19)
haStorageCidr = "192.168.4.0/29"; # storage subnet — internal to pve1 only
haStoragePrefixLength = 29; # storage subnet prefix length (/29)
haServer1Ip = "192.168.2.228"; # LAN IP, node 1 (vmbr0 / ens18)
haServer2Ip = "192.168.2.227"; # LAN IP, node 2 (vmbr0 / ens18)
haServer1StorageIp = "192.168.10.228"; # cluster-net IP, node 1 (vmbr1 / ens19, VLAN 10)
haServer2StorageIp = "192.168.10.227"; # cluster-net IP, node 2 (vmbr1 / ens19, VLAN 10)
haStorageCidr = "192.168.10.224/29"; # cluster subnet — VLAN 10, internal to pve1 only
haStoragePrefixLength = 29; # cluster subnet prefix length (/29)
haServer1ClientIp = "192.168.20.228"; # storage-client IP, node 1 (vmbr2 / ens20, VLAN 20)
haServer2ClientIp = "192.168.20.227"; # storage-client IP, node 2 (vmbr2 / ens20, VLAN 20)
haServerVip = "192.168.20.229"; # storage-client floating VIP on vmbr2 (Pacemaker IPaddr2 vip-storage, VLAN 20)
haServerLanVip = "192.168.2.229"; # LAN floating VIP on vmbr0 (Pacemaker IPaddr2 vip-lan) — NFS access
dockerStorageIp = "192.168.20.225"; # docker CT storage-client IP (vmbr2 / eth1, VLAN 20)
haClientCidr = "192.168.20.0/24"; # storage-client subnet — VLAN 20, internal to pve1 only
haClientPrefixLength = 24; # storage-client subnet prefix length (/24)
haStorageRoot = "/srv/ha-data"; # XFS-over-DRBD mount point on the Active node
haStorageNfsFqdn = "nfs.storage.home"; # NFS VIP FQDN (storage.home zone) — resolves to haServerVip; use this in fileSystems device strings
haIscsiIqn = "iqn.2026-01.home.sweet:ha-storage";
# DRBD backing disk — identified by SCSI controller path so it resolves to the
# correct block device regardless of OS-level naming (sda vs sdb can differ
# between Proxmox VMs depending on disk-add order). drive-scsi1 is always the
# dedicated data disk on all HA nodes; drive-scsi0 is the OS disk.
haServerDrbdDisk = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1";
# Storage
storageRoot = "/tank"; # ZFS pool root on `server`
# NFS datasets exported from `storageRoot` on `nfsServerHost` and mounted
# by client hosts. `subpath` is relative to `storageRoot` — combined with
# it to build both the export line in modules/build-types/server.nix and
# the "<nfsServerHost>:<storageRoot>/<subpath>" device string each client
# mount uses in modules/docker/mount-data.nix. `mountpoint` is the
# absolute local path clients mount it at, referenced by that same file's
# fileSystems attribute name plus every other place that needs to know
# where the share lives locally (modules/build-types/docker.nix's
# tmpfiles rules, modules/traefik/rotate-logs.nix's log path). Renaming a
# dataset or moving where it's mounted only needs changing it here — the
# export and every client reference follow automatically.
# NFS share definitions — used by ha-server.nix (exports), docker/mount-data.nix,
# and pxe-boot/mount-pxe-images.nix (mounts). `subpath` is relative to
# haStorageRoot; `mountpoint` is the absolute local path on each client.
# Renaming a share only needs changing it here — exports and all client
# mounts follow automatically.
nfsShares = {
options = "(rw,sync,no_subtree_check,no_root_squash)";
dockerConfig = {
@@ -1 +0,0 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPW/X9Mdrqs0wLR7XbEDTihk7TEkNZ3LcCeXoa2ITSDA linode-server
@@ -1,14 +0,0 @@
{
"data": "ENC[AES256_GCM,data:hzI2cJ81+Psv35MMZO06UHpC8F9FLVkfUzn0ReuR+WoUds8D5zhDFbQHeR6ByeHTuK8Hlj712okhoXh6Vm7l3WmhLJZkr8IGLsV9W+P2PYityjuOtBPrphrKUeSqxJfjQ31/EhZLOqw/508XUmQDNcs1/n2g0TtQ2UQSTAuO8r1OlRfRSPizvVdj7lu+Vqg3dDBRETSOJAYIh8XQXoQWl3M4dS7jOYUvgc4EsYOWrvPPPS+8xXzctBPeToasY0IyWtLzoeNajCs5EpH5u7S5K9S40/vFPa45Ic2IxEkb1HHClyN6lXWbnc5QlJ5HPrq+2YBU2y9dJHd9DQiwEbcEBIvR3/Lo9puVDppXx9kez7i4XR1UHU7WVu39nQZuP50BQQazj+eM0/HJapXfjoqtaaq0qAGN4onnmRd0vEmrZZl4WMXmx8m9HhxdNJoZ4C2Wb2skdHZok88tPzjd4r7xKzVhAbB+wqS4LML6FmCNk+m0USQXa25USXs0JDV7lgiTFQJPE/67JGa820WFicGs,iv:YmOeK2Ha3yBXumVO9strgLgqNmPOcnqwUDJv5QNR1WM=,tag:Z7kZvQLDOKx3uX4dRXJqeQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBOSWl5ajRSVndlMHJJQTFp\nYzRRU3RQMEVzazB0Sm5FdWZmMzBRaUl5RUFvCjgwRUZWdUpvUjkzWHFVdThIZ1ZV\nM2RMNy9pUWF4VVBCVHBGWHgwakhKQ2cKLS0tIERIOTVVZm55QWkvL25SRGZkZmtI\nT2o3ZEJNQ2hmTDNoRUx2Z3UxcmlyekUKLOajmvRfLdCJL74PKSgBtIXDuAVd8NwM\nh4BtDs0hONOz82JaBqFw8Uz28hVFG/gcS80br1o2klqPd2gN62PCVw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:33Z",
"mac": "ENC[AES256_GCM,data:JJC+12gTCsDVvMWRtL5cj50kf1n6Xn2j1hNBnwvZXUA9Pdd96SXt61U/Q80h8GZ7Ycs/slsV7h5f3g6+8tV0PcdBM/vy0vPM5qX1zMySyMv2p+dkJb9MwQpPg2xAQ9jjYM9237p5n9nysgu74h4V7ccBqmBzp764bL9wx6hEzS4=,iv:U6u1GFvoxaqxmHv1zCht24nW0ZMJR2b4pZqCG3bGNzs=,tag:MocwywOBg/1rUuoSADuJVA==,type:str]",
"version": "3.13.2"
}
}
@@ -1 +0,0 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILVRdEddGd+AMNe4kXbmA9UXK8JRsPEuxPx1vhNT9ZG8 lxc-server
@@ -1,14 +0,0 @@
{
"data": "ENC[AES256_GCM,data:JTmbN++hNMddBdMKIdzqZ6hLD1So9hyAzwbce/QvqWBsa1At85cMRUx+P1Yi4Sq2aXCg1Tf2g8XC9DXsVfMusUGEHrHbJApwkia/xHNMcFOohscUkPltXnm/lRilm4hwJUv6ay6UyLvIqMQAzzC8YWebHkKo00rxRxQWQ77wjmUG4dJeujlK5wMH0zTXUkrf7tIdJ20Zzg1BrAU6s4FYLvRvcUSW2ROu+sF2Sxgy1qVNJfDrRnY4REh9013tTSB5IlgJzUSQLs1vHr7EpIfmA5MCCSkZjplGCKvxkcNHB3aaOkrxkGa18JeWoklRen31UPU9zMUhTjAZ6VbbIagxzaFGKVP27cOiHQPNpEIMXwaRzoprjBX2PJ/Bs6pxe6hBpmiKOb40XdtxFQx2rofPcbQwklTpM5SFwvK+/bgawy1m8O3NHMhcw1qvd+6KjUdqy++/ivFC2Nyy+VfUDyGu0JLK3X+YqU6JXXMcUWOq79pVQIL3q1ofbhVHQ8p17+V3wQQUB3kPo4EEcKikI7Z9,iv:TVYVDOiTsgXaIcuJdnd3djPWXCMDDGjpefAW0MR+7Es=,tag:jIYO1oatbvInp77d4fC8mQ==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5QlNzNEFqSkhKTVhWVisr\nS21zdVBZSnpITDdrb243c0pUdDR5WXl1bFVrCmkxaWR6U01sLzhLL0MwRGh1VG5D\nNEFiU1FtNXRBdGtPellFTXYraitCUTgKLS0tIFdwWGFnMWFCN1ZvK2VBMWR2VTJZ\nRlQ4dEpmczA4bEdvUkNhaC93OXhDTkEK0QBkaLV1mbTMlSnjmN4x+qljGipHM/DQ\nUlmBYyi3nEOrI36I/Mm8yoBZai/qWqdg6IG+sxDE49ZOLp9PhpAWww==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:44Z",
"mac": "ENC[AES256_GCM,data:/l6pa3LE7+kFYqH2pv2RIcYBycLrZfpb92Al3SIU2tMeFUUvh+C8q8P9CgmAaiQuQ8S2dfYIJVx67zc7cRqI/UL/dFvDSv0YaMTGQ8Wn4fIXSh01EL0f/QVaIfb+uuvyEsdjy2ScWTWcCf2ICnC/zaMmp+xP+MmR1DRBM8KPIkY=,iv:xsmaYZ6dyHLU3BVfT3jxbfWWeBvYKMT+D9MtRxF9jlo=,tag:VzOiutHQJGHqb5UCI+cxSA==,type:str]",
"version": "3.13.2"
}
}
@@ -1 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILdvVsj5cw/y6s2q5cPqKnULgnkEnsIQahrhm9DyCdd/ proxmox-ha-server-1
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN7IABG7x9ejHSENmHsL/BF9MQadGuBnHZa1eUEFMYnH ha-server-1
@@ -1 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILjgkgUFZfpTLkFQeEbqXlvktE641EexWLv2I4iVKyQi proxmox-ha-server-2
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAINWuFwPKW2jwQEj6c//xb95DbtandPkJ7M0ceyTNd35G ha-server-2
@@ -1 +0,0 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGQHSubxjvaIV9Xp5ABJSKsajCZBGyGmjsdaA2TWCFP8 proxmox-server
@@ -1,14 +0,0 @@
{
"data": "ENC[AES256_GCM,data:GdVHSekFtRtppf8xxMPsDE0cu8Hwq3ol0iJTfrwKwaAB0IW8YS8TFAeRnfOI5AE3P1JUaGDZ1Od4UXu4Jk2+uYCgwz074qc1WukzUsd8Q3G03ojdrTA13uH8YCUTRsm3bQfB+kmH5qefn9j88UJq7TYiTLe3U0DUU/B83lgyiFIygP8CKWDgFH3jWZOWPmyVa4Z1fMWVsiNhjO4hHduamN/I2FOY7U2TI5+zcYgGeunjnniLHcQdiO0iK9TOP5SckOVPZOy6daKzGcckEq03NV4y+miSucEUHlcQqMUyvsJwETIaCq28xR2MDTyDz42SL2HIodb7FyREtxd91W6gPN1j51yUZxt4WA9R8/WczDcUgmROK+HInPp+ktjWqv2ymXXV5xidxh7m+vrGTPC+YQqyFD4eIp3+h2S/vyzHimBa2rdEp9NNXTQlH3CfL2aLC95FOme+uKTDqWA2PHfB0rQ5uhHAuAgaX5vb/GTiQii5dq4k2O4MDf9u+fdPuOSCaGdlirWmSE45SvaXLJ3M2zQq7M/UTE+wSFyc,iv:EaIsYnGxf42LWQ3hzBU/HsBda/FHInhjZkwsPLcZMEY=,tag:16OuAs3Ram+xd4RfWEBXkw==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBETE4vOHhldGRFRjdYaXpi\nSmgzTFlqL3Qzak1CelRBR29jZzR1V0pOQWw0CkhSWDkvbkJWaDVHMzlKNDZUdzhH\nWGoyWXQ1KzJZQU1kQzFvaDk2V2lWQmsKLS0tIHlTY2tua2F1TXBhamsvcDZ0bmI1\nZVFwUGtQWW1HNTM3UUs4bHRZU0xKRDgKBG2iI9JP0lhU5VCWXrpN1b2rYEYk8sOZ\n9FUO14KKMg9QRfSa2iHOa84DByx2hxVRc9wLukBUpkKOAjGSxeGMeg==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-25T11:45:58Z",
"mac": "ENC[AES256_GCM,data:jG7ohweKKndoadidquejYG/1w9iL+9Xb5/IsU/C9fn/Tq5RXEjrxO37COY8sAD7dvQf7iBNsly7upsjtHaMK8ybjQDaa6IQhoGBlfSOA2O968klaJZRQRiLPzCRet252KXzOtvDscrBvyYItvyqjnW5qBbw7lfT4y2J4OA2ieC0=,iv:mMo7Y6XEjQcchNOY6eaw5LOmjFKaQKx84o9bnGNXCKI=,tag:aHkPjs7uVfv8h0REHN8YmA==,type:str]",
"version": "3.13.2"
}
}