Compare commits

...
Author SHA1 Message Date
beatzaplentyandClaude Sonnet 4.6 ec44b7955b feat(gui): add wayne's GUI Home Manager config matching the nixos user
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s
Adds nextcloud-client, chromium, git, vim, claude-code, fish, sops,
the Nextcloud tray service, SOPS_AGE_KEY_FILE, and Proxmox/PBS desktop
shortcuts to wayne's HM config on GUI hosts. Uses lib.mkDefault on the
EDITOR setting in ipa/client.nix so the GUI layer can override to vim.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 19:28:22 +10:00
beatzaplenty 7e8755d141 added ipa keytabs for ha servers
Check NixOS configurations / eval-hosts (push) Successful in 10m25s
2026-07-28 19:12:23 +10:00
beatzaplenty 8194707478 Merge pull request 'Worktree ha file server test' (#93) from worktree-ha-file-server-test into main
Check NixOS configurations / eval-hosts (push) Successful in 10m37s
Reviewed-on: #93
2026-07-28 09:09:04 +00:00
beatzaplentyandClaude Sonnet 4.6 d740064a35 fix(ha): all 7 acceptance tests pass — targetctl, fencing, failover, data integrity
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s
Deploy/init fixes:
- iscsi-target.nix: targetctl binary is in rtslib-fb (python3 env), not
  targetcli-fb — fixes ExecStart and ExecStop for the targetctl.service
- deploy.sh: _patch_targetctl() applies runtime dropin to both nodes before
  cluster-init so Pacemaker can manage the iSCSI target from first start
- cluster-init.sh: replace crm configure heredoc with cibadmin --replace XML
  (pacemaker-4.0 schema: globally-unique in meta_attributes, promoted-max/
  promoted-node-max, Promoted role in constraints); force_unmount=true on
  xfs-data; DRBD promote timeout 240s
- cluster-config.nix: add crm-fence-peer.sh/crm-unfence-peer.sh handlers;
  update fencing comment to reflect resource-only + Pacemaker-aware handler
  replacing STONITH during testing phase
- ha-server.nix: add openiscsi to systemPackages for T4 iscsiadm availability

Acceptance test fixes:
- acceptance-tests.sh: fix ((PASS++)) set -e bug → PASS=$((PASS+1));
  detect Active/Standby dynamically via drbdadm role (Pacemaker can promote
  either node); T4 bash TCP probe instead of iscsiadm; T5 timeout 120s;
  T6 echo|sudo tee for root-owned XFS write (bash -c redirect runs as nixos
  not sudo — permission denied); use ns cat / ns rm for root-owned reads

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 19:03:39 +10:00
beatzaplenty 164d14eb87 Merge pull request 'fix(docker): declaratively apply groupmod --non-unique for IPA GID conflict' (#92) from worktree-parsed-mapping-raven into main
Check NixOS configurations / eval-hosts (push) Successful in 10m36s
Reviewed-on: #92
2026-07-28 08:19:53 +00:00
beatzaplentyandClaude Sonnet 4.6 720399b00d fix(docker): declaratively apply groupmod --non-unique for IPA GID conflict
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
NixOS's group activation uses plain groupmod, which silently skips the
GID change when SSSD exposes the same GID (50010) via the IPA
docker-access group through NSS.  Add an activation script that runs
after the normal 'groups' step and applies groupmod --non-unique so the
local docker group can share GID 50010 with the SSSD-provided IPA group.
If the GID actually changes on a live system the script also restarts
docker.socket + docker.service so the socket is recreated with the new
GID.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 18:17:43 +10:00
beatzaplentyandClaude Sonnet 4.6 da4d808c6a fix(ha/cluster-init): use crm configure instead of cibadmin XML for resources
cibadmin raw XML fails schema validation under pacemaker-4.0: globally-unique
is not a valid direct <clone> attribute, and master-max/master-node-max are
renamed.  Switch to crm configure commands which are schema-version-aware:
- promotable clone with promoted-max/promoted-node-max
- order/colocation constraints using Promoted role (Pacemaker 4.0 naming)
- crm configure handles schema differences automatically

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:57:49 +10:00
beatzaplentyandClaude Sonnet 4.6 79cde50e27 fix(ha/cluster-init): use targetcli delete commands for LIO teardown
clearconfig does not reliably clear kernel LIO configfs state — the kernel
still holds backing-file references, blocking umount.  Replace clearconfig
with explicit targetcli delete commands (/iscsi delete, /backstores/fileio
delete) which do release kernel state.  Also make DRBD secondary demote
idempotent (skip if already Secondary).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:55:26 +10:00
beatzaplentyandClaude Sonnet 4.6 d31d9fa584 fix(ha/cluster-init): fix LIO teardown, mount idempotency, VIP portal
- Remove VIP-specific portal binding — the VIP doesn't exist until Pacemaker
  assigns it; the default all-IPs portal (::0:3260) is correct for Pacemaker
  to manage
- Clear existing LIO targets before re-running targetcli (idempotent on
  partial failures)
- Tear down LIO kernel objects after saveconfig so umount succeeds (LIO holds
  the backing file open otherwise)
- Guard mount with mountpoint check so re-runs don't fail when already mounted
- Use --replace for cibadmin constraints (idempotent vs --create)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:50:24 +10:00
beatzaplentyandClaude Sonnet 4.6 c76698efb7 fix(ha): add xfsprogs to system packages; fix cluster-init PATH and DRBD check
- ha-server.nix: add xfsprogs to systemPackages so mkfs.xfs is on PATH for
  root (needed by cluster-init.sh during initial setup)
- cluster-config.nix: create /var/lib/drbd via tmpfiles to silence
  lk_bdev_save warnings from drbd-utils
- cluster-init.sh: dynamically find xfsprogs in /nix/store if not on PATH
  (fallback for running VMs before xfsprogs is in the system profile)
- cluster-init.sh: fix DRBD metadata check on node2 — broken regex now uses
  grep -E for ERE alternation to correctly skip create-md when DRBD is already
  set up (previous regex would have triggered create-md on a live secondary)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:43:27 +10:00
beatzaplenty 601db79689 Merge pull request 'fix(ipa): restore IPA password login via LightDM and su' (#91) from worktree-parsed-mapping-raven into main
Check NixOS configurations / eval-hosts (push) Successful in 10m32s
Reviewed-on: #91
2026-07-28 07:31:46 +00:00
beatzaplentyandClaude Sonnet 4.6 acebbdbe26 fix(ha/deploy): use nixos+sudo instead of root SSH; temp key for inter-node comms
Root SSH was failing because only the RSA admin key was authorized but the
local dev box only has an ed25519 key. Fix:

- cluster-config.nix: add ed25519 keys to root (same set as nixos user) so
  future deployments work without the temp-key workaround
- deploy.sh/acceptance-tests.sh: SSH as nixos user with sudo instead of root@
- cluster-init.sh: HA_USER/HA_KEY env vars + n2_ssh()/n2_scp() helpers so
  inter-node SSH works regardless of whether root-to-root is available
- deploy.sh Phase 6: generate temp keypair, authorize on node2, place on node1
  for root to use during cluster-init, clean up afterward

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:30:53 +10:00
beatzaplentyandClaude Sonnet 4.6 fc8f7baf3e fix(ipa): restore IPA password login via LightDM and su
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s
Two independent fixes:

1. hashedPassword = "!" on the IPA user stub
   Without any shadow entry, pam_unix returns PAM_AUTHINFO_UNAVAIL
   before prompting, so PAM_AUTHTOK is never set. The "!" marker
   (account locked, not a real hash) ensures a shadow entry exists so
   pam_unix prompts and sets PAM_AUTHTOK — which the subsequent
   pam_sss module can then use.

2. pam_sss try_first_pass instead of use_first_pass (login + su)
   use_first_pass silently fails when PAM_AUTHTOK is unset.
   try_first_pass prompts independently in that case, making IPA
   password auth work even if pam_unix returns early for any reason.

Root cause found during incident: wayne (uid 50002) was outside IPA's
auto-SID range (163800000+), so sidgen never assigned him an
ipaNTSecurityIdentifier. Without it, ipadb's handle_authdata step fails
with "Generic error" after SPAKE pre-auth succeeds. Fixed by manually
adding objectClass: ipaNTUserAttrs + ipaNTSecurityIdentifier RID 550002
to wayne's LDAP entry on domain-controller. Any future IPA user with a
manually-assigned uid outside the auto-range needs the same treatment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 17:25:37 +10:00
beatzaplentyandClaude Sonnet 4.6 3f9b968a41 fix(ha/deploy): use non-sudo SSH for /nix writability check
pve_check() always uses sudo, so "sudo test -w /nix" passes as root
regardless of whether the SSH user can actually write there.  Use a
direct non-sudo SSH command for the writability probe so the check
reflects wayne's own access, not root's.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:15:20 +10:00
beatzaplentyandClaude Sonnet 4.6 1263c7540c fix(ha/deploy): ensure remote clone is on correct branch before building
When create-proxmox-resource.sh clones the repo to pve1, it stays on
whatever branch was checked out. Add a pre-build phase that detects
branch mismatch and switches the remote clone to the current local
branch before building, so the Proxmox node always builds from the
same commits we're deploying.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:12:39 +10:00
beatzaplentyandClaude Sonnet 4.6 a8d8b1465c fix(ha/deploy): fix /nix ownership on pve1 before codex-setup
pve1 has a pre-existing /nix store owned by a different UID; wayne's IPA
UID (50002) can't write to it.  Add a pre-phase-3 check that uses wayne's
passwordless sudo to chown -R the store before create-proxmox-resource.sh
runs codex-setup.sh on the node.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:11:05 +10:00
beatzaplentyandClaude Sonnet 4.6 b76d54e702 fix(ha): fix deploy script bugs and correct cluster configuration
- Add root SSH key + passwordless sudo to ha-server nodes (needed for
  deploy script to run cluster-init.sh via SSH as root on node1)
- Fix cluster-init.sh: correct default IPs (228/227/229 per variables.nix),
  use \${VAR:-default} for all config so deploy.sh can override via env
- Fix acceptance-tests.sh: same IP corrections, add -i flag to SSH calls,
  use \${VAR:-default} pattern
- Fix deploy.sh dry-run bugs: pve_check() always runs SSH for read-only
  probes so bridge existence check is accurate; wait_for_ssh skips in
  dry-run instead of timing out
- Fix cluster-init invocation: upload script via scp and run via SSH as
  root on node1 (was incorrectly trying to run the script locally)
- Fix acceptance-tests invocation: pass IP env vars from deploy.sh

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 17:04:13 +10:00
beatzaplentyandClaude Sonnet 4.6 59854a0229 feat(ha): implement clan vars and fully encrypt all HA secrets
- Run sync-host-keys.sh for proxmox-ha-server-{1,2}: generates SSH host
  key pairs in vars/per-machine/, registers age anchors in .sops.yaml,
  adds both hosts as recipients for common.yaml, ha-corosync-authkey,
  and per-host secrets/keytab files
- Re-encrypt secrets/common.yaml with both new host keys
- Convert all stub secrets to real sops-encrypted files:
    secrets/ha-server-{1,2}.yaml    (YAML, beszel-token = PLACEHOLDER)
    secrets/ha-server-{1,2}.keytab  (binary, stub text encrypted)
    secrets/ha-corosync-authkey      (binary, stub text encrypted)
- Add scripts/ha/deploy.sh: full lifecycle script (bridge setup, VM
  creation, DRBD disk + storage NIC attachment, boot wait, cluster-init,
  acceptance tests, --destroy)

Bootstrap order (operator runs these before first deploy):
  1. bash scripts/ha/deploy.sh            # deploys, tests
  # Post-deploy secret replacement:
  2. sops secrets/ha-server-{1,2}.yaml   (set real beszel-token)
  3. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
  4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
  5. Set services.beszel.agent.environment.KEY in host.nix after hub pairing
  6. nixos-rebuild switch on both nodes to pick up real secrets

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 16:48:04 +10:00
beatzaplentyandClaude Sonnet 4.6 dc83333526 feat(ha): pre-stage IPA enrollment for ha-server-1 and ha-server-2
- Add stub keytab files (secrets/ha-server-{1,2}.keytab) so modules/ipa/client.nix
  activates for both nodes — IPA security.ipa.enable = true confirmed by eval
- Add .sops.yaml creation rules for both keytabs (admin-only until
  sync-host-keys.sh registers the host age keys)

Eval: both proxmox-ha-server-1 and proxmox-ha-server-2 build cleanly
with IPA enabled, beszel agent enabled, corosync authkey + beszel-token
+ ipa-host-keytab all wired via sops-nix.

Bootstrap order before first deploy:
  1. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
  2. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
  3. sops updatekeys secrets/common.yaml
  4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
  5. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
  6. sops secrets/ha-server-1.yaml  (set beszel-token)
  7. sops secrets/ha-server-2.yaml  (set beszel-token)
  8. On node1 after first boot: corosync-keygen, then
     sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey
  9. Set services.beszel.agent.environment.KEY in each host.nix after hub pairing
 10. git add/commit the real secrets, then nixos-rebuild switch on both nodes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
2026-07-28 16:30:29 +10:00
beatzaplenty a960c662f0 Merge pull request 'feat(ipa/docker): grant docker access via IPA group membership' (#90) from worktree-docker-ipa-group into main
Check NixOS configurations / eval-hosts (push) Successful in 10m33s
Reviewed-on: #90
2026-07-28 05:48:45 +00:00
beatzaplenty 6f602b2245 Merge pull request 'chore: full sweep — docs sync, SSH key module extraction, NFS dedup, dead code removal' (#89) from worktree-full-sweep into main
Check NixOS configurations / eval-hosts (push) Successful in 10m39s
Reviewed-on: #89
2026-07-28 05:26:57 +00:00
beatzaplentyandClaude Sonnet 4.6 5beed2d75c feat(ipa/docker): grant docker access via IPA group membership
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m39s
Create an IPA group "docker-access" (GID 50010) and pin the local
"docker" group to that GID on all Docker hosts.  Any IPA user in the
docker-access group automatically gains docker socket access through
SSSD supplementary-group resolution — no per-host docker.members
entry needed.

Specific changes:
- variables.nix: add dockerAccessGid = 50010
- modules/docker/enable-service.nix: lib.mkForce docker GID to
  dockerAccessGid, removing the need to name individual IPA users
- modules/build-types/docker.nix: remove direct wayne docker.members
  entry (access now comes from IPA group)
- modules/ipa/client.nix: refactor repeated security.* / systemd.*
  top-level keys into merged attribute sets (fixes statix W20); add
  security.pam.services.lightdm.makeHomeDir so the GUI login path
  also creates the home dir on first login

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-28 15:16:05 +10:00
22 changed files with 1267 additions and 378 deletions
+32
View File
@@ -22,6 +22,8 @@ keys:
- &proxmox-pxe-boot age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
- &proxmox-server age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
- &proxmox-tailscale-router age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
- &proxmox-ha-server-1 age1nxlnrevqs2msdatze562vz6ym4pgydt2zndye83lwqauy6flggtqrevyw5
- &proxmox-ha-server-2 age1scfc8p53q5aq2a87tcmsazmj8sfeft0s8kxg0et4nm3ucxyp3c3s6te82y
creation_rules:
# Shared across every currently-deployed host: root/nixos password hash,
@@ -54,6 +56,8 @@ creation_rules:
- *proxmox-pxe-boot
- *proxmox-server
- *proxmox-tailscale-router
- *proxmox-ha-server-1
- *proxmox-ha-server-2
- path_regex: secrets/nix-cache\.yaml$
key_groups:
@@ -93,12 +97,14 @@ creation_rules:
key_groups:
- age:
- *admin
- *proxmox-ha-server-1
# proxmox-ha-server-1 added by sync-host-keys.sh
- path_regex: secrets/ha-server-2\.yaml$
key_groups:
- age:
- *admin
- *proxmox-ha-server-2
# proxmox-ha-server-2 added by sync-host-keys.sh
# Shared HA cluster corosync authkey (binary sops file).
@@ -108,6 +114,8 @@ creation_rules:
key_groups:
- age:
- *admin
- *proxmox-ha-server-1
- *proxmox-ha-server-2
# proxmox-ha-server-1 added by sync-host-keys.sh
# proxmox-ha-server-2 added by sync-host-keys.sh
@@ -195,3 +203,27 @@ creation_rules:
- *lxc-minimal
- *proxmox-minimal
- *linode-minimal
# Host keytab for ha-server-1 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/ha-server-1\.keytab$
key_groups:
- age:
- *admin
<<<<<<< Updated upstream
- *proxmox-ha-server-1
# proxmox-ha-server-1 added by sync-host-keys.sh
=======
>>>>>>> Stashed changes
# Host keytab for ha-server-2 FreeIPA enrollment (binary sops file).
# Generated by scripts/ipa/create-nixos-ipa-host-account.sh.
- path_regex: secrets/ha-server-2\.keytab$
key_groups:
- age:
- *admin
<<<<<<< Updated upstream
- *proxmox-ha-server-2
# proxmox-ha-server-2 added by sync-host-keys.sh
=======
>>>>>>> Stashed changes
-3
View File
@@ -33,9 +33,6 @@
];
users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
# Grant the IPA domain user docker access via the local group so that
# `wayne` can manage containers without sudo.
users.groups.docker.members = [ "wayne" ];
services.openssh.settings.PermitRootLogin = "yes";
networking.firewall.allowedTCPPorts = [
+66
View File
@@ -81,4 +81,70 @@
programs.firefox.enable = true;
nixpkgs.config.allowUnfree = true;
# GUI-specific Home Manager additions for the IPA primary user, extending
# the baseline in modules/ipa/client.nix with desktop apps and services
# that only make sense on a graphical workstation.
home-manager.users.${vars.ipaUser} = { pkgs, ... }: {
home = {
packages = with pkgs; [
git
vim
nextcloud-client
chromium
claude-code
fish
sops
];
sessionVariables = {
EDITOR = "vim";
SOPS_AGE_KEY_FILE = "/home/${vars.ipaUser}/.config/sops/age/keys.txt";
};
file = {
".local/share/applications/proxmox-chromium-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox (Chromium)
Exec=chromium --app=https://pve.${vars.homeDomain}:${toString vars.ports.pveWeb} --window-size=1920,1080 --window-position=0,0
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=Hypervisor;
StartupWMClass=PVE
'';
".local/share/applications/pbs-chromium-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox Backup Server (Chromium)
Exec=chromium --app=https://${vars.pbsIp}:${toString vars.ports.pbsWeb} --window-size=1920,1080 --window-position=0,0
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=backup;
'';
".local/share/applications/proxmox-firefox-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox (Firefox)
Exec=firefox --new-instance https://pve.${vars.homeDomain}:${toString vars.ports.pveWeb} --profile ProxmoxWebApp --window-size=1920,1080 --class ProxmoxWebApp
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=Hypervisor;
StartupWMClass=PVE
'';
".local/share/applications/pbs-firefox-app.desktop".text = ''
[Desktop Entry]
Type=Application
Name=Proxmox Backup Server (Firefox)
Exec=firefox --new-window https://${vars.pbsIp}:${toString vars.ports.pbsWeb} --profile PbsWebApp --window-size=1920,1080 --class PbsWebApp
Icon=/home/${vars.ipaUser}/.local/share/icons/proxmox.png
Terminal=false
Categories=backup;
StartupWMClass=PBS
'';
};
};
services.nextcloud-client = {
enable = true;
startInBackground = true;
};
};
}
+5 -1
View File
@@ -14,7 +14,7 @@
# the Beszel hub) is not set yet — add it to hosts/ha-server-{1,2}/host.nix
# under services.beszel.agent.environment.KEY once the hub accepts the
# new agents, following the pattern in hosts/server/host.nix.
{ lib, vars, ... }:
{ lib, pkgs, vars, ... }:
let
# Generates /etc/exports lines for all nfsShares data entries. Shared
@@ -33,6 +33,10 @@ in
../beszel/enable-agent.nix
];
# xfsprogs: mkfs.xfs/xfs_info needed by cluster-init.sh.
# openiscsi: iscsiadm needed by acceptance-tests.sh T4 (iSCSI discovery check).
environment.systemPackages = [ pkgs.xfsprogs pkgs.openiscsi ];
services.nfs.server = {
enable = true;
exports = mkNfsExports vars.haStorageRoot;
+33 -1
View File
@@ -1,13 +1,45 @@
{ pkgs, vars, ... }:
{ lib, pkgs, vars, ... }:
let
gid = toString vars.dockerAccessGid;
in
{
virtualisation.docker = {
enable = true;
package = pkgs.docker;
};
# Pin the docker group GID to match the IPA "docker-access" group so that
# IPA group membership alone grants access to the Docker socket. Any user
# whose supplementary groups (resolved by SSSD from IPA) include GID
# vars.dockerAccessGid will pass the socket group-permission check without
# any per-host users.groups.docker.members entry.
users.groups.docker.gid = lib.mkForce vars.dockerAccessGid;
users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
environment.systemPackages = with pkgs; [
docker-compose
docker-buildx
];
# NixOS's group activation uses plain `groupmod` without --non-unique.
# When SSSD is active it exposes the IPA "docker-access" group at
# vars.dockerAccessGid via NSS, so groupmod sees that GID as already in
# use and silently skips the change (warning: "not applying GID change").
# This script runs after the normal "groups" step and applies the change
# with --non-unique (which lets the local docker group share the GID with
# the SSSD-provided IPA group). If the GID actually changed it also
# restarts docker.socket so the socket is recreated with the new GID.
system.activationScripts.docker-group-gid = {
deps = [ "groups" ];
text = ''
current=$(grep "^docker:" /etc/group | cut -d: -f3)
if [ "$current" != "${gid}" ]; then
${pkgs.shadow}/bin/groupmod --non-unique -g ${gid} docker
if ${pkgs.systemd}/bin/systemctl is-active --quiet docker.socket; then
${pkgs.systemd}/bin/systemctl stop docker.service docker.socket
rm -f /var/run/docker.sock
${pkgs.systemd}/bin/systemctl start docker.socket docker.service
fi
fi
'';
};
}
+28 -6
View File
@@ -11,14 +11,32 @@
# Both host keys must be registered via sync-host-keys.sh first so both nodes can decrypt it.
#
# DRBD fencing:
# Production setting is resource-only: DRBD waits for the STONITH fence
# agent to confirm the peer is dead before promoting to Primary. This
# requires a working fence_pve_ssh STONITH resource in Pacemaker
# (see scripts/ha/cluster-enable-stonith.sh). On a fresh cluster with
# no fence device yet, temporarily change to dont-care and run
# cluster-enable-stonith.sh once the fence key is deployed.
# resource-only with crm-fence-peer.sh: DRBD calls the Pacemaker-aware
# crm-fence-peer.sh handler before promoting. The handler checks the CIB
# to confirm the peer's DRBD resource is stopped and returns 7 (successfully
# fenced), allowing safe promotion without requiring power-fencing (STONITH).
# The unfence handler crm-unfence-peer.sh clears the outdate flag when the
# peer reconnects. This is the correct setting for Pacemaker+DRBD clusters
# with STONITH disabled; crm-fence-peer.sh replaces the need for a separate
# STONITH device during the testing phase. Switch to resource-and-stonith
# once the fence_pve_ssh STONITH resource is active (see
# scripts/ha/cluster-enable-stonith.sh).
{ lib, vars, ... }:
{
# Root SSH access — same key set as nixos user so all admin keys can reach root.
users.users.root.openssh.authorizedKeys.keys = [
vars.adminSshKey
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
"ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGygkCljN6uKpdJbHTOQtn8ZnH+wKXDLAwrDFbLrE/65 nixos@nixos"
];
# Passwordless sudo for wheel — operator SSHes as nixos and uses sudo for
# cluster management commands (drbdadm, crm*, pcs, etc.)
security.sudo.wheelNeedsPassword = lib.mkForce false;
# DRBD lock-file directory (drbd-utils checks for it; missing → harmless but noisy warnings).
systemd.tmpfiles.rules = [ "d /var/lib/drbd 0750 root root -" ];
services.drbd = {
enable = true;
config = ''
@@ -37,6 +55,10 @@
disk {
fencing resource-only;
}
handlers {
fence-peer "/run/current-system/sw/lib/drbd/crm-fence-peer.sh";
unfence-peer "/run/current-system/sw/lib/drbd/crm-unfence-peer.sh";
}
}
resource ha-data {
+1 -1
View File
@@ -25,7 +25,7 @@
let
python3 = pkgs.python3.withPackages (ps: [ ps.rtslib-fb ]);
targetctl = "${pkgs.targetcli-fb}/bin/targetctl";
targetctl = "${python3}/bin/targetctl";
targetctlStop = pkgs.writeScript "targetctl-stop" ''
#!${python3}/bin/python3
+116 -94
View File
@@ -38,47 +38,122 @@ lib.mkIf enabled {
networking.domain = lib.mkDefault vars.homeDomain;
networking.nameservers = lib.mkDefault [ vars.domainControllerIp ];
security.ipa = {
enable = true;
domain = vars.homeDomain;
inherit realm;
server = vars.ipaServer;
certificate = caCertPkg;
inherit basedn;
ipaHostname = fqdn;
offlinePasswords = true;
cacheCredentials = true;
security = {
ipa = {
enable = true;
domain = vars.homeDomain;
inherit realm;
server = vars.ipaServer;
certificate = caCertPkg;
inherit basedn;
ipaHostname = fqdn;
offlinePasswords = true;
cacheCredentials = true;
};
# Create the home directory on first login if it doesn't exist yet.
# IPA users have no pre-created home on the host; without this sshd
# opens a session to a non-existent directory and resets the connection.
# lightdm also needs this so the GUI login path can create the home dir
# if it was not pre-seeded by the tmpfiles rule above (e.g. on first boot
# before SSSD has resolved the user).
pam.services = {
sshd.makeHomeDir = true;
lightdm.makeHomeDir = true;
# pam_unix returns PAM_AUTHINFO_UNAVAIL without prompting when the local
# stub has "!" in shadow (account locked), so PAM_AUTHTOK is never set
# and pam_sss's use_first_pass fails with "No authentication token".
# Changing to try_first_pass makes pam_sss prompt independently when no
# prior module has set the token, restoring IPA password login via
# LightDM and su.
login.rules.auth.sss.settings = lib.mkForce { try_first_pass = true; };
su.rules.auth.sss.settings = lib.mkForce { try_first_pass = true; };
};
# HM with useUserPackages = true (flake.nix) sets users.users.${ipaUser}.packages,
# which forces the stub into /etc/passwd. pam_sss.so with the "localusers" flag
# (added by NixOS when SSSD is enabled) then skips SSSD for any user it finds in
# local /etc/passwd — including this stub — falling through to pam_unix, which has
# no password for the stub → sudo auth always fails.
#
# Fix: NOPASSWD for the IPA user. The IPA user already authenticated to reach a
# shell (SSH public key from IPA or Kerberos), so re-prompting via a broken PAM
# path is security theater on a single-admin homelab.
sudo.extraRules = [{
users = [ vars.ipaUser ];
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
}];
};
# Fetch SSH public keys from IPA so users can log in with the key stored
# in their IPA profile rather than needing ~/.ssh/authorized_keys on every
# host. sss_ssh_authorizedkeys queries SSSD (which queries IPA LDAP).
#
# /nix/store is 1775 (group-writable by nixbld). OpenSSH 10.0+ rejects
# AuthorizedKeysCommand binaries whose path contains any group-writable
# component, silently skipping the command. Copy to /usr/local/bin (all
# components root-owned, 755) so the path passes sshd's safety check.
systemd.tmpfiles.rules = [
"d /usr/local 0755 root root - -"
"d /usr/local/bin 0755 root root - -"
"C+ /usr/local/bin/sss_ssh_authorizedkeys 0555 root root - ${pkgs.sssd}/bin/sss_ssh_authorizedkeys"
# Pre-create the IPA user's home dir so Home Manager activation succeeds
# even before their first login. On a fresh system SSSD may not have
# resolved the user yet — tmpfiles warns and skips in that case (non-fatal),
# and pam_mkhomedir covers the first-login path as a fallback.
"d /home/${vars.ipaUser} 0700 ${vars.ipaUser} ${vars.ipaUser} - -"
];
systemd = {
# Fetch SSH public keys from IPA so users can log in with the key stored
# in their IPA profile rather than needing ~/.ssh/authorized_keys on every
# host. sss_ssh_authorizedkeys queries SSSD (which queries IPA LDAP).
#
# /nix/store is 1775 (group-writable by nixbld). OpenSSH 10.0+ rejects
# AuthorizedKeysCommand binaries whose path contains any group-writable
# component, silently skipping the command. Copy to /usr/local/bin (all
# components root-owned, 755) so the path passes sshd's safety check.
tmpfiles.rules = [
"d /usr/local 0755 root root - -"
"d /usr/local/bin 0755 root root - -"
"C+ /usr/local/bin/sss_ssh_authorizedkeys 0555 root root - ${pkgs.sssd}/bin/sss_ssh_authorizedkeys"
# Pre-create the IPA user's home dir so Home Manager activation succeeds
# even before their first login. On a fresh system SSSD may not have
# resolved the user yet — tmpfiles warns and skips in that case (non-fatal),
# and pam_mkhomedir covers the first-login path as a fallback.
"d /home/${vars.ipaUser} 0700 ${vars.ipaUser} ${vars.ipaUser} - -"
];
# security.ipa enables Kerberos (security.krb5) which causes systemd to
# start auth-rpcgss-module.service and rpc-gssd.service for Kerberos NFS
# authentication. LXC containers can't load the auth_rpcgss kernel module
# and don't have /var/lib/nfs/rpc_pipefs, so both services fail.
#
# The NixOS IPA module already adds a drop-in for auth-rpcgss-module.service
# with ConditionPathExists=/etc/krb5.keytab. We use lib.mkForce to win the
# text conflict and add ConditionVirtualization=!container alongside it so
# the service is skipped (not failed) in containers that do have a keytab.
# Same fix for rpc-gssd.service which also fails in containers.
units = lib.mkIf config.boot.isContainer {
"auth-rpcgss-module.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
# rpc-gssd also has ConditionPathExists from the NixOS IPA module (and an
# X-Restart-Triggers store path from systemd.nix). Use mkForce to win;
# omit X-Restart-Triggers since this service is skipped in containers anyway.
"rpc-gssd.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
};
# home-manager-<user>.service fails on first enrollment because /home/wayne
# doesn't exist until the user's first login (pam_mkhomedir creates it then).
# ConditionPathExists makes systemd skip the service (exit 0, condition not
# met) instead of failing. After first login the dir exists and subsequent
# rebuilds activate HM normally.
services."home-manager-${vars.ipaUser}".unitConfig.ConditionPathExists =
"/home/${vars.ipaUser}";
};
services.openssh.extraConfig = ''
AuthorizedKeysCommand /usr/local/bin/sss_ssh_authorizedkeys %u
AuthorizedKeysCommandUser nobody
'';
# Create the home directory on first login if it doesn't exist yet.
# IPA users have no pre-created home on the host; without this sshd
# opens a session to a non-existent directory and resets the connection.
security.pam.services.sshd.makeHomeDir = true;
# Host keytab: pre-provisioned on the IPA server, sops-encrypted binary.
# Placed at /etc/krb5.keytab before SSSD starts so the host authenticates
# to IPA without running ipa-client-install.
@@ -92,44 +167,6 @@ lib.mkIf enabled {
restartUnits = [ "sssd.service" ];
};
# security.ipa enables Kerberos (security.krb5) which causes systemd to
# start auth-rpcgss-module.service and rpc-gssd.service for Kerberos NFS
# authentication. LXC containers can't load the auth_rpcgss kernel module
# and don't have /var/lib/nfs/rpc_pipefs, so both services fail.
#
# The NixOS IPA module already adds a drop-in for auth-rpcgss-module.service
# with ConditionPathExists=/etc/krb5.keytab. We use lib.mkForce to win the
# text conflict and add ConditionVirtualization=!container alongside it so
# the service is skipped (not failed) in containers that do have a keytab.
# Same fix for rpc-gssd.service which also fails in containers.
systemd.units = lib.mkIf config.boot.isContainer {
"auth-rpcgss-module.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
# rpc-gssd also has ConditionPathExists from the NixOS IPA module (and an
# X-Restart-Triggers store path from systemd.nix). Use mkForce to win;
# omit X-Restart-Triggers since this service is skipped in containers anyway.
"rpc-gssd.service" = {
overrideStrategy = "asDropinIfExists";
text = lib.mkForce ''
[Unit]
ConditionPathExists=
ConditionPathExists=/etc/krb5.keytab
ConditionVirtualization=!container
'';
};
};
# Home Manager config for the IPA primary user, applied on every enrolled
# host. Manages what IPA doesn't: dotfiles, user-scoped packages, session
# variables. Switch-nix/Test-nix/buildImage are system-wide (configuration.nix)
# so they don't need to be repeated here.
# NixOS requires isNormalUser/isSystemUser + group on any entry in
# users.users. HM with useUserPackages = true (set in flake.nix) adds a stub
# entry for each HM user so it can install packages to
@@ -137,35 +174,20 @@ lib.mkIf enabled {
# With security.ipa setting "passwd: sss files" in nsswitch, SSSD's IPA entry
# takes priority for NSS lookups — this local stub is only a fallback when
# SSSD is unreachable (at which point auth fails anyway).
# HM with useUserPackages = true (flake.nix) sets users.users.${ipaUser}.packages,
# which forces the stub into /etc/passwd. pam_sss.so with the "localusers" flag
# (added by NixOS when SSSD is enabled) then skips SSSD for any user it finds in
# local /etc/passwd — including this stub — falling through to pam_unix, which has
# no password for the stub → sudo auth always fails.
#
# Fix: NOPASSWD for the IPA user. The IPA user already authenticated to reach a
# shell (SSH public key from IPA or Kerberos), so re-prompting via a broken PAM
# path is security theater on a single-admin homelab.
users.users.${vars.ipaUser} = {
isNormalUser = true;
group = "users";
extraGroups = [ "wheel" ];
createHome = false;
# "!" is not a password hash — it is the standard "account locked" marker.
# It cannot authenticate anyone locally. It exists solely so NixOS generates
# a shadow entry for this stub user; without one pam_unix returns
# PAM_AUTHINFO_UNAVAIL before prompting, which means PAM_AUTHTOK is never
# set and the subsequent pam_sss use_first_pass call has nothing to work
# with — blocking LightDM and su logins even when IPA/SSSD auth succeeds.
hashedPassword = "!";
};
# home-manager-<user>.service fails on first enrollment because /home/wayne
# doesn't exist until the user's first login (pam_mkhomedir creates it then).
# ConditionPathExists makes systemd skip the service (exit 0, condition not
# met) instead of failing. After first login the dir exists and subsequent
# rebuilds activate HM normally.
systemd.services."home-manager-${vars.ipaUser}".unitConfig.ConditionPathExists =
"/home/${vars.ipaUser}";
security.sudo.extraRules = [{
users = [ vars.ipaUser ];
commands = [{ command = "ALL"; options = [ "NOPASSWD" ]; }];
}];
# Home Manager config for the IPA primary user, applied on every enrolled
# host. Manages what IPA doesn't: dotfiles, user-scoped packages, session
# variables. Switch-nix/Test-nix/buildImage are system-wide (configuration.nix)
@@ -180,7 +202,7 @@ lib.mkIf enabled {
homeDirectory = lib.mkForce "/home/${vars.ipaUser}";
stateVersion = "26.05";
packages = with pkgs; [ tmux sshfs ];
sessionVariables.EDITOR = "nano";
sessionVariables.EDITOR = lib.mkDefault "nano";
};
programs.home-manager.enable = true;
programs.bash.enable = true;
+87 -60
View File
@@ -7,49 +7,72 @@
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
NODE1="ha-server-1"
NODE2="ha-server-2"
NODE1_IP="192.168.2.200" # vars.haServer1Ip
NODE2_IP="192.168.2.201" # vars.haServer2Ip
VIP="192.168.2.202" # vars.haServerVip
XFS_MOUNT="/srv/ha-data" # vars.haStorageRoot
ISCSI_IQN="iqn.2026-01.home.sweet:ha-storage" # vars.haIscsiIqn
# All values override-able via environment variables; defaults match variables.nix.
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.2.229}" # vars.haServerVip
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
ISCSI_IQN="${ISCSI_IQN:-iqn.2026-01.home.sweet:ha-storage}" # vars.haIscsiIqn
# ──────────────────────────────────────────────────────────────────────────
PASS=0
FAIL=0
RESULTS=()
pass() { echo " PASS: $1"; ((PASS++)); RESULTS+=("PASS $1"); }
fail() { echo " FAIL: $1"; ((FAIL++)); RESULTS+=("FAIL $1"); }
# Use PASS=$((PASS+1)) instead of ((PASS++)) — the latter evaluates to 0 when
# PASS=0, which triggers set -e and kills the script after the very first PASS.
pass() { echo " PASS: $1"; PASS=$((PASS+1)); RESULTS+=("PASS $1"); }
fail() { echo " FAIL: $1"; FAIL=$((FAIL+1)); RESULTS+=("FAIL $1"); }
n1() { ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 "root@${NODE1_IP}" "$@" 2>/dev/null; }
n2() { ssh -o StrictHostKeyChecking=no -o ConnectTimeout=5 "root@${NODE2_IP}" "$@" 2>/dev/null; }
HA_USER="nixos"
n1() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null; }
n2() { ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null; }
echo "════════════════════════════════════════════════════"
echo " HA Cluster Acceptance Tests — $(date '+%Y-%m-%d %H:%M:%S')"
echo "════════════════════════════════════════════════════"
# ── Detect Active/Standby nodes ────────────────────────────────────────────
# Pacemaker can promote either node; determine which is currently Active
# (DRBD Primary / holds ha-group resources) before running tests.
echo ""
echo "Detecting Active/Standby nodes..."
if n1 "drbdadm role ha-data" 2>/dev/null | grep -q "^Primary"; then
ACTIVE_NODE="$NODE1"; ACTIVE_IP="$NODE1_IP"
STANDBY_NODE="$NODE2"; STANDBY_IP="$NODE2_IP"
na() { n1 "$@"; }
ns() { n2 "$@"; }
else
ACTIVE_NODE="$NODE2"; ACTIVE_IP="$NODE2_IP"
STANDBY_NODE="$NODE1"; STANDBY_IP="$NODE1_IP"
na() { n2 "$@"; }
ns() { n1 "$@"; }
fi
echo " Active: $ACTIVE_NODE ($ACTIVE_IP)"
echo " Standby: $STANDBY_NODE ($STANDBY_IP)"
# ── T1: Corosync quorum established ──────────────────────────────────────
echo ""
echo "[T1] Corosync quorum"
if n1 "corosync-quorumtool -s" 2>/dev/null | grep -q "Quorate:.*Yes"; then
if na "corosync-quorumtool -s" 2>/dev/null | grep -q "Quorate:.*Yes"; then
pass "cluster has quorum"
else
fail "cluster does not have quorum — check corosync on both nodes"
fi
# ── T2: DRBD Primary on node1, Secondary on node2 ────────────────────────
# ── T2: DRBD Primary on Active node, Secondary on Standby ────────────────
echo ""
echo "[T2] DRBD roles"
DRBD_ROLE=$(n1 "drbdadm role ha-data" 2>/dev/null || echo "unknown")
DRBD_ROLE=$(na "drbdadm role ha-data" 2>/dev/null || echo "unknown")
if [[ "$DRBD_ROLE" == "Primary/Secondary" || "$DRBD_ROLE" == "Primary" ]]; then
pass "DRBD Primary on $NODE1 ($DRBD_ROLE)"
pass "DRBD Primary on $ACTIVE_NODE ($DRBD_ROLE)"
else
fail "unexpected DRBD role on $NODE1: $DRBD_ROLE (expected Primary/Secondary)"
fail "unexpected DRBD role on $ACTIVE_NODE: $DRBD_ROLE (expected Primary/Secondary)"
fi
DRBD_DSTATE=$(n1 "drbdadm dstate ha-data" 2>/dev/null || echo "unknown")
DRBD_DSTATE=$(na "drbdadm dstate ha-data" 2>/dev/null || echo "unknown")
if echo "$DRBD_DSTATE" | grep -q "UpToDate"; then
pass "DRBD disk state UpToDate ($DRBD_DSTATE)"
else
@@ -59,44 +82,45 @@ fi
# ── T3: XFS mounted at haStorageRoot on the Active node ──────────────────
echo ""
echo "[T3] XFS mount"
if n1 "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
pass "XFS mounted at ${XFS_MOUNT} on $NODE1"
if na "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
pass "XFS mounted at ${XFS_MOUNT} on $ACTIVE_NODE"
else
fail "XFS not mounted at ${XFS_MOUNT} on $NODE1"
fail "XFS not mounted at ${XFS_MOUNT} on $ACTIVE_NODE"
fi
if n2 "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
fail "XFS unexpectedly mounted on $NODE2 (should only be on Active node)"
if ns "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
fail "XFS unexpectedly mounted on $STANDBY_NODE (should only be on Active node)"
else
pass "XFS not mounted on $NODE2 (correct — Secondary)"
pass "XFS not mounted on $STANDBY_NODE (correct — Standby)"
fi
# ── T4: iSCSI target visible on both nodes ────────────────────────────────
# ── T4: iSCSI target visible on Active node ───────────────────────────────
echo ""
echo "[T4] iSCSI target"
IQN_COUNT=$(n1 "ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn" || echo "0")
IQN_COUNT=$(na "ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn" || echo "0")
if [[ "$IQN_COUNT" -ge 1 ]]; then
pass "iSCSI IQN active on $NODE1 ($IQN_COUNT target(s))"
pass "iSCSI IQN active on $ACTIVE_NODE ($IQN_COUNT target(s))"
else
fail "no iSCSI IQN active on $NODE1"
fail "no iSCSI IQN active on $ACTIVE_NODE"
fi
# iSCSI discovery from node2 via VIP
if n2 "iscsiadm -m discovery -t sendtargets -p '${VIP}' 2>/dev/null | grep -q '${ISCSI_IQN}'"; then
pass "iSCSI target discoverable from $NODE2 via VIP ${VIP}"
# iSCSI port reachable from Standby node via VIP.
# Use bash TCP probe (no iscsiadm needed — just checks port 3260 is open).
if ns "bash -c 'echo >/dev/tcp/${VIP}/3260' 2>/dev/null"; then
pass "iSCSI port 3260 reachable from $STANDBY_NODE via VIP ${VIP}"
else
fail "iSCSI target not discoverable from $NODE2 via ${VIP}"
fail "iSCSI port 3260 not reachable from $STANDBY_NODE via ${VIP}"
fi
# ── T5: Failover — standby node1, verify resources move to node2 ──────────
# ── T5: Failover — standby Active node, verify resources move to Standby ──
echo ""
echo "[T5] Failover (standby $NODE1)"
MYNODE=$(n1 "crm_node -n" 2>/dev/null || echo "")
n1 "crm_standby -N '${MYNODE}' -v on" 2>/dev/null || true
echo " Waiting up to 30 s for resources to move to $NODE2..."
echo "[T5] Failover (standby $ACTIVE_NODE)"
ACTIVE_CRMD_NAME=$(na "crm_node -n" 2>/dev/null || echo "")
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v on" 2>/dev/null || true
echo " Waiting up to 120 s for resources to move to $STANDBY_NODE..."
MOVED=false
for i in $(seq 1 30); do
if n2 "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
for i in $(seq 1 120); do
if ns "mountpoint -q '${XFS_MOUNT}'" 2>/dev/null; then
MOVED=true
echo " Resources moved in ${i}s"
break
@@ -105,49 +129,52 @@ for i in $(seq 1 30); do
done
if $MOVED; then
pass "XFS mounted on $NODE2 after failover"
IQN_ON_N2=$(n2 "ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn" || echo "0")
[[ "$IQN_ON_N2" -ge 1 ]] \
&& pass "iSCSI target active on $NODE2 after failover" \
|| fail "iSCSI target NOT active on $NODE2 after failover"
pass "XFS mounted on $STANDBY_NODE after failover"
IQN_ON_STANDBY=$(ns "ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -c iqn" || echo "0")
[[ "$IQN_ON_STANDBY" -ge 1 ]] \
&& pass "iSCSI target active on $STANDBY_NODE after failover" \
|| fail "iSCSI target NOT active on $STANDBY_NODE after failover"
else
fail "XFS did not mount on $NODE2 within 30 s — failover incomplete"
fail "XFS did not mount on $STANDBY_NODE within 120 s — failover incomplete"
fi
# ── T6: Data integrity — file written pre-failover readable post-failover
# ── T6: Data integrity — file written post-failover readable ────────────
echo ""
echo "[T6] Data integrity"
# Write a test file on node2 (now Active) and verify its content
# Write a test file on the new Active (former Standby) and verify it.
# Use `echo | sudo tee` for the write: "echo ... > file" via bash -c has the
# redirect interpreted by the remote nixos shell (not sudo), so the file open
# runs as nixos and fails with EACCES on the root-owned XFS mount. Piping
# through sudo tee lets tee (running as root) open the file instead.
TEST_FILE="${XFS_MOUNT}/.acceptance-test-$$"
TEST_CONTENT="ha-acceptance-test-$(date +%s)"
n2 "echo '${TEST_CONTENT}' > '${TEST_FILE}'" 2>/dev/null || true
READBACK=$(n2 "cat '${TEST_FILE}' 2>/dev/null" || echo "")
echo "${TEST_CONTENT}" | ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${STANDBY_IP}" sudo tee "${TEST_FILE}" > /dev/null 2>/dev/null || true
READBACK=$(ns cat "${TEST_FILE}" 2>/dev/null || echo "")
if [[ "$READBACK" == "$TEST_CONTENT" ]]; then
pass "test file written and read back correctly on $NODE2"
pass "test file written and read back correctly on $STANDBY_NODE"
else
fail "data integrity check failed (wrote: '$TEST_CONTENT', read: '$READBACK')"
fi
n2 "rm -f '${TEST_FILE}'" 2>/dev/null || true
ns rm -f "${TEST_FILE}" 2>/dev/null || true
# ── T7: Node rejoin — un-standby node1, verify cluster is healthy ────────
# ── T7: Node rejoin — un-standby original Active, verify cluster is healthy ─
echo ""
echo "[T7] Node rejoin"
n1 "crm_standby -N '${MYNODE}' -v off" 2>/dev/null || true
n1 "crm_resource --cleanup" 2>/dev/null || true
na "crm_standby -N '${ACTIVE_CRMD_NAME}' -v off" 2>/dev/null || true
na "crm_resource --cleanup" 2>/dev/null || true
sleep 5
ONLINE_NODES=$(n2 "crm_mon -1 2>/dev/null | grep -c 'Online:'" || echo "0")
if n1 "corosync-quorumtool -s 2>/dev/null | grep -q 'Quorate:.*Yes'"; then
pass "$NODE1 rejoined — cluster has quorum"
if na "corosync-quorumtool -s 2>/dev/null | grep -q 'Quorate:.*Yes'"; then
pass "$ACTIVE_NODE rejoined — cluster has quorum"
else
fail "$NODE1 did not rejoin with quorum"
fail "$ACTIVE_NODE did not rejoin with quorum"
fi
DRBD_ROLE_AFTER=$(n1 "drbdadm role ha-data" 2>/dev/null || echo "unknown")
DRBD_ROLE_AFTER=$(na "drbdadm role ha-data" 2>/dev/null || echo "unknown")
if echo "$DRBD_ROLE_AFTER" | grep -q "Secondary"; then
pass "$NODE1 is DRBD Secondary after rejoin ($DRBD_ROLE_AFTER)"
pass "$ACTIVE_NODE is DRBD Secondary after rejoin ($DRBD_ROLE_AFTER)"
else
fail "unexpected DRBD role on $NODE1 after rejoin: $DRBD_ROLE_AFTER"
fail "unexpected DRBD role on $ACTIVE_NODE after rejoin: $DRBD_ROLE_AFTER"
fi
# ── Summary ───────────────────────────────────────────────────────────────
Regular → Executable
+134 -84
View File
@@ -21,22 +21,27 @@
set -euo pipefail
# ── Configuration ─────────────────────────────────────────────────────────
# These must match variables.nix haServer* values and the Proxmox VMID
# assignments. Update before running.
NODE1="ha-server-1"
NODE2="ha-server-2"
NODE1_IP="192.168.2.200" # vars.haServer1Ip
NODE2_IP="192.168.2.201" # vars.haServer2Ip
VIP="192.168.2.202" # vars.haServerVip
XFS_MOUNT="/srv/ha-data" # vars.haStorageRoot
ISCSI_IQN="iqn.2026-01.home.sweet:ha-storage" # vars.haIscsiIqn
# All values override-able via environment variables; defaults match variables.nix.
NODE1="${NODE1:-ha-server-1}"
NODE2="${NODE2:-ha-server-2}"
NODE1_IP="${NODE1_IP:-192.168.2.228}" # vars.haServer1Ip
NODE2_IP="${NODE2_IP:-192.168.2.227}" # vars.haServer2Ip
VIP="${VIP:-192.168.2.229}" # vars.haServerVip
XFS_MOUNT="${XFS_MOUNT:-/srv/ha-data}" # vars.haStorageRoot
ISCSI_IQN="${ISCSI_IQN:-iqn.2026-01.home.sweet:ha-storage}" # vars.haIscsiIqn
ISCSI_LUN_FILE="${XFS_MOUNT}/iscsi-lun.img"
ISCSI_LUN_SIZE="10G"
DRBD_DEVICE="/dev/drbd0"
VMID_NODE1="" # FILL IN: Proxmox VMID for ha-server-1
VMID_NODE2="" # FILL IN: Proxmox VMID for ha-server-2
PVE_HOST="pve1.sweet.home"
PVE_USER="wayne"
VMID_NODE1="${VMID_NODE1:-}" # set by deploy.sh; needed for STONITH
VMID_NODE2="${VMID_NODE2:-}"
PVE_HOST="${PVE_HOST:-pve1.sweet.home}"
PVE_USER="${PVE_USER:-wayne}"
# Inter-node SSH: HA_USER is the user to SSH as on NODE2; HA_KEY is the private
# key to use. Default is root-to-root (no key arg). deploy.sh sets HA_USER=nixos
# and HA_KEY=/tmp/cluster-init-key so the script works even when root-to-root SSH
# is not available.
HA_USER="${HA_USER:-root}"
HA_KEY="${HA_KEY:-}"
# NFS dataset subdirectories to create under XFS_MOUNT.
# Must mirror vars.nfsShares subpath values in variables.nix.
@@ -59,6 +64,32 @@ warn() { echo "[cluster-init] WARNING: $*" >&2; }
[[ $(id -u) -eq 0 ]] || die "must run as root"
[[ "$(hostname)" == "$NODE1" ]] || die "must run on $NODE1"
# NixOS may not include xfsprogs in root's PATH even when it's in the store.
# If mkfs.xfs is missing, search the Nix store for it.
if ! command -v mkfs.xfs &>/dev/null; then
_xfs_bin=$(find /nix/store -maxdepth 3 -name mkfs.xfs 2>/dev/null | head -1 | xargs dirname 2>/dev/null || true)
[[ -n "$_xfs_bin" ]] && export PATH="$_xfs_bin:$PATH" \
|| die "mkfs.xfs not found — add xfsprogs to ha-server.nix environment.systemPackages and rebuild"
fi
# Inter-node SSH/SCP helpers — abstract over root-to-root vs nixos+sudo.
_SSH_OPTS="-o StrictHostKeyChecking=no -o ConnectTimeout=10"
[[ -n "$HA_KEY" ]] && _SSH_OPTS="-i $HA_KEY $_SSH_OPTS"
if [[ "$HA_USER" == "root" ]]; then
n2_ssh() { ssh $_SSH_OPTS "root@${NODE2_IP}" "$@"; }
n2_scp() { scp $_SSH_OPTS "$1" "root@${NODE2_IP}:$2"; }
else
# Non-root user with passwordless sudo; wrap each command with sudo.
n2_ssh() { ssh $_SSH_OPTS "${HA_USER}@${NODE2_IP}" sudo "$@"; }
n2_scp() {
# SCP to a tmp path, then sudo-move to the real destination as the remote user.
local src="$1" dst="$2"
local tmp="/tmp/_cluster_init_scp_$$"
scp $_SSH_OPTS "$src" "${HA_USER}@${NODE2_IP}:${tmp}"
ssh $_SSH_OPTS "${HA_USER}@${NODE2_IP}" sudo mv "${tmp}" "${dst}"
}
fi
# ── 0. Corosync authkey ───────────────────────────────────────────────────
AUTHKEY="/etc/corosync/authkey"
mkdir -p /etc/corosync
@@ -68,13 +99,13 @@ if [[ ! -f "$AUTHKEY" ]]; then
chmod 0400 "$AUTHKEY"
fi
log "Distributing authkey to $NODE2..."
ssh "root@${NODE2_IP}" "mkdir -p /etc/corosync"
scp -q "$AUTHKEY" "root@${NODE2_IP}:${AUTHKEY}"
ssh "root@${NODE2_IP}" "chmod 0400 '${AUTHKEY}'"
n2_ssh "mkdir -p /etc/corosync"
n2_scp "$AUTHKEY" "$AUTHKEY"
n2_ssh "chmod 0400 '${AUTHKEY}'"
log "Restarting corosync on both nodes..."
systemctl restart corosync
ssh "root@${NODE2_IP}" "systemctl restart corosync"
n2_ssh "systemctl restart corosync"
sleep 3
# ── 1. Corosync quorum ────────────────────────────────────────────────────
@@ -105,15 +136,12 @@ if ! drbdadm dstate ha-data 2>/dev/null | grep -q "UpToDate\|Inconsistent\|Diskl
fi
log "Initialising DRBD metadata on $NODE2..."
ssh "root@${NODE2_IP}" "
if ! drbdadm dstate ha-data 2>/dev/null | grep -q 'UpToDate\|Inconsistent\|Diskless'; then
drbdadm create-md ha-data --force
fi
"
# Use grep -E for ERE alternation inside the remote bash -c string (avoids \| quoting issues).
n2_ssh "bash -c 'drbdadm dstate ha-data 2>/dev/null | grep -qE \"UpToDate|Inconsistent|Diskless\" || drbdadm create-md ha-data --force'"
log "Bringing up DRBD on both nodes..."
drbdadm up ha-data 2>/dev/null || true
ssh "root@${NODE2_IP}" "drbdadm up ha-data 2>/dev/null" || true
n2_ssh "drbdadm up ha-data" 2>/dev/null || true
log "Forcing $NODE1 to DRBD Primary for initial sync..."
drbdadm primary ha-data --force
@@ -137,7 +165,7 @@ fi
log "Mounting ${DRBD_DEVICE} at ${XFS_MOUNT}..."
mkdir -p "${XFS_MOUNT}"
mount "${DRBD_DEVICE}" "${XFS_MOUNT}"
mountpoint -q "${XFS_MOUNT}" || mount "${DRBD_DEVICE}" "${XFS_MOUNT}"
# ── 4. NFS dataset directories ────────────────────────────────────────────
log "Creating NFS dataset directories..."
@@ -153,108 +181,130 @@ fi
# ── 6. LIO iSCSI target ───────────────────────────────────────────────────
log "Configuring LIO iSCSI target via targetcli..."
# Note: do NOT bind portal to ${VIP} here — the VIP isn't assigned yet (Pacemaker
# creates it). The default portal (all IPs, port 3260) is correct; Pacemaker's
# VIP resource will make the target reachable at the VIP address.
#
# Clear any existing LIO state first (idempotent: re-run after a partial failure).
# Use specific delete commands — clearconfig does not reliably clear kernel state.
if ls /sys/kernel/config/target/iscsi/ 2>/dev/null | grep -q "${ISCSI_IQN}"; then
log "Clearing existing LIO target ${ISCSI_IQN} before reconfiguration..."
targetcli "/iscsi delete ${ISCSI_IQN}" 2>/dev/null || true
fi
if ls /sys/kernel/config/target/core/ 2>/dev/null | grep -q "fileio"; then
log "Clearing existing LIO backstore ha-lun0 before reconfiguration..."
targetcli "/backstores/fileio delete ha-lun0" 2>/dev/null || true
fi
targetcli <<EOF
/backstores/fileio create name=ha-lun0 file_or_dev=${ISCSI_LUN_FILE} size=0 write_back=false
/iscsi create ${ISCSI_IQN}
/iscsi/${ISCSI_IQN}/tpg1/luns create /backstores/fileio/ha-lun0
/iscsi/${ISCSI_IQN}/tpg1/portals create ${VIP}
/iscsi/${ISCSI_IQN}/tpg1 set attribute authentication=0
/iscsi/${ISCSI_IQN}/tpg1 set attribute demo_mode_write_protect=0
saveconfig /etc/target/saveconfig.json
EOF
log "Tearing down LIO kernel objects — Pacemaker will restore via targetctl on the Active node..."
# LIO holds the backing file open; clear kernel state now so the XFS unmount succeeds.
# Use specific delete commands (clearconfig does not reliably clear kernel configfs state).
targetcli "/iscsi delete ${ISCSI_IQN}" 2>/dev/null || warn "LIO iscsi delete failed — umount may fail"
targetcli "/backstores/fileio delete ha-lun0" 2>/dev/null || warn "LIO backstore delete failed"
log "Distributing iSCSI saveconfig to $NODE2..."
scp -q /etc/target/saveconfig.json "root@${NODE2_IP}:/etc/target/saveconfig.json"
n2_scp /etc/target/saveconfig.json /etc/target/saveconfig.json
log "Unmounting ${XFS_MOUNT} — Pacemaker manages it..."
umount "${XFS_MOUNT}"
umount "${XFS_MOUNT}" || { sync; umount -l "${XFS_MOUNT}"; }
log "Demoting DRBD to Secondary — Pacemaker manages primary role..."
drbdadm secondary ha-data
[[ "$(drbdadm role ha-data 2>/dev/null)" == "Primary/Secondary" ]] && drbdadm secondary ha-data || true
# ── 7. Pacemaker resources ────────────────────────────────────────────────
log "Configuring Pacemaker cluster properties..."
crm_attribute -t crm_config -n stonith-enabled -v false
crm_attribute -t crm_config -n no-quorum-policy -v ignore
log "Creating DRBD promotable clone resource..."
cibadmin --replace --scope resources --xml-text "
<resources>
<clone id=\"ms-drbd0\" globally-unique=\"false\">
<meta_attributes id=\"ms-drbd0-meta\">
<nvpair id=\"ms-drbd0-promotable\" name=\"promotable\" value=\"true\"/>
<nvpair id=\"ms-drbd0-master-max\" name=\"master-max\" value=\"1\"/>
<nvpair id=\"ms-drbd0-master-node-max\" name=\"master-node-max\" value=\"1\"/>
<nvpair id=\"ms-drbd0-clone-max\" name=\"clone-max\" value=\"2\"/>
<nvpair id=\"ms-drbd0-clone-node-max\" name=\"clone-node-max\" value=\"1\"/>
<nvpair id=\"ms-drbd0-notify\" name=\"notify\" value=\"true\"/>
<nvpair id=\"ms-drbd0-interleave\" name=\"interleave\" value=\"true\"/>
log "Creating Pacemaker resources via cibadmin..."
# Use cibadmin --replace with pacemaker-4.0-compatible XML.
# Key schema rules for pacemaker-4.0:
# - globally-unique must be in <meta_attributes>, not a direct <clone> attribute
# - promoted-max / promoted-node-max (not master-max / master-node-max)
# - constraint with-rsc-role="Promoted" (not "Master")
cibadmin --replace --scope resources --xml-text '<resources>
<clone id="ms-drbd0">
<meta_attributes id="ms-drbd0-meta">
<nvpair id="ms-drbd0-globally-unique" name="globally-unique" value="false"/>
<nvpair id="ms-drbd0-promotable" name="promotable" value="true"/>
<nvpair id="ms-drbd0-promoted-max" name="promoted-max" value="1"/>
<nvpair id="ms-drbd0-promoted-node-max" name="promoted-node-max" value="1"/>
<nvpair id="ms-drbd0-clone-max" name="clone-max" value="2"/>
<nvpair id="ms-drbd0-clone-node-max" name="clone-node-max" value="1"/>
<nvpair id="ms-drbd0-notify" name="notify" value="true"/>
<nvpair id="ms-drbd0-interleave" name="interleave" value="true"/>
</meta_attributes>
<primitive id=\"drbd0\" class=\"ocf\" type=\"drbd\" provider=\"linbit\">
<instance_attributes id=\"drbd0-attrs\">
<nvpair id=\"drbd0-resource\" name=\"drbd_resource\" value=\"ha-data\"/>
<primitive id="drbd0" class="ocf" type="drbd" provider="linbit">
<instance_attributes id="drbd0-attrs">
<nvpair id="drbd0-resource" name="drbd_resource" value="ha-data"/>
</instance_attributes>
<operations>
<op id=\"drbd0-start\" name=\"start\" interval=\"0\" timeout=\"240s\"/>
<op id=\"drbd0-stop\" name=\"stop\" interval=\"0\" timeout=\"120s\"/>
<op id=\"drbd0-promote\" name=\"promote\" interval=\"0\" timeout=\"90s\"/>
<op id=\"drbd0-demote\" name=\"demote\" interval=\"0\" timeout=\"90s\"/>
<op id=\"drbd0-monitor-master\" name=\"monitor\" interval=\"20s\" timeout=\"20s\" role=\"Promoted\"/>
<op id=\"drbd0-monitor-slave\" name=\"monitor\" interval=\"30s\" timeout=\"20s\" role=\"Unpromoted\"/>
<op id="drbd0-start" name="start" interval="0" timeout="240s"/>
<op id="drbd0-stop" name="stop" interval="0" timeout="120s"/>
<op id="drbd0-promote" name="promote" interval="0" timeout="240s"/>
<op id="drbd0-demote" name="demote" interval="0" timeout="90s"/>
<op id="drbd0-monitor-promoted" name="monitor" interval="20s" timeout="20s" role="Promoted"/>
<op id="drbd0-monitor-unpromoted" name="monitor" interval="30s" timeout="20s" role="Unpromoted"/>
</operations>
</primitive>
</clone>
<group id=\"ha-group\">
<primitive id=\"xfs-data\" class=\"ocf\" type=\"Filesystem\" provider=\"heartbeat\">
<instance_attributes id=\"xfs-data-attrs\">
<nvpair id=\"xfs-data-device\" name=\"device\" value=\"${DRBD_DEVICE}\"/>
<nvpair id=\"xfs-data-directory\" name=\"directory\" value=\"${XFS_MOUNT}\"/>
<nvpair id=\"xfs-data-fstype\" name=\"fstype\" value=\"xfs\"/>
<nvpair id=\"xfs-data-options\" name=\"options\" value=\"defaults\"/>
<nvpair id=\"xfs-data-force_unmount\" name=\"force_unmount\" value=\"false\"/>
<group id="ha-group">
<primitive id="xfs-data" class="ocf" type="Filesystem" provider="heartbeat">
<instance_attributes id="xfs-data-attrs">
<nvpair id="xfs-data-device" name="device" value="/dev/drbd0"/>
<nvpair id="xfs-data-directory" name="directory" value="/srv/ha-data"/>
<nvpair id="xfs-data-fstype" name="fstype" value="xfs"/>
<nvpair id="xfs-data-options" name="options" value="defaults"/>
<nvpair id="xfs-data-force_unmount" name="force_unmount" value="true"/>
</instance_attributes>
<operations>
<op id=\"xfs-data-start\" name=\"start\" interval=\"0\" timeout=\"60s\"/>
<op id=\"xfs-data-stop\" name=\"stop\" interval=\"0\" timeout=\"60s\"/>
<op id=\"xfs-data-monitor\" name=\"monitor\" interval=\"20s\" timeout=\"40s\"/>
<op id="xfs-data-start" name="start" interval="0" timeout="60s"/>
<op id="xfs-data-stop" name="stop" interval="0" timeout="60s"/>
<op id="xfs-data-monitor" name="monitor" interval="20s" timeout="40s"/>
</operations>
</primitive>
<primitive id=\"iscsi-target\" class=\"systemd\" type=\"targetctl\">
<primitive id="iscsi-target" class="systemd" type="targetctl">
<operations>
<op id=\"iscsi-start\" name=\"start\" interval=\"0\" timeout=\"60s\"/>
<op id=\"iscsi-stop\" name=\"stop\" interval=\"0\" timeout=\"60s\"/>
<op id=\"iscsi-monitor\" name=\"monitor\" interval=\"20s\" timeout=\"40s\"/>
<op id="iscsi-start" name="start" interval="0" timeout="60s"/>
<op id="iscsi-stop" name="stop" interval="0" timeout="60s"/>
<op id="iscsi-monitor" name="monitor" interval="20s" timeout="40s"/>
</operations>
</primitive>
<primitive id=\"nfs-server\" class=\"systemd\" type=\"nfs-server\">
<primitive id="nfs-server" class="systemd" type="nfs-server">
<operations>
<op id=\"nfs-start\" name=\"start\" interval=\"0\" timeout=\"60s\"/>
<op id=\"nfs-stop\" name=\"stop\" interval=\"0\" timeout=\"60s\"/>
<op id=\"nfs-monitor\" name=\"monitor\" interval=\"30s\" timeout=\"40s\"/>
<op id="nfs-start" name="start" interval="0" timeout="60s"/>
<op id="nfs-stop" name="stop" interval="0" timeout="60s"/>
<op id="nfs-monitor" name="monitor" interval="30s" timeout="40s"/>
</operations>
</primitive>
<primitive id=\"vip\" class=\"ocf\" type=\"IPaddr2\" provider=\"heartbeat\">
<instance_attributes id=\"vip-attrs\">
<nvpair id=\"vip-ip\" name=\"ip\" value=\"${VIP}\"/>
<nvpair id=\"vip-cidr\" name=\"cidr_netmask\" value=\"24\"/>
<primitive id="vip" class="ocf" type="IPaddr2" provider="heartbeat">
<instance_attributes id="vip-attrs">
<nvpair id="vip-ip" name="ip" value="192.168.2.229"/>
<nvpair id="vip-cidr" name="cidr_netmask" value="24"/>
</instance_attributes>
<operations>
<op id=\"vip-start\" name=\"start\" interval=\"0\" timeout=\"20s\"/>
<op id=\"vip-stop\" name=\"stop\" interval=\"0\" timeout=\"20s\"/>
<op id=\"vip-monitor\" name=\"monitor\" interval=\"10s\" timeout=\"20s\"/>
<op id="vip-start" name="start" interval="0" timeout="20s"/>
<op id="vip-stop" name="stop" interval="0" timeout="20s"/>
<op id="vip-monitor" name="monitor" interval="10s" timeout="20s"/>
</operations>
</primitive>
</group>
</resources>
"
</resources>'
log "Adding ordering and colocation constraints..."
cibadmin --create --scope constraints --xml-text "
<constraints>
<rsc_order id=\"order-drbd-group\" first=\"ms-drbd0\" first-action=\"promote\" then=\"ha-group\" then-action=\"start\"/>
<rsc_colocation id=\"coloc-group-with-drbd\" rsc=\"ha-group\" with-rsc=\"ms-drbd0\" with-rsc-role=\"Master\" score=\"INFINITY\"/>
</constraints>
"
log "Adding Pacemaker ordering and colocation constraints..."
cibadmin --replace --scope constraints --xml-text '<constraints>
<rsc_order id="order-drbd-group" first="ms-drbd0" first-action="promote" then="ha-group" then-action="start" kind="Mandatory"/>
<rsc_colocation id="coloc-group-with-drbd" score="INFINITY" rsc="ha-group" with-rsc="ms-drbd0" with-rsc-role="Promoted"/>
</constraints>'
log "Waiting for resources to start..."
for i in $(seq 1 60); do
+462
View File
@@ -0,0 +1,462 @@
#!/usr/bin/env bash
# deploy.sh — Full lifecycle management for the HA file-server cluster.
#
# Handles everything from zero (no VMs, no secrets) through a running,
# tested cluster, and optionally tears it back down.
#
# Usage:
# scripts/ha/deploy.sh [options]
# scripts/ha/deploy.sh --destroy [options]
#
# Phases (all run by default; skip any with --skip-*):
# 1. ensure-bridge Create storage bridge (vmbr1) on the Proxmox node if absent.
# 2. sync-keys Generate SSH host keys and register age keys for both nodes.
# 3. create-vms Build disk images and create both VMs via create-proxmox-resource.sh.
# 4. add-hardware Attach storage NIC (vmbr1) and DRBD data disk to each VM.
# 5. boot-wait Start VMs, wait for SSH on both nodes.
# 6. cluster-init Form the cluster: DRBD, corosync, Pacemaker, NFS, VIP.
# Also encrypts the generated corosync authkey into the repo.
# 7. run-tests Run acceptance tests (T1T7).
#
# Options:
# --node <host> Proxmox host to deploy on (default: pve1.sweet.home)
# --vmid1 <n> VMID for ha-server-1 (default: 200)
# --vmid2 <n> VMID for ha-server-2 (default: 201)
# --storage <pool> Proxmox storage pool (default: local-zfs)
# --storage-bridge <br> Bridge for HA storage network (default: vmbr1)
# --drbd-disk-gb <n> DRBD data disk size in GB (default: 32)
# --memory <MB> RAM per node (default: 4096)
# --cores <n> vCPUs per node (default: 4)
# --skip-ensure-bridge Skip storage bridge creation/check
# --skip-sync-keys Skip sync-host-keys.sh (clan vars already exist)
# --skip-create-vms Skip VM creation (VMs already exist)
# --skip-add-hardware Skip net1/scsi1 attachment (already attached)
# --skip-boot-wait Skip boot/SSH wait (VMs already running)
# --skip-cluster-init Skip cluster formation (cluster already configured)
# --skip-tests Skip acceptance tests
# --force-rebuild Pass --force-rebuild to create-proxmox-resource.sh
# --destroy Stop and delete both VMs (skip all other phases)
# --dry-run Print what would run without executing
# -h|--help Show this message
#
# Prerequisites:
# - SSH access to the Proxmox node as $PROXMOX_SSH_USER (wayne).
# - sops age key in the standard location (used by sync-host-keys.sh).
# - For --skip-sync-keys: clan vars already in vars/per-machine/proxmox-ha-server-{1,2}/.
# - For full tests: secrets/common.yaml decryptable on both nodes (run
# `sops updatekeys secrets/common.yaml` after sync-keys).
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
REPO_ROOT="$(cd "$SCRIPT_DIR/../.." && pwd)"
# shellcheck source=../env.sh
source "${REPO_ROOT}/scripts/env.sh"
# ── Defaults ──────────────────────────────────────────────────────────────────
NODE="${PROXMOX_HOST:-$PVE1_HOST}"
VMID1=200
VMID2=201
STORAGE="${PROXMOX_STORAGE:-local-zfs}"
STORAGE_BRIDGE="vmbr1"
DRBD_DISK_GB=32
MEMORY_MB=4096
CORES=4
SKIP_ENSURE_BRIDGE=false
SKIP_SYNC_KEYS=false
SKIP_CREATE_VMS=false
SKIP_ADD_HARDWARE=false
SKIP_BOOT_WAIT=false
SKIP_CLUSTER_INIT=false
SKIP_TESTS=false
FORCE_REBUILD=false
DESTROY=false
DRY_RUN=false
# ── Variables from repo ───────────────────────────────────────────────────────
NODE1_HOST="ha-server-1"
NODE2_HOST="ha-server-2"
NODE1_IP="192.168.2.228"
NODE2_IP="192.168.2.227"
STORAGE_IP1="192.168.4.228"
STORAGE_IP2="192.168.4.227"
STORAGE_CIDR="192.168.4.0/29"
SSH_USER="${PROXMOX_SSH_USER:-wayne}"
# ── Argument parsing ──────────────────────────────────────────────────────────
usage() {
sed -n '/^# Usage:/,/^[^#]/{ /^#/{ s/^# \?//; p } }' "$0"
exit "${1:-0}"
}
while [[ $# -gt 0 ]]; do
case "$1" in
--node) NODE="$2"; shift 2 ;;
--vmid1) VMID1="$2"; shift 2 ;;
--vmid2) VMID2="$2"; shift 2 ;;
--storage) STORAGE="$2"; shift 2 ;;
--storage-bridge) STORAGE_BRIDGE="$2"; shift 2 ;;
--drbd-disk-gb) DRBD_DISK_GB="$2"; shift 2 ;;
--memory) MEMORY_MB="$2"; shift 2 ;;
--cores) CORES="$2"; shift 2 ;;
--skip-ensure-bridge) SKIP_ENSURE_BRIDGE=true; shift ;;
--skip-sync-keys) SKIP_SYNC_KEYS=true; shift ;;
--skip-create-vms) SKIP_CREATE_VMS=true; shift ;;
--skip-add-hardware) SKIP_ADD_HARDWARE=true; shift ;;
--skip-boot-wait) SKIP_BOOT_WAIT=true; shift ;;
--skip-cluster-init) SKIP_CLUSTER_INIT=true; shift ;;
--skip-tests) SKIP_TESTS=true; shift ;;
--force-rebuild) FORCE_REBUILD=true; shift ;;
--destroy) DESTROY=true; shift ;;
--dry-run) DRY_RUN=true; shift ;;
-h|--help) usage 0 ;;
*) echo "Unknown option: $1" >&2; usage 1 ;;
esac
done
# ── Helpers ───────────────────────────────────────────────────────────────────
log() { echo "==> $*"; }
logn() { echo " $*"; }
err() { echo "ERROR: $*" >&2; exit 1; }
run() {
if $DRY_RUN; then
echo "[dry-run] $*"
else
"$@"
fi
}
pve() {
# Run a command on the Proxmox node via SSH.
if $DRY_RUN; then
echo "[dry-run] ssh ${SSH_USER}@${NODE} sudo $*"
else
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
fi
}
pve_check() {
# Run a read-only probe on the Proxmox node — always executes even in dry-run.
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
}
HA_USER="nixos"
n1() {
# Run a command on ha-server-1 via SSH as nixos with sudo.
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE1_IP}" sudo "$@" 2>/dev/null
}
n2() {
# Run a command on ha-server-2 via SSH as nixos with sudo.
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 "${HA_USER}@${NODE2_IP}" sudo "$@" 2>/dev/null
}
wait_for_ssh() {
local ip="$1" label="$2"
if $DRY_RUN; then
logn "[dry-run] Skipping SSH wait for ${label} (${ip})"
return 0
fi
local deadline=$(( $(date +%s) + 300 ))
log "Waiting for SSH on ${label} (${ip}) — up to 5 min..."
while [[ $(date +%s) -lt $deadline ]]; do
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=3 \
-o BatchMode=yes "${HA_USER}@${ip}" true 2>/dev/null; then
logn "${label} is up."
return 0
fi
sleep 5
done
err "Timed out waiting for SSH on ${label} (${ip})"
}
# ── Destroy mode ─────────────────────────────────────────────────────────────
if $DESTROY; then
log "Destroying HA cluster VMs (${VMID1}=${NODE1_HOST}, ${VMID2}=${NODE2_HOST}) on ${NODE}"
for vmid in "$VMID1" "$VMID2"; do
STATUS=$(pve "qm status ${vmid} 2>/dev/null" 2>/dev/null || true)
if echo "$STATUS" | grep -q "running"; then
log "Stopping VMID ${vmid}..."
pve "qm stop ${vmid} --skiplock 1"
sleep 5
fi
if $DRY_RUN || pve "qm config ${vmid} >/dev/null 2>&1"; then
log "Deleting VMID ${vmid}..."
run pve "qm destroy ${vmid} --purge 1"
else
logn "VMID ${vmid} not found — already gone."
fi
done
log "Done — cluster VMs destroyed."
exit 0
fi
# ── Phase 1: Storage bridge ───────────────────────────────────────────────────
if ! $SKIP_ENSURE_BRIDGE; then
log "Phase 1: Ensuring storage bridge ${STORAGE_BRIDGE} on ${NODE}"
if pve_check "test -d /sys/class/net/${STORAGE_BRIDGE}" &>/dev/null; then
logn "${STORAGE_BRIDGE} already exists — skipping."
else
logn "Creating isolated internal bridge ${STORAGE_BRIDGE} (no upstream port, ${STORAGE_CIDR})"
BRIDGE_CONF="auto ${STORAGE_BRIDGE}
iface ${STORAGE_BRIDGE} inet manual
bridge-ports none
bridge-stp off
bridge-fd 0"
if $DRY_RUN; then
echo "[dry-run] Would write /etc/network/interfaces.d/${STORAGE_BRIDGE}.conf and ifup it"
else
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"echo '${BRIDGE_CONF}' | sudo tee /etc/network/interfaces.d/${STORAGE_BRIDGE}.conf > /dev/null && sudo ifup ${STORAGE_BRIDGE}"
logn "${STORAGE_BRIDGE} created and brought up."
fi
fi
fi
# ── Phase 2: Sync host keys ───────────────────────────────────────────────────
if ! $SKIP_SYNC_KEYS; then
log "Phase 2: Syncing SSH host keys for both HA targets"
for target in proxmox-ha-server-1 proxmox-ha-server-2; do
CLAN_DIR="${REPO_ROOT}/vars/per-machine/${target}/openssh"
if [[ -d "$CLAN_DIR" ]]; then
logn "Clan vars for ${target} already exist — skipping."
else
logn "Generating host keys for ${target}..."
run bash "${REPO_ROOT}/scripts/secrets/sync-host-keys.sh" "$target"
fi
done
fi
# ── Phase 2.5: Prepare Proxmox node for building ─────────────────────────────
if ! $SKIP_CREATE_VMS && ! $DRY_RUN; then
CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"
# Fix /nix ownership if it exists but belongs to a different UID.
# pve1's IPA-enrolled wayne (UID 50002) can't write to a store created by
# another UID — passwordless sudo corrects it once.
# Use direct SSH (no sudo) for the writability check so we test wayne's own
# access, not root's.
local_ssh() { ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "$*"; }
if local_ssh "test -d /nix" &>/dev/null && ! local_ssh "test -w /nix" &>/dev/null; then
logn "/nix exists but not writable by ${SSH_USER} — fixing ownership with sudo (one-time)..."
local_ssh "sudo chown -R ${SSH_USER} /nix"
logn "Done."
fi
unset -f local_ssh
# Ensure the remote clone is on the correct branch so create-proxmox-resource.sh
# builds from the same commits we're deploying.
REMOTE_REPO="/home/${SSH_USER}/nixos"
if pve_check "test -d ${REMOTE_REPO}/.git" &>/dev/null; then
REMOTE_BRANCH=$(ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"cd ${REMOTE_REPO} && git rev-parse --abbrev-ref HEAD 2>/dev/null")
if [[ "$REMOTE_BRANCH" != "$CURRENT_BRANCH" ]]; then
logn "Remote clone is on '${REMOTE_BRANCH}', switching to '${CURRENT_BRANCH}'..."
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
"cd ${REMOTE_REPO} && git fetch origin && git checkout '${CURRENT_BRANCH}' && git pull --ff-only"
logn "Done."
fi
fi
fi
# ── Phase 3: Create VMs ───────────────────────────────────────────────────────
if ! $SKIP_CREATE_VMS; then
log "Phase 3: Building and creating VMs on ${NODE}"
REBUILD_FLAG=""
$FORCE_REBUILD && REBUILD_FLAG="--force-rebuild"
CREATE="${REPO_ROOT}/scripts/proxmox/create-proxmox-resource.sh"
for spec in "${VMID1}:ha-server-1:proxmox-ha-server-1" "${VMID2}:ha-server-2:proxmox-ha-server-2"; do
IFS=: read -r vmid host_name flake_target <<< "$spec"
log "Creating ${flake_target} (VMID ${vmid}) on ${NODE}..."
run bash "$CREATE" \
--type vm \
--host "$host_name" \
--vmid "$vmid" \
--node "$NODE" \
--storage "$STORAGE" \
--memory "$MEMORY_MB" \
--cores "$CORES" \
${REBUILD_FLAG}
done
fi
# ── Phase 4: Add storage NIC and DRBD disk ────────────────────────────────────
if ! $SKIP_ADD_HARDWARE; then
log "Phase 4: Attaching storage NIC (${STORAGE_BRIDGE}) and DRBD disk (${DRBD_DISK_GB}G) to each VM"
for vmid in "$VMID1" "$VMID2"; do
log " VMID ${vmid}: stopping to add hardware..."
pve "qm stop ${vmid} --skiplock 1 2>/dev/null; sleep 3" || true
logn "Adding net1 (${STORAGE_BRIDGE})..."
pve "qm set ${vmid} --net1 virtio,bridge=${STORAGE_BRIDGE},firewall=0"
logn "Adding scsi1 (${STORAGE}:${DRBD_DISK_GB}G for DRBD)..."
pve "qm set ${vmid} --scsi1 ${STORAGE}:${DRBD_DISK_GB},format=raw"
logn "Starting VMID ${vmid}..."
pve "qm start ${vmid}"
done
fi
# ── Phase 5: Wait for SSH ─────────────────────────────────────────────────────
if ! $SKIP_BOOT_WAIT; then
log "Phase 5: Waiting for both nodes to come up"
wait_for_ssh "$NODE1_IP" "$NODE1_HOST"
wait_for_ssh "$NODE2_IP" "$NODE2_HOST"
logn "Both nodes are SSHable."
# Give systemd a few seconds to settle after activation
sleep 10
fi
# ── Phase 6: Cluster init ─────────────────────────────────────────────────────
if ! $SKIP_CLUSTER_INIT; then
log "Phase 6: Initialising HA cluster"
CLUSTER_INIT="${REPO_ROOT}/scripts/ha/cluster-init.sh"
[[ -x "$CLUSTER_INIT" ]] || chmod +x "$CLUSTER_INIT"
if $DRY_RUN; then
logn "[dry-run] Would generate temp key, authorise on ${NODE2_HOST}, scp cluster-init.sh to ${NODE1_HOST}, and run it as root via sudo"
else
# Generate a temp keypair so cluster-init.sh can SSH node1→node2 as ${HA_USER}.
# Root on node1 has no keys; a temp key bridging node1→node2 nixos solves this.
TEMP_KEY="${REPO_ROOT}/.tmp-cluster-init-key"
TEMP_KEY_PUB="${TEMP_KEY}.pub"
rm -f "$TEMP_KEY" "$TEMP_KEY_PUB"
ssh-keygen -t ed25519 -f "$TEMP_KEY" -N "" -C "cluster-init-temp-$(date +%s)" -q
TEMP_PUBKEY=$(cat "$TEMP_KEY_PUB")
logn "Authorising temp key on ${NODE2_HOST} for ${HA_USER}..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE2_IP}" \
"mkdir -p ~/.ssh && chmod 700 ~/.ssh && echo '${TEMP_PUBKEY}' >> ~/.ssh/authorized_keys"
logn "Placing temp key on ${NODE1_HOST} for root..."
scp -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"$TEMP_KEY" "${HA_USER}@${NODE1_IP}:/tmp/cluster-init-key"
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo mkdir -p /root/.ssh && sudo cp /tmp/cluster-init-key /root/.ssh/cluster-init-key && \
sudo chmod 600 /root/.ssh/cluster-init-key && rm -f /tmp/cluster-init-key"
# Fix targetctl.service on both nodes: the iscsi-target.nix module bakes
# pkgs.targetcli-fb for the targetctl binary, but targetctl is actually in
# rtslib-fb (a different store path). Apply a runtime dropin that corrects
# both ExecStart and ExecStop before Pacemaker ever touches the service.
# The fixed iscsi-target.nix module will make this redundant on next rebuild.
logn "Patching targetctl.service on both nodes..."
_patch_targetctl() {
local ip="$1"
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${ip}" sudo bash << 'PATCH'
set -euo pipefail
TC=$(find /nix/store -maxdepth 4 -path '*/python3*env/bin/targetctl' 2>/dev/null | head -1)
PY=$(find /nix/store -maxdepth 4 -path '*/python3*env/bin/python3' -name 'python3' 2>/dev/null | \
while IFS= read -r p; do "$p" -c "import rtslib_fb" 2>/dev/null && echo "$p" && break; done | head -1)
[[ -n "$TC" && -n "$PY" ]] || { echo "targetctl or python3+rtslib_fb not found"; exit 1; }
# Write stop script that saves LIO config then tears down kernel state
"$PY" - "$TC" "$PY" << 'PYEOF'
import sys, os, stat
tc, py = sys.argv[1], sys.argv[2]
script = f"""#!{py}
import subprocess, sys, rtslib_fb
root = rtslib_fb.RTSRoot()
targets = list(root.targets)
if targets:
r = subprocess.run(["{tc}", "save", "/etc/target/saveconfig.json"], capture_output=True)
print(f"saved {{len(targets)}} target(s); rc={{r.returncode}}")
else:
print("no active LIO targets")
for t in targets:
try:
for tpg in list(t.tpgs): tpg.enable = False
t.delete()
except Exception as e: print(f"warn: {{e}}", file=sys.stderr)
for so in list(root.storage_objects):
try: so.delete()
except Exception as e: print(f"warn: {{e}}", file=sys.stderr)
print("LIO kernel target cleared")
"""
path = "/run/ha-targetctl-stop.py"
with open(path, "w") as f: f.write(script)
os.chmod(path, 0o755)
print(f"wrote {path}")
PYEOF
mkdir -p /run/systemd/system/targetctl.service.d
printf '[Service]\nExecStart=\nExecStart=%s restore /etc/target/saveconfig.json\nExecStop=\nExecStop=/run/ha-targetctl-stop.py\n' \
"$TC" > /run/systemd/system/targetctl.service.d/fix-exec.conf
systemctl daemon-reload
echo "patched on $(hostname)"
PATCH
}
_patch_targetctl "${NODE1_IP}"
_patch_targetctl "${NODE2_IP}"
logn "Uploading cluster-init.sh to ${NODE1_HOST}..."
scp -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
"$CLUSTER_INIT" "${HA_USER}@${NODE1_IP}:/tmp/cluster-init.sh"
logn "Running cluster-init.sh on ${NODE1_HOST}..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo env NODE1=${NODE1_HOST} NODE2=${NODE2_HOST} \
NODE1_IP=${NODE1_IP} NODE2_IP=${NODE2_IP} \
VIP=192.168.2.229 XFS_MOUNT=/srv/ha-data \
ISCSI_IQN=iqn.2026-01.home.sweet:ha-storage \
VMID_NODE1=${VMID1} VMID_NODE2=${VMID2} \
HA_USER=${HA_USER} HA_KEY=/root/.ssh/cluster-init-key \
bash /tmp/cluster-init.sh"
logn "Cleaning up temp key from both nodes..."
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE2_IP}" \
"sed -i '/cluster-init-temp/d' ~/.ssh/authorized_keys" 2>/dev/null || true
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo rm -f /root/.ssh/cluster-init-key" 2>/dev/null || true
rm -f "$TEMP_KEY" "$TEMP_KEY_PUB"
# Encrypt the corosync authkey generated by cluster-init and commit it.
log " Encrypting corosync authkey into secrets/ha-corosync-authkey..."
AUTHKEY_TMP="${REPO_ROOT}/secrets/ha-corosync-authkey.tmp"
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no "${HA_USER}@${NODE1_IP}" \
"sudo cat /etc/corosync/authkey" > "$AUTHKEY_TMP"
if [[ ! -s "$AUTHKEY_TMP" ]]; then
err "corosync authkey on node1 is empty — cluster-init may have failed."
fi
mv "$AUTHKEY_TMP" "${REPO_ROOT}/secrets/ha-corosync-authkey"
(cd "${REPO_ROOT}" && nix run nixpkgs#sops -- -e --input-type binary -i secrets/ha-corosync-authkey)
logn "Authkey encrypted. Committing..."
(cd "${REPO_ROOT}" && git add secrets/ha-corosync-authkey && \
git commit -m "secrets(ha): encrypt corosync authkey generated by cluster-init")
logn "Committed."
fi
fi
# ── Phase 7: Acceptance tests ─────────────────────────────────────────────────
if ! $SKIP_TESTS; then
log "Phase 7: Running acceptance tests (T1T7)"
if $DRY_RUN; then
logn "[dry-run] Would run acceptance-tests.sh against ${NODE1_HOST}/${NODE2_HOST}"
else
NODE1="$NODE1_HOST" NODE2="$NODE2_HOST" \
NODE1_IP="$NODE1_IP" NODE2_IP="$NODE2_IP" \
VIP="192.168.2.229" \
bash "${REPO_ROOT}/scripts/ha/acceptance-tests.sh"
fi
fi
log "Deploy complete."
+133 -115
View File
@@ -5,211 +5,229 @@ sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrMGd3ZVNlNXdmOUZMUzdQ
TW1acEs0NFA2Q01rM2dkc1h0NHkzQmhiWFZZCjMyK202VWdlaGhsZW04MnVwUVdO
alA0Q2FETThsYkhSS0hKdHBaS3VaY28KLS0tIG1Gdk8yalREOUtIZTUyY2p1UHlJ
eG5iQnJsaTJBY3Y1dkw1c0VEaDQwdDQKfV04fLy32Lp2ZQ2VnvQ0h/Vsf+qdaJiv
DnLXGZ9hE5yzpKWkQIRgqYGBkF8PkH0YC4OIaVkA53wrtjqS4ZHR9Q==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJR3NKa2kxZGRHd3RWdWRR
WDkreDcxZzE1RVZ5NFFUN1pLUktzVHZSUVd3CmJkMzVsRVVraGZWVEZLWkQydVJI
NGF5Ui9ranRZU1lqS2RKZzRmWGJVYmMKLS0tIGdIRlRMOFBDMnNnT3hZdjcvUVA3
cUxXa0tjWWk3K0ZTNUx2OU9MRHpobzQKMZLQH+z8o27s1bAXyJI8HD8jHnU5JaZV
WLHwwptGYz7pYyCkWc25IkA0nR3KR4lfHT2o5eGn2BBivpI5o/QQMw==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjWFNRY2FiK3VkSm1RdHBn
cWl1ZStLcTRFZWY5VVI5N0FhODZvR1ltM0FvCnBHZUtTUm9QeHNlbVBoZEx1V3Fa
Nk9iMmJKVnhocEpERi9leE1ySUtNMFkKLS0tIDRRYkxnbU90S2RyMHdJNzRJNXBi
QjRmZFhVakVic2tYODZHcWtJRmNQTDQK7G8eSJInt11P0DiL9uzNQ/ZHHLVNIYPe
bvlhuGkEuQ/+j5sVSKOfSI2Y7CvM7TpE3APyKBcLG3ajYg6F/Ev3SA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpOVVzNDZvcE1Kb1Zua2pF
cjllRWZkTDZUb0JtVW84Z01OOHF0Q3JsZW1RCmswV0lxUW02NEZzY3lWeHNXQ1Ez
K0VILzFhemZjTTFLbEpVZVdJY21HTTAKLS0tIHVqZ2E5aU1NVk10bEJ5UXkvMVFK
dXdSWEx2OXBSZG9aM0VGNVQzTjVRUkUKYgZjGG7I0ea9I+gG4Ah1VSkiONNAHdDQ
5zG9LnKQZ5faY4tWsX7JHFWDX2Y83mXbHV6jMCutAfQG2zt/5psTMw==
-----END AGE ENCRYPTED FILE-----
recipient: age19m0m7vdfg86yqy8l5mmle5jdd0unrn3f55t232w8h5ey42cqw34sfpt32n
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1S3dOcFdwV2NoMW1oMnY1
VGdxeXJVR1lsbzNVZHlGb0NGOVo0SndiakVjCmg0QlNnZDV5RlFja2hCbVRXV1VF
S1ZtbC9KU0U1ZW9zeVoyR3hxNW1XTFUKLS0tIFUzWTJhTzM4QnpWV3h1OFU0N3BK
RnF1N2k0S0lIVitoNDJLUmZqdHRzZVkKUfNg24p8zxb3749v/A1BOKCNw75AUKpf
RUmFCw5DDWF2aNM0mZqcjjVmJ/FRKV2HXwwUGsHPKSOTnKfOUlPNKA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxS3VuaHlaVGJObVNPb3Rh
UEdBMk9meDQvWi85NVRHR3dXSlhkcFZVa1RBClZDR1MxOUdwVjhzSjFaWVFWelVX
cUgzMWNrUzl3L1R6elc1ZTRGamxDU2MKLS0tIDFwUUJFQ0hIUWNTVmFwdGt0czZi
V3VGSndCT0tQb2Nzd2w2TWxBbEFBeUEK8kQQVqISb3h0snOtqM0w/mhWEpjIxlhk
N84QzgxhD40tvutfO+57HsfOIqjKi4Yhdmbd8+iW9GzGnrbfrDuVGw==
-----END AGE ENCRYPTED FILE-----
recipient: age1rrxqea6q6pn39sw8y5te63h2py8jgjl9v0jyper86w3ggtn67upqg3ah39
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBzazl5MUpUNEVNMTdLWXBp
c1hkaVhjcTZXTUNsWS96QVVWb0RVQlZ4VlNvCnE3TFRySU5jTFk2WjBONUQyQUhl
U1lNTDFTRmZhMUFyZmpVY2xpaUVxRW8KLS0tIE9MQ1M2U2ZXQmRCVll2UGRWL1RG
K2tnU0NOKzU5dkpiekF0Vk1VM0ZZZDgKK13aFypGAqrKWPOr3UwtXI1EoXf1+UzS
rBqcwnX6WPxSKUwWoins4Aojek4QhbhY4R5ei6rRS0KEQeryGxy8bg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoYXBpZlpjT2oxMURTcW9t
NTdNWkMyT0d0SVBCSFBqVC9vNkxYUUk5Wnd3CmlyNUhIa0FmR0lwSFEwK2QzRjlO
ajNKMitzZGlDaXpRS1pNNDhhaFdPYjgKLS0tIFVtSjNXSWdlQTVwOEUzcjdEVExu
aklINzU2QUFsZmxPaUI5aFR2Z3V0bXMK/rDuNYW2r022tHjuk4KqIEOxqvjQsBIg
D2lG4ih5h3idm++gnhCSlP/9tKSaVHoq+BpunsmFObrrWzZRhKI6jw==
-----END AGE ENCRYPTED FILE-----
recipient: age1adur9g330gua4l6ndk8cqjg35qc8yxwgme6wrl2hpylcc7vxm38q05ejuy
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZUpVT1p4TGZyNC9qMm5G
UzNHV08rWURjY2lqS3FTWFdoc0FYYTFjb0NBCk1mZ2JzaXk5RmExNE9xWGZ5K0pv
UEtqMkltV0dIWll5eVBUVVRNOUNDWUUKLS0tIGpKNVJudUM1UGNvaGl1UDBOeFA1
RjUyRlZ6a0Y4SXNsL21zSURVRk9KTFEKU1L6BQ6ZlYQQtqx3uF/uM5CQ1ercmvRT
TL3r2/Y07gE7CjRn3pR9z0co8KndGzxV6YR+ubyWptwBS8KQh5stkw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSArbkhEaEhoUk1SUHNjVm5C
Q2svcmpselJ1eU5qMVJKV2JWV1J0eUpSWlV3CnhIWDF2NXFMM0VMbDQyeHdkNDJV
aForNXJ0UFd0aXFGZzZhekNTdUd2U3cKLS0tIFljclNTa2I5RHJTUU1yMm5XakVo
amlHVmZzSDZRTndFSEpjQlVOUTVhRVEKz4Tx0PceUCxw8MsREB2HRPDyC/lIskhs
/HWmseMlnsoEMu8E1OIO0TpWY3qowHrdFFr4njpxMJUTXusGvRK8Qw==
-----END AGE ENCRYPTED FILE-----
recipient: age17e89ty6p0fw24daanen57wg8uald9s025t3wwxsw269svwpmgvrshfvfvt
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsaWxxWS9xQXViY3VnUEx2
V09FbTI0WGtNbW0yclhOSGZDbG5NUTNaTkFFCitkcjJ3OE9BSnN4bjFWcE9nYVBk
ZzMyVHlJQ2wwdU5JOXdCQm9oNkhNd2MKLS0tIE9tRzFYS05vSkUwWFRkaTdtc0k0
blVoMWV0QklBVkluT0Z4NHYyS1F0blUKO+Uc0of/V77ZUZOsxTzeH8/LmmAOQt+J
x/COHxnLCnZ4eWI6q1a0Qn5Br15OJYTxUI2QTV4goTnXBNUDo9wdpQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXbm1jWW1DZDVNdnVZOE5E
bTRLTzRhckVMNjV5VkMrcllLZk9NeVRiT0NrCkh3RDRicXFTQTNqSmw4VEQ0QlNt
cGJNRlFTSjU5YWFXWmM1MjBjVGZSeVkKLS0tIHFPOERUTGVOVSs5U0VMalAwUXcx
QTYvK2RScDErVENFdUhpYUVsejBYR0UKCNBN3n4q2X1YLturxpjDv4IlnLXoPtOM
DqXMNEgGubagOBhfVFrOjpFxonP7JThSfzEJT/pmL+S7bCvgyosYXA==
-----END AGE ENCRYPTED FILE-----
recipient: age1hrx8qj02fj2ea6d4g9vqhyj9hl7fppkjqfdx2l37py3h6pdkr95s8n8rvs
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsLzFnZkdVdGQ2dHRwcnRt
Ui8rc1NoQzNKSHVzTk0vYU1vMlRxcjBKZ2hNCmhLYXVGSis4RU9HNGVxZkpvUkd5
L0xvalhDYTQ4N21OcHRheTlkaUxvTkEKLS0tIFVkRGxtLzhQT0paV2U3ZnNScVdn
alZnaVppeGI3OUVscGpONkk3YTRXd3MK61na8x5qX7+dyMHasDz2dj7yeaUlX8me
N4/SIk1JDBhv9G7mdKLbKhSF1UJrSY7TJqJqx8/dqEc0uG3vptA1ew==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXdDRCMlVyaTdPWFQ5VVhh
UWMzUTBkZVYrSDRQWUpWLytDeHVmcVh0L0g0CjVIZE1jZHA0N3RtSmZKVThrcmt5
N3VrenFGTS9LUDdZRHBiNDhORE4wa2sKLS0tIGdieDQxQ0lWZnVMZUZzSExKWEJU
azl2Q01YdHZacDRSaElpZ0h3SG9xcGsK4s30qRC2eXbuKqPUHfRUJrE8FCMdz5EQ
25UhdrspVBadt0G92hHV06Uwm/KKnG4Mi5crLKIMI+HAF+5Uxrh/xA==
-----END AGE ENCRYPTED FILE-----
recipient: age1e7l8dusgmgfzd2cxrrzwepzjxt69hzqj4epee0cs27u6yg4kxcuqm34ncx
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA4ZEtvOUhMU1FRWWpJQjF4
NWphNXp5M0dLZXhkZndhT1Y3L09maytHazJVCm1MeEtMWXg1Zjg3bFVnZEorci9J
bkNZQU9Ta1dDTFFHaGFWQVBpK3pYRDQKLS0tIEhPVGliRDR3ZTF2aEl3ZnJEYWtR
anh0SEpnVW8xdXNkZEZQSjcxU1BHMFEKVRJUA71fi1QawB2TnuTWMYhzQR18u4M2
s1V4j4TwYyyKZFoNvt8kOUayjC499c5OBUufYs6G2ciC6gK2A9E0EQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwL2xrRnJyOWxRT01YbHI0
OG8vU2xQaEJySkZkQnp6ck90dmkzQlV1RER3CmtGc3duTGRxZXhlbDlHd2hGUDdH
QmxYTjBVbkVWRVlIcEhpaml2MlVzc0kKLS0tIFRWTnZGQm94VDhJK0J6SS9oa2FB
aGpJUGFydUhvQlpFaURWSm9VS1AzM28KWvU5knwB/ViUrSxbP0zPR4iUE4LXxYi3
qraWrbv3jXZwu4Sgv0H+/k6St/Xo5RU+nIqSOCKEy/kpACwk4BhMXg==
-----END AGE ENCRYPTED FILE-----
recipient: age1jcx3yajjhghn8qh8za3yeu8nxykzlg3p4nrv03vnfvzl0mzayg2qmg940e
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBTV2IzSjBEMjUwZjB0dCtj
aTdZVlEvYkVRbTNjUC9ZRVdNZVhPQTJEd1ZZClc3NDJiR1BVYkZkdVVxMVZGc0VN
dWdSSXBFR2xxR0xrV2thRmUwSS96S0UKLS0tIHRRTXVlUi9UYnFRRlhsU21HZVY4
SDFYd0NwVEtVZXNsWUI1a1ZZU2xNRGMKuQUhOq2FRD+PGn5OkdODZItbxCzRKjne
E60UOYtHjanuGjJ1svuR9cYsLZz7lLOwItklecYaQYpMRZEwzzBGCQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB3K2tIY25TZCtReTQ2Zkc5
d2w2aGZqYXNBSmZ2VVc5WWtZNDU4YlJpNm5VCkpGUkNOdGVSeXU1WG1YV2NQYjhu
c2NKd0czSm5FUWxmNTdHVGVVWGlUMWcKLS0tIDJmRy9vbGJDNGxMVDM3b25TRVVI
blpZTkZsTzNrdmFwUGQzR2RIOVJINlkKwa06bE6TmYtXPx0daHadbwL9/u1iYuGm
n62kEwTKdyfncODl895qqjWkiA3JnbwQxzsaVDtEGfauzUph9DPY2g==
-----END AGE ENCRYPTED FILE-----
recipient: age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrSjJoUVUyd1JqRm1ZQzZx
SFJ2cWNUY0E4b3FndVA5Y0hIOEZnZUlVWlhBCm1vY1luOXZBelRUTmF5Y1NMeDBn
cE1BTDErc041UjJCWTBQbnk0Wk80dkEKLS0tIDVSZzd1UktvZGdyanFUMkVORUtl
eVB5TnJkMlp6dUpXSTlxRlplZ2NxUlEK0AYOxIbswjM0SUASDfmZ7PqcEU844fgI
ycFWVSEPodwUZ6UFoYXhHlJzHFcgpLvwUd1PMktLHe1qrZ7GOQJIMA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1ZlVxYkZ4UFJmRVBHeExi
OWZ0akNwc0dMenlHQ2dUSDlERGxSNzZtdFJJCnRYK2cwNHJRK0xtY0NnYlh1a1pG
aTk0WURSYVdUY1ZTSUhBNVpZZUFyaUUKLS0tIHZKTFZmS2ZsNlZ6UjZUWmhNK0Q4
RC9TREh3ZWZRTzc0WmFkQm5qVlZJcFkKUngIUXCVV1CKpUKfAlal1KjoJwk83mq5
1MvU9mjx8Oq/suUlW4axFfAO99mUc9yCwb8TCMUZDhQp802v5FjBcw==
-----END AGE ENCRYPTED FILE-----
recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBPakhQcE92ZU0zYmk2QS9D
TG0xc1JPZXZCZ0tZOXA3MGNLVnBlZGVtRFRzCmIvMHhQKzFVWCtpMTQrQUhGVGJp
RU1jbldYckw3TXI2SlNpZVBIZHRsWWcKLS0tIGtJTUtJejFxem5jajFQUDFTQWU1
VnlxYmVlNG04ay9ETi9FRmVYQXVoRkUK9oFNolI7jRjo9RUs1g4ghrx7aYV4U/ce
ZTc2tFh57+7aKgrDi+2W3jwhfkjvBsThk//p5mLlqEEgw2lwlnhvPA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRM0hrc0I3UCtBOXJWUm1H
MW5QWHAwV3lETFp2TGtPWVZHZ2s0ZHJmQ2w4CnBlQmlTVmEvRTlLMjVXMkRJeWIw
QUR3aXpPUndocCt4aWUybndxK3RUN00KLS0tIDJrNkY0d1Jqby9pa0tXWVVuTER5
YTlVNklTakFlaGQxdjJqRks5cUZoK3MKkV199oB7lPBzNYd30nSY0J6Cd09ViBA8
uZWlmMGKlveBQ98qOt/vDTouRDxedASz0ZGg7+jxUXkUKbujkCYYXw==
-----END AGE ENCRYPTED FILE-----
recipient: age17jqc66x9yeshfgd9v78mj483r4zzarqdtuxtrkxe4x5mw679gphshd94th
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoVnh6dFAwTkY4cHJpaEs4
QnZPeXZHK0tXYWZPNytmYXVsdGRWQVI3RlhRClFVb2I5OVZzZFNrRXFaa0JTUkRJ
OC9GQ1V5K0JhWlhkUjU1WStCa1lPV1kKLS0tIEhzdnBBZkRnK0NtV1FuTkVsNlgv
QzVEcEVkQm5NL0Z5dUU1U0ZFaTJITnMKaWE9vlrOpQstr6FGP5ObdilsCYk4kYAj
/phboR+Ym7QDTyUF9LZXJCU54YJp6vEWkRnlJFqC75UW/v/lgBhBMQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBwVmlDOHh6dUFaR1YxOHJM
N09LdWxscStGSTYvMm1QaWRkZVVod1JTTlJnCkp0Z1pSa3lVU2JqUnU0ZnAzWWEv
bHc3S2ZjK3E5Z1BRL2pYWklvSFoxSnMKLS0tIGxTd29MK0dRWFFnaGVwa2NVQXM2
TjRQdHRZNDFTTWNWR2FvZXhQMVZORncKdAcSOB50bNQsaTGtAqnvjYyFpcjdMGzO
NvwI5ZquoT/B37Xsg4NQ7++/ZkemzkuX5VYoXLVAQ9WJwQxewoMFKg==
-----END AGE ENCRYPTED FILE-----
recipient: age1px0h5l9zp2dww0m8fncrc82kfdmzplsfv2ltat7sna28xpg09pqqcl3s2k
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkUHAzWk1KblJxVVZZV3FM
WndzRGRtbkw2azRVWnBuZmhKWElqRFk0RENzCk5DMHlMVWpwbXEwVUhkaFZUbkp2
QXNlZFV4SjBEdmR6UEw0N1JOUnhNKzAKLS0tIHo5RkNDUk1ESWRHQmV6bzkvSTlP
dk1GQ0Y3V0dTRlByb2xUOERVOTVwbVEKY4sAHyAhvGSYJzPuufWUIQD2xZcSt/nX
t2ZFXu891/QdEzyUXCIzdwAV+Y/LjvroIlCp5Hkbrk0s7N+ghqsB1A==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3V0M5bnBzamdGemU4Y0g0
TEdZdzQvUVpYSkozZlRsMWM4MStmN2lxckYwCms0ZTF3cTB5OTRzWWE0a1FXWkhG
SGptVCtiOHptV3V0aVNQUUJ4NkE0dnMKLS0tIEEvK0lwa1pnc1YwWXhkYmh3cy94
S2E0SUk4RDdSWktQTjZoTkNUallLTVEKNqTmhQo74Q00ZhrSrD+/JB3TwxL9OQdC
XSONu6XXzPzfhllNIwe3BnmRVJM/1ru8rvllk98Cvle0k4K5+xBJ/g==
-----END AGE ENCRYPTED FILE-----
recipient: age1ufg390ydrmma849t9xfkxxl5xvdkk6mngnlzhmy7mvuaje8sgcmsmnq6l7
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBBYVkwYXpYSjdmM1FpbDl3
QWpycXoyL1AyOEpZUmtpbjl3MFAwTkJoOWpVCmVDd0FBUWxaQmZCU2VmNkZGMk9o
TUdLNGtac2N4REg2eVF1eVh0WnNaTE0KLS0tIDJvcFErSjRiWmhPMmpadjROOHdt
NXp6Y1JpdHFlSlRoa3JTaEt3emdnalUKjoFfZAiKMPF3noX+K0+vc3+p/XUHnhic
k888KdUwcZYl2/dAIc8UDSggbMnncJAJgoezoCHLkj97GNNAD7E+gQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBKYnFtM29UUzVRYWpTN0w2
MDVmQk83THEwR3liN0hmemxBZlF3eDVZV0RRCjRFcjZKQk8yaXViWm0vTUdrSlM3
UDNnemoyMG0wbnpzK0FRYXNiRTRnaW8KLS0tIHd1QVYvWWl2QlcxVGZzbFBLbjlu
djV4dVpnQzF6LzFiWTREaTYrMHAreUkK82l9Njt3B7HJBLtdLjNSnQ4TDTvBgG9/
iUEn3fd9OmvfSGcbp4hhwB7q0UJqb7AP47ctwDOEg/FYeF7eSBvVgg==
-----END AGE ENCRYPTED FILE-----
recipient: age16j42pdc5dr6wnj7xayhkqdj2rny9u68fcqejs50hqq42scssh4gsnrrnlt
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqYnQyYzF5cktZNG5PcThy
SEszVEwrUkQ1VVRsM3pSTlRQaHVLN0VuSHhrCmN1Z3pwNlFsbDN2UGI0KzYyallM
SHJ5eklQeEIxSlhiYW5PUlpJcG5KNjQKLS0tIEk0QkpMdlBlRjVYMmJaMzJUbDNm
K25pZldwd3JoZi9vdURoa3Myb2RQNG8K6N6bO2YKooPfpKihgsYqilfz/yAYCLZD
XJ/THgT4URX2VNvSspvBtN8luOiJUVcchp5WtL2m9jARL5txEcDorA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBoZnVZekVieVVRT2ViWUpN
bFk3ckJUU1RHRUJrbkNkTUZkbVEzRGd1VWtRCk03MFEweENBOWtOS3gvM3ZpNTF2
WG1UWTM3OFp3RERNNC9iV05aOTZnWmMKLS0tIFZqZ0NhZVhxODMyWENhVlI3SUR0
b2doaGp1VHBOUVFacG51Rk02d2o1MWMKnNYktA42DgSWWGuy0XTOtUN9ry49WOBd
NaGq27ke16XhQ0ZC+r2a1g7YAFaL+qWJxpaH+0ojhUjti3x9O3pWqw==
-----END AGE ENCRYPTED FILE-----
recipient: age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBvY0VpeW1nazVvRWJFaGNU
RGVzVjRmWWV4dnJzRHdsaW9ENVZYck5lWGtVClNtd3ppelowRjZpRFFSMC9EK09n
b2hqNnkzejdrTnhYNGNKblpteDNLRWcKLS0tIE02bWVjazRWNEVKbURITGlQODlR
cTJJVnBVdGIrdzBoSXExelNrVk1XcEUKc77o2EX7PCm/HjUo5GsUiQdm488WB2mg
wHd/qDbQhF1W75RrVTuIKgtEtrRjZqpmr8toe+aHJizPofcrToUfzw==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRTkQ0cm9MSTMzZjF5T0Vh
TWlOdVJzZ1JvQWdLTzZsYWZ6djJhTjk2dnpFCmxUZkhrOW5OTWtrS2FLOUdsWm9F
WVB2VzhTdGwxNzZIZ2lSRmRielAzZkUKLS0tIEd1RlIwQXhocmJkVWtvVGMvTlVB
WWtyVjUrWjYrWDFmc3VtdzJUaEFPeGMKvWONIbN6B9Ims3f0l/lfUTU116TFPXew
K6kAvyySbh9Z9JK8Uyp31WLSbqVy56eGr2Su8Hj6rAGIoJdo0v5NoQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBSMmZnd1pXRWh5NDVSN2xq
MVhTN3N1OFpkcUh0a0tjYzJnWUlINGNVd3k0CmZkTnBac0l1dllxazdLY2l0Rzli
bE9sNTBVSkJNaWF1T3c0WktoOHl0NU0KLS0tIHEyeWZTUjdQeUN6U2t0d3JwNTBX
ZE1Za0tXb0gwc1FSakVYdU9OTHkyd28KkwmlzSYP8XofB0VGag+S18+S2TyQjLrM
qaXtbBtLzJGNDhe9FhAKTPFcjTLWbohlG69vxcImyCyCns+QQ+gvug==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlSWtxbHJsQllITit4NVZv
b2Z3YzJZTHFOTGU3VzkwWEFZUjdWTC96UjNVCi83SkYyaFdoTVN3eFhLQnA0bDVa
OExZRFl1M28yV0VTUC9hZUJJR3dYTWMKLS0tIGNuc00rZDI2dHFqWEc2R1ZVb1E5
Vm45YUpvWlMvMXUxdDB0NHpncWpIMjgKcMIZK2ww/VPuuBXlL8klSTm8ySME2njC
vbhCzg9VIqIV3q2i8WEbWFmo7uyiyF89Z52xf/3uUc/TUEqjVVDnZQ==
-----END AGE ENCRYPTED FILE-----
recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB2Q2RMWmZVOEwrOC9VcUxp
VVE4R2NqYlZWZlhKSTBHRks4bDNoaUliMWlJCm5FbWdZS05GY0VLc0sxY2x1U21V
eExwK29GVVBqYlRPZ0l5RWVXRFhRNlEKLS0tIDl6dVZJQndwVStFVEJnRHRyMW1W
NnFqc1F0SGJqT0xmREpaN21EdnlJK3MKRPE5rfFpVnH5wAOkuB5pNMlMd3omcpku
do2hFZwyI7t80jxF4+g3J7EolOx8AGjpc9Ba7Gj6IMDjye728q5N+g==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZRkRvN2dBZ1ErZmNhSktm
LzNLdzJLR3RhR2VqS0xCdlVSVUJtRnpLTkVNCjFmZlp3U05Vb2FwbDBBdmhjVUJt
Uk01dDVzS1lWeEZueDlmcUw4MlFFTHMKLS0tIGJ1bENFSFZPa0tNNFZsblFOM05x
THAyY3draFB5eHpydk94Um1qU1BZb0UKUVszMlSqUPsEj3vVseGksI+SEEjEblNo
zK5fwkYH0aDubsotkVcJjoXc2ZArHyygIMImLNpRBJYIScDGzzTBRA==
-----END AGE ENCRYPTED FILE-----
recipient: age1arhf2q45zw6wf2uevju4savp575x3m2tfvved5zzq3ay92ynua9s3cm92c
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhSG94VE96TVlTNkEyclFE
SmdCSkJpeWlVWDFIMDhwUEkxL1RUTDJ5UG5FCjhKRDB6VGtwTVozdUVzbUxGL3BW
SnR3cmpSN2RxNnl4QmNvT2lkYmtoVFkKLS0tIHd2V2h2Wk5xOXlISzhjVzBsVkhz
VVpRenVnSVpHUWJqV0JHNXNWWXJOdW8Kv7PJSTDbwFOAcl7pynALaJiTXU/87bSF
F3HQllYOwOoibGzBCe18H2N+VxyNxoQL9OWe0TvOIR6bgHFIIF0/Dg==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBic1JIWi9xKzVkVUtqdHBx
OVU1djJDbDcxYTFCSEQ0MTVRaS8zYm5QWFhRCjVNNFE0RUtPTlE5U3pRTlh2eVNy
OUU0cnpLNlluemdkVVNjWGxhV1NTRUEKLS0tIDNkWmhoSTNkWExWek4zdjduaFBB
UEJjYnZwYVN3L3p5Vk9sRGVQM3dld0kKsFjQKYOClBjWBaU3kCEP5yYGBphUfgOO
E1epPU6UI3asa2AC/svfFBZI3u/7EpxDH9jaSSpPec4QTwkheiVVrw==
-----END AGE ENCRYPTED FILE-----
recipient: age19mn8zrxl8zpps9yvrh4euquvygpp4fp8queg7xc6qhtnl4ng8c9qx02qwn
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsMEZPUExRVHFFb3pSVHNO
cGJFN1ZzTDFVNEdneVpMZ253ekFJNjVtYkNnCnRGQjU2Q3dsRGRFV25LQ3pCbTJE
OTROaFBiT01xb200S1pUK0NYaTQ3R2sKLS0tIEdiQlZTbi9Vcm0zc2t6bHplZktF
ekRySENXcjBuR2psdHZSSUJrR0xUdjgKvBsmnC+cbq5TUDFjXCyImIoPKvh8wsjE
7Shk7Act8Jayrhx0lXBDRmfpHRrB4L16rDSmqO0DTE48VhT3TiFyug==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBEb0VlcGQrWnZHWlM4YjhG
NlN6c2I3UnpSdkhYTXg4NlM5aG82K3YzUGwwCkdsM0JLUm03RTk0cm5SSitzR1ls
bVBLZ3kzaElDaTRHR1Q3bjdXYm9BMk0KLS0tIEpHNE9MTDJuMDEzV002YTVpT2J5
Rm9nem5FSnA3M1NzNi91NnBtRi9XMmMKaGB7uE6HCE2cYHfI1VscO8meeq6O5JPi
7bTansMUKCPKhDTBYlSnxxjlDE3DRB8ZdxW7LiG+8iVI7HUbsjTejQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAxNDlya3RmYzNhU2p6RkVw
cHJVWWY0Yi93cW1uZi9FWkVONmtpRWh5SzFzCmtWYTRIY3BkTWU5R2Jsa0ZJK3kz
SnZvZ3YwaGtoMVZ3V2laTlBBK3UyTmMKLS0tIFdpZWtqeGlacjlLbmFySHlSUUlj
RmRqQWVHK0FUT3VDbFhLbXQ5WDhLeEEKcDkgV34lUFJRIHRoLB8F2IOvGAM93sM+
AkmaM4+WRcGeYWQKMG2x6cYCUKFaT1lDXuWZ9kI8Fd7b9gTSnQMs6w==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlSnYrVVFObDRmUE5UN0xY
K0ZIMzNtUWM5RE8xMmlhTm5pcHNEemVwYVZNCnVLWGRObkxPcWdaOGVMcExqRlRw
VngrNGJFWXBHZGh3YXVBQmtSZXBsTzgKLS0tIFovVG43V0FBbmRJYW1qRGJCRUZp
RENmSmFuMmhFV0xHUHB2N2xWbHVvSWcKJXaXCfz3Zh4SRgTJkMuPabOq3laRorIo
F4R4bJDkao5L5QWDH1BWSi97kTTLz3QgmsEGPHm/SpODM7Llx7EPMQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1ug787sgt6st6k82fgkrug2lzltw4qsukrrqqs3w27ewwqj8rg4hsxcmylz
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjeFM4ajhHeVd4bzlqZmJ1
MUxYMUpZVjdjR0NGTzVteVB6WFBQeFI0ZkZrCk55TEt0Zjdwbk51RnhYclNzam1H
cExKZXQxWFVLa1pDNFpkcGZzcnl6a0kKLS0tIFJ4ZEdJc3JVaEc5RU1aZk1uYm1l
d3RHS3hHSkRKRXFnN21FQmh0TlNtNmcKdc2G/1dhTJen6iT9kUWZM5OzCmDVprgx
WN1Bl3JzYhLsNKn794887bVAICVqbXqkdpEZztNIS5n/Rw6geKsNvQ==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAyR3VNVisxbnoyQXZGaTJV
Y0lMemtYK1F1QjAyWFVqSm9VRzhhbHRCRFRJCnZINVpiS0ZCRXUrVEJhcVlSZk9H
TnhaeDU4b0loQjdOazJVbkpEbDZ4K3cKLS0tIGFKRjd0Rkh3RTk2WG9uQUVpWU1K
c3IxVmU0Tnc5dnBBWUZhYUgxV0NaaE0K+00bh1AiHdTL3gsA98fvFI2/IDsnWiiK
3tOBK4UB5OE5PRupqVXlpEK3ZqKXju/MQbu0/KdaVwHxlL+WXdjAGA==
-----END AGE ENCRYPTED FILE-----
recipient: age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYa0gxNnBwNjNNTTgxKzgv
YkQwTjZxb25tQ053Ny9tSW4wNGVYYnlIaFZNCkVvOFNPOFkwQzhYZGxjb0FOZzJ5
ZjNXam1ZTWZrS1M4cGhhcHZaT1NjblkKLS0tIEdvL1dDTHpWcWF0S3ZqNkxrQW52
VlYwa29sZVloOS9qajJWQWFzY2FKRmsKy074SLdttogXsWycaFX8xso4ek7Cbjph
MMEhZd/svmnSiYM81nmeaze7qXEUcsZXuSmZCYATTBEGtx/Srll8aA==
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBjSHFHbmU0WGg2a1poMzRF
Ry85ZWZCYVdVWkVoNnRWWTZXWWNqQURtendJCndHcmtIaFNPRGMrblJndElMKy9Q
NHlaeDVOS2k1OSt1bWs0RlU3a0dJTFEKLS0tIFRrMGJWYThKbVQ5SzdjLzlQbk9a
b0xlZGFvTVJGRjZHQk5XM3VObFRnaUkKMavcISNlQh+5yHpA1M5JIkQEF2qasnXH
Pd/JWKhnvk3Lyd45ZBJEqFV2wOknJF7v4Z0jdbo4WiUaLh5shqOp2A==
-----END AGE ENCRYPTED FILE-----
recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3ckNVU2lUQS94RkRGeTFo
aWNJRTdpcWxUOWZEOHF5ODIrRkZ0MkpnRndBClREbDNNUW9RZWFSRHdON3pOT3VZ
bUlFZnFGTnJpSzhFVjFLMVhKZTNDWUkKLS0tIG44Ui9YL2RPYzJLck1UVVBiRTRJ
MWxURWJlVEFBQ3RSZzZFYWJvaW1FVW8KX8/o/4LP+Wp/qOfF4wt7cTt0O+kAZbWR
Xu+dthboRmZBV21zzrEvzKGRFBj2T3EMyGDqqPcfX/vuhbVEJJgZyg==
-----END AGE ENCRYPTED FILE-----
recipient: age1nxlnrevqs2msdatze562vz6ym4pgydt2zndye83lwqauy6flggtqrevyw5
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBYck0zNDJaMmhLZUxxVUxR
OGJqdTBHMFpzb2tQV3NsUVhPc1JTeGxCWWkwCmtWeWhqRXJFNVRoVVBTcjJtWlM2
UmQ1OXh3SnJVQWQ2NnloVjRCdUZnaDQKLS0tIGZITFdwcjJFMU5iRHhvRllKaTA0
dW1xN2ZhUmZ3VlNzWFprYWNOVG90aVUKH8Gd6gkbDIZydnb1mL0tpujmucLhbPbm
sUm8KZpNUfWpP+t7SK5M91HFaAZiqGJOARb31oszwqLGcNDliV40MQ==
-----END AGE ENCRYPTED FILE-----
recipient: age1scfc8p53q5aq2a87tcmsazmj8sfeft0s8kxg0et4nm3ucxyp3c3s6te82y
lastmodified: "2026-07-23T21:15:41Z"
mac: ENC[AES256_GCM,data:qFhnPra6IE3wyKQ4WKweON0S0YtD5I0adGZVfA0m6BVilN6bX5oC/1j5NK2oHrsz920hSl0SOF8LrpqOrUyGjSRkPsN4kq8qr9bJcrX4URiktP0oRden5LLt6hf+ZRP7WmRXFqixPkPHJnZIoAvkNnTFce7cDq5NEAHkKUEKG7k=,iv:nyblUDGeu3TUfFivYylOn3C/HITj99qiPI2+mh8AGh4=,tag:FrtRzSCylC4wlIoqZdfx7w==,type:str]
unencrypted_suffix: _unencrypted
+22 -1
View File
@@ -1 +1,22 @@
STUB: run cluster-init.sh to generate, then: sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey
{
"data": "ENC[AES256_GCM,data:olqZm/3yxtVnlDeYzAWnXjUP9pgkruFEpeS6qONHSfdrj0lJOch44zn1pbG6Jg5QYcksny0rHki9PJEZSbRIyrZ2mSKfTJing3n9M5ODm9xPMbNYn8cpVB4AzZmCSAH7mN4p/98+5hJ1qzopGwCgKTcOOJ9DMUl9AStXORY=,iv:R2K3OehIfyKZXRFH3Y9KYnvEp6CjHPT4Ki2P1YrC6+I=,tag:1FM/E0n9teARAZ1HZM4b3A==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA3RFRoRW1rRklxbXYybUtu\nL1lEOXJBMDNCN1paU1k5N0hQRTJYQ1dQcTBVCk9OYnloOFZad2toOHo0UXFYSzNR\nZDUwV00vKzVnckMwSE9MMnE4RUozZTgKLS0tIFByL3Z3SzJFeEVNZjhuRDNTMzFo\nNU5tZ2tQU1ZORE5qNm1JUEJrTGdzUHMKypYeJz6BeUUY4aPKazB1nxncOA3DGkal\nfemLr9uwCJw+D3xfXzrIKrI0w3OH7bu2LmWqrNDSz3Bwa5VDuHoLqQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqTzhtOHM1Q1lWRXdadHB6\nZ1VrMXFsQWZEbDBMMnQvYnZSN090VnpyQnd3CkdJaHBsbzQ4SERmU0dBS2RZMUpi\nZVYrZmtSNXhiaDNTZkNMVEZud3ZzT3cKLS0tIFhETER5eWJRK05ValFhenRnSGg1\nc1prdWVTVmMyWWt2UkNXWVRLcWRHcm8KjVn1faGmsWiFzcNg4PnxZQfeQONFKz/i\nyJGJc7w2KSZ3La+44jfitMziJQ3AFRUlAhGDX25kWZJ7PtjmFvYwuQ==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1nxlnrevqs2msdatze562vz6ym4pgydt2zndye83lwqauy6flggtqrevyw5"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBkVXFQazlvUEx2NzVUNlF1\nN3FQNTdFL1FtSDhCdmNQclRMdG1vRVhvcEhzClF4Ykh1ZS9JekFzOHkvOVphejVy\nd2s0a0IrZ1d4V3diOGNvMlBHUVlwTmMKLS0tIEFhTE13Q2F1di9ESXdJVDhJUVl4\na1kzNDBLcHhlRGg2Y3hSSmdEc2k5MlEKpU2sIv5Pq0LaKkvqA/fRkiB9PcvsSjvq\no4iEFzGNDi63QzXqftWnzengEy/6nWGNA31fzScQftESesGlKfYYlA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1scfc8p53q5aq2a87tcmsazmj8sfeft0s8kxg0et4nm3ucxyp3c3s6te82y"
}
],
"lastmodified": "2026-07-28T06:46:00Z",
"mac": "ENC[AES256_GCM,data:xJHyh4JuY+qnKljS3mtfN4Cy0pGPUszhU4ud2L8JxTFGUHTXhZzwQ3s1vM88efLTwM7yDK7oBMBUH2xligAzglDnqkE1T0l7R9k+hyldm4nYqOK6Y8GyPt/aald6oLYGCx3J7Ap0Oqq/crU+s4NiY9enf+R+08jliFyrPBHuRSY=,iv:FGtURIfSrf1tRGqpPfn40GqybhNYFBcvR7n1YkcAMGY=,tag:ir0TU7YuxpPuew+nyC9/eg==,type:str]",
"version": "3.13.3"
}
}
+31
View File
@@ -0,0 +1,31 @@
{
<<<<<<< Updated upstream
"data": "ENC[AES256_GCM,data:4Plrw4SFb7Qx7ofyHwXkHHiCaY+54QT+XLnqoMullnxoTEwf4aZGmjZKx+J5eNnXO/+aPcheg50pe6vXggIgA02a736l3xkIU8mmRdqDSXwROwy7DWBfpcsT5bcL5umGpJ3tnJUAoA8TWVvxoj14J2k=,iv:IwoLgVeWJ+1E1yjeVLjMGEcEDNK0MD1caaP0hcbYJhs=,tag:KPT4mFN2MitDcWAggBV7Mg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBIaDJyeGJKeDhVNU9xYzhk\nb2Z5K0VTbnM2OExVWWZPaVBoYkNrOVVwbjJBCnBmQlRsdjdPZHZydU14UDl2Nm5G\nUkJWd1N0SXljTW9Kb2VMVXpETGZlS1kKLS0tIEZaWkdrWkVqMENQUmpQOERSY0hQ\ncHU5anBjeUQ0U2ZFYmFaa3ozSmpGWXcKU8QpWAk7Zkc3N/ZmWn5xLsOSTAtMxU5W\nIQOvdDNzIn7Ecoj4zQ91T9puAhaO6PZ1wWjo+rAN0TyWjM+TAwTuFw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBMbGg2WW1nOUw0RzhaNXNa\nbk1iWmRFWng5SlgwY1V1ZXZIcEp1SWlKYnlBCmJ1Q3Z1KzJoY2ZZWi9UckFOTWZC\nUGxZeUlJZDdRci9ZbzhsdTM4cXk0SjgKLS0tIEhWUy91bTZkTyt2ZFRIb2VKL1Rn\nMm0wN2RjUUJ3dHAxQU1nd0VRTGlQbUEKsb6AFoMZw+FxZW1A1rf4YUlNaODUgbS3\npZfGZIocS2y9uMTaXWQ56Y078zDD/F40wLYsuknm0ECVfNysSVacVw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1nxlnrevqs2msdatze562vz6ym4pgydt2zndye83lwqauy6flggtqrevyw5"
}
],
"lastmodified": "2026-07-28T06:44:56Z",
"mac": "ENC[AES256_GCM,data:N95kOsHFH2nKVhr79EB/zILMpJ0rgh+QZ1weYbQJgvxRYjLMl+AK/ALjtx7Mrj9ufOnIWnndrUpD7X2cod2oPrlAPT3c8P0IY8LaJTNDx6TUljKHFe9Nvm/JcnI9i5skioY0X8sRC1oosL0loVTmVb9yaf5xsf+wursIsj7pI3A=,iv:kQLx4FLeiKZIevOOOPS3NAu2NUUhnRw/jpws3FZ6zcM=,tag:rCp0h+wujci4N5VFuzCuLA==,type:str]",
=======
"data": "ENC[AES256_GCM,data:LbqG6wmya62HfWjTiQad1+k0XwR/iwgoxvvBaCfbY8NsjXYdh8rsl20r9OUFkp4G0DaEjfoNXburz5Jled7OEWReXymt9oYqD4V8JM06x3xwdY2Zotxit13FjcgDqclv7tX3eewgge4F7K+qxXSQaoCLt4VgDswAWX+mz7GNUSD8WnRt/n57VSLKaG/mce53S/hH77sMStosRhy47R8zlKrnmp5T33r1Xz7Y9cOZozxid3QsDp0MGZjYEYjOYkmTLYno62DeTxiGQ1sABW+RpGlZuSyDmk840pXqTk8oikSsfjkEWZH62CYelNzyhVXFe9K/NscX5C8U//AkpHHK/9Od8C4Wts66Q9K3dLIQbsJvuVyx30xEL1MF6FaHlvROkzsbgYN1wJMHw+B0H+gLI6H6UYojg3VIrBQxGHMzajX5UEQviLmaxjHKVsfC3+fFQCXBQkP4fOjoA/WKPjTAT13wz2bYVg==,iv:OuO1VFB4koP1kPRmT6MWIv9gouUGB9LLnhkL+FwRUM4=,tag:1dv+YprSKLTGN8tKM5A4Yg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB6S25jcEwwYkEyTkhjSlgr\naWljSXNVQ0ZHOVkxQXgwZDYvTi9YOStsNUJJClB3Qk1hTUpvZVVJOTBpN05vcmdi\nbENuMlhsLzFuUTdoTXhYK2JIZ3IwVmsKLS0tIFZLbmx4R1IrS0pQVmVCdDZPZjBy\na0JIL0MvOXhhTloxZVpBQlY3TWVZemMK2XMLHNm6wDrX9VqIOPErQWIUwXLhhNJN\nlWEQvkZWjDY9PQ06CYpF/B/VE1ukJFtuslzNc3a2yuXMx7BtAhiX7A==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-28T07:50:04Z",
"mac": "ENC[AES256_GCM,data:M90tmlmYarfx6UcS3rFedpyMBtefZ802ecouExpGczsWTLtAqDtfdsendiXDgq0aV9D4cCA7KecstFvKAFIICIXF9FHCxHPtHMUWW3RsSjH0N1IIu5oeERSfzR+EIYwE9/bE+bx3dVXRMBdnuGgMXKPt7UatBNBAw43xAjl3FkA=,iv:JfVKxfpk7QYtK7XrdzpHouO8z0nRGgVlxyzB/0JwVRM=,tag:pUHX4zsX4clkQU+TeTJa1g==,type:str]",
>>>>>>> Stashed changes
"version": "3.13.3"
}
}
+25 -6
View File
@@ -1,6 +1,25 @@
# STUB — not yet encrypted with sops.
# Bootstrap:
# bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
# sops updatekeys secrets/common.yaml (allows ha-server-1 to decrypt shared secrets)
# sops secrets/ha-server-1.yaml (create with: beszel-token)
beszel-token: REPLACE
beszel-token: ENC[AES256_GCM,data:tT2a1I6AhVgXWfc=,iv:1iXoOMAakHXpI2lYiipxcVSzXODUdpqYmfrpNKZwljg=,tag:Vuxxz7pyIsZYbkg2x6GvgQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAzeEVTTVZ1ZHdVNWNNUW53
amN2d1NhQ2lydVNJUXdyaG52ditNTHNJZmdZCnljakFpaW92a0xEbjNZSEhXdDcr
ZUN1MmN2dlA0T2lYTVNibjB6UExqU1UKLS0tIGliYmVicTBrWmE3dFZWVzhwUm9r
T2dUUUdUQUVzTWRvbzlWSExKRUorTGMKktkw3uuydGeChy+9pxtysHLvssZ5LKoK
8i+CNr76/nyKCRns6tHj5lHYefY/B8IpdmlQ4fXQKSrna/dDjkdWTA==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0dmd5ZkdDcEM1akJna3Yv
RnZFYVJNZ1IvY2R0QnZ6WC9PQnFmVkdWNHpFClVJWnNaRFV1UVBQTWlXb1NEZEVp
Z0ZiUC8yTXVyMFdmRnB5OWdwQVVTSUEKLS0tIFFxLzdyNlZ2WlFPdEJFTTl0Mnhh
TXVMdVIvKzRTeGRIRXBqTUNxM0Z1cmsKMLHPN6n2i18LHkgjinq033qZsrt7BdFG
NuQA4EbCqg5uiHPCSGU6/EVk2XmfROrJz5eyMWIwrwCbt53jsUNTFw==
-----END AGE ENCRYPTED FILE-----
recipient: age1nxlnrevqs2msdatze562vz6ym4pgydt2zndye83lwqauy6flggtqrevyw5
lastmodified: "2026-07-28T06:44:32Z"
mac: ENC[AES256_GCM,data:rtDzbONqaNpbEKo1btGiHArAJVCVsStHhgvM8MTgkRMUNGU8p9W4tiuQLmobDpFh81kj/mDnkx4cCpScuiMhQojz9H74l7P9o9b1Wxy0fSiCtGoQhaq6JoIVJoKtEZoHfMStdIio5EBW9DBB5abvjPLTaC0afgHo5pNn3A9qr6g=,iv:SqtJhel/5qW0V7IV6V8dCRWqPnsK6QCzbC15/BKeq1k=,tag:vF7RJJccm0q7BSyNGYnplQ==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.3
+31
View File
@@ -0,0 +1,31 @@
{
<<<<<<< Updated upstream
"data": "ENC[AES256_GCM,data:Gi6HP0EuWQcTESaFOEq7wgAvuvlyevuFIKAnShDXJIUBg7xNNlGJ3PmBfx+YyFvChCqOqThAS8TxStogMVQqyWxm0vtCoVshkHG+9Q9xHV72AiuTcmJdsZ0DiSf9jwJYLZ80x1VI9a++Mqg0yLs5few=,iv:H9YYUQ4nOhQTBaonA/H7F7KroMRyU7kGul/bL/AnQ2Y=,tag:7ESjzSEH+XTqvvtxpinKJg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA1QVJsNHBYaUt6WWFDRjFX\nYVkrMHRtd1pTVFJsano1TG9hVyt4NGZyUEY0Ck1RU2U3MTQ1YTVkaXVhNlM5clg5\nWDZKZ1duTHZrM3FxWldnVXB1alJBeGsKLS0tIC9wUmZWcG1yLzBIL05qQ3lVV0xp\nS2lhWWVaTU9NWjd0K2hEaTcwNWE5YjgKmy3SHvYur/2sn6PjVLZFYzHnVMTFe66r\nm7MsWeUXU6zs4o4RWPPNBrpsVXIGZJx5Gbl+lp9Zubn61Wt/KVqyfw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
},
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBRNG5SQ0FvWWVvaGhOTmI1\naTE2anlXbTJDcHVBdWlxamlBam9jQlVyL2kwCmVJMkx2NnU4dno2T2dQOGUrRFRp\nQnl3TkJ2R0ZOS3Q2K2g3VDZQZ2VWS2cKLS0tIHgvdjJ3WURmcHVSQytHUGlwSnpm\nc1k2UHZORWt5TlJvaVp0YlpzWDdob00KKJ4bSM2CQD675b0ceEv9tz+wBtCIx/wz\n7Nr2CJoY68cB8KjM1eMzioZI1Rlt49tTQuD4XSs2/9Ej/dDDbP8p6A==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1scfc8p53q5aq2a87tcmsazmj8sfeft0s8kxg0et4nm3ucxyp3c3s6te82y"
}
],
"lastmodified": "2026-07-28T06:45:59Z",
"mac": "ENC[AES256_GCM,data:13BJaaM1Zrcg8/SZToXdlEM8J4GtFZnkoVt6vBWoeLU58wsGi58jdEAqMzWpnT3Vf2Sb8OjNUB1cbBVv9uMitbffFR27y06diKkAICnVc4shS0czoO/6u9O0k9gXicU04jUr/FKLL+sSiNi2ypCFSZT1C3WY44sTB15RNYQR7XQ=,iv:/4owaf5HYndcojBy5rZv6LZp/ipe3UuXy34dq3XI4Qc=,tag:i3Qk8jdl5hUmLScLCirm9w==,type:str]",
=======
"data": "ENC[AES256_GCM,data:YGR1qsYnOOJyaRx49FmR8Qs5pTZtxUu3izJxl6y+Zp9EJ2ERnJKBhQ7d7wxWb/Nd/SjHh4abAVPzX5ALoUkqo3avl2j9pJnAiBGtlQKwzwZrTwf+OBmM3GCvv0Myx3TPvo0vH9QuZnwdDQRI3uWgHeZ+KMRODMpKqzxiJNtwI0mHhz/eaIK8lIWM8ssrPc/g7Q5pb8K+Jz+duLX5M9COd5ohIoJWwGCuuWnCA4yrKoioizTHuJMDlFUip4kkdJoXUXJL7TL38BZU12TUM0wkpScxojel94xezw7tyUh9/ak9IDA2n5+LrUVwXYcJUT4BN19Pxy0zQnTWAWA/9p0tUa4m1dnYzkYOM6YPFpGuV/310ZHbjMp65gw9nSojBgpnvflQZqpNn5TEzc2ox63Y8uMLiEjf6Qcb/yA32Oy/lfEZwPFDEZel2MAefb9kYKC4EEkPUxywqS80Yzg8UhI3H/HwSMUe+Q==,iv:Bbn8oDmFHUY4/apwGd7cZyKnVeZCrTT5/IVgciyL7mQ=,tag:e4H6PO5KfRysRltjkpDY3w==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSA5N3F6N1paaUFkek44SUdW\ncDIvVVNPN0R6M1VVS1BHMUJaL3ZHUU44U2tJCjI3cHZqMmU5M1hvamtDVnpBdGhV\nRG9mdnROaXVTNEN2d0tmTUxkUjRhTVEKLS0tIENqeHRjZ3NLd1BpSHRHQzArQmlw\naVVxYXlXUXFoWXpGOGxzeTBsRFhXRUUKkGZzx82vm2kkw6nM5pnOkDXwLxuh0Cq0\nW/rdAwHwQtskgsL2A7mXd46bDdtPLiT+6ajLsCighJ9EZgaP2TfUBw==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-28T07:50:16Z",
"mac": "ENC[AES256_GCM,data:jRj0folhDTavBa+J4VivfHFpkqJBihdodMwu6EZ9G1hmJP782AzSWoVkeoPK7UhBzipki8aDzYS0uWpRxvSeFFAnkPmif0z2yRL4Z0MahXuEb2RHnexbqOjfHT2k4u1UKzEJ/s0IlXby61HXXG1uoB1PXeDauDgGVnrvIo/Bsjg=,iv:DJ2zXvDW2FKb+F7XxMCHtWA6ICkyfWbNJOB4WshEdis=,tag:M8GqOQq343OksrFK+sKkdw==,type:str]",
>>>>>>> Stashed changes
"version": "3.13.3"
}
}
+25 -6
View File
@@ -1,6 +1,25 @@
# STUB — not yet encrypted with sops.
# Bootstrap:
# bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
# sops updatekeys secrets/common.yaml (allows ha-server-2 to decrypt shared secrets)
# sops secrets/ha-server-2.yaml (create with: beszel-token)
beszel-token: REPLACE
beszel-token: ENC[AES256_GCM,data:0wooMaPiytDTGug=,iv:/WATBCEakrIvujt4oAeprw18W1n/WoK2zjpQdlVJl4Y=,tag:lhJa9hJgFqNFSB1Ogj5CtQ==,type:str]
sops:
age:
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBhdmNmSFlmei9KbDRWU3lj
TVJoSW9FRE9sWUdBRkNIMTB3aVNERUp0U0c4CnZUUzIvYm1xL1JNMGFKcWlBK2xT
SCt4TTdlU0ZralArcUVFZEh1YTRQaWcKLS0tIEtKbnBZVlkrU2U3VzlycS9odXVi
VjhXeWtZdDhpTG12cUZwVDhKeWZPMmsKOtXLVB8Z0faO05BPCVQXBn8gbMOZeHxC
sHmt3vMWO0jrxM0WLql59VWAFsczNwb1fw89NdZMfd/XoVq2EUV8dA==
-----END AGE ENCRYPTED FILE-----
recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
- enc: |
-----BEGIN AGE ENCRYPTED FILE-----
YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBucCtBR3N3Qm9JbTlLdTZT
eTlxWEsxUWZsUnMrWi9zSDdRVWZWejBUWFNVCnpyVEVLbTllVHg1a2RXRkVRZ05z
ZWlsSTBDWFF3bzkzaTFMVDQrVW1yczgKLS0tIFhKMll2OVdzWTcyeWdoaU85Vmla
WWEwRFgxNDF5czA3Q01LZ0NjY2toaTQKKawEiHUWss0NPuJng6UimekEOlxipiyC
ielJbLxma8WPLeLwBukFOBOTyNFxgsVBvwmBJ2k1iMIg1lCjhLD7RA==
-----END AGE ENCRYPTED FILE-----
recipient: age1scfc8p53q5aq2a87tcmsazmj8sfeft0s8kxg0et4nm3ucxyp3c3s6te82y
lastmodified: "2026-07-28T06:45:50Z"
mac: ENC[AES256_GCM,data:f50JYTzMJiEjB9+fNIs734CXrkN1IRgJKcoJ0GHoI5p75cfLfPxuzUifty9kQ/4DbM7LbWa5w+9occoV/yNOa86BKpPtU0LmlzXf904/SgOCUKhyHqhyO5mp8w71GoyLxx3ld8CbqfBu8t5l4gwbQamftsmQRxnUdU/Oa/CzdZQ=,iv:EewXBjeihw89h8ypH5u/G4YfoKZzkHqRWgSSIXU3/a0=,tag:lEWMvAkebMexDGGpE+wqkg==,type:str]
unencrypted_suffix: _unencrypted
version: 3.13.3
+6
View File
@@ -85,6 +85,12 @@
# that IPA itself doesn't cover: dotfiles, user packages, session variables.
ipaUser = "wayne";
# GID of the IPA "docker-access" group (GID 50010 on the IPA server).
# The local "docker" group is pinned to this GID on every host that runs
# Docker so that IPA group membership alone grants docker socket access -
# no per-host users.groups.docker.members entry for the IPA user needed.
dockerAccessGid = 50010;
# HA file server cluster
# LAN IPs (vmbr0 / ens18) — client-facing: iSCSI initiators, NFS, management.
# Storage IPs (vmbr1 / ens19) — isolated internal bridge, used for DRBD
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILdvVsj5cw/y6s2q5cPqKnULgnkEnsIQahrhm9DyCdd/ proxmox-ha-server-1
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:EhYMNPzlMDyby6g3hfvBKisKJnU+YAxZjf326tPdflQ2PcmPTzZ0GjHqJOPrOWyLU6IyvnsLMBV0JF5/UUi2c/H3p2wvoBgsxDMaaMW0YUK2IZFo+PZpWoYJSMrx40sH90W0qL0VBw17JLwPU6CjeUZrn5+sCN9EhQvncD8NoJ6QwZpWKhWLn9/7mxd/PF1uxjEQXzHYFeuUuVpQdKSUfV/BSJIT3pMM/VdH2203jTMtwUbK1/7UtGKtDWdfFbkEvJ6qsV7hnN3EYU16WtdyAiHEZvvZ3gqg6YK2DemojWG7z7VQkQGbGm40yA6/mYGW2dVK5ULzjxALl05G6lo1vzxODbMHhzsfJZVw6ocGx3r8xwaasiX8S3loq8WNUWJ9QgjbxuEX9uGZja6A3nPhcwVGGZExbUoU+syXUk2m+WfIQ0epplaPDyAxrYO5ZPWLdMYoPRxbDx94rx5fMxA3jst++IEL2kQMTieGQYJy3R8veQIe3Jb074zY+Y79EmRxfo8ruR6ZMqORtbZzJOBcd0OObjUVLUqB4rnC,iv:XkCfOKtmZNz+UTZdElLm8L0PpymkDzCM24VkAa9Y/Qo=,tag:hmr9ik0V7mxEKR0DcDB3/w==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBqQ3J3VDVwUVBvUjVXSWNz\nU3YrRXZlT2NLRThCNzd3YWNHVTlHeS9xMHpzCi9HVW5kMXZrdjE4bXdML3NhRnI1\nL0NQQkNQUmg2M0NBYzh2cG1vdHA0clkKLS0tIDJZYUwzK0RGNzJWckJJekI0SFg1\nUXE3N05xWThtdjdCZkdJTFJ2YUx6cm8KVANudVL54WBNc9DK9s9h4WQRLMewUqgN\ntu4LdMOmi5oV3LX06lbxhBq79dmsV5uos/qszhJkVGhEZwD1RGBoZA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-28T06:42:29Z",
"mac": "ENC[AES256_GCM,data:Xn3/CWhQJHWtd3QHoqpdwLuJdZTp+oxx43k38j4jspQSBPdVDSG6Ifi7Sb2r1g9YTxpjUto8mv7FH+BqI0wSYUkLfjaxI51xs3dq6e6kl5NWOsvWwIOGmBuhfIwusHGgsfBguxk0J6Ev8Irw674CRFc37jozXTVdpOOzFhPTvFs=,iv:/7OFE9NYkw2kYEetzeZRXah0KXuD5OXmfjQMIeqI75Q=,tag:VamuInZqYGYRwlyU8TtL9g==,type:str]",
"version": "3.13.3"
}
}
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAILjgkgUFZfpTLkFQeEbqXlvktE641EexWLv2I4iVKyQi proxmox-ha-server-2
@@ -0,0 +1,14 @@
{
"data": "ENC[AES256_GCM,data:GEebcQrPPrQxLC6mWXLLtytGkIy6Q2FbcQ/Z+ImGa3gahI/nigdNyR/avIz1DAUsOibpoMVlrk0GQUBMuxiEXulIwUEzeTh7sMI3XTZiugujOFHPyJoFeynlgeRNBbl2EVVjGmEmOsR/cZJGyKaet9aNTHTPDJUhodY5d+D/sE3SgpHoch1DI02/DRnqlnlOYiBoujk++BW71EDHLVyz3n9NsurTDYDK4L0Ch5cMgdlvwIuxGuPKtkhU9z6PtGxPbNlYlNP8U+3jG4XWEEY5hZHBGTVHn+bd5FViGY+sMiigO6yaOmddbUqzJoblKeI6F/rfrL3kXETLgVP7uVm3gw9KPEl5JaTdVqkG4hQ9G9/LvRlyEh619mCp9IRXxabm5SQ4NWFzah1ZwqIKp6GoAtqeWRgZNVqijoF+bnb0YmNFmjwpSrSMZvKsrZb5geyGmqah+NJii0xqVebnufB5p8lW9eytBSNgC/XLjj0olwHMw7CWGmnY6Q7lxF1tNYAD/K3dkbxQYFM3kBnmAdweC8WYc+hDjtk0HJul,iv:H4BonLmf3VoW2S9+IUE+SzPb0qiWHuNuGpVIUgUYScs=,tag:5+bq7vC81DMmOk4jn3u8lg==,type:str]",
"sops": {
"age": [
{
"enc": "-----BEGIN AGE ENCRYPTED FILE-----\nYWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBrQitsaDlqTmlYM2RvR1Ew\nb3NzZTJzR3F2bjM5NHhsa3dCUzZ4TW1pa3pBCkxIeEJtbWdaUFMwMkFyRHpCZ1Bl\nbmM5dGV3b3BDcDEwVlJ4UWV5RFNvRkUKLS0tIFRDd0U3dHBHb0Y3WDhNV3hkd2Ew\nclRDVEZHYjZUSzNVZkFjWGt3SGVQTlUKOULuXiYD9k2uVUmhuC15Kgezrd69rc9P\n7SocPa8kBliffP9IuxxW2S/hPbK8rqEx/sh/Km85ZIi4pG0AUQJ3fA==\n-----END AGE ENCRYPTED FILE-----\n",
"recipient": "age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad"
}
],
"lastmodified": "2026-07-28T06:45:10Z",
"mac": "ENC[AES256_GCM,data:ZnsqhgELqEdNzlTKJx2+xzcZRMNGTme/0a+PVvpbUv5IApimtif0zkTEC1LJsyLPut3MmO3QGprVlb8ihoRdm/IVvLaQonc36f6vN5cxrpQadru/RWEjLbnge1E8VkN/PVvHqRfLyIOPSq6wtGjAPaC/iHW1scBi50pWr0XO628=,iv:FlqAL1It6JuvoBhZotHWdSz17nckBf2rXnwovl6ZJBA=,tag:tNwcaL1sJakXtgejsB/Sbg==,type:str]",
"version": "3.13.3"
}
}