fix(gc-hosts): filter discovered guests against flake to exclude non-NixOS hosts

pve1 runs non-NixOS guests (FreeIPA domain-controller, Proxmox Data Manager, etc.) alongside NixOS ones. Validate each discovered hostname against the set of hostnames defined in nixosConfigurations before adding it to the gc list. Brings back the flake eval but uses it correctly: extract all hostname values (not target-name keys) to build a filter set, then only include pve1 guests whose name matches a flake-managed NixOS host. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
fix(gc-hosts): use pct/qm list output directly as hostnames
2026-07-29 14:06:17 +10:00 · 2026-07-29 13:58:45 +10:00 · 2026-07-29 13:53:02 +10:00 · 2026-07-29 13:45:48 +10:00 · 2026-07-29 13:44:26 +10:00 · 2026-07-29 13:35:24 +10:00
26 changed files with 415 additions and 267 deletions
@@ -23,3 +23,5 @@ host-keys/
 # Temporary Milestone 1 audit checklist (remove-sensetive-info-refactor.md)
 # - working notes only, never committed, deleted once every row is rotated.
 secrets-inventory.md
+.claude/worktrees/                                                                                                  
+.claude/settings.local.json 
@@ -21,9 +21,7 @@ machines when deployed.
  `modules/installer/common.nix` (the auto-installer's own root/nixos login —
  a deliberate, documented choice, see `docs/auto-installer.md`, not
  accidental tech debt) and **SSH public keys** in `variables.nix`
-  (`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`) plus a couple of
-  per-host `KEY` values for beszel-agent auth (`hosts/server/host.nix`,
-  `hosts/nix-cache/host.nix`). Don't use the installer's hardcoded hash as a
+  (`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`, `vars.beszelHubKey`). Don't use the installer's hardcoded hash as a
  template for a *real* host — every other host uses sops-nix
  (`hashedPasswordFile`, see "Security Notes" in `README.md`). Flag any *new*
  secret-like string you encounter instead of committing it.
@@ -375,9 +373,8 @@ removing a host.

 - `hosts/<name>/host.nix` — per-machine identity **only**: hostname, hostId,
  per-machine secrets, `system.stateVersion`. These files carry no `imports`
-  of their own beyond narrow parameterized helpers (see
-  `modules/beszel/host-token.nix` below) — all shared behavior comes from the
-  platform/build-type modules composed in `flake.nix`, not from the host file.
+  of their own — all shared behavior comes from the platform/build-type modules
+  composed in `flake.nix`, not from the host file.
 - `modules/platforms/{linode,proxmox,lxc,baremetal}.nix` — platform-specific
  config: boot method, guest tooling, and the hardware config, imported
  directly by the platform module itself — **not** wired in from
@@ -434,10 +431,10 @@ removing a host.
  substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the
  full design (per-host local stores, no shared `/nix/store`, and how the
  `nixremote` signing/SSH keys fit together).
- `modules/beszel/host-token.nix` — parameterized helper module
-  (`{ name, sopsFile }`) that wires a host's beszel-agent sops secret/template
-  and `environmentFile`; used by `hosts/server/host.nix` and
-  `hosts/nix-cache/host.nix` to avoid duplicating that boilerplate.
+- `modules/beszel/enable-agent.nix` — enables beszel-agent, sets `HUB_URL`,
+  fixes the upstream `StateDirectory` bug, and wires the universal
+  `beszel-token` sops secret (from `secrets/common.yaml`) into the agent's
+  `environmentFile`; see `docs/beszel.md` for the full setup guide.
 - `modules/tailscale/`, `modules/docker/`, `modules/networking/`,
  `modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose,
  single-host
@@ -0,0 +1,113 @@
+# Beszel agent
+
+[Beszel](https://github.com/henrygd/beszel) is the monitoring dashboard used
+in this LAN. The hub runs as a Docker container on `docker.sweet.home` (port
+`vars.ports.beszelHub`, 8090). Each monitored NixOS host runs a
+`beszel-agent` that connects back to the hub.
+
+---
+
+## How it works
+
+Everything is handled by a single module:
+
+**`modules/beszel/enable-agent.nix`** — imported by a build type. It:
+- Enables `beszel-agent`
+- Sets `HUB_URL` to `docker.sweet.home:8090`
+- Sets `KEY` from `vars.beszelHubKey` (`variables.nix`) — the hub's SSH
+  public key, shared by every agent. Update `beszelHubKey` if the docker
+  host is ever rebuilt and the hub generates a new keypair.
+- Reads the universal `beszel-token` from `secrets/common.yaml` via sops
+  and passes it to the agent as `TOKEN` in an env file
+- Fixes an upstream bug where the agent couldn't persist its hub-pairing
+  fingerprint across restarts (adds a real `StateDirectory`)
+
+A host file needs no beszel configuration at all — just import the module
+in the build type and add the system in the hub UI.
+
+---
+
+## Adding beszel to a new build type
+
+Add `../beszel/enable-agent.nix` to the `imports` list in
+`modules/build-types/<type>.nix`:
+
+```nix
+imports = [
+  ../beszel/enable-agent.nix
+  # ... other imports
+];
+```
+
+That's the only change required. The host file needs nothing.
+
+---
+
+## Adding a new system to the hub
+
+1. Rebuild and deploy the host with its build type importing `enable-agent.nix`.
+2. Open the beszel hub (`http://docker.sweet.home:8090`).
+3. Go to **Systems → Add system**, enter the host's IP and the default port
+   (45876). The agent will connect and the system will appear as active.
+
+---
+
+## One-time setup: add the token to `secrets/common.yaml`
+
+The universal token is stored once in the common secrets file, shared by all
+agents. Only needed once, not per-host:
+
+```sh
+sops secrets/common.yaml
+```
+
+Add:
+```yaml
+beszel-token: <token from the beszel hub Settings → Keys>
+```
+
+`secrets/common.yaml` is already a sops recipient for every host via their
+SSH host keys, so no additional sops recipient setup is needed.
+
+---
+
+## Optional: monitoring extra filesystems
+
+To report disk usage for a mount beyond the root filesystem, add
+`EXTRA_FILESYSTEMS` in the host file:
+
+```nix
+services.beszel.agent.environment = {
+  EXTRA_FILESYSTEMS = "/mnt/data";   # colon-separated for multiple paths
+};
+```
+
+The `server` host uses this to expose its ZFS data pool:
+
+```nix
+services.beszel.agent.environment = {
+  EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
+  LOG_LEVEL = "debug";
+};
+```
+
+---
+
+## Optional: monitoring Docker containers
+
+`enable-agent.nix` has a commented-out line for Docker monitoring:
+
+```nix
+#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
+```
+
+Uncomment it if the host runs docker-socket-proxy and you want per-container
+stats. Hosts without Docker should leave it commented out.
+
+---
+
+## If the hub key changes
+
+If the docker host is ever rebuilt and beszel generates a new SSH keypair,
+update `beszelHubKey` in `variables.nix` and rebuild all beszel-enabled hosts.
+The new key is visible in the beszel hub under **Settings → Keys**.
@@ -259,11 +259,11 @@
    },
    "nixpkgs_2": {
      "locked": {
-        "lastModified": 1785104993,
-        "narHash": "sha256-eKbrvPoAOFutbYMdbB3r5EQVmFxKv24iKqHPPUXA0gM=",
+        "lastModified": 1785133411,
+        "narHash": "sha256-Yjv0WEg39KRYS0rBdTbu6Fc/or/ihAKk13W9sQ6VWd0=",
        "owner": "NixOS",
        "repo": "nixpkgs",
-        "rev": "8623c4c20aa4ca2f5fb81510d2944066c3fb0d96",
+        "rev": "2f5a153c270b70cb0f8c11f46d96d6d3bc39f4e3",
        "type": "github"
      },
      "original": {
@@ -1,12 +1,5 @@
 { vars, ... }:
 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "ha-server-1";
-      sopsFile = ../../secrets/ha-server-1.yaml;
-    })
-  ];
-
  networking = {
    hostName = vars.haServer1Host;
    hostId = "3a4b5c6d";
@@ -23,8 +16,5 @@
    nameservers = [ vars.domainControllerIp ];
  };

-  # Set KEY after pairing this host with the beszel hub; the token is sops-managed.
-  # services.beszel.agent.environment.KEY = "";
-
  system.stateVersion = "26.05";
 }
@@ -1,12 +1,5 @@
 { vars, ... }:
 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "ha-server-2";
-      sopsFile = ../../secrets/ha-server-2.yaml;
-    })
-  ];
-
  networking = {
    hostName = vars.haServer2Host;
    hostId = "7e8f9a0b";
@@ -23,8 +16,5 @@
    nameservers = [ vars.domainControllerIp ];
  };

-  # Set KEY after pairing this host with the beszel hub; the token is sops-managed.
-  # services.beszel.agent.environment.KEY = "";
-
  system.stateVersion = "26.05";
 }
@@ -1,13 +1,6 @@
 { vars, ... }:

 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "nix-cache";
-      sopsFile = ../../secrets/nix-cache.yaml;
-    })
-  ];
-
  networking = {
    hostName = vars.nixCacheHost;
    useDHCP = false;
@@ -19,10 +12,6 @@
    nameservers = [ vars.domainControllerIp ];
  };

-  services.beszel.agent.environment = {
-    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
-  };
-
  # Preserved from the pre-refactor `nix-cache` target — stateVersion must
  # never be bumped on an already-installed machine.
  system.stateVersion = "25.05";
@@ -26,7 +26,7 @@

    # Optional: set environment vars
    sessionVariables = {
-      EDITOR = "vim";
+      EDITOR = "nano";
      SOPS_AGE_KEY_FILE = "${config.home.homeDirectory}/.config/sops/age/keys.txt";
    };

@@ -11,7 +11,9 @@
    defaultGateway = { address = vars.lanGateway; interface = vars.lxcLanInterface; };
    nameservers = [ vars.domainControllerIp ];
  };
-
+services.beszel.agent.environment = {
+#    KEY = "";
+  };
  # Preserved from the pre-refactor `pxe-boot` target — stateVersion must
  # never be bumped on an already-installed machine.
  system.stateVersion = "25.05";
@@ -1,13 +1,6 @@
 { vars, ... }:

 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "server";
-      sopsFile = ../../secrets/server.yaml;
-    })
-  ];
-
  networking = {
    hostName = vars.nfsServerHost;
    hostId = "6689f93e";
@@ -21,7 +14,6 @@
  };

  services.beszel.agent.environment = {
-    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
    EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
    LOG_LEVEL = "debug";
  };
@@ -1,13 +1,6 @@
 { vars, ... }:

 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "tailscale-router";
-      sopsFile = ../../secrets/tailscale-router.yaml;
-    })
-  ];
-
  networking = {
    hostName = "tailscale-router";
    useDHCP = false;
@@ -19,10 +12,6 @@
    nameservers = [ vars.domainControllerIp ];
  };

-  services.beszel.agent.environment = {
-    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
-  };
-
  # No networking.hostId: only ZFS-touching hosts (server, docker) need one
  # for pool-import safety, and this host does neither.

@@ -1,13 +1,6 @@
 { vars, ... }:

 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "tor-relay";
-      sopsFile = ../../secrets/tor-relay.yaml;
-    })
-  ];
-
  networking = {
    hostName = "tor-relay";
    useDHCP = false;
@@ -22,10 +15,6 @@
  # No networking.hostId: only ZFS-touching hosts need one for pool-import
  # safety, and this host does neither.

-  services.beszel.agent.environment = {
-    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
-  };
-
  # A genuinely new host (not a pre-refactor carry-over), so it tracks the
  # flake's current nixpkgs release rather than being pinned to an older one.
  system.stateVersion = "26.05";
@@ -1,10 +1,23 @@
-{ vars, ... }:
+{ config, vars, ... }:

 {
-  services.beszel.agent.enable = true;
-  services.beszel.agent.environment = {
-    #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
-    HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
+  # Universal token shared by all beszel agents. Add to secrets/common.yaml:
+  #   sops secrets/common.yaml
+  #   beszel-token: <value from the beszel hub UI>
+  sops.secrets."beszel-token" = { };
+
+  sops.templates."beszel.env".content = ''
+    TOKEN=${config.sops.placeholder."beszel-token"}
+  '';
+
+  services.beszel.agent = {
+    enable = true;
+    environmentFile = config.sops.templates."beszel.env".path;
+    environment = {
+      #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
+      HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
+      KEY = vars.beszelHubKey;
+    };
  };

  # The upstream module runs beszel-agent under DynamicUser with
@@ -1,11 +0,0 @@
-{ name, sopsFile }:
-
-{ config, ... }:
-
-{
-  sops.secrets."beszel-token".sopsFile = sopsFile;
-  sops.templates."${name}-beszel.env".content = ''
-    TOKEN=${config.sops.placeholder."beszel-token"}
-  '';
-  services.beszel.agent.environmentFile = config.sops.templates."${name}-beszel.env".path;
-}
@@ -30,7 +30,7 @@ in
    ../ha/pacemaker-stack.nix
    ../ha/iscsi-target.nix
    ../ha/cluster-config.nix
-    # ../beszel/enable-agent.nix
+    ../beszel/enable-agent.nix
  ];

  # xfsprogs: mkfs.xfs/xfs_info needed by cluster-init.sh.
@@ -45,4 +45,7 @@ in
  # Pacemaker controls nfs-server — prevent systemd from starting it at boot
  # on both nodes (only the Active node should be serving NFS).
  systemd.services.nfs-server.wantedBy = lib.mkForce [ ];
+
+  # Same reason as server.nix: exports use standard auth, not Kerberos.
+  systemd.services.rpc-svcgssd.enable = false;
 }
@@ -314,6 +314,7 @@ in
  imports = [
    ../pxe-boot/stage-installer-artifacts.nix
    ../pxe-boot/mount-pxe-images.nix
+    ../beszel/enable-agent.nix
  ];

  environment.systemPackages = with pkgs; [
@@ -102,6 +102,12 @@ in
    requires = [ "zfs-mount.service" ];
  };

+  # rpc-svcgssd handles Kerberos/GSS-API for NFS. Not needed: exports use
+  # standard auth, not sec=krb5. On IPA-joined hosts the keytab exists (host/
+  # principal only) but has no nfs/ principal, causing spurious failure.
+  # Mask it so nfs-server's Wants= can't pull it in.
+  systemd.services.rpc-svcgssd.enable = false;
+
  services.nfs.server = {
    enable = true;
    exports = mkNfsExports vars.storageRoot;
@@ -0,0 +1,237 @@
+#!/usr/bin/env bash
+# gc-hosts.sh — Run nix-collect-garbage -d on all live NixOS hosts.
+#
+# The host list is rebuilt on every run:
+#   1. This workstation (nixos) — always first
+#   2. pve1 — always second (non-NixOS Proxmox node with Nix installed)
+#   3. Every NixOS guest currently running on pve1 (discovered via pct/qm list)
+#
+# nix-cache is excluded: gc-ing the shared binary cache evicts store paths
+# that other hosts depend on for substitution.
+#
+# NixOS hosts: tries "sudo -n nix-collect-garbage -d" first (works when
+# wheelNeedsPassword = false, e.g. the HA cluster).  Falls back to user-level
+# "nix-collect-garbage -d" if sudo needs a password — still collects
+# unreferenced store paths and old nixos-user profile generations, but leaves
+# old system generations in place.
+# pve1: runs "bash -l -c nix-collect-garbage -d" as the login user so
+# /etc/profile is sourced and the Nix daemon's PATH is set up automatically.
+#
+# Usage (from repo root):
+#   bash scripts/gc-hosts.sh [--dry-run]
+
+set -euo pipefail
+cd "$(dirname "$0")/.."
+source scripts/env.sh 2>/dev/null || true
+source scripts/lib/nix-eval.sh 2>/dev/null || true
+
+# ── config ────────────────────────────────────────────────────────────────────
+
+: "${MAX_JOBS:=8}"
+: "${NIXOS_USER:=nixos}"
+: "${PVE1_SSH_USER:=${PROXMOX_SSH_USER:-wayne}}"
+
+# GC connections use BatchMode — no interactive prompts, just succeed or fail.
+SSH_OPTS=(-o StrictHostKeyChecking=no -o BatchMode=yes -o ConnectTimeout=10)
+# Discovery connections do NOT use BatchMode so that sudo can prompt if needed
+# (pct/qm list require root access on Proxmox).
+SSH_QUERY_OPTS=(-o StrictHostKeyChecking=no -o ConnectTimeout=10)
+
+DRY_RUN=0
+for arg in "$@"; do
+  case "$arg" in
+    --dry-run) DRY_RUN=1 ;;
+    *) echo "Unknown option: $arg" >&2; exit 1 ;;
+  esac
+done
+
+# ── build the host list ───────────────────────────────────────────────────────
+
+# ORDERED_HOSTS: names in display/execution order.
+# HOST_TARGET[name]: SSH target string (user@host).
+# HOST_TYPE[name]: "nixos" (try sudo gc, fallback user) | "nix" (login-shell gc).
+declare -a  ORDERED_HOSTS=()
+declare -A  HOST_TARGET=()
+declare -A  HOST_TYPE=()
+declare -A  _SEEN_HOSTNAMES=()   # dedup tracker
+
+_add_host() {
+  local name="$1" target="$2" type="$3"
+  if [[ -n "${_SEEN_HOSTNAMES[$name]+_}" ]]; then return; fi
+  _SEEN_HOSTNAMES[$name]=1
+  ORDERED_HOSTS+=("$name")
+  HOST_TARGET[$name]="$target"
+  HOST_TYPE[$name]="$type"
+}
+
+# 1. Workstation (hard-wired first)
+_add_host "nixos" "${NIXOS_USER}@nixos" "nixos"
+
+# 2. pve1 (hard-wired second; non-NixOS, no system generations)
+_add_host "pve1" "${PVE1_SSH_USER}@${PVE1_HOST}" "nix"
+
+# 3. Dynamically discover running NixOS guests on pve1
+#
+# create-proxmox-resource.sh names every guest after its NixOS hostname:
+#   pct create ... --hostname <nixos-hostname>  (LXC)
+#   qm  create ... --name    <nixos-hostname>   (VM)
+# So pct/qm list output already contains the NixOS hostname directly.
+# We validate against the flake to filter out non-NixOS guests on pve1
+# (e.g. FreeIPA, Proxmox Backup Server) that share the same Proxmox node.
+echo "Discovering running guests on ${PVE1_HOST}..."
+
+# Eval the flake once to get the set of hostnames that are actually NixOS.
+# Values are NixOS hostnames (e.g. "docker"); keys are flake targets ("lxc-docker").
+nixos_hostnames=""
+nixos_hostnames="$(
+  nix eval --json "${NIX_EVAL_FLAGS[@]}" .#nixosConfigurations \
+    --apply 'cfgs: builtins.attrValues (builtins.mapAttrs (_: cfg: cfg.config.networking.hostName) cfgs)' \
+    2>/dev/null | jq -r '.[]' | sort -u
+)" || { echo "  warning: flake eval failed — non-NixOS guests will not be filtered" >&2; }
+
+# SSH_QUERY_OPTS (no BatchMode) so sudo can prompt if needed for pct/qm.
+if ssh "${SSH_QUERY_OPTS[@]}" "${PVE1_SSH_USER}@${PVE1_HOST}" "true" 2>/dev/null; then
+  running_guests="$(
+    ssh "${SSH_QUERY_OPTS[@]}" "${PVE1_SSH_USER}@${PVE1_HOST}" bash -s <<'DISCOVER'
+sudo pct list 2>/dev/null | awk 'NR>1 && $2=="running" { print $NF }'
+sudo qm  list 2>/dev/null | awk 'NR>1 && $3=="running" { print $2  }'
+DISCOVER
+  )" || running_guests=""
+
+  while IFS= read -r hostname; do
+    [[ -z "$hostname" ]] && continue
+    # Exclude nix-cache.
+    case "$hostname" in *nix-cache*) continue ;; esac
+    # Skip if not a flake-managed NixOS host (filters non-NixOS pve1 guests).
+    if [[ -n "$nixos_hostnames" ]] && ! grep -qxF "$hostname" <<< "$nixos_hostnames"; then
+      continue
+    fi
+    # Skip if already in the list (e.g. a proxmox-gui guest whose hostname is nixos).
+    if [[ -n "${_SEEN_HOSTNAMES[$hostname]+_}" ]]; then continue; fi
+
+    echo "  + $hostname"
+    _add_host "$hostname" "${NIXOS_USER}@${hostname}" "nixos"
+  done <<< "$(echo "$running_guests" | sort -u)"
+else
+  echo "  warning: ${PVE1_HOST} unreachable — skipping dynamic host discovery" >&2
+fi
+
+echo ""
+echo "Hosts: ${ORDERED_HOSTS[*]}"
+echo ""
+
+# ── dry-run ───────────────────────────────────────────────────────────────────
+
+if [[ "$DRY_RUN" -eq 1 ]]; then
+  echo "[dry-run] commands that would run:"
+  for host in "${ORDERED_HOSTS[@]}"; do
+    target="${HOST_TARGET[$host]}"
+    type="${HOST_TYPE[$host]}"
+    if [[ "$type" == "nixos" ]]; then
+      echo "  ssh ${SSH_OPTS[*]} $target 'sudo -n nix-collect-garbage -d'"
+      echo "  # fallback: ssh ... $target 'nix-collect-garbage -d'"
+    else
+      echo "  ssh ${SSH_OPTS[*]} $target 'bash -l -c nix-collect-garbage -d'"
+    fi
+  done
+  exit 0
+fi
+
+# ── gc worker ─────────────────────────────────────────────────────────────────
+
+gc_one() {
+  local host="$1" target="${HOST_TARGET[$1]}" type="${HOST_TYPE[$1]}" logfile="$2"
+
+  if ! ssh "${SSH_OPTS[@]}" "$target" "true" 2>>"$logfile"; then
+    echo "unreachable"; return
+  fi
+
+  if [[ "$type" == "nixos" ]]; then
+    if ssh "${SSH_OPTS[@]}" "$target" "sudo -n nix-collect-garbage -d" \
+        >>"$logfile" 2>>"$logfile"; then
+      echo "ok(sudo)"; return
+    fi
+    echo "[sudo needs password — falling back to user-level gc]" >>"$logfile"
+    if ssh "${SSH_OPTS[@]}" "$target" "nix-collect-garbage -d" \
+        >>"$logfile" 2>>"$logfile"; then
+      echo "ok(user)"; return
+    fi
+  else
+    # Non-NixOS node: use a login shell so /etc/profile is sourced and the
+    # Nix daemon's bin dir is on PATH (set up by /etc/profile.d/nix-daemon.sh
+    # which the Nix installer adds to /etc/profile).
+    if ssh "${SSH_OPTS[@]}" "$target" "bash -l -c 'nix-collect-garbage -d'" \
+        >>"$logfile" 2>>"$logfile"; then
+      echo "ok"; return
+    fi
+  fi
+
+  echo "failed:$?"
+}
+
+# ── parallel execution ────────────────────────────────────────────────────────
+
+echo "Running gc on ${#ORDERED_HOSTS[@]} hosts (up to ${MAX_JOBS} parallel)..."
+echo ""
+
+TMPDIR_GC="$(mktemp -d)"
+trap 'rm -rf "$TMPDIR_GC"' EXIT
+
+declare -A LOGS=()
+job_count=0
+
+for host in "${ORDERED_HOSTS[@]}"; do
+  logfile="${TMPDIR_GC}/${host}.log"
+  resultfile="${TMPDIR_GC}/${host}.result"
+  LOGS[$host]="$logfile"
+  : > "$logfile"
+
+  ( result="$(gc_one "$host" "$logfile")"; echo "$result" > "$resultfile" ) &
+
+  (( job_count++ )) || true
+  if [[ "$job_count" -ge "$MAX_JOBS" ]]; then
+    wait -n 2>/dev/null || wait
+    (( job_count-- )) || true
+  fi
+done
+
+wait
+
+# ── summary ───────────────────────────────────────────────────────────────────
+
+echo "Results:"
+echo "──────────────────────────────"
+
+ok_hosts=()
+warn_hosts=()
+fail_hosts=()
+
+for host in "${ORDERED_HOSTS[@]}"; do
+  result="$(cat "${TMPDIR_GC}/${host}.result" 2>/dev/null || echo "failed:missing")"
+  case "$result" in
+    ok|"ok(sudo)"|"ok(user)")
+      printf "  %-22s  %s\n" "$host" "$result"
+      ok_hosts+=("$host") ;;
+    unreachable)
+      printf "  %-22s  UNREACHABLE\n" "$host"
+      warn_hosts+=("$host") ;;
+    *)
+      printf "  %-22s  FAILED (%s)\n" "$host" "$result"
+      fail_hosts+=("$host") ;;
+  esac
+done
+
+echo ""
+echo "  ${#ok_hosts[@]} succeeded, ${#warn_hosts[@]} unreachable, ${#fail_hosts[@]} failed"
+
+for host in "${warn_hosts[@]+"${warn_hosts[@]}"}" "${fail_hosts[@]+"${fail_hosts[@]}"}"; do
+  logfile="${LOGS[$host]}"
+  if [[ -s "$logfile" ]]; then
+    echo ""
+    echo "── $host ──"
+    cat "$logfile"
+  fi
+done
+
+echo ""
+[[ "${#fail_hosts[@]}" -eq 0 ]]
@@ -1,6 +1,7 @@
 root-hashedPassword: ENC[AES256_GCM,data:Kp0nOZI7vDoLhJHiOJBwJn0rQZ5yhnwapGnAcA+qh8vlDETtFs/iQdetF/2ZxmANf62SviTNd+Ag0q5JIF1996x7onZGXqxgSMCuVzZLBdUlsO5IR0BslWWz47khYGTe4WkUg4NB1itBfQ==,iv:5Sra5vJ79V8hxQT3g9qJ+dOj2W2sumIhqpitqnHjJdk=,tag:3Igu0+8GeUZHqS3fKUVwog==,type:str]
 nixos-hashedPassword: ENC[AES256_GCM,data:pT7tVRN6X4a+DNUgB7fIUUE3CbnetkjxmoSL1PxSU+ktsFU+fB0mEvJjA1uujsGH5Rcztg7YM815+M0Z67ILmHaXbza5DtFacrqhi4/b277xly0SHRX4yOvBwQh6mJG1jn/0O/wvUUIYdw==,iv:bp2nfhC8nFbk6o5iWDAugvbzu7J/a1xayFnBEtkhNpE=,tag:HqWgkIpSrSM/K9OK2WO+VQ==,type:str]
 nix-github-token: ENC[AES256_GCM,data:k1vYz7SqVhzpWa6jTL6NUD8lKOCpHCgTm+HT4IcnbzbSTUZP/bJUYw==,iv:UqAULZnr/4+VcioUDfTwvOSuwM8K9JgGhiApvYQPyoc=,tag:1LKHXhWAO/AHPDIZFBb04A==,type:str]
+beszel-token: ENC[AES256_GCM,data:ds7OFjIXpOe/OIiEIydK9qsYkq3rMShK+jCTRRHKLzxUV0Bl,iv:nEt5FxkQaiTmAPFbj7vGJIMAEBjXmx+XYcdqaAxGzo8=,tag:+ql/UsgBR+TPySG9VDZi1g==,type:str]
 sops:
    age:
        - enc: |
@@ -228,7 +229,7 @@ sops:
            BBYPJMn3lfDgiQPvfXINfhJI6O+bWyjt0WnfodCGFM6EJLARXiTaqA==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
-    lastmodified: "2026-07-23T21:15:41Z"
-    mac: ENC[AES256_GCM,data:qFhnPra6IE3wyKQ4WKweON0S0YtD5I0adGZVfA0m6BVilN6bX5oC/1j5NK2oHrsz920hSl0SOF8LrpqOrUyGjSRkPsN4kq8qr9bJcrX4URiktP0oRden5LLt6hf+ZRP7WmRXFqixPkPHJnZIoAvkNnTFce7cDq5NEAHkKUEKG7k=,iv:nyblUDGeu3TUfFivYylOn3C/HITj99qiPI2+mh8AGh4=,tag:FrtRzSCylC4wlIoqZdfx7w==,type:str]
+    lastmodified: "2026-07-29T01:49:58Z"
+    mac: ENC[AES256_GCM,data:+m1nB00cOyr5IuxUMwvumkEIPKbYSw30UQVFyLPe+4VSOniKm8zxb52DTTaZyFyxUNAQOWewdZfvPRpTHBAgqt/HE8dMcALqPLAneNzpxIYr7oUh3TkRw9Qkk8NVsZnqTiar6C0n9xsA23tfVt5FSByqYzuAirNdwbKi2pkH59c=,iv:jonNGWQP0plAL/lrpiBKwpVcXULWl2+ZOnzBI47J1Ss=,tag:DjBN2/V5SUYq0zeYkdJylQ==,type:str]
    unencrypted_suffix: _unencrypted
-    version: 3.13.2
+    version: 3.13.3
@@ -1,25 +0,0 @@
-beszel-token: ENC[AES256_GCM,data:tT2a1I6AhVgXWfc=,iv:1iXoOMAakHXpI2lYiipxcVSzXODUdpqYmfrpNKZwljg=,tag:Vuxxz7pyIsZYbkg2x6GvgQ==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBLT0oxSDR6WHJGT0VTY211
-            d3hYcU42SzRIQzhvdUxWSnZhRitoNGFlclVBClFEZ0xXVnU2OFV3TmRXZHJyOVp4
-            aDZjQ0g0RlV5bnlzU0Y4ZnYxWm4zK1UKLS0tIDJYZ1F6OG8vdUxnS2NuaG9SL1gv
-            bURjNjRWQXIvQlNyVVZieGpCblV6bmsKQrCQQz7b4+w2WB3E56lEZoZa8/Yx8mn5
-            vWtJa6nvqlE4miJdRpbjQWUeFHghr0A6FsYDg5yf+Mcpf2otTw8psQ==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB5cVFzS3hFVkZpcGRWcUc3
-            YnRqVWUrSU1mUGRaaWhZTFQyRGRLMm9BSWdFCnhzYVhtWExGMFlRdU40K05TV0li
-            L09BK1h6SFlWSGRtQ2F1QW9iM3lDVWMKLS0tIGNQSE9aN0xHNytlOHBzM3MvMVpB
-            VEdyUFc0WWgzVEswRG1XSyt4dThTWGMKRkR+FifFDDKzrMrlJgZj/lbJDjfHsvjI
-            BGfJj1x0whWiozg/SZ6BsYYYlVkcCBwVTmPYx2iQHk4jLqHcMF36dw==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1k73g8x47hs93wcv7qh92n3htz8pl295g49hyvlrf3570mts0hgys5g04d6
-    lastmodified: "2026-07-28T06:44:32Z"
-    mac: ENC[AES256_GCM,data:rtDzbONqaNpbEKo1btGiHArAJVCVsStHhgvM8MTgkRMUNGU8p9W4tiuQLmobDpFh81kj/mDnkx4cCpScuiMhQojz9H74l7P9o9b1Wxy0fSiCtGoQhaq6JoIVJoKtEZoHfMStdIio5EBW9DBB5abvjPLTaC0afgHo5pNn3A9qr6g=,iv:SqtJhel/5qW0V7IV6V8dCRWqPnsK6QCzbC15/BKeq1k=,tag:vF7RJJccm0q7BSyNGYnplQ==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.3
@@ -1,25 +0,0 @@
-beszel-token: ENC[AES256_GCM,data:0wooMaPiytDTGug=,iv:/WATBCEakrIvujt4oAeprw18W1n/WoK2zjpQdlVJl4Y=,tag:lhJa9hJgFqNFSB1Ogj5CtQ==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlMWJCTUNSVVA0Rk84VWl5
-            emJ5ck0rMTFRMmljUmVOTlpUT1dlM2VFekZnCmRZbWcvL214YjUzakNDYkovQm5q
-            cVJQZnJvVGpIMkpqUnZEenkxZHNDSkkKLS0tIE5tUFl4a0FtZFNNTXFudlFNbjcv
-            a1dEbnl1NTR2T1lWN2wweVZ2ZnErZEEKbn9bLqIsMwSGKBjR97kMfFDuh1l7Hu5o
-            zdLjnAHzCEAsTW6V6wPB1rvi++FO04lqq1RfEiWXJkB2KWSRf+jdTw==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBDYUhMZWZCVS9MbnlscVBQ
-            aENENGFLTjJ3MVg5SGJCeXZsNVZ2ZFM4aVZZClYyRUhYTFZENEkvQjJqNGp6b3Q2
-            SWthQ01EMDdQYkVBQ0ZRdmRIMTFnUE0KLS0tIEJHV2YzRmw1MDVsVWhHRW9LMnVH
-            cU9qdUQzcjlNWFdQZmVydHdpamdQQ2MKndSN6wQbwyGqnyS819tHJRqDLRryXonC
-            CPuBYrTwxosRPwgh+nyJto4lj1uZqCmFVMXGjn87em8ykY5arp9o8Q==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1fefy6dk8zn5c3edwmrs9vwx79quftnt784m628t9e34q3ft3cehqz8u72r
-    lastmodified: "2026-07-28T06:45:50Z"
-    mac: ENC[AES256_GCM,data:f50JYTzMJiEjB9+fNIs734CXrkN1IRgJKcoJ0GHoI5p75cfLfPxuzUifty9kQ/4DbM7LbWa5w+9occoV/yNOa86BKpPtU0LmlzXf904/SgOCUKhyHqhyO5mp8w71GoyLxx3ld8CbqfBu8t5l4gwbQamftsmQRxnUdU/Oa/CzdZQ=,iv:EewXBjeihw89h8ypH5u/G4YfoKZzkHqRWgSSIXU3/a0=,tag:lEWMvAkebMexDGGpE+wqkg==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.3
@@ -1,4 +1,3 @@
-beszel-token: ENC[AES256_GCM,data:meuzUP/6wCssJDVTgbC0XwiLZPMGyDl55HEIiON9xOXCD9k6,iv:TDqWcp+8Mxd8wN09r5otQRQXq3XTeQphaTWxvvuLTAs=,tag:cRPZQGlwB/dTguBAheWPQg==,type:str]
 cache-priv-key: ENC[AES256_GCM,data:6vQKIf7eS0WNL2Eptoi4VWr18SRMZfN/H/aFUUtXdMYQY5LLyBp2EHRKqZcGFuh1nZhUdAxUztq/CVXx+QFxKW+ElHxCxUSp0QqI1fdSkBkKZb8hlit5SoX9JtLzZGg0HBNM3nJu,iv:0J+xmrPJhInHhFR/c41ACjuTfaIoMkQFSfbL2KkgFa8=,tag:f4s9Szs5oprVVRSyXaX48A==,type:str]
 sops:
    age:
@@ -38,7 +37,7 @@ sops:
            IqA477+noQ/Rjrszb2hEvxID7keogcDUWMWQzQvdMc22+3mvAr9qIg==
            -----END AGE ENCRYPTED FILE-----
          recipient: age1jlltcv5jcnm40z5k0q6hv053k2rqpqvemtuecdwn527uw8uqz4es3x7m68
-    lastmodified: "2026-07-19T23:30:21Z"
-    mac: ENC[AES256_GCM,data:kLGE2xawQT7mx+sfw68hmGk5nCEGiEjZrqTEl9B1dtQmTrMwmoVr/1RISi4LfJrwxy31mDgff4lcIL4wIJuM373uk3X8j4RNyYQNTfKEkORT6r8NHeepNs267O77pKGd7OmcM4MT/BqOnB8ELS7Wlf2ect7CAlvUUVyc8icxgZE=,iv:EYLDsHYHZ1XOQXafOTqHHWpk/OBNq/R6IJnOBYV33E4=,tag:thxrCPC5oGvDjhK7Dz87YA==,type:str]
+    lastmodified: "2026-07-29T01:59:11Z"
+    mac: ENC[AES256_GCM,data:/nbcfause6G6F8IvMoyPZtkWS1XRLAivhwTFu6y5P0Mm0eCcO6M7/rgioN9dngKzPXrCUl3Dx/EvhrrWKe2/Saq9WEOFgvS2V05pTRbQgYjuugVzW2paPq1fgmoDYNjHz2yFYWAovbIFtjVxMR9tmcASMTe6r/FocvMXbFCJh7Y=,iv:n3KoaPPtDhWK8mxJNJRk7WPVUUNR59nZSA8JyysWNDc=,tag:tYhkNXsj2ohVjHkvxife2w==,type:str]
    unencrypted_suffix: _unencrypted
-    version: 3.13.1
+    version: 3.13.3
@@ -1,43 +0,0 @@
-beszel-token: ENC[AES256_GCM,data:cbQOXhLzNk4g9d6hvm2DH7Q5ApTPCTzsW2txflDT2dD/UPIE,iv:V19MI1GEo5/0205Hrt7JImfkjduFiZ7f9aIkDVaI8mU=,tag:WCArgdrnIOudVe/Tw+oxRw==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBZeDQyNFh3dm1YZ2FTMjdK
-            cDBWcDE2a1c1S0s1enhWRnVuMlVJc0gyTXk0CmxWNkRJMDhkeGpRTGltaitnVkZS
-            a3Q5TW4zYm5Ja2FETEhJcGF0N2ZKbmcKLS0tIDN0a1FqRGNOY3Y1UWxvUU8zWU1m
-            bE9DVzZESG1HTEhVWUdJOTF0bDhRVGcKP6OoyDAGLB9jQ69jpFyho5eaeK9XtZgN
-            RlSJpBm2Jo19h/crpH9AWXUAIG0BWueyr8mwBu12cQdFIU3IyZT6gg==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSAwRVFQcHpkQkNlSDhhM1Zp
-            bUZyOFduQXg1cDd6ZlNObFgzL2hmUmJhM1JFCnprMldPQXJNVW10dVRqQTdWcGlv
-            RnBYWWFsaVNrMkJpS0pkOGlQQzlJVVUKLS0tIFNsMEEwZTREZ1lwWFJGdE5YSVVU
-            ZEZ1bVpFMEQ5N0g0L2RacUpLMWQrVDQKxPzq6f960purgAmUJw6IZnZSnhkzNE8r
-            CSrFDowKTZI2KRdCtQ5fGhEoWO0ZPgVNxYV0KH7JBttylcpRLm6r5w==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1sweerhrga9yf8x6sv0apz4ed4g48rnlcq34rpv20t0rcelwgpgeqwvndzz
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSB0czVoT3ZyYTZhOEQ3RWtR
-            bVl1MW5vNERxUEpmNXV0MGVBRE9ySnBjT3pzCkM2aUVpZjg0SkNVTnRRMlhyMTN6
-            NlFrZDVKV09Yc0tuKzFzR0ZtQ2t6WkkKLS0tIG1mbUNFdHBycS9UOGc2cjNpeHVm
-            NUd1NThRQlZXeG1WbmR5Y3pTYXRKc3MKwSnE+0bGmxOAQUje6jHxuzIIyD6ZAwVz
-            b5AAYwbGRagKj6fimsHBUmi4ohyG1huIGGOU8HiUYpu4PGJgOscztg==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1nruncs4l0ufk7yuc4des8p99c0alfndl0lhsws8tycl5pplfp56s30af5f
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlVGRDMWNDZUR6c1VpUjRK
-            WDY3L2lNcWFFcm1UV3RPMjlqYnBGVEJLcFFzCjFxck4wdlp0Wmtzc1RKNS82MXpK
-            ZHBzOGhkc3ZuZUE2UmpUSTgycWdLSGMKLS0tIFUyYjczeUFWU2FyMlBTdzAxMTBE
-            VzhaVzlSL05nZzNmR0ZjNEFPTXYycHcKfiJ0KjdxtLWsXxsWKzAL+H3hYYjHrYO9
-            BjKknq1ZQJM0sB/Tid+GLqDwKi966MQK+AwHF5MqbsHW7eE5bO1nwg==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1529taqdwr6t0w7cvzmty0d5y5593wffl0krt48j6uc4u39k56g2qf6ywtp
-    lastmodified: "2026-07-19T02:30:40Z"
-    mac: ENC[AES256_GCM,data:rKHZjU/MH08ASTlu32HZO9uWmsBYuMCEC6M8gwVhzuWvmablnP05tS2z13XfaWaCEUXk6kmGJKuU0zu5+IKVZgamCF6DAMtxQb6bVCaLsoAm/GSqWQ5VI9eHqgnSSdN/o3ul/33Rf8iBQo4aw8FFAmDVuNz8bfAn0QefFTj0ByI=,iv:JD2gtqRinOY77etg6PUmZNovkYl1Q3F6ZvRi4x7RznQ=,tag:/5IMpWKRVt+l1luCTQE0BA==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.1
@@ -1,43 +0,0 @@
-beszel-token: ENC[AES256_GCM,data:qg2eb7Eq/w7vIuiiDeK/s6h9nmHWuooyrd93GvdLD2hGOvka,iv:FWRBQTIY9PzV2V8J7z5ftEb5dvjN/S0hP0H9JV+Uuq8=,tag:WP7DHOu4elfqdLoW7vo9qg==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBCZmozK3VjcENGb3l2MDNj
-            QVE3VVI3Z3R0MjduSlJGL3BFWkg5NHZmZms0CkNBTWZKZFJVNkJMZ0tpNXhXWW5q
-            WndvbmhuOVh5ZHhJMmJwenh0M2dEVlUKLS0tIGpXQ3dhRXVWUDZzdGg4RHN4V3p0
-            WDZnK1lMUHVIS09hRUEzVHRHQ0EwcWsKEd5NtstHSPH8okXcFLW7BId5xPGmgyC3
-            dAodIGkLBlJlaL1bZuRy6Vaac+mJZL5KJYD/o1kYSXRiHJfNpaulTg==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBXWWhYOFU3VHZ3S1VuRTF2
-            QlZvMkJUNDZmT1RDWWZsMkxnZjJldjdrRG44CkU0NEJTR0NhQUpjMEY4RXBLSVBJ
-            bDNtZ1UrRDNUbzA0NDJ1cWJKdlBBMWMKLS0tIE9vTmEzTUxpODF4SG1MQ1czM1N5
-            dVgrMERMRXJwVmhtMldTZUlHNUJVTEkKzUf9cETbo0K9bCPkrlbUkgkhRIPvjCHR
-            GFVwgb2fcaOzZHcDwlLDLaP6HbRPtzJfVbLpQ1XU0N/xijSr9ENB6g==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1f7usptjx9rv4rxauasve200gxtdt9jkqhhdqstlf20wvlm7u75rsjfw50m
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBlOVdWN0tHNWZmaWlWQ2hJ
-            MVFYd0pMNmJTdTlJQ3h1c0dzbDE3U2VqdzNzCitCcFdhbjRPYnl3Q214WnRxa2Jq
-            eTFnc2VBcXlGcTVYdWpWTFZiWXBMUDAKLS0tIC9YdVkrTU5EeXNlNzliLzBlYUR2
-            Qk1ITW9tMElzUHF4MVJyWS9uRGs1QncKj8OgxWYabf8MccWD6zYLxl+FvYPAvLCW
-            p/FJ8uMa9YWp0RjICvWIwyO0gK7oaxTh/JzfBjABngEZnZazNYP7Uw==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1k7d2du5mejsmv5rzavm4xwgpthqvcfsehduquv28nzs53zppa3kqngfxq2
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBsbzFKckFQVDg0VEpJMFlv
-            WSs1ODVoYlZZK3YrSXlMU1hIQjdLTUhNdHhJCkRnbHY5Z2NGQVMvOHV4R2hTd0xu
-            LzdheXdZVVBsTjhVOU8zZGFKN0lPWjQKLS0tIFdaaG5SeElDN3YvMjFxdHVLWlNX
-            M1BQQmtJTUdsK2x5TEo5TmNrMEZidkEKhi4jNKhrkBAPhJoeYNg1d60LQTqmJt+G
-            Z/2X5uWzh12zF5keL8cNHH6sD0z0MOChtd4UF37yr+lb/LFJkia6LQ==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1zhfyuzlq40reuqlr34gf77852nhs3t6mqfzrqmas8z6sxk7tcfhsungrm0
-    lastmodified: "2026-07-26T01:12:11Z"
-    mac: ENC[AES256_GCM,data:0r68y6XmdiW7p/h9QmTR5h37TxKp+0wF44LP8QfG6cGP0tvaasuvv+V9EC3msnREYjwJhlnHjRd8hJ31HGVkOHbYWI4V5cF4ZnaiGgrZJhsGcJvOJSroAvQwev+vIlXZpAPgZz3Z35FDvOpgb71l5c0so0l4EJhc70JOh2FxhvQ=,iv:Or6l/v/E01+9wtZwyl+pCttiuReR6V0kmBzE9KhsfgQ=,tag:cAEUgj/9tKDToKFr00g5UQ==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2
@@ -1,25 +0,0 @@
-beszel-token: ENC[AES256_GCM,data:gjbT3uROiVKQOJaUeafTxjVknQO1Tvbyx/Pl2bTad7DezByX,iv:3ikf7OaT2omO8yd6G6UwYbaRBSzyvbn+NghxAe5bcgI=,tag:Zuc2EP8rUtdDhr5CzSW2Pw==,type:str]
-sops:
-    age:
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBJL25EUUZack1FMzlnMmdk
-            Z1hnejZRNnMwVHpmWkFNdGcyeHVsNnRsSGtZCk9OdDhhcnR1WW9ZMEZ1OUVYbm1n
-            RmZRVy8wb1J3emJBK3Rrd1d4U1dYUDAKLS0tIGJaaElvSk1sOTBOM0lKck16OUtu
-            NnRZb3U0ZndmaHBZTm8zczhWdE1oaEUKkf6fLomAHoKPhuM4e9q96YmmH+h4VrEj
-            2x0rnwBwOoRzYWutB2MVtlsphAZmZ/PK0tEecT2MM0XXayVG/33qdg==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age1njap586hc0q43kr03g6c8eqhdsmk8zcafkl3f83xwlc2gqhlmfgs4tmwad
-        - enc: |
-            -----BEGIN AGE ENCRYPTED FILE-----
-            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBpeEJvaHl4akc1TFdEVGV2
-            czg3OVprU3p4ejNpTktXZEpneDgrOEhrZGlZCmJwT1dhSkZneHE5UmR0WTd5UENq
-            VHJEWG1EekJLY2pRZldtVGtxTHlGaGMKLS0tIEYrWHE0WTgyUlIwdktmNzNIS3FW
-            ZVRvT1dHa1Vzc2RSakVISzdMTlpnVGsKeT+edn4+LUkVtpRUNd/gKX3H1HG2bvNo
-            c8iI6qr/l6oxfP85OrKYFDU9IGvDMxSSdbixHtojPEb5OKVurV0WPQ==
-            -----END AGE ENCRYPTED FILE-----
-          recipient: age16kqfmvz4e23hmdlqresnyw69ej604s320mmd49h4hm3fhqchtgyqrws0k2
-    lastmodified: "2026-07-23T23:32:57Z"
-    mac: ENC[AES256_GCM,data:l9a/yNRoxY1hvSkLuR4N7deeKue/1JPlSvZvJfCSNbQ21p1qR433BbSDYvfW+kXQXS8GVcfgXSd9ywNzgVvkA5lR1++uYsZBLbYxJ+s3TKWs6/yECAZ0eM1KBA0BEm7cLSsHTOwd+2WspvmCYir++FDO9XRuS3guiMnQBglDf/E=,iv:eE6GVqexQNSiLYfmTTUdUx5AO//wyjSIsr96xAX1pcI=,tag:CNsoyDZYLUt5Seu7W5wJrw==,type:str]
-    unencrypted_suffix: _unencrypted
-    version: 3.13.2
@@ -40,17 +40,24 @@
  # from scratch.
  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICuHUxGNH6ei3BZD+EfZs3l4X8uJNcjQiOsM/G4yo4O/ lxc-nix-cache";

+  # Beszel hub's SSH public key — used by every agent to authenticate the
+  # hub's incoming connection. Update if the docker host is ever rebuilt and
+  # the hub generates a new keypair.
+  beszelHubKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
+
  # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
  # host (modules/nix-cache/server.nix) — one per client host that's allowed
  # to use it as a distributed builder.
  remoteBuilderAuthorizedKeys = [
-    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFDEA1S2ikpObREgbP5uVBWMxIOGbY8B+Wx7VTZK1m6t root@server"
-    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPAYIT9ormlmxZ0SziyDQaUntnKI8HK9/s3Qac1ZKjP2 root@docker"
-    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKKKzoEPl/ZW9KBRHBcp6/ThOngGpwMv5EhkTlgC4aDf root@nixos"
-    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIGtOWOCS+ImHc7NehguoyD7PbonGosKMZqc9+QR3v/h root@nixos"
-    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHxXTQxFnArK5HXG7czeoybZebCGfxpUdusJkPn+BCSp root@server"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+ioWPhHixlgCB9KIQ0QTHTz6A+Oo2F3uKiINLip5rO root@docker"
    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
-    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEj26SL/emsVjW2YhRucJVp2kTz8WgcEQgjBEBLRikk root@claude"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII/rLceRhnDobVXQYiPceuhDHHvVjFQ1pc9A6un/eUlA root@server"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHj11bLPpRzH2oslnwFzEvY9cSgfEFtSZbLQaDm4nZMK root@pxe-boot"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH4/Sesm8NYpj73R0cbGhI0Ubvz73vIVWAnbEDTlBTdh root@tor-relay"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIBCFUUtePndW7pqtlawft1QCdHmBVs3O/c8EJO+RcXV root@tailscale-router"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBZ9WKKAlP9Z7GQdgaZ1Xgw9C+vja2lqEZO5rJFpVqYN root@ha-server-1"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEGNKlaaMckd8nLWNGz4B2QokXjnnIvM+rEUv+R6h0sp root@ha-server-2"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+XeMco7OxUpjrjZm54HogMs9QB5xlcKmElASRvrmlW root@nixos"
  ];

  # Admin SSH public key, authorized on the primary user of every host and
Author	SHA1	Message	Date
beatzaplentyandClaude Sonnet 4.6	4ba9b141fe	fix(gc-hosts): filter discovered guests against flake to exclude non-NixOS hosts Check NixOS configurations / eval-hosts (pull_request) Successful in 10m32s Details pve1 runs non-NixOS guests (FreeIPA domain-controller, Proxmox Data Manager, etc.) alongside NixOS ones. Validate each discovered hostname against the set of hostnames defined in nixosConfigurations before adding it to the gc list. Brings back the flake eval but uses it correctly: extract all hostname values (not target-name keys) to build a filter set, then only include pve1 guests whose name matches a flake-managed NixOS host. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 14:06:17 +10:00
beatzaplentyandClaude Sonnet 4.6	a5308a7ee5	fix(gc-hosts): use pct/qm list output directly as hostnames Check NixOS configurations / eval-hosts (pull_request) Successful in 10m32s Details The flake hostname map approach was wrong: create-proxmox-resource.sh names every guest after its NixOS hostname (pct create --hostname <nixos-hostname>, qm create --name <nixos-hostname>), so pct/qm list already output the final hostname directly. Looking those names up in a map keyed by flake target names (lxc-docker, proxmox-server, ...) found nothing and the dynamic list was always empty. Drop the flake eval entirely; use the pct/qm list output as-is, filtering only nix-cache and already-seen entries. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 13:58:45 +10:00
beatzaplentyandClaude Sonnet 4.6	7b4794211d	fix(gc-hosts): fix pve1 PATH and discovery sudo prompts Check NixOS configurations / eval-hosts (pull_request) Successful in 10m45s Details Two bugs: 1. Dynamic discovery used BatchMode=yes for the pct/qm list SSH call, which silently suppressed any sudo password prompt and returned empty output. Fix: split SSH_OPTS into SSH_OPTS (BatchMode, for gc) and SSH_QUERY_OPTS (no BatchMode, for discovery) so sudo can prompt when needed. 2. pve1 gc used a non-login SSH session which doesn't source /etc/profile, so nix-collect-garbage was never on PATH. The Nix installer registers itself via /etc/profile.d/nix-daemon.sh which only runs in login shells. Fix: use "bash -l -c 'nix-collect-garbage -d'" instead. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 13:53:02 +10:00
beatzaplentyandClaude Sonnet 4.6	0ef8259225	fix(gc-hosts): source nix-daemon profile on pve1 before running gc Check NixOS configurations / eval-hosts (pull_request) Successful in 10m34s Details BatchMode SSH sessions don't source /etc/profile on non-NixOS hosts, so nix-collect-garbage isn't on PATH for the wayne user. Source the nix-daemon profile script explicitly, matching the pattern in scripts/lib/nix-bootstrap.sh. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 13:45:48 +10:00
beatzaplentyandClaude Sonnet 4.6	629c1457a9	refactor(gc-hosts): discover running pve1 guests dynamically each run Replace the static host list with dynamic discovery: workstation (nixos) and pve1 are hard-wired first and second; remaining hosts are discovered on every run by SSHing to pve1, listing running VMs/containers via pct/qm list, and resolving their NixOS hostnames from a single flake eval. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 13:44:26 +10:00
beatzaplentyandClaude Sonnet 4.6	decf3ddff9	feat(scripts): add gc-hosts.sh for parallel nix gc across all live hosts Check NixOS configurations / eval-hosts (pull_request) Successful in 10m29s Details Runs nix-collect-garbage -d on all deployed NixOS hosts and pve1 in parallel, skipping nix-cache to avoid evicting shared cache paths. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 13:35:24 +10:00
beatzaplenty	15bc5bd369	Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos Check NixOS configurations / eval-hosts (push) Successful in 10m30s Details	2026-07-29 13:02:03 +10:00
beatzaplenty	5a4fbbf7ad	enable beszel agent	2026-07-29 13:01:53 +10:00
beatzaplenty	e720bbb018	Merge pull request 'fix(nfs): mask rpc-svcgssd on server and ha-server build types' (#99 ) from worktree-docs-beszel-guide into main Check NixOS configurations / eval-hosts (push) Failing after 9m46s Details Reviewed-on: #99	2026-07-29 02:54:12 +00:00
beatzaplentyandClaude Sonnet 4.6	40cdf724b4	fix(nfs): mask rpc-svcgssd on server and ha-server build types Check NixOS configurations / eval-hosts (pull_request) Failing after 9m43s Details nfs-utils changed ConditionPathExists from !/etc/krb5.keytab to /etc/krb5.keytab, so on IPA-joined hosts the service now starts instead of skipping. Exports use standard auth (no sec=krb5) so the nfs/ Kerberos principal is never provisioned and the service fails. enable = false masks the unit so nfs-server's Wants= can't pull it in. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 12:51:50 +10:00
beatzaplenty	aaa193645b	Merge pull request 'refactor(beszel): move KEY and TOKEN fully into enable-agent.nix' (#98 ) from worktree-docs-beszel-guide into main Check NixOS configurations / eval-hosts (push) Successful in 10m45s Details Reviewed-on: #98	2026-07-29 02:38:19 +00:00
beatzaplentyandClaude Sonnet 4.6	f22ff7db79	refactor(beszel): move KEY and TOKEN fully into enable-agent.nix Check NixOS configurations / eval-hosts (pull_request) Successful in 10m59s Details KEY (hub's SSH public key) is now stored in variables.nix as beszelHubKey and set by enable-agent.nix — no host file ever needs to touch it. TOKEN was already universal; this removes the last per-host beszel config. Importing enable-agent.nix in a build type is now the only step needed to add a new host to beszel monitoring. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 12:37:54 +10:00
beatzaplenty	0da3c3070f	updated flake.lock and added beszel agent environment Check NixOS configurations / eval-hosts (push) Failing after 16m56s Details	2026-07-29 12:18:38 +10:00
beatzaplenty	e1551feefd	enabled beszel-agent Check NixOS configurations / eval-hosts (push) Successful in 10m34s Details	2026-07-29 12:14:49 +10:00
beatzaplentyandClaude Sonnet 4.6	b4bc30cb2c	chore(secrets): remove per-host beszel-token secrets Check NixOS configurations / eval-hosts (push) Successful in 10m27s Details Five files contained only beszel-token and are deleted entirely. nix-cache.yaml retains cache-priv-key; beszel-token removed from it. The universal token in secrets/common.yaml replaces all of these. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 12:00:38 +10:00
beatzaplenty	5c8d55bd78	Merge branch 'main' of https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos Check NixOS configurations / eval-hosts (push) Failing after 4m33s Details	2026-07-29 11:50:44 +10:00
beatzaplenty	999c9a3151	add beszel universal token	2026-07-29 11:50:01 +10:00
beatzaplenty	1e0fff1b26	Merge pull request 'refactor(beszel): switch to universal token via secrets/common.yaml' (#97 ) from worktree-docs-beszel-guide into main Check NixOS configurations / eval-hosts (push) Failing after 13m26s Details Reviewed-on: #97	2026-07-29 01:47:15 +00:00
beatzaplentyandClaude Sonnet 4.6	24c6469f10	refactor(beszel): switch to universal token via secrets/common.yaml Check NixOS configurations / eval-hosts (pull_request) Successful in 10m40s Details Replace per-host host-token.nix imports with a single beszel-token secret in secrets/common.yaml, wired once in enable-agent.nix. Host files now only need services.beszel.agent.environment.KEY — no imports block required. Delete modules/beszel/host-token.nix (no longer referenced anywhere). Action needed: run `sops secrets/common.yaml` and add `beszel-token: <value>` from the beszel hub UI before deploying. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 10:58:15 +10:00
beatzaplenty	c9458ac8a6	Merge pull request 'docs: add beszel agent setup guide' (#96 ) from worktree-docs-beszel-guide into main Check NixOS configurations / eval-hosts (push) Successful in 10m24s Details Reviewed-on: #96	2026-07-29 00:41:22 +00:00
beatzaplentyandClaude Sonnet 4.6	99ba0ed52b	docs: add beszel agent setup guide Check NixOS configurations / eval-hosts (pull_request) Successful in 10m37s Details Documents how to wire beszel-agent into any flake build: the two-module split (enable-agent.nix in build types, host-token.nix in host files), sops secret creation, hub pairing flow, and optional EXTRA_FILESYSTEMS / Docker monitoring knobs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>	2026-07-29 10:41:03 +10:00
beatzaplenty	1e4072029e	update rmote-builder keys Check NixOS configurations / eval-hosts (push) Successful in 10m38s Details	2026-07-29 10:39:53 +10:00
beatzaplenty	46649fc7e0	update remote-builder client keys Check NixOS configurations / eval-hosts (push) Successful in 10m32s Details	2026-07-29 10:29:40 +10:00
beatzaplenty	4cbcc3beb9	updated .gitignore and change default editor in home.nix Check NixOS configurations / eval-hosts (push) Failing after 13m5s Details	2026-07-29 10:17:37 +10:00
beatzaplenty	35f696ccd5	Merge pull request 'feat(ha): add resize-data-disk.sh for online data disk expansion' (#95 ) from worktree-ha-resize-disk into main Check NixOS configurations / eval-hosts (push) Successful in 10m27s Details Reviewed-on: #95	2026-07-29 00:01:49 +00:00