Merge pull request 'docs: add beszel agent setup guide' (#96) from worktree-docs-beszel-guide into main

Reviewed-on: #96

CLAUDE.md

@@ -21,7 +21,9 @@ machines when deployed.
   `modules/installer/common.nix` (the auto-installer's own root/nixos login —
   a deliberate, documented choice, see `docs/auto-installer.md`, not
   accidental tech debt) and **SSH public keys** in `variables.nix`
-  (`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`, `vars.beszelHubKey`). Don't use the installer's hardcoded hash as a
+  (`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`) plus a couple of
+  per-host `KEY` values for beszel-agent auth (`hosts/server/host.nix`,
+  `hosts/nix-cache/host.nix`). Don't use the installer's hardcoded hash as a
   template for a *real* host — every other host uses sops-nix
   (`hashedPasswordFile`, see "Security Notes" in `README.md`). Flag any *new*
   secret-like string you encounter instead of committing it.
@@ -373,8 +375,9 @@ removing a host.
 
 - `hosts/<name>/host.nix` — per-machine identity **only**: hostname, hostId,
   per-machine secrets, `system.stateVersion`. These files carry no `imports`
-  of their own — all shared behavior comes from the platform/build-type modules
-  composed in `flake.nix`, not from the host file.
+  of their own beyond narrow parameterized helpers (see
+  `modules/beszel/host-token.nix` below) — all shared behavior comes from the
+  platform/build-type modules composed in `flake.nix`, not from the host file.
 - `modules/platforms/{linode,proxmox,lxc,baremetal}.nix` — platform-specific
   config: boot method, guest tooling, and the hardware config, imported
   directly by the platform module itself — **not** wired in from
@@ -431,10 +434,10 @@ removing a host.
   substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the
   full design (per-host local stores, no shared `/nix/store`, and how the
   `nixremote` signing/SSH keys fit together).
-- `modules/beszel/enable-agent.nix` — enables beszel-agent, sets `HUB_URL`,
-  fixes the upstream `StateDirectory` bug, and wires the universal
-  `beszel-token` sops secret (from `secrets/common.yaml`) into the agent's
-  `environmentFile`; see `docs/beszel.md` for the full setup guide.
+- `modules/beszel/host-token.nix` — parameterized helper module
+  (`{ name, sopsFile }`) that wires a host's beszel-agent sops secret/template
+  and `environmentFile`; used by `hosts/server/host.nix` and
+  `hosts/nix-cache/host.nix` to avoid duplicating that boilerplate.
 - `modules/tailscale/`, `modules/docker/`, `modules/networking/`,
   `modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose,
   single-host

docs/beszel.md

@@ -5,32 +5,30 @@ in this LAN. The hub runs as a Docker container on `docker.sweet.home` (port
 `vars.ports.beszelHub`, 8090). Each monitored NixOS host runs a
 `beszel-agent` that connects back to the hub.
 
+This document explains how to wire beszel-agent into any build in the flake.
+
 ---
 
-## How it works
+## How it is split across modules
 
-Everything is handled by a single module:
+There are two modules and they have separate jobs:
 
-**`modules/beszel/enable-agent.nix`** — imported by a build type. It:
-- Enables `beszel-agent`
-- Sets `HUB_URL` to `docker.sweet.home:8090`
-- Sets `KEY` from `vars.beszelHubKey` (`variables.nix`) — the hub's SSH
-  public key, shared by every agent. Update `beszelHubKey` if the docker
-  host is ever rebuilt and the hub generates a new keypair.
-- Reads the universal `beszel-token` from `secrets/common.yaml` via sops
-  and passes it to the agent as `TOKEN` in an env file
-- Fixes an upstream bug where the agent couldn't persist its hub-pairing
-  fingerprint across restarts (adds a real `StateDirectory`)
+| Module | What it does | Where it goes |
+|---|---|---|
+| `modules/beszel/enable-agent.nix` | Enables the systemd service, sets `HUB_URL`, fixes upstream StateDirectory bug | **Build type** (`modules/build-types/*.nix`) |
+| `modules/beszel/host-token.nix` | Declares the sops secret + env-file template that holds `TOKEN` | **Host file** (`hosts/<name>/host.nix`) |
 
-A host file needs no beszel configuration at all — just import the module
-in the build type and add the system in the hub UI.
+`KEY` (the hub's public key for this agent slot) is a plain string set
+directly on the host under `services.beszel.agent.environment.KEY` — it comes
+from the hub UI, not from sops, so it belongs in the host file rather than a
+secret.
 
 ---
 
 ## Adding beszel to a new build type
 
-Add `../beszel/enable-agent.nix` to the `imports` list in
-`modules/build-types/<type>.nix`:
+Open `modules/build-types/<type>.nix` and add `../beszel/enable-agent.nix` to
+the `imports` list:
 
 ```nix
 imports = [
@@ -39,75 +37,156 @@ imports = [
 ];
 ```
 
-That's the only change required. The host file needs nothing.
+That is the only change needed at the build-type level. The module sets:
+- `services.beszel.agent.enable = true`
+- `HUB_URL` pointing at `docker.sweet.home:8090`
+- a persistent `StateDirectory` for the hub-pairing fingerprint
 
 ---
 
-## Adding a new system to the hub
+## Wiring the host file
 
-1. Rebuild and deploy the host with its build type importing `enable-agent.nix`.
-2. Open the beszel hub (`http://docker.sweet.home:8090`).
-3. Go to **Systems → Add system**, enter the host's IP and the default port
-   (45876). The agent will connect and the system will appear as active.
+Every host whose build type includes `enable-agent.nix` needs two things in
+its `hosts/<name>/host.nix`:
+
+### 1 — Import `host-token.nix`
+
+```nix
+imports = [
+  (import ../../modules/beszel/host-token.nix {
+    name = "<hostname>";            # must match the sops secret file name below
+    sopsFile = ../../secrets/<hostname>.yaml;
+  })
+];
+```
+
+This creates a sops-managed env file at runtime containing `TOKEN=<value>` and
+passes it to the agent via `environmentFile`. The `name` argument also becomes
+the sops template name (`<name>-beszel.env`), so it must be unique per host.
+
+### 2 — Set `KEY` once the hub has paired the agent
+
+```nix
+services.beszel.agent.environment = {
+  KEY = "ssh-ed25519 AAAA...";
+};
+```
+
+`KEY` is the hub's SSH public key for this agent slot — copy it from the hub
+UI after the agent first connects (see "Pairing with the hub" below).
+
+Until the hub has been paired, leave `KEY` commented out or omit it. The agent
+will still start and appear in the hub as an unpaired entry; fill in the key
+after that first connection.
 
 ---
 
-## One-time setup: add the token to `secrets/common.yaml`
+## Adding the sops secret
 
-The universal token is stored once in the common secrets file, shared by all
-agents. Only needed once, not per-host:
+The `beszel-token` value in `secrets/<hostname>.yaml` is the token the agent
+uses to authenticate to the hub. To add it for a new host:
 
-```sh
-sops secrets/common.yaml
-```
+1. Decrypt and edit the host's secrets file:
+   ```sh
+   sops secrets/<hostname>.yaml
+   ```
+2. Add the token:
+   ```yaml
+   beszel-token: <token from the beszel hub UI>
+   ```
+3. Save and close — sops re-encrypts the file on exit.
 
-Add:
-```yaml
-beszel-token: <token from the beszel hub Settings → Keys>
-```
+If the host does not yet have a `secrets/<hostname>.yaml`, create one using
+the same sops recipients as the other secrets files (check `.sops.yaml`), or
+run `scripts/secrets/sync-host-keys.sh <target>` first so the host's own
+SSH key is already a valid recipient.
 
-`secrets/common.yaml` is already a sops recipient for every host via their
-SSH host keys, so no additional sops recipient setup is needed.
+The token is found in the beszel hub under **Settings → Keys** or when you
+add a new system via the hub UI.
 
 ---
 
 ## Optional: monitoring extra filesystems
 
-To report disk usage for a mount beyond the root filesystem, add
-`EXTRA_FILESYSTEMS` in the host file:
+To have the agent report disk usage for a mount point that is not the root
+filesystem, add `EXTRA_FILESYSTEMS` in the host file alongside `KEY`:
 
 ```nix
 services.beszel.agent.environment = {
-  EXTRA_FILESYSTEMS = "/mnt/data";   # colon-separated for multiple paths
+  KEY = "ssh-ed25519 AAAA...";
+  EXTRA_FILESYSTEMS = "/mnt/data";   # colon-separated if more than one
 };
 ```
 
-The `server` host uses this to expose its ZFS data pool:
+The `server` host uses this to expose the ZFS data pool:
 
 ```nix
-services.beszel.agent.environment = {
-  EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
-  LOG_LEVEL = "debug";
-};
+EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
 ```
 
 ---
 
 ## Optional: monitoring Docker containers
 
-`enable-agent.nix` has a commented-out line for Docker monitoring:
+The `enable-agent.nix` module has a commented-out line for Docker monitoring:
 
 ```nix
 #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
 ```
 
 Uncomment it if the host runs docker-socket-proxy and you want per-container
-stats. Hosts without Docker should leave it commented out.
+stats in the hub. Hosts that do not run Docker should leave this commented out.
 
 ---
 
-## If the hub key changes
+## Pairing with the hub
 
-If the docker host is ever rebuilt and beszel generates a new SSH keypair,
-update `beszelHubKey` in `variables.nix` and rebuild all beszel-enabled hosts.
-The new key is visible in the beszel hub under **Settings → Keys**.
+1. Deploy the host with `host-token.nix` imported and `enable-agent.nix`
+   present in the build type. Leave `KEY` unset for now.
+2. Open the beszel hub (`http://docker.sweet.home:8090`).
+3. Go to **Systems → Add system**. The new host should appear as an
+   unpaired entry — copy the `KEY` value shown there.
+4. In `hosts/<name>/host.nix`, set:
+   ```nix
+   services.beszel.agent.environment.KEY = "<copied key>";
+   ```
+5. Rebuild and deploy the host. The agent will now pair permanently with the
+   hub.
+
+---
+
+## Example: complete host file
+
+For reference, `hosts/tor-relay/host.nix` is the minimal case — one
+filesystem, no Docker, LXC platform:
+
+```nix
+{ vars, ... }:
+{
+  imports = [
+    (import ../../modules/beszel/host-token.nix {
+      name = "tor-relay";
+      sopsFile = ../../secrets/tor-relay.yaml;
+    })
+  ];
+
+  networking = { ... };
+
+  services.beszel.agent.environment = {
+    KEY = "ssh-ed25519 AAAA...";
+  };
+
+  system.stateVersion = "26.05";
+}
+```
+
+`hosts/server/host.nix` is the fuller case — extra filesystem path, debug
+logging:
+
+```nix
+services.beszel.agent.environment = {
+  KEY = "ssh-ed25519 AAAA...";
+  EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
+  LOG_LEVEL = "debug";
+};
+```

hosts/ha-server-1/host.nix

@@ -1,5 +1,12 @@
 { vars, ... }:
 {
+  imports = [
+    (import ../../modules/beszel/host-token.nix {
+      name = "ha-server-1";
+      sopsFile = ../../secrets/ha-server-1.yaml;
+    })
+  ];
+
   networking = {
     hostName = vars.haServer1Host;
     hostId = "3a4b5c6d";
@@ -16,5 +23,8 @@
     nameservers = [ vars.domainControllerIp ];
   };
 
+  # Set KEY after pairing this host with the beszel hub; the token is sops-managed.
+  # services.beszel.agent.environment.KEY = "";
+
   system.stateVersion = "26.05";
 }

hosts/ha-server-2/host.nix

@@ -1,5 +1,12 @@
 { vars, ... }:
 {
+  imports = [
+    (import ../../modules/beszel/host-token.nix {
+      name = "ha-server-2";
+      sopsFile = ../../secrets/ha-server-2.yaml;
+    })
+  ];
+
   networking = {
     hostName = vars.haServer2Host;
     hostId = "7e8f9a0b";
@@ -16,5 +23,8 @@
     nameservers = [ vars.domainControllerIp ];
   };
 
+  # Set KEY after pairing this host with the beszel hub; the token is sops-managed.
+  # services.beszel.agent.environment.KEY = "";
+
   system.stateVersion = "26.05";
 }

hosts/nix-cache/host.nix

@@ -1,6 +1,13 @@
 { vars, ... }:
 
 {
+  imports = [
+    (import ../../modules/beszel/host-token.nix {
+      name = "nix-cache";
+      sopsFile = ../../secrets/nix-cache.yaml;
+    })
+  ];
+
   networking = {
     hostName = vars.nixCacheHost;
     useDHCP = false;
@@ -12,6 +19,10 @@
     nameservers = [ vars.domainControllerIp ];
   };
 
+  services.beszel.agent.environment = {
+    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
+  };
+
   # Preserved from the pre-refactor `nix-cache` target — stateVersion must
   # never be bumped on an already-installed machine.
   system.stateVersion = "25.05";

hosts/server/host.nix

@@ -1,6 +1,13 @@
 { vars, ... }:
 
 {
+  imports = [
+    (import ../../modules/beszel/host-token.nix {
+      name = "server";
+      sopsFile = ../../secrets/server.yaml;
+    })
+  ];
+
   networking = {
     hostName = vars.nfsServerHost;
     hostId = "6689f93e";
@@ -14,6 +21,7 @@
   };
 
   services.beszel.agent.environment = {
+    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
     EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
     LOG_LEVEL = "debug";
   };

hosts/tailscale-router/host.nix

@@ -1,6 +1,13 @@
 { vars, ... }:
 
 {
+  imports = [
+    (import ../../modules/beszel/host-token.nix {
+      name = "tailscale-router";
+      sopsFile = ../../secrets/tailscale-router.yaml;
+    })
+  ];
+
   networking = {
     hostName = "tailscale-router";
     useDHCP = false;
@@ -12,6 +19,10 @@
     nameservers = [ vars.domainControllerIp ];
   };
 
+  services.beszel.agent.environment = {
+    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
+  };
+
   # No networking.hostId: only ZFS-touching hosts (server, docker) need one
   # for pool-import safety, and this host does neither.
 

hosts/tor-relay/host.nix

@@ -1,6 +1,13 @@
 { vars, ... }:
 
 {
+  imports = [
+    (import ../../modules/beszel/host-token.nix {
+      name = "tor-relay";
+      sopsFile = ../../secrets/tor-relay.yaml;
+    })
+  ];
+
   networking = {
     hostName = "tor-relay";
     useDHCP = false;
@@ -15,6 +22,10 @@
   # No networking.hostId: only ZFS-touching hosts need one for pool-import
   # safety, and this host does neither.
 
+  services.beszel.agent.environment = {
+    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
+  };
+
   # A genuinely new host (not a pre-refactor carry-over), so it tracks the
   # flake's current nixpkgs release rather than being pinned to an older one.
   system.stateVersion = "26.05";

modules/beszel/enable-agent.nix

@@ -1,23 +1,10 @@
-{ config, vars, ... }:
+{ vars, ... }:
 
 {
-  # Universal token shared by all beszel agents. Add to secrets/common.yaml:
-  #   sops secrets/common.yaml
-  #   beszel-token: <value from the beszel hub UI>
-  sops.secrets."beszel-token" = { };
-
-  sops.templates."beszel.env".content = ''
-    TOKEN=${config.sops.placeholder."beszel-token"}
-  '';
-
-  services.beszel.agent = {
-    enable = true;
-    environmentFile = config.sops.templates."beszel.env".path;
-    environment = {
-      #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
-      HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
-      KEY = vars.beszelHubKey;
-    };
+  services.beszel.agent.enable = true;
+  services.beszel.agent.environment = {
+    #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
+    HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
   };
 
   # The upstream module runs beszel-agent under DynamicUser with

modules/beszel/host-token.nix

@@ -0,0 +1,11 @@
+{ name, sopsFile }:
+
+{ config, ... }:
+
+{
+  sops.secrets."beszel-token".sopsFile = sopsFile;
+  sops.templates."${name}-beszel.env".content = ''
+    TOKEN=${config.sops.placeholder."beszel-token"}
+  '';
+  services.beszel.agent.environmentFile = config.sops.templates."${name}-beszel.env".path;
+}

modules/build-types/ha-server.nix

@@ -45,7 +45,4 @@ in
   # Pacemaker controls nfs-server — prevent systemd from starting it at boot
   # on both nodes (only the Active node should be serving NFS).
   systemd.services.nfs-server.wantedBy = lib.mkForce [ ];
-
-  # Same reason as server.nix: exports use standard auth, not Kerberos.
-  systemd.services.rpc-svcgssd.enable = false;
 }

modules/build-types/server.nix

@@ -102,12 +102,6 @@ in
     requires = [ "zfs-mount.service" ];
   };
 
-  # rpc-svcgssd handles Kerberos/GSS-API for NFS. Not needed: exports use
-  # standard auth, not sec=krb5. On IPA-joined hosts the keytab exists (host/
-  # principal only) but has no nfs/ principal, causing spurious failure.
-  # Mask it so nfs-server's Wants= can't pull it in.
-  systemd.services.rpc-svcgssd.enable = false;
-
   services.nfs.server = {
     enable = true;
     exports = mkNfsExports vars.storageRoot;

variables.nix

@@ -40,24 +40,19 @@
   # from scratch.
   nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICuHUxGNH6ei3BZD+EfZs3l4X8uJNcjQiOsM/G4yo4O/ lxc-nix-cache";
 
-  # Beszel hub's SSH public key — used by every agent to authenticate the
-  # hub's incoming connection. Update if the docker host is ever rebuilt and
-  # the hub generates a new keypair.
-  beszelHubKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
-
   # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
   # host (modules/nix-cache/server.nix) — one per client host that's allowed
   # to use it as a distributed builder.
   remoteBuilderAuthorizedKeys = [
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+ioWPhHixlgCB9KIQ0QTHTz6A+Oo2F3uKiINLip5rO root@docker"
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICMJhrfFayLBG+gWtO6oAvgambw5nWWgztiTFEaaaVRH debian@surface"
-    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIJEj26SL/emsVjW2YhRucJVp2kTz8WgcEQgjBEBLRikk root@claude"
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII/rLceRhnDobVXQYiPceuhDHHvVjFQ1pc9A6un/eUlA root@server"
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHj11bLPpRzH2oslnwFzEvY9cSgfEFtSZbLQaDm4nZMK root@pxe-boot"
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH4/Sesm8NYpj73R0cbGhI0Ubvz73vIVWAnbEDTlBTdh root@tor-relay"
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIBCFUUtePndW7pqtlawft1QCdHmBVs3O/c8EJO+RcXV root@tailscale-router"
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBZ9WKKAlP9Z7GQdgaZ1Xgw9C+vja2lqEZO5rJFpVqYN root@ha-server-1"
     "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIEGNKlaaMckd8nLWNGz4B2QokXjnnIvM+rEUv+R6h0sp root@ha-server-2"
+    "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+XeMco7OxUpjrjZm54HogMs9QB5xlcKmElASRvrmlW root@nixos"
   ];
 
   # Admin SSH public key, authorized on the primary user of every host and