fix(nfs): mask rpc-svcgssd on server and ha-server build types

nfs-utils changed ConditionPathExists from !/etc/krb5.keytab to
/etc/krb5.keytab, so on IPA-joined hosts the service now starts instead
of skipping. Exports use standard auth (no sec=krb5) so the nfs/ Kerberos
principal is never provisioned and the service fails.

enable = false masks the unit so nfs-server's Wants= can't pull it in.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

CLAUDE.md

@@ -21,9 +21,7 @@ machines when deployed.
   `modules/installer/common.nix` (the auto-installer's own root/nixos login —
   a deliberate, documented choice, see `docs/auto-installer.md`, not
   accidental tech debt) and **SSH public keys** in `variables.nix`
-  (`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`) plus a couple of
-  per-host `KEY` values for beszel-agent auth (`hosts/server/host.nix`,
-  `hosts/nix-cache/host.nix`). Don't use the installer's hardcoded hash as a
+  (`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`, `vars.beszelHubKey`). Don't use the installer's hardcoded hash as a
   template for a *real* host — every other host uses sops-nix
   (`hashedPasswordFile`, see "Security Notes" in `README.md`). Flag any *new*
   secret-like string you encounter instead of committing it.
@@ -375,9 +373,8 @@ removing a host.
 
 - `hosts/<name>/host.nix` — per-machine identity **only**: hostname, hostId,
   per-machine secrets, `system.stateVersion`. These files carry no `imports`
-  of their own beyond narrow parameterized helpers (see
-  `modules/beszel/host-token.nix` below) — all shared behavior comes from the
-  platform/build-type modules composed in `flake.nix`, not from the host file.
+  of their own — all shared behavior comes from the platform/build-type modules
+  composed in `flake.nix`, not from the host file.
 - `modules/platforms/{linode,proxmox,lxc,baremetal}.nix` — platform-specific
   config: boot method, guest tooling, and the hardware config, imported
   directly by the platform module itself — **not** wired in from
@@ -434,10 +431,10 @@ removing a host.
   substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the
   full design (per-host local stores, no shared `/nix/store`, and how the
   `nixremote` signing/SSH keys fit together).
-- `modules/beszel/host-token.nix` — parameterized helper module
-  (`{ name, sopsFile }`) that wires a host's beszel-agent sops secret/template
-  and `environmentFile`; used by `hosts/server/host.nix` and
-  `hosts/nix-cache/host.nix` to avoid duplicating that boilerplate.
+- `modules/beszel/enable-agent.nix` — enables beszel-agent, sets `HUB_URL`,
+  fixes the upstream `StateDirectory` bug, and wires the universal
+  `beszel-token` sops secret (from `secrets/common.yaml`) into the agent's
+  `environmentFile`; see `docs/beszel.md` for the full setup guide.
 - `modules/tailscale/`, `modules/docker/`, `modules/networking/`,
   `modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose,
   single-host

docs/beszel.md

@@ -5,30 +5,32 @@ in this LAN. The hub runs as a Docker container on `docker.sweet.home` (port
 `vars.ports.beszelHub`, 8090). Each monitored NixOS host runs a
 `beszel-agent` that connects back to the hub.
 
-This document explains how to wire beszel-agent into any build in the flake.
-
 ---
 
-## How it is split across modules
+## How it works
 
-There are two modules and they have separate jobs:
+Everything is handled by a single module:
 
-| Module | What it does | Where it goes |
-|---|---|---|
-| `modules/beszel/enable-agent.nix` | Enables the systemd service, sets `HUB_URL`, fixes upstream StateDirectory bug | **Build type** (`modules/build-types/*.nix`) |
-| `modules/beszel/host-token.nix` | Declares the sops secret + env-file template that holds `TOKEN` | **Host file** (`hosts/<name>/host.nix`) |
+**`modules/beszel/enable-agent.nix`** — imported by a build type. It:
+- Enables `beszel-agent`
+- Sets `HUB_URL` to `docker.sweet.home:8090`
+- Sets `KEY` from `vars.beszelHubKey` (`variables.nix`) — the hub's SSH
+  public key, shared by every agent. Update `beszelHubKey` if the docker
+  host is ever rebuilt and the hub generates a new keypair.
+- Reads the universal `beszel-token` from `secrets/common.yaml` via sops
+  and passes it to the agent as `TOKEN` in an env file
+- Fixes an upstream bug where the agent couldn't persist its hub-pairing
+  fingerprint across restarts (adds a real `StateDirectory`)
 
-`KEY` (the hub's public key for this agent slot) is a plain string set
-directly on the host under `services.beszel.agent.environment.KEY` — it comes
-from the hub UI, not from sops, so it belongs in the host file rather than a
-secret.
+A host file needs no beszel configuration at all — just import the module
+in the build type and add the system in the hub UI.
 
 ---
 
 ## Adding beszel to a new build type
 
-Open `modules/build-types/<type>.nix` and add `../beszel/enable-agent.nix` to
-the `imports` list:
+Add `../beszel/enable-agent.nix` to the `imports` list in
+`modules/build-types/<type>.nix`:
 
 ```nix
 imports = [
@@ -37,156 +39,75 @@ imports = [
 ];
 ```
 
-That is the only change needed at the build-type level. The module sets:
-- `services.beszel.agent.enable = true`
-- `HUB_URL` pointing at `docker.sweet.home:8090`
-- a persistent `StateDirectory` for the hub-pairing fingerprint
+That's the only change required. The host file needs nothing.
 
 ---
 
-## Wiring the host file
+## Adding a new system to the hub
 
-Every host whose build type includes `enable-agent.nix` needs two things in
-its `hosts/<name>/host.nix`:
-
-### 1 — Import `host-token.nix`
-
-```nix
-imports = [
-  (import ../../modules/beszel/host-token.nix {
-    name = "<hostname>";            # must match the sops secret file name below
-    sopsFile = ../../secrets/<hostname>.yaml;
-  })
-];
-```
-
-This creates a sops-managed env file at runtime containing `TOKEN=<value>` and
-passes it to the agent via `environmentFile`. The `name` argument also becomes
-the sops template name (`<name>-beszel.env`), so it must be unique per host.
-
-### 2 — Set `KEY` once the hub has paired the agent
-
-```nix
-services.beszel.agent.environment = {
-  KEY = "ssh-ed25519 AAAA...";
-};
-```
-
-`KEY` is the hub's SSH public key for this agent slot — copy it from the hub
-UI after the agent first connects (see "Pairing with the hub" below).
-
-Until the hub has been paired, leave `KEY` commented out or omit it. The agent
-will still start and appear in the hub as an unpaired entry; fill in the key
-after that first connection.
+1. Rebuild and deploy the host with its build type importing `enable-agent.nix`.
+2. Open the beszel hub (`http://docker.sweet.home:8090`).
+3. Go to **Systems → Add system**, enter the host's IP and the default port
+   (45876). The agent will connect and the system will appear as active.
 
 ---
 
-## Adding the sops secret
+## One-time setup: add the token to `secrets/common.yaml`
 
-The `beszel-token` value in `secrets/<hostname>.yaml` is the token the agent
-uses to authenticate to the hub. To add it for a new host:
+The universal token is stored once in the common secrets file, shared by all
+agents. Only needed once, not per-host:
 
-1. Decrypt and edit the host's secrets file:
-   ```sh
-   sops secrets/<hostname>.yaml
-   ```
-2. Add the token:
-   ```yaml
-   beszel-token: <token from the beszel hub UI>
-   ```
-3. Save and close — sops re-encrypts the file on exit.
+```sh
+sops secrets/common.yaml
+```
 
-If the host does not yet have a `secrets/<hostname>.yaml`, create one using
-the same sops recipients as the other secrets files (check `.sops.yaml`), or
-run `scripts/secrets/sync-host-keys.sh <target>` first so the host's own
-SSH key is already a valid recipient.
+Add:
+```yaml
+beszel-token: <token from the beszel hub Settings → Keys>
+```
 
-The token is found in the beszel hub under **Settings → Keys** or when you
-add a new system via the hub UI.
+`secrets/common.yaml` is already a sops recipient for every host via their
+SSH host keys, so no additional sops recipient setup is needed.
 
 ---
 
 ## Optional: monitoring extra filesystems
 
-To have the agent report disk usage for a mount point that is not the root
-filesystem, add `EXTRA_FILESYSTEMS` in the host file alongside `KEY`:
+To report disk usage for a mount beyond the root filesystem, add
+`EXTRA_FILESYSTEMS` in the host file:
 
 ```nix
 services.beszel.agent.environment = {
-  KEY = "ssh-ed25519 AAAA...";
-  EXTRA_FILESYSTEMS = "/mnt/data";   # colon-separated if more than one
+  EXTRA_FILESYSTEMS = "/mnt/data";   # colon-separated for multiple paths
 };
 ```
 
-The `server` host uses this to expose the ZFS data pool:
+The `server` host uses this to expose its ZFS data pool:
 
 ```nix
-EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
+services.beszel.agent.environment = {
+  EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
+  LOG_LEVEL = "debug";
+};
 ```
 
 ---
 
 ## Optional: monitoring Docker containers
 
-The `enable-agent.nix` module has a commented-out line for Docker monitoring:
+`enable-agent.nix` has a commented-out line for Docker monitoring:
 
 ```nix
 #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
 ```
 
 Uncomment it if the host runs docker-socket-proxy and you want per-container
-stats in the hub. Hosts that do not run Docker should leave this commented out.
+stats. Hosts without Docker should leave it commented out.
 
 ---
 
-## Pairing with the hub
+## If the hub key changes
 
-1. Deploy the host with `host-token.nix` imported and `enable-agent.nix`
-   present in the build type. Leave `KEY` unset for now.
-2. Open the beszel hub (`http://docker.sweet.home:8090`).
-3. Go to **Systems → Add system**. The new host should appear as an
-   unpaired entry — copy the `KEY` value shown there.
-4. In `hosts/<name>/host.nix`, set:
-   ```nix
-   services.beszel.agent.environment.KEY = "<copied key>";
-   ```
-5. Rebuild and deploy the host. The agent will now pair permanently with the
-   hub.
-
----
-
-## Example: complete host file
-
-For reference, `hosts/tor-relay/host.nix` is the minimal case — one
-filesystem, no Docker, LXC platform:
-
-```nix
-{ vars, ... }:
-{
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "tor-relay";
-      sopsFile = ../../secrets/tor-relay.yaml;
-    })
-  ];
-
-  networking = { ... };
-
-  services.beszel.agent.environment = {
-    KEY = "ssh-ed25519 AAAA...";
-  };
-
-  system.stateVersion = "26.05";
-}
-```
-
-`hosts/server/host.nix` is the fuller case — extra filesystem path, debug
-logging:
-
-```nix
-services.beszel.agent.environment = {
-  KEY = "ssh-ed25519 AAAA...";
-  EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
-  LOG_LEVEL = "debug";
-};
-```
+If the docker host is ever rebuilt and beszel generates a new SSH keypair,
+update `beszelHubKey` in `variables.nix` and rebuild all beszel-enabled hosts.
+The new key is visible in the beszel hub under **Settings → Keys**.

hosts/ha-server-1/host.nix

@@ -1,12 +1,5 @@
 { vars, ... }:
 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "ha-server-1";
-      sopsFile = ../../secrets/ha-server-1.yaml;
-    })
-  ];
-
   networking = {
     hostName = vars.haServer1Host;
     hostId = "3a4b5c6d";
@@ -23,8 +16,5 @@
     nameservers = [ vars.domainControllerIp ];
   };
 
-  # Set KEY after pairing this host with the beszel hub; the token is sops-managed.
-  # services.beszel.agent.environment.KEY = "";
-
   system.stateVersion = "26.05";
 }

hosts/ha-server-2/host.nix

@@ -1,12 +1,5 @@
 { vars, ... }:
 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "ha-server-2";
-      sopsFile = ../../secrets/ha-server-2.yaml;
-    })
-  ];
-
   networking = {
     hostName = vars.haServer2Host;
     hostId = "7e8f9a0b";
@@ -23,8 +16,5 @@
     nameservers = [ vars.domainControllerIp ];
   };
 
-  # Set KEY after pairing this host with the beszel hub; the token is sops-managed.
-  # services.beszel.agent.environment.KEY = "";
-
   system.stateVersion = "26.05";
 }

hosts/nix-cache/host.nix

@@ -1,13 +1,6 @@
 { vars, ... }:
 
 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "nix-cache";
-      sopsFile = ../../secrets/nix-cache.yaml;
-    })
-  ];
-
   networking = {
     hostName = vars.nixCacheHost;
     useDHCP = false;
@@ -19,10 +12,6 @@
     nameservers = [ vars.domainControllerIp ];
   };
 
-  services.beszel.agent.environment = {
-    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
-  };
-
   # Preserved from the pre-refactor `nix-cache` target — stateVersion must
   # never be bumped on an already-installed machine.
   system.stateVersion = "25.05";

hosts/server/host.nix

@@ -1,13 +1,6 @@
 { vars, ... }:
 
 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "server";
-      sopsFile = ../../secrets/server.yaml;
-    })
-  ];
-
   networking = {
     hostName = vars.nfsServerHost;
     hostId = "6689f93e";
@@ -21,7 +14,6 @@
   };
 
   services.beszel.agent.environment = {
-    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
     EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
     LOG_LEVEL = "debug";
   };

hosts/tailscale-router/host.nix

@@ -1,13 +1,6 @@
 { vars, ... }:
 
 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "tailscale-router";
-      sopsFile = ../../secrets/tailscale-router.yaml;
-    })
-  ];
-
   networking = {
     hostName = "tailscale-router";
     useDHCP = false;
@@ -19,10 +12,6 @@
     nameservers = [ vars.domainControllerIp ];
   };
 
-  services.beszel.agent.environment = {
-    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
-  };
-
   # No networking.hostId: only ZFS-touching hosts (server, docker) need one
   # for pool-import safety, and this host does neither.
 

hosts/tor-relay/host.nix

@@ -1,13 +1,6 @@
 { vars, ... }:
 
 {
-  imports = [
-    (import ../../modules/beszel/host-token.nix {
-      name = "tor-relay";
-      sopsFile = ../../secrets/tor-relay.yaml;
-    })
-  ];
-
   networking = {
     hostName = "tor-relay";
     useDHCP = false;
@@ -22,10 +15,6 @@
   # No networking.hostId: only ZFS-touching hosts need one for pool-import
   # safety, and this host does neither.
 
-  services.beszel.agent.environment = {
-    KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
-  };
-
   # A genuinely new host (not a pre-refactor carry-over), so it tracks the
   # flake's current nixpkgs release rather than being pinned to an older one.
   system.stateVersion = "26.05";

modules/beszel/enable-agent.nix

@@ -1,10 +1,23 @@
-{ vars, ... }:
+{ config, vars, ... }:
 
 {
-  services.beszel.agent.enable = true;
-  services.beszel.agent.environment = {
-    #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
-    HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
+  # Universal token shared by all beszel agents. Add to secrets/common.yaml:
+  #   sops secrets/common.yaml
+  #   beszel-token: <value from the beszel hub UI>
+  sops.secrets."beszel-token" = { };
+
+  sops.templates."beszel.env".content = ''
+    TOKEN=${config.sops.placeholder."beszel-token"}
+  '';
+
+  services.beszel.agent = {
+    enable = true;
+    environmentFile = config.sops.templates."beszel.env".path;
+    environment = {
+      #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
+      HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
+      KEY = vars.beszelHubKey;
+    };
   };
 
   # The upstream module runs beszel-agent under DynamicUser with

modules/beszel/host-token.nix

@@ -1,11 +0,0 @@
-{ name, sopsFile }:
-
-{ config, ... }:
-
-{
-  sops.secrets."beszel-token".sopsFile = sopsFile;
-  sops.templates."${name}-beszel.env".content = ''
-    TOKEN=${config.sops.placeholder."beszel-token"}
-  '';
-  services.beszel.agent.environmentFile = config.sops.templates."${name}-beszel.env".path;
-}

modules/build-types/ha-server.nix

@@ -45,4 +45,7 @@ in
   # Pacemaker controls nfs-server — prevent systemd from starting it at boot
   # on both nodes (only the Active node should be serving NFS).
   systemd.services.nfs-server.wantedBy = lib.mkForce [ ];
+
+  # Same reason as server.nix: exports use standard auth, not Kerberos.
+  systemd.services.rpc-svcgssd.enable = false;
 }

modules/build-types/server.nix

@@ -102,6 +102,12 @@ in
     requires = [ "zfs-mount.service" ];
   };
 
+  # rpc-svcgssd handles Kerberos/GSS-API for NFS. Not needed: exports use
+  # standard auth, not sec=krb5. On IPA-joined hosts the keytab exists (host/
+  # principal only) but has no nfs/ principal, causing spurious failure.
+  # Mask it so nfs-server's Wants= can't pull it in.
+  systemd.services.rpc-svcgssd.enable = false;
+
   services.nfs.server = {
     enable = true;
     exports = mkNfsExports vars.storageRoot;

variables.nix

@@ -40,6 +40,11 @@
   # from scratch.
   nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICuHUxGNH6ei3BZD+EfZs3l4X8uJNcjQiOsM/G4yo4O/ lxc-nix-cache";
 
+  # Beszel hub's SSH public key — used by every agent to authenticate the
+  # hub's incoming connection. Update if the docker host is ever rebuilt and
+  # the hub generates a new keypair.
+  beszelHubKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
+
   # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
   # host (modules/nix-cache/server.nix) — one per client host that's allowed
   # to use it as a distributed builder.