From 24c6469f107245ca5126f2d6192a05cedcff4ca3 Mon Sep 17 00:00:00 2001 From: beatzaplenty Date: Wed, 29 Jul 2026 10:58:15 +1000 Subject: [PATCH] refactor(beszel): switch to universal token via secrets/common.yaml MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace per-host host-token.nix imports with a single beszel-token secret in secrets/common.yaml, wired once in enable-agent.nix. Host files now only need services.beszel.agent.environment.KEY — no imports block required. Delete modules/beszel/host-token.nix (no longer referenced anywhere). Action needed: run `sops secrets/common.yaml` and add `beszel-token: ` from the beszel hub UI before deploying. Co-Authored-By: Claude Sonnet 4.6 --- CLAUDE.md | 18 ++--- docs/beszel.md | 131 ++++++++++++-------------------- hosts/ha-server-1/host.nix | 9 +-- hosts/ha-server-2/host.nix | 9 +-- hosts/nix-cache/host.nix | 7 -- hosts/server/host.nix | 7 -- hosts/tailscale-router/host.nix | 7 -- hosts/tor-relay/host.nix | 7 -- modules/beszel/enable-agent.nix | 22 ++++-- modules/beszel/host-token.nix | 11 --- 10 files changed, 74 insertions(+), 154 deletions(-) delete mode 100644 modules/beszel/host-token.nix diff --git a/CLAUDE.md b/CLAUDE.md index b0113f0..6351261 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -21,9 +21,8 @@ machines when deployed. `modules/installer/common.nix` (the auto-installer's own root/nixos login — a deliberate, documented choice, see `docs/auto-installer.md`, not accidental tech debt) and **SSH public keys** in `variables.nix` - (`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`) plus a couple of - per-host `KEY` values for beszel-agent auth (`hosts/server/host.nix`, - `hosts/nix-cache/host.nix`). Don't use the installer's hardcoded hash as a + (`vars.adminSshKey`, `vars.remoteBuilderAuthorizedKeys`) plus per-host + `KEY` values for beszel-agent auth (see `docs/beszel.md`). Don't use the installer's hardcoded hash as a template for a *real* host — every other host uses sops-nix (`hashedPasswordFile`, see "Security Notes" in `README.md`). Flag any *new* secret-like string you encounter instead of committing it. @@ -375,9 +374,8 @@ removing a host. - `hosts//host.nix` — per-machine identity **only**: hostname, hostId, per-machine secrets, `system.stateVersion`. These files carry no `imports` - of their own beyond narrow parameterized helpers (see - `modules/beszel/host-token.nix` below) — all shared behavior comes from the - platform/build-type modules composed in `flake.nix`, not from the host file. + of their own — all shared behavior comes from the platform/build-type modules + composed in `flake.nix`, not from the host file. - `modules/platforms/{linode,proxmox,lxc,baremetal}.nix` — platform-specific config: boot method, guest tooling, and the hardware config, imported directly by the platform module itself — **not** wired in from @@ -434,10 +432,10 @@ removing a host. substituter + SSH remote-builder wiring; see `docs/nix-cache.md` for the full design (per-host local stores, no shared `/nix/store`, and how the `nixremote` signing/SSH keys fit together). -- `modules/beszel/host-token.nix` — parameterized helper module - (`{ name, sopsFile }`) that wires a host's beszel-agent sops secret/template - and `environmentFile`; used by `hosts/server/host.nix` and - `hosts/nix-cache/host.nix` to avoid duplicating that boilerplate. +- `modules/beszel/enable-agent.nix` — enables beszel-agent, sets `HUB_URL`, + fixes the upstream `StateDirectory` bug, and wires the universal + `beszel-token` sops secret (from `secrets/common.yaml`) into the agent's + `environmentFile`; see `docs/beszel.md` for the full setup guide. - `modules/tailscale/`, `modules/docker/`, `modules/networking/`, `modules/traefik/`, `modules/tor/`, `modules/services/*` — single-purpose, single-host diff --git a/docs/beszel.md b/docs/beszel.md index 67f4962..d52d490 100644 --- a/docs/beszel.md +++ b/docs/beszel.md @@ -5,30 +5,29 @@ in this LAN. The hub runs as a Docker container on `docker.sweet.home` (port `vars.ports.beszelHub`, 8090). Each monitored NixOS host runs a `beszel-agent` that connects back to the hub. -This document explains how to wire beszel-agent into any build in the flake. - --- -## How it is split across modules +## How it works -There are two modules and they have separate jobs: +Everything is handled by a single module: -| Module | What it does | Where it goes | -|---|---|---| -| `modules/beszel/enable-agent.nix` | Enables the systemd service, sets `HUB_URL`, fixes upstream StateDirectory bug | **Build type** (`modules/build-types/*.nix`) | -| `modules/beszel/host-token.nix` | Declares the sops secret + env-file template that holds `TOKEN` | **Host file** (`hosts//host.nix`) | +**`modules/beszel/enable-agent.nix`** — imported by a build type. It: +- Enables `beszel-agent` +- Sets `HUB_URL` to `docker.sweet.home:8090` +- Reads the universal `beszel-token` from `secrets/common.yaml` via sops and + passes it to the agent as `TOKEN` in an env file +- Fixes an upstream bug where the agent couldn't persist its hub-pairing + fingerprint across restarts (adds a real `StateDirectory`) -`KEY` (the hub's public key for this agent slot) is a plain string set -directly on the host under `services.beszel.agent.environment.KEY` — it comes -from the hub UI, not from sops, so it belongs in the host file rather than a -secret. +The only thing a host file ever needs to add is `KEY` — the hub's public key +for that agent slot, which comes from the beszel hub UI after first pairing. --- ## Adding beszel to a new build type -Open `modules/build-types/.nix` and add `../beszel/enable-agent.nix` to -the `imports` list: +Add `../beszel/enable-agent.nix` to the `imports` list in +`modules/build-types/.nix`: ```nix imports = [ @@ -37,34 +36,14 @@ imports = [ ]; ``` -That is the only change needed at the build-type level. The module sets: -- `services.beszel.agent.enable = true` -- `HUB_URL` pointing at `docker.sweet.home:8090` -- a persistent `StateDirectory` for the hub-pairing fingerprint +That's the only build-type change required. --- ## Wiring the host file -Every host whose build type includes `enable-agent.nix` needs two things in -its `hosts//host.nix`: - -### 1 — Import `host-token.nix` - -```nix -imports = [ - (import ../../modules/beszel/host-token.nix { - name = ""; # must match the sops secret file name below - sopsFile = ../../secrets/.yaml; - }) -]; -``` - -This creates a sops-managed env file at runtime containing `TOKEN=` and -passes it to the agent via `environmentFile`. The `name` argument also becomes -the sops template name (`-beszel.env`), so it must be unique per host. - -### 2 — Set `KEY` once the hub has paired the agent +No `imports` are needed in the host file. Just set `KEY` once you've paired +the agent with the hub: ```nix services.beszel.agent.environment = { @@ -72,53 +51,47 @@ services.beszel.agent.environment = { }; ``` -`KEY` is the hub's SSH public key for this agent slot — copy it from the hub -UI after the agent first connects (see "Pairing with the hub" below). - -Until the hub has been paired, leave `KEY` commented out or omit it. The agent -will still start and appear in the hub as an unpaired entry; fill in the key -after that first connection. +Leave `KEY` commented out until after the first pairing (see "Pairing with +the hub" below). --- -## Adding the sops secret +## One-time setup: add the token to `secrets/common.yaml` -The `beszel-token` value in `secrets/.yaml` is the token the agent -uses to authenticate to the hub. To add it for a new host: +The universal token is stored once in the common secrets file, shared by all +agents. You only need to do this once, not per-host: -1. Decrypt and edit the host's secrets file: - ```sh - sops secrets/.yaml - ``` -2. Add the token: - ```yaml - beszel-token: - ``` -3. Save and close — sops re-encrypts the file on exit. +```sh +sops secrets/common.yaml +``` -If the host does not yet have a `secrets/.yaml`, create one using -the same sops recipients as the other secrets files (check `.sops.yaml`), or -run `scripts/secrets/sync-host-keys.sh ` first so the host's own -SSH key is already a valid recipient. +Add: +```yaml +beszel-token: +``` -The token is found in the beszel hub under **Settings → Keys** or when you -add a new system via the hub UI. +The token is found in the beszel hub under **Settings → Keys** or in the +"Add system" flow. + +`secrets/common.yaml` is already a sops recipient for every host via their +SSH host keys, so no additional sops recipient setup is needed for hosts that +are already provisioned. --- ## Optional: monitoring extra filesystems -To have the agent report disk usage for a mount point that is not the root -filesystem, add `EXTRA_FILESYSTEMS` in the host file alongside `KEY`: +To report disk usage for a mount beyond the root filesystem, add +`EXTRA_FILESYSTEMS` alongside `KEY` in the host file: ```nix services.beszel.agent.environment = { KEY = "ssh-ed25519 AAAA..."; - EXTRA_FILESYSTEMS = "/mnt/data"; # colon-separated if more than one + EXTRA_FILESYSTEMS = "/mnt/data"; # colon-separated for multiple paths }; ``` -The `server` host uses this to expose the ZFS data pool: +The `server` host uses this to expose its ZFS data pool: ```nix EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}"; @@ -128,48 +101,39 @@ EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath} ## Optional: monitoring Docker containers -The `enable-agent.nix` module has a commented-out line for Docker monitoring: +`enable-agent.nix` has a commented-out line for Docker monitoring: ```nix #DOCKER_HOST = "tcp://docker-socket-proxy:2375"; ``` Uncomment it if the host runs docker-socket-proxy and you want per-container -stats in the hub. Hosts that do not run Docker should leave this commented out. +stats. Hosts without Docker should leave it commented out. --- ## Pairing with the hub -1. Deploy the host with `host-token.nix` imported and `enable-agent.nix` - present in the build type. Leave `KEY` unset for now. +1. Deploy the host with its build type importing `enable-agent.nix`. Leave + `KEY` unset (commented out) for now. 2. Open the beszel hub (`http://docker.sweet.home:8090`). -3. Go to **Systems → Add system**. The new host should appear as an - unpaired entry — copy the `KEY` value shown there. +3. Go to **Systems → Add system**. The new host should appear as an unpaired + entry — copy the `KEY` value shown there. 4. In `hosts//host.nix`, set: ```nix services.beszel.agent.environment.KEY = ""; ``` -5. Rebuild and deploy the host. The agent will now pair permanently with the - hub. +5. Rebuild and deploy the host. The agent will now pair permanently. --- ## Example: complete host file -For reference, `hosts/tor-relay/host.nix` is the minimal case — one -filesystem, no Docker, LXC platform: +Minimal case (`hosts/tor-relay/host.nix` — one filesystem, no Docker, LXC): ```nix { vars, ... }: { - imports = [ - (import ../../modules/beszel/host-token.nix { - name = "tor-relay"; - sopsFile = ../../secrets/tor-relay.yaml; - }) - ]; - networking = { ... }; services.beszel.agent.environment = { @@ -180,8 +144,7 @@ filesystem, no Docker, LXC platform: } ``` -`hosts/server/host.nix` is the fuller case — extra filesystem path, debug -logging: +Fuller case (`hosts/server/host.nix` — extra filesystem, debug logging): ```nix services.beszel.agent.environment = { diff --git a/hosts/ha-server-1/host.nix b/hosts/ha-server-1/host.nix index f1b91b0..f923f5d 100644 --- a/hosts/ha-server-1/host.nix +++ b/hosts/ha-server-1/host.nix @@ -1,12 +1,5 @@ { vars, ... }: { - imports = [ - (import ../../modules/beszel/host-token.nix { - name = "ha-server-1"; - sopsFile = ../../secrets/ha-server-1.yaml; - }) - ]; - networking = { hostName = vars.haServer1Host; hostId = "3a4b5c6d"; @@ -23,7 +16,7 @@ nameservers = [ vars.domainControllerIp ]; }; - # Set KEY after pairing this host with the beszel hub; the token is sops-managed. + # Set KEY after pairing this host with the beszel hub (see docs/beszel.md). # services.beszel.agent.environment.KEY = ""; system.stateVersion = "26.05"; diff --git a/hosts/ha-server-2/host.nix b/hosts/ha-server-2/host.nix index 4439a33..5dc038a 100644 --- a/hosts/ha-server-2/host.nix +++ b/hosts/ha-server-2/host.nix @@ -1,12 +1,5 @@ { vars, ... }: { - imports = [ - (import ../../modules/beszel/host-token.nix { - name = "ha-server-2"; - sopsFile = ../../secrets/ha-server-2.yaml; - }) - ]; - networking = { hostName = vars.haServer2Host; hostId = "7e8f9a0b"; @@ -23,7 +16,7 @@ nameservers = [ vars.domainControllerIp ]; }; - # Set KEY after pairing this host with the beszel hub; the token is sops-managed. + # Set KEY after pairing this host with the beszel hub (see docs/beszel.md). # services.beszel.agent.environment.KEY = ""; system.stateVersion = "26.05"; diff --git a/hosts/nix-cache/host.nix b/hosts/nix-cache/host.nix index d314f59..e886fb5 100644 --- a/hosts/nix-cache/host.nix +++ b/hosts/nix-cache/host.nix @@ -1,13 +1,6 @@ { vars, ... }: { - imports = [ - (import ../../modules/beszel/host-token.nix { - name = "nix-cache"; - sopsFile = ../../secrets/nix-cache.yaml; - }) - ]; - networking = { hostName = vars.nixCacheHost; useDHCP = false; diff --git a/hosts/server/host.nix b/hosts/server/host.nix index 344c2d8..775e451 100644 --- a/hosts/server/host.nix +++ b/hosts/server/host.nix @@ -1,13 +1,6 @@ { vars, ... }: { - imports = [ - (import ../../modules/beszel/host-token.nix { - name = "server"; - sopsFile = ../../secrets/server.yaml; - }) - ]; - networking = { hostName = vars.nfsServerHost; hostId = "6689f93e"; diff --git a/hosts/tailscale-router/host.nix b/hosts/tailscale-router/host.nix index 6198bf4..9f3dae7 100644 --- a/hosts/tailscale-router/host.nix +++ b/hosts/tailscale-router/host.nix @@ -1,13 +1,6 @@ { vars, ... }: { - imports = [ - (import ../../modules/beszel/host-token.nix { - name = "tailscale-router"; - sopsFile = ../../secrets/tailscale-router.yaml; - }) - ]; - networking = { hostName = "tailscale-router"; useDHCP = false; diff --git a/hosts/tor-relay/host.nix b/hosts/tor-relay/host.nix index 8cdcad3..7e894ce 100644 --- a/hosts/tor-relay/host.nix +++ b/hosts/tor-relay/host.nix @@ -1,13 +1,6 @@ { vars, ... }: { - imports = [ - (import ../../modules/beszel/host-token.nix { - name = "tor-relay"; - sopsFile = ../../secrets/tor-relay.yaml; - }) - ]; - networking = { hostName = "tor-relay"; useDHCP = false; diff --git a/modules/beszel/enable-agent.nix b/modules/beszel/enable-agent.nix index 0770f74..8e97657 100644 --- a/modules/beszel/enable-agent.nix +++ b/modules/beszel/enable-agent.nix @@ -1,10 +1,22 @@ -{ vars, ... }: +{ config, vars, ... }: { - services.beszel.agent.enable = true; - services.beszel.agent.environment = { - #DOCKER_HOST = "tcp://docker-socket-proxy:2375"; - HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}"; + # Universal token shared by all beszel agents. Add to secrets/common.yaml: + # sops secrets/common.yaml + # beszel-token: + sops.secrets."beszel-token" = { }; + + sops.templates."beszel.env".content = '' + TOKEN=${config.sops.placeholder."beszel-token"} + ''; + + services.beszel.agent = { + enable = true; + environmentFile = config.sops.templates."beszel.env".path; + environment = { + #DOCKER_HOST = "tcp://docker-socket-proxy:2375"; + HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}"; + }; }; # The upstream module runs beszel-agent under DynamicUser with diff --git a/modules/beszel/host-token.nix b/modules/beszel/host-token.nix deleted file mode 100644 index 5339b4b..0000000 --- a/modules/beszel/host-token.nix +++ /dev/null @@ -1,11 +0,0 @@ -{ name, sopsFile }: - -{ config, ... }: - -{ - sops.secrets."beszel-token".sopsFile = sopsFile; - sops.templates."${name}-beszel.env".content = '' - TOKEN=${config.sops.placeholder."beszel-token"} - ''; - services.beszel.agent.environmentFile = config.sops.templates."${name}-beszel.env".path; -}