Add buildImage shell function for building lxc-* tarballs with host keys

lxc-* hosts need NIXOS_HOST_KEYS_DIR + --impure to bake in a pre-seeded
SSH host key, otherwise sops-nix's .sops.yaml recipient never matches
and every secret permanently fails to decrypt on first boot. That
invocation is easy to forget, so wrap it as `buildImage <flake-target>`
alongside the existing Switch-nix/Test-nix helpers.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CLAUDE.md

@@ -133,9 +133,10 @@ nixosSystem {
 ```
 
 Platforms: `linode`, `proxmox`, `lxc`. Build types: `minimal`, `nix-cache`,
-`server`, `docker`, `gui`, `pxe-boot`. Not every combination is built — e.g.
-`pxe-boot` has no `linode` variant (PXE/DHCP/TFTP need LAN L2 adjacency a
-Linode VPS doesn't have). Treat `flake.nix`'s `generatedTargets` as the source
+`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`. Not every
+combination is built — e.g. `pxe-boot` has no `linode` variant (PXE/DHCP/TFTP
+need LAN L2 adjacency a Linode VPS doesn't have). Treat `flake.nix`'s
+`generatedTargets` as the source
 of truth for which hosts exist — `README.md`, `AGENTS.md`,
 `docs/flake-lock-automation.md`, and the CI eval workflows
 (`.github/workflows/check-nixos.yml`, `.gitea/workflows/check-nixos.yml`) list

README.md

@@ -10,7 +10,7 @@ pieces composed in `flake.nix`:
 
 - **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`
 - **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
-  `gui`, `pxe-boot`
+  `gui`, `pxe-boot`, `tailscale-exit-node`
 
 Not every combination exists — `pxe-boot` has no `linode` variant, since
 PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have. The full
@@ -26,6 +26,7 @@ list:
 | `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack (`proxmox-docker` is the real, deployed one — previously the flat `docker` target) |
 | `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation (`proxmox-gui` is the real, deployed one — previously the flat `nixos` target) |
 | `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host (`proxmox-pxe-boot` is the real, deployed one — previously the flat `pxe-boot` target) |
+| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node (no deployed target yet; `lxc-tailscale-exit-node` is the one planned for actual use) |
 
 The "(real, deployed)" targets above are also tracked machine-readably in
 `variables.nix`'s `deployedTargets` — keep both in sync when a deployment

docs/auto-installer.md

@@ -77,11 +77,33 @@ system profile) — there's no separate activation step to run yourself.
 of this (build, host-key handling, upload, `pct create` with the flags
 above) — see its `--help`.
 
-Host keys still need pre-seeding the same way as any other host (see "Host
-keys" below) — the sops-nix activation-vs-first-boot race is identical
-regardless of how the image reaches the machine. `NIXOS_HOST_KEYS_DIR=...
-nix build ... --impure` bakes the matching key into the tarball the same way
-it does for the ISO/PXE installer images.
+Host keys still need pre-seeding the same way as any other host — the
+sops-nix activation-vs-first-boot race is identical regardless of how the
+image reaches the machine. Unlike the ISO/PXE installer (where
+`modules/installer/host-keys.nix` bakes *every* `host-keys/` entry into
+`/etc/host-keys/` for `auto-install.sh` to pick from and copy at install
+time — see "Host keys" below), an `lxc-*` tarball has no install step to
+copy anything during, so `modules/platforms/lxc.nix` bakes this *one*
+target's key straight into `/etc/ssh/ssh_host_ed25519_key(.pub)` directly,
+keyed by its own exact flake target name (`config.environment.etc` can't
+be read back from within a module still contributing to it, so this comes
+in via `specialArgs.flakeTarget`, set by `flake.nix`'s `mkTarget`):
+
+```sh
+NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
+  nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure
+```
+
+Confirmed the hard way: without this, the tarball's own built-in system
+just generates a fresh host key at first boot like any host would, which
+can never match whatever `.sops.yaml` actually trusts for that target —
+`sops-install-secrets` fails with `Error getting data key: 0 successful
+groups required, got 0`, and *every* secret (including this host's own
+login) permanently fails to decrypt, silently — no error in the boot log
+at all, since the activation step that would install secrets only runs on
+a from-scratch first activation and skips silently once `/run/current-system`
+already exists. `scripts/create-proxmox-resource.sh` always builds with
+`NIXOS_HOST_KEYS_DIR` set for this reason.
 
 ## Layout
 

flake.nix

@@ -33,6 +33,9 @@
       # nix-cache itself consumes the nix-cache substituter and remote
       # builder.
       mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix }:
+        let
+          flakeTarget = "${platform}-${buildType}";
+        in
         nixpkgs.lib.nixosSystem {
           inherit system;
           modules = [
@@ -42,7 +45,7 @@
             ./modules/platforms/${platform}.nix
             ./modules/build-types/${buildType}.nix
             hostPath
-            { environment.etc."flake-target".text = "${platform}-${buildType}"; }
+            { environment.etc."flake-target".text = flakeTarget; }
             home-manager.nixosModules.home-manager
             {
               home-manager = {
@@ -56,7 +59,13 @@
             ./modules/nix-cache/client.nix
             ./modules/nix-cache/remote-builder-client.nix
           ];
-          specialArgs = { inherit inputs vars netbootSystem; };
+          # flakeTarget is passed via specialArgs (not read back from
+          # config.environment.etc."flake-target" above) specifically so
+          # modules/platforms/lxc.nix can use it to select its own host key
+          # file without a same-option circular dependency (a module
+          # contributing to environment.etc can't read the merged
+          # environment.etc it's itself contributing to).
+          specialArgs = { inherit inputs vars netbootSystem flakeTarget; };
         };
 
       # Generated platform x build-type matrix. pxe-boot has no linode
@@ -85,6 +94,10 @@
 
         proxmox-pxe-boot = mkTarget { platform = "proxmox"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
         lxc-pxe-boot = mkTarget { platform = "lxc"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
+
+        linode-tailscale-exit-node = mkTarget { platform = "linode"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
+        proxmox-tailscale-exit-node = mkTarget { platform = "proxmox"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
+        lxc-tailscale-exit-node = mkTarget { platform = "lxc"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
       };
 
       # Auto-install environments (migrated from the former nix-auto-installer

hosts/tailscale-exit-node/host.nix

@@ -0,0 +1,12 @@
+_:
+
+{
+  networking.hostName = "exit-node";
+
+  # No networking.hostId: only ZFS-touching hosts (server, docker) need one
+  # for pool-import safety, and this host does neither.
+
+  # A genuinely new host (not a pre-refactor carry-over), so it tracks the
+  # flake's current nixpkgs release rather than being pinned to an older one.
+  system.stateVersion = "26.05";
+}

modules/beszel/enable-agent.nix

@@ -6,4 +6,13 @@
     #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
     HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
   };
+
+  # The upstream module runs beszel-agent under DynamicUser with
+  # ProtectSystem = "strict" and no StateDirectory, so /var/lib/beszel-agent
+  # (where the agent persists its hub-pairing fingerprint, per
+  # https://github.com/henrygd/beszel/discussions/1542) isn't writable --
+  # every restart silently fails to save it and regenerates a fresh one in
+  # memory, permanently desyncing from whatever the hub has on record after
+  # the very first successful pairing. Give it real persistent storage.
+  systemd.services.beszel-agent.serviceConfig.StateDirectory = "beszel-agent";
 }

modules/build-types/tailscale-exit-node.nix

@@ -0,0 +1,21 @@
+{ ... }:
+
+{
+  imports = [
+    ../tailscale/exit-node.nix
+  ];
+
+  # "server", not "both": this build type only ever advertises itself as an
+  # exit node (see ../tailscale/exit-node.nix) -- it doesn't advertise LAN
+  # subnet routes, so it doesn't need the "client"-side loose reverse-path
+  # filtering that "both" would also turn on. Deliberately left unbundled
+  # from LAN-subnet-route advertisement so this build type stays valid on
+  # every platform, including linode (a remote VPS with no network path to
+  # the home LAN at all).
+  services.tailscale.useRoutingFeatures = "server";
+
+  # Forwarded exit-node traffic arrives on tailscale0 already
+  # tailscale-authenticated -- the firewall's normal per-port allow-list
+  # would otherwise drop it. Standard NixOS/Tailscale exit-node guidance.
+  networking.firewall.trustedInterfaces = [ "tailscale0" ];
+}

modules/common/aliases.nix

@@ -17,6 +17,26 @@ let
       --refresh \
       --flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
   '';
+
+  # lxc-* hosts pre-seed their SSH host key at build time (see
+  # modules/platforms/lxc.nix) so sops-nix's .sops.yaml recipient matches on
+  # first boot -- without it, secrets permanently fail to decrypt (see that
+  # file's comment for the confirmed failure). That requires --impure plus
+  # NIXOS_HOST_KEYS_DIR pointing at the repo's host-keys/ dir, same pattern
+  # docs/auto-installer.md uses for the installer ISO. A function, not a
+  # shellAlias, since the target name has to interpolate into the middle of
+  # the flake attribute path, not just append after it. Must be run from the
+  # repo root, same as every other host-keys/ command in this repo.
+  buildImageFn = ''
+    buildImage() {
+      if [ -z "$1" ]; then
+        echo "usage: buildImage <flake-target>  (e.g. lxc-docker)" >&2
+        return 1
+      fi
+      NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
+        ".#nixosConfigurations.$1.config.system.build.tarball"
+    }
+  '';
 in
 {
   programs.bash = {
@@ -25,5 +45,6 @@ in
       "Switch-nix" = mySwitchCmd;
       "Test-nix" = myTestCmd;
     };
+    initExtra = buildImageFn;
   };
 }

modules/platforms/lxc.nix

@@ -1,5 +1,44 @@
-{ lib, modulesPath, ... }:
+{ lib, modulesPath, flakeTarget, ... }:
 
+let
+  # Bakes this exact flake target's pre-generated SSH host key straight
+  # into /etc/ssh/ -- mirrors modules/installer/host-keys.nix's
+  # builtins.getEnv pattern (impure and empty under normal `nix
+  # build`/`nix eval`, so this is a no-op unless explicitly opted into
+  # with NIXOS_HOST_KEYS_DIR=... --impure), but places the key directly
+  # rather than staging it under /etc/host-keys/ for a later manual copy
+  # -- this is the whole system for a `lxc-*` host, built straight to a
+  # pct-restorable tarball with no install step, so there's no later copy
+  # step to stage for.
+  #
+  # Without this, config.system.build.tarball's built-in system just
+  # generates a fresh host key at first boot like any other host would --
+  # but sops-nix derives its decryption key from *this* file, and
+  # .sops.yaml only trusts whatever key scripts/sync-host-keys.sh already
+  # registered for this exact target name. A freshly-generated key can
+  # never match that, so every secret (including this host's own login)
+  # permanently fails to decrypt. Confirmed live: sops-install-secrets
+  # errored with "Error getting data key: 0 successful groups required,
+  # got 0" -- the container's actual host key's age fingerprint didn't
+  # match the one registered in .sops.yaml at all.
+  hostKeysDirStr = builtins.getEnv "NIXOS_HOST_KEYS_DIR";
+  hasHostKeysDir = hostKeysDirStr != "" && builtins.pathExists hostKeysDirStr;
+  hostKeysDir = /. + hostKeysDirStr;
+
+  # flakeTarget ("${platform}-${buildType}") comes in via specialArgs from
+  # flake.nix's mkTarget -- exactly the name scripts/sync-host-keys.sh
+  # registers keys under. Deliberately not read back from
+  # config.environment.etc."flake-target" (which is set to the same value)
+  # -- this module also *contributes* to environment.etc below, and a
+  # module reading the merged value of an option it's still defining is a
+  # circular dependency (confirmed: "infinite recursion encountered").
+  privKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key";
+  pubKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key.pub";
+  hasKeyForThisTarget =
+    hasHostKeysDir
+    && builtins.pathExists privKeyFile
+    && builtins.pathExists pubKeyFile;
+in
 {
   # LXC containers share the host kernel — Proxmox starts them by exec'ing
   # /sbin/init directly, no bootloader/initrd involved — and Proxmox has its
@@ -35,4 +74,75 @@
   # for the same reason; it just doesn't disable NetworkManager itself,
   # which modules/common/configuration.nix enables for every host.
   networking.networkmanager.enable = lib.mkForce false;
+
+  environment.etc = lib.mkIf hasKeyForThisTarget {
+    "ssh/ssh_host_ed25519_key" = {
+      source = privKeyFile;
+      mode = "0600";
+    };
+    "ssh/ssh_host_ed25519_key.pub" = {
+      source = pubKeyFile;
+      mode = "0644";
+    };
+  };
+
+  # virtualisation/proxmox-lxc.nix (imported above) registers the Nix
+  # store DB via a systemd service (register-nix-paths) -- it never runs
+  # an activation script at all. Confirmed live this means neither
+  # sops-nix's "for users" secrets (password hashes -- installed by the
+  # activation script itself, not a systemd service, since they need to
+  # exist *before* user creation) nor the user-creation step that
+  # consumes them ever run on a real lxc-* boot. Regular secrets
+  # (nix-serve's key, beszel's token, etc.) work anyway because sops-nix
+  # provides its own systemd service for those.
+  #
+  # A systemd service, not boot.postBootCommands: tried that first (it's
+  # a genuine, generally-invoked hook -- nixos/modules/system/boot/stage-2-init.sh,
+  # which becomes this container's actual /sbin/init, unconditionally
+  # runs it) but switch-to-configuration behaves differently that early in
+  # boot (raw stage-2-init.sh, before systemd itself has even started) --
+  # confirmed live it silently failed to rewrite /etc/shadow from there
+  # even in "test" mode, despite the exact same command working reliably
+  # every time when run post-boot (i.e. as a normal systemd service, which
+  # is what this is). Not fully root-caused why the early context
+  # specifically breaks it; a real systemd service sidesteps needing to.
+  #
+  # /etc/shadow already has PLACEHOLDER entries for every declared user
+  # baked in at build time (part of constructing the system closure).
+  # update-users-groups.pl deliberately never overwrites an *existing*
+  # shadow entry -- a correct safety property in general (don't clobber a
+  # real user's real password on a config rebuild) -- but on a genuine
+  # first boot that only means the real hashedPasswordFile-derived hash
+  # never gets the chance to be applied either, since the placeholder is
+  # already "seen". Safe to clear here specifically: there is no real
+  # password yet to protect on a first boot.
+  #
+  # "test" mode, not "boot": confirmed live "boot" mode aborts partway
+  # through (before rewriting /etc/shadow) on a warning that "/boot" is on
+  # a different filesystem -- a real check for a host with a bootloader to
+  # update, meaningless for a container that has none
+  # (boot.loader.{grub,systemd-boot}.enable are both false above), but it
+  # still aborts the script. "test" runs every activation step without
+  # touching boot-loader state at all.
+  #
+  # ConditionPathExists (systemd-native, not a bash-level check) means
+  # this only ever runs once, on the genuine first boot -- systemd itself
+  # skips even starting it on every later boot once the marker exists.
+  # switch-to-configuration is otherwise the operator's call per this
+  # repo's own safety rules, not something to run on every boot.
+  systemd.services.nixos-lxc-first-boot-activate = {
+    description = "Complete first-boot NixOS activation (users, secrets) for this LXC container";
+    wantedBy = [ "multi-user.target" ];
+    unitConfig.ConditionPathExists = "!/var/lib/nixos-lxc-first-boot-activated";
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+    };
+    script = ''
+      rm -f /etc/shadow
+      /run/current-system/bin/switch-to-configuration test
+      mkdir -p /var/lib
+      touch /var/lib/nixos-lxc-first-boot-activated
+    '';
+  };
 }

modules/tailscale/exit-node.nix

@@ -1,12 +1,19 @@
-{ vars, ... }:
+_:
 
 {
   services.tailscale = {
     enable = true;
 
-    extraUpFlags = [
+    # extraSetFlags (tailscale set, via the always-on tailscaled-set
+    # service), not extraUpFlags -- extraUpFlags is only ever applied by
+    # tailscaled-autoconnect, which itself only runs when
+    # services.tailscale.authKeyFile is set (nothing in this repo sets one,
+    # so tailscale up is a manual, one-time operator step on every host that
+    # uses this service). extraSetFlags has no such gate, so
+    # --advertise-exit-node self-reapplies on every boot once the operator
+    # has authenticated the node once.
+    extraSetFlags = [
       "--advertise-exit-node"
-      "--advertise-routes=${vars.lanCidr}"
     ];
   };
 }

scripts/create-proxmox-resource.sh

@@ -40,8 +40,12 @@ Create mode (default):
                         config.networking.hostName (server, docker,
                         nix-cache, nixos, pxe-boot, nix-minimal). Use
                         --list to see what's available for --type.
-  --name <name>          Proxmox display name/hostname (default: the flake
-                          target name, e.g. lxc-server)
+  --name <name>          Proxmox display name/hostname (default: --host's
+                          value, e.g. nix-cache -- for lxc this becomes the
+                          guest's real networking.hostName too, since
+                          proxmoxLXC.manageHostName pulls it from Proxmox's
+                          own container config, so it must match host.nix
+                          regardless of build type)
   --vmid <n>              Numeric VMID (default: next free, via
                           \`pvesh get /cluster/nextid\` on the node).
                           Refuses to run if this ID already exists.
@@ -274,7 +278,13 @@ if [[ -z "$flake_target" ]]; then
   exit 1
 fi
 
-[[ -z "$name" ]] && name="$flake_target"
+# The container/VM's real identity is --host (e.g. "nix-cache"), validated
+# above against config.networking.hostName -- not the flake target name
+# (e.g. "lxc-nix-cache"), which is build-type-specific and only exists to
+# pick which platform variant to build. Defaulting --name to the flake
+# target would make lxc's --hostname (which proxmoxLXC.manageHostName
+# feeds straight into the guest's real hostname) disagree with host.nix.
+[[ -z "$name" ]] && name="$host"
 
 # --- refuse to duplicate a host that's already really deployed ----------
 # Checked by hostName, not exact flake target: proxmox-server being
@@ -383,9 +393,19 @@ else
 fi
 
 if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
+  # Mirrors the real build commands' "${NIX_OPTS[@]}" below -- nix_extra_opts
+  # (called earlier, once) has already decided whether nix-cache is in play,
+  # and the dry-run preview needs to reflect that decision instead of always
+  # printing the same command regardless of outcome.
+  nix_opts_display=""
+  if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
+    printf -v nix_opts_display '%q ' "${NIX_OPTS[@]}"
+    nix_opts_display=" ${nix_opts_display% }"
+  fi
   if [[ "$type" == "lxc" ]]; then
     if [[ "$dry_run" -eq 1 ]]; then
       echo "[dry-run] would build: NIXOS_HOST_KEYS_DIR=${repo_root}/host-keys nix build --impure \\"
+      echo "[dry-run]   --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
       echo "[dry-run]   .#nixosConfigurations.${flake_target}.config.system.build.tarball"
       local_image="<built-tarball>"
     else
@@ -399,7 +419,8 @@ if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
     fi
   else
     if [[ "$dry_run" -eq 1 ]]; then
-      echo "[dry-run] would build: nix build .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
+      echo "[dry-run] would build: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
+      echo "[dry-run]   .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
       echo "[dry-run] would run:   sudo ./result-${flake_target} \\"
       echo "[dry-run]                --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\"
       echo "[dry-run]                --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\"