Fix duplicate-host check reporting false SSH failures

The remote bash script run over SSH ended with a for-loop whose last
statement was `[[ "$n" == "$target" ]] && echo ...`. When the last
VM/CT checked on the node didn't match --host, that test evaluated
false and became the exit status of the whole remote script (1) --
which the wrapper then misreported as "couldn't reach the node",
even though SSH connectivity and the check itself were both fine.
The actual signal is the script's stdout, not its exit code, so end
it with an explicit exit 0.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CLAUDE.md

@@ -93,9 +93,10 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
   (`--force-rebuild` to skip that and always rebuild), and probes
   nix-cache's substituter/remote-builder reachability once up front rather
   than letting every `nix build` call retry against it individually.
-  Refuses to create a target whose host identity already has a real
+  Refuses to create a target whose host identity already exists live on
-  deployment elsewhere (`variables.nix`'s `deployedTargets`) unless
+  the node (checked directly via `qm`/`pct`, not any file in this repo)
-  `--allow-duplicate-host` is passed. `--dry-run` throughout both modes.
+  unless `--allow-duplicate-host` is passed. `--dry-run` throughout both
+  modes.
 - `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
   default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
   cross-script config here instead of duplicating it per-script.
@@ -133,9 +134,10 @@ nixosSystem {
 ```
 
 Platforms: `linode`, `proxmox`, `lxc`. Build types: `minimal`, `nix-cache`,
-`server`, `docker`, `gui`, `pxe-boot`. Not every combination is built — e.g.
+`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`. Not every
-`pxe-boot` has no `linode` variant (PXE/DHCP/TFTP need LAN L2 adjacency a
+combination is built — e.g. `pxe-boot` has no `linode` variant (PXE/DHCP/TFTP
-Linode VPS doesn't have). Treat `flake.nix`'s `generatedTargets` as the source
+need LAN L2 adjacency a Linode VPS doesn't have). Treat `flake.nix`'s
+`generatedTargets` as the source
 of truth for which hosts exist — `README.md`, `AGENTS.md`,
 `docs/flake-lock-automation.md`, and the CI eval workflows
 (`.github/workflows/check-nixos.yml`, `.gitea/workflows/check-nixos.yml`) list

README.md

@@ -10,7 +10,7 @@ pieces composed in `flake.nix`:
 
 - **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`
 - **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
-  `gui`, `pxe-boot`
+  `gui`, `pxe-boot`, `tailscale-exit-node`
 
 Not every combination exists — `pxe-boot` has no `linode` variant, since
 PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have. The full
@@ -18,19 +18,23 @@ list:
 
 | Target | Purpose |
 | --- | --- |
-| `linode-minimal` | Minimal NixOS host profile on a Linode VPS (real, deployed) |
+| `linode-minimal` | Minimal NixOS host profile on a Linode VPS |
-| `proxmox-minimal` | Minimal NixOS host profile on Proxmox (real, deployed — previously the flat `nix-minimal` target) |
+| `proxmox-minimal` | Minimal NixOS host profile on Proxmox — previously the flat `nix-minimal` target |
 | `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container |
-| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder (`proxmox-nix-cache` is the real, deployed one — previously the flat `nix-cache` target) |
+| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder — previously the flat `nix-cache` target |
-| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host (`proxmox-server` is the real, deployed one — previously the flat `server` target) |
+| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host — previously the flat `server` target |
-| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack (`proxmox-docker` is the real, deployed one — previously the flat `docker` target) |
+| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack — previously the flat `docker` target |
-| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation (`proxmox-gui` is the real, deployed one — previously the flat `nixos` target) |
+| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target |
-| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host (`proxmox-pxe-boot` is the real, deployed one — previously the flat `pxe-boot` target) |
+| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host — previously the flat `pxe-boot` target |
+| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node |
 
-The "(real, deployed)" targets above are also tracked machine-readably in
+Which variant of a given buildtype is actually deployed isn't tracked
-`variables.nix`'s `deployedTargets` — keep both in sync when a deployment
+anywhere in this repo — that's live infrastructure state, not something a
-changes. `scripts/create-proxmox-resource.sh` reads that list to refuse
+committed file can keep accurate, and it changes independently of the code.
-creating a same-identity duplicate of an already-deployed host by accident.
+Check the Proxmox node itself, or `/etc/flake-target` on a running host (see
+below), if you need to know what's really out there right now.
+`scripts/create-proxmox-resource.sh`'s duplicate-host guard works the same
+way: it checks the Proxmox node directly rather than any file here.
 
 Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
 (hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay

docs/auto-installer.md

@@ -77,11 +77,33 @@ system profile) — there's no separate activation step to run yourself.
 of this (build, host-key handling, upload, `pct create` with the flags
 above) — see its `--help`.
 
-Host keys still need pre-seeding the same way as any other host (see "Host
+Host keys still need pre-seeding the same way as any other host — the
-keys" below) — the sops-nix activation-vs-first-boot race is identical
+sops-nix activation-vs-first-boot race is identical regardless of how the
-regardless of how the image reaches the machine. `NIXOS_HOST_KEYS_DIR=...
+image reaches the machine. Unlike the ISO/PXE installer (where
-nix build ... --impure` bakes the matching key into the tarball the same way
+`modules/installer/host-keys.nix` bakes *every* `host-keys/` entry into
-it does for the ISO/PXE installer images.
+`/etc/host-keys/` for `auto-install.sh` to pick from and copy at install
+time — see "Host keys" below), an `lxc-*` tarball has no install step to
+copy anything during, so `modules/platforms/lxc.nix` bakes this *one*
+target's key straight into `/etc/ssh/ssh_host_ed25519_key(.pub)` directly,
+keyed by its own exact flake target name (`config.environment.etc` can't
+be read back from within a module still contributing to it, so this comes
+in via `specialArgs.flakeTarget`, set by `flake.nix`'s `mkTarget`):
+
+```sh
+NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
+  nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure
+```
+
+Confirmed the hard way: without this, the tarball's own built-in system
+just generates a fresh host key at first boot like any host would, which
+can never match whatever `.sops.yaml` actually trusts for that target —
+`sops-install-secrets` fails with `Error getting data key: 0 successful
+groups required, got 0`, and *every* secret (including this host's own
+login) permanently fails to decrypt, silently — no error in the boot log
+at all, since the activation step that would install secrets only runs on
+a from-scratch first activation and skips silently once `/run/current-system`
+already exists. `scripts/create-proxmox-resource.sh` always builds with
+`NIXOS_HOST_KEYS_DIR` set for this reason.
 
 ## Layout
 

docs/nix-cache.md

@@ -59,6 +59,15 @@ On `nix-cache`, install the matching public key used by `nixremote` authorized k
 The committed `nixremote` authorized keys are public SSH keys only. Keep the
 matching private keys on client hosts and out of the repository.
 
+nix-cache's own SSH *host* key is trusted declaratively via
+`programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
+sourced from `vars.nixCacheHostKey` (`variables.nix`) — every client rebuild
+picks it up automatically, so distributed builds don't fail with "Host key
+verification failed" on a client that has never manually SSH'd to nix-cache
+before. If nix-cache's host key is ever rotated or the host rebuilt from
+scratch, update `vars.nixCacheHostKey` to match its new
+`/etc/ssh/ssh_host_ed25519_key.pub`.
+
 ## Manual verification
 
 After deployment:

flake.nix

@@ -33,6 +33,9 @@
       # nix-cache itself consumes the nix-cache substituter and remote
       # builder.
       mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix }:
+        let
+          flakeTarget = "${platform}-${buildType}";
+        in
         nixpkgs.lib.nixosSystem {
           inherit system;
           modules = [
@@ -42,7 +45,7 @@
             ./modules/platforms/${platform}.nix
             ./modules/build-types/${buildType}.nix
             hostPath
-            { environment.etc."flake-target".text = "${platform}-${buildType}"; }
+            { environment.etc."flake-target".text = flakeTarget; }
             home-manager.nixosModules.home-manager
             {
               home-manager = {
@@ -56,7 +59,13 @@
             ./modules/nix-cache/client.nix
             ./modules/nix-cache/remote-builder-client.nix
           ];
-          specialArgs = { inherit inputs vars netbootSystem; };
+          # flakeTarget is passed via specialArgs (not read back from
+          # config.environment.etc."flake-target" above) specifically so
+          # modules/platforms/lxc.nix can use it to select its own host key
+          # file without a same-option circular dependency (a module
+          # contributing to environment.etc can't read the merged
+          # environment.etc it's itself contributing to).
+          specialArgs = { inherit inputs vars netbootSystem flakeTarget; };
         };
 
       # Generated platform x build-type matrix. pxe-boot has no linode
@@ -85,6 +94,10 @@
 
         proxmox-pxe-boot = mkTarget { platform = "proxmox"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
         lxc-pxe-boot = mkTarget { platform = "lxc"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
+
+        linode-tailscale-exit-node = mkTarget { platform = "linode"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
+        proxmox-tailscale-exit-node = mkTarget { platform = "proxmox"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
+        lxc-tailscale-exit-node = mkTarget { platform = "lxc"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
       };
 
       # Auto-install environments (migrated from the former nix-auto-installer

hosts/tailscale-exit-node/host.nix

@@ -0,0 +1,12 @@
+_:
+
+{
+  networking.hostName = "exit-node";
+
+  # No networking.hostId: only ZFS-touching hosts (server, docker) need one
+  # for pool-import safety, and this host does neither.
+
+  # A genuinely new host (not a pre-refactor carry-over), so it tracks the
+  # flake's current nixpkgs release rather than being pinned to an older one.
+  system.stateVersion = "26.05";
+}

modules/beszel/enable-agent.nix

@@ -6,4 +6,13 @@
     #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
     HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
   };
+
+  # The upstream module runs beszel-agent under DynamicUser with
+  # ProtectSystem = "strict" and no StateDirectory, so /var/lib/beszel-agent
+  # (where the agent persists its hub-pairing fingerprint, per
+  # https://github.com/henrygd/beszel/discussions/1542) isn't writable --
+  # every restart silently fails to save it and regenerates a fresh one in
+  # memory, permanently desyncing from whatever the hub has on record after
+  # the very first successful pairing. Give it real persistent storage.
+  systemd.services.beszel-agent.serviceConfig.StateDirectory = "beszel-agent";
 }

modules/build-types/tailscale-exit-node.nix

@@ -0,0 +1,21 @@
+{ ... }:
+
+{
+  imports = [
+    ../tailscale/exit-node.nix
+  ];
+
+  # "server", not "both": this build type only ever advertises itself as an
+  # exit node (see ../tailscale/exit-node.nix) -- it doesn't advertise LAN
+  # subnet routes, so it doesn't need the "client"-side loose reverse-path
+  # filtering that "both" would also turn on. Deliberately left unbundled
+  # from LAN-subnet-route advertisement so this build type stays valid on
+  # every platform, including linode (a remote VPS with no network path to
+  # the home LAN at all).
+  services.tailscale.useRoutingFeatures = "server";
+
+  # Forwarded exit-node traffic arrives on tailscale0 already
+  # tailscale-authenticated -- the firewall's normal per-port allow-list
+  # would otherwise drop it. Standard NixOS/Tailscale exit-node guidance.
+  networking.firewall.trustedInterfaces = [ "tailscale0" ];
+}

modules/common/aliases.nix

@@ -17,6 +17,26 @@ let
       --refresh \
       --flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
   '';
+
+  # lxc-* hosts pre-seed their SSH host key at build time (see
+  # modules/platforms/lxc.nix) so sops-nix's .sops.yaml recipient matches on
+  # first boot -- without it, secrets permanently fail to decrypt (see that
+  # file's comment for the confirmed failure). That requires --impure plus
+  # NIXOS_HOST_KEYS_DIR pointing at the repo's host-keys/ dir, same pattern
+  # docs/auto-installer.md uses for the installer ISO. A function, not a
+  # shellAlias, since the target name has to interpolate into the middle of
+  # the flake attribute path, not just append after it. Must be run from the
+  # repo root, same as every other host-keys/ command in this repo.
+  buildImageFn = ''
+    buildImage() {
+      if [ -z "$1" ]; then
+        echo "usage: buildImage <flake-target>  (e.g. lxc-docker)" >&2
+        return 1
+      fi
+      NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
+        ".#nixosConfigurations.$1.config.system.build.tarball"
+    }
+  '';
 in
 {
   programs.bash = {
@@ -25,5 +45,6 @@ in
       "Switch-nix" = mySwitchCmd;
       "Test-nix" = myTestCmd;
     };
+    initExtra = buildImageFn;
   };
 }

modules/nix-cache/remote-builder-client.nix

@@ -5,6 +5,14 @@
   #   sudo install -d -m 0700 /root/.ssh
   #   sudo install -m 0600 ./nixremote /root/.ssh/nixremote
   #   sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
+  # Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
+  # can connect the first time without a manual ssh-keyscan/known_hosts
+  # step on every new client.
+  programs.ssh.knownHosts.${vars.nixCacheHost} = {
+    hostNames = [ vars.nixCacheHost ];
+    publicKey = vars.nixCacheHostKey;
+  };
+
   nix = {
     distributedBuilds = true;
 

modules/platforms/lxc.nix

@@ -1,5 +1,44 @@
-{ lib, modulesPath, ... }:
+{ lib, modulesPath, flakeTarget, ... }:
 
+let
+  # Bakes this exact flake target's pre-generated SSH host key straight
+  # into /etc/ssh/ -- mirrors modules/installer/host-keys.nix's
+  # builtins.getEnv pattern (impure and empty under normal `nix
+  # build`/`nix eval`, so this is a no-op unless explicitly opted into
+  # with NIXOS_HOST_KEYS_DIR=... --impure), but places the key directly
+  # rather than staging it under /etc/host-keys/ for a later manual copy
+  # -- this is the whole system for a `lxc-*` host, built straight to a
+  # pct-restorable tarball with no install step, so there's no later copy
+  # step to stage for.
+  #
+  # Without this, config.system.build.tarball's built-in system just
+  # generates a fresh host key at first boot like any other host would --
+  # but sops-nix derives its decryption key from *this* file, and
+  # .sops.yaml only trusts whatever key scripts/sync-host-keys.sh already
+  # registered for this exact target name. A freshly-generated key can
+  # never match that, so every secret (including this host's own login)
+  # permanently fails to decrypt. Confirmed live: sops-install-secrets
+  # errored with "Error getting data key: 0 successful groups required,
+  # got 0" -- the container's actual host key's age fingerprint didn't
+  # match the one registered in .sops.yaml at all.
+  hostKeysDirStr = builtins.getEnv "NIXOS_HOST_KEYS_DIR";
+  hasHostKeysDir = hostKeysDirStr != "" && builtins.pathExists hostKeysDirStr;
+  hostKeysDir = /. + hostKeysDirStr;
+
+  # flakeTarget ("${platform}-${buildType}") comes in via specialArgs from
+  # flake.nix's mkTarget -- exactly the name scripts/sync-host-keys.sh
+  # registers keys under. Deliberately not read back from
+  # config.environment.etc."flake-target" (which is set to the same value)
+  # -- this module also *contributes* to environment.etc below, and a
+  # module reading the merged value of an option it's still defining is a
+  # circular dependency (confirmed: "infinite recursion encountered").
+  privKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key";
+  pubKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key.pub";
+  hasKeyForThisTarget =
+    hasHostKeysDir
+    && builtins.pathExists privKeyFile
+    && builtins.pathExists pubKeyFile;
+in
 {
   # LXC containers share the host kernel — Proxmox starts them by exec'ing
   # /sbin/init directly, no bootloader/initrd involved — and Proxmox has its
@@ -35,4 +74,75 @@
   # for the same reason; it just doesn't disable NetworkManager itself,
   # which modules/common/configuration.nix enables for every host.
   networking.networkmanager.enable = lib.mkForce false;
+
+  environment.etc = lib.mkIf hasKeyForThisTarget {
+    "ssh/ssh_host_ed25519_key" = {
+      source = privKeyFile;
+      mode = "0600";
+    };
+    "ssh/ssh_host_ed25519_key.pub" = {
+      source = pubKeyFile;
+      mode = "0644";
+    };
+  };
+
+  # virtualisation/proxmox-lxc.nix (imported above) registers the Nix
+  # store DB via a systemd service (register-nix-paths) -- it never runs
+  # an activation script at all. Confirmed live this means neither
+  # sops-nix's "for users" secrets (password hashes -- installed by the
+  # activation script itself, not a systemd service, since they need to
+  # exist *before* user creation) nor the user-creation step that
+  # consumes them ever run on a real lxc-* boot. Regular secrets
+  # (nix-serve's key, beszel's token, etc.) work anyway because sops-nix
+  # provides its own systemd service for those.
+  #
+  # A systemd service, not boot.postBootCommands: tried that first (it's
+  # a genuine, generally-invoked hook -- nixos/modules/system/boot/stage-2-init.sh,
+  # which becomes this container's actual /sbin/init, unconditionally
+  # runs it) but switch-to-configuration behaves differently that early in
+  # boot (raw stage-2-init.sh, before systemd itself has even started) --
+  # confirmed live it silently failed to rewrite /etc/shadow from there
+  # even in "test" mode, despite the exact same command working reliably
+  # every time when run post-boot (i.e. as a normal systemd service, which
+  # is what this is). Not fully root-caused why the early context
+  # specifically breaks it; a real systemd service sidesteps needing to.
+  #
+  # /etc/shadow already has PLACEHOLDER entries for every declared user
+  # baked in at build time (part of constructing the system closure).
+  # update-users-groups.pl deliberately never overwrites an *existing*
+  # shadow entry -- a correct safety property in general (don't clobber a
+  # real user's real password on a config rebuild) -- but on a genuine
+  # first boot that only means the real hashedPasswordFile-derived hash
+  # never gets the chance to be applied either, since the placeholder is
+  # already "seen". Safe to clear here specifically: there is no real
+  # password yet to protect on a first boot.
+  #
+  # "test" mode, not "boot": confirmed live "boot" mode aborts partway
+  # through (before rewriting /etc/shadow) on a warning that "/boot" is on
+  # a different filesystem -- a real check for a host with a bootloader to
+  # update, meaningless for a container that has none
+  # (boot.loader.{grub,systemd-boot}.enable are both false above), but it
+  # still aborts the script. "test" runs every activation step without
+  # touching boot-loader state at all.
+  #
+  # ConditionPathExists (systemd-native, not a bash-level check) means
+  # this only ever runs once, on the genuine first boot -- systemd itself
+  # skips even starting it on every later boot once the marker exists.
+  # switch-to-configuration is otherwise the operator's call per this
+  # repo's own safety rules, not something to run on every boot.
+  systemd.services.nixos-lxc-first-boot-activate = {
+    description = "Complete first-boot NixOS activation (users, secrets) for this LXC container";
+    wantedBy = [ "multi-user.target" ];
+    unitConfig.ConditionPathExists = "!/var/lib/nixos-lxc-first-boot-activated";
+    serviceConfig = {
+      Type = "oneshot";
+      RemainAfterExit = true;
+    };
+    script = ''
+      rm -f /etc/shadow
+      /run/current-system/bin/switch-to-configuration test
+      mkdir -p /var/lib
+      touch /var/lib/nixos-lxc-first-boot-activated
+    '';
+  };
 }

modules/tailscale/exit-node.nix

@@ -1,12 +1,19 @@
-{ vars, ... }:
+_:
 
 {
   services.tailscale = {
     enable = true;
 
-    extraUpFlags = [
+    # extraSetFlags (tailscale set, via the always-on tailscaled-set
+    # service), not extraUpFlags -- extraUpFlags is only ever applied by
+    # tailscaled-autoconnect, which itself only runs when
+    # services.tailscale.authKeyFile is set (nothing in this repo sets one,
+    # so tailscale up is a manual, one-time operator step on every host that
+    # uses this service). extraSetFlags has no such gate, so
+    # --advertise-exit-node self-reapplies on every boot once the operator
+    # has authenticated the node once.
+    extraSetFlags = [
       "--advertise-exit-node"
-      "--advertise-routes=${vars.lanCidr}"
     ];
   };
 }

scripts/create-proxmox-resource.sh

@@ -10,7 +10,9 @@
 #
 # SAFETY:
 #   - The default (create) mode only ever creates a NEW resource -- it
-#     refuses to run if the target VMID already exists on the node.
+#     refuses to run if the target VMID already exists on the node, or if
+#     a VM/CT identified as --host already exists under any other VMID
+#     (checked live against the node; --allow-duplicate-host overrides).
 #   - --modify only ever touches a resource you name explicitly via
 #     --vmid, shows exactly what will change first, and (outside
 #     --dry-run) always requires typing that VMID back to confirm before
@@ -40,8 +42,12 @@ Create mode (default):
                         config.networking.hostName (server, docker,
                         nix-cache, nixos, pxe-boot, nix-minimal). Use
                         --list to see what's available for --type.
-  --name <name>          Proxmox display name/hostname (default: the flake
+  --name <name>          Proxmox display name/hostname (default: --host's
-                          target name, e.g. lxc-server)
+                          value, e.g. nix-cache -- for lxc this becomes the
+                          guest's real networking.hostName too, since
+                          proxmoxLXC.manageHostName pulls it from Proxmox's
+                          own container config, so it must match host.nix
+                          regardless of build type)
   --vmid <n>              Numeric VMID (default: next free, via
                           \`pvesh get /cluster/nextid\` on the node).
                           Refuses to run if this ID already exists.
@@ -52,10 +58,11 @@ Create mode (default):
   --force-rebuild                Skip the "does the node already have this
                                   image" check -- always build fresh and
                                   overwrite what's there.
-  --allow-duplicate-host        Required if --host already has a real
+  --allow-duplicate-host        Required if a VM/CT identified as --host
-                                deployment elsewhere (variables.nix's
+                                already exists on the node (checked live via
-                                deployedTargets) -- otherwise refused, since
+                                qm/pct, not any file in this repo) --
-                                it'd share that host's hostName/hostId.
+                                otherwise refused, since it'd share that
+                                host's hostName/hostId.
 
 Modify mode (reconfigure an EXISTING resource -- requires --modify):
   --modify                Switch to modify mode.
@@ -274,27 +281,66 @@ if [[ -z "$flake_target" ]]; then
   exit 1
 fi
 
-[[ -z "$name" ]] && name="$flake_target"
+# The container/VM's real identity is --host (e.g. "nix-cache"), validated
+# above against config.networking.hostName -- not the flake target name
+# (e.g. "lxc-nix-cache"), which is build-type-specific and only exists to
+# pick which platform variant to build. Defaulting --name to the flake
+# target would make lxc's --hostname (which proxmoxLXC.manageHostName
+# feeds straight into the guest's real hostname) disagree with host.nix.
+[[ -z "$name" ]] && name="$host"
 
-# --- refuse to duplicate a host that's already really deployed ----------
+# --- refuse to duplicate a host that's already live on the node ---------
-# Checked by hostName, not exact flake target: proxmox-server being
+# Queries the node itself (qm/pct's own name/hostname config), not any
-# deployed also blocks --type lxc --host server, since both would carry
+# static list in this repo -- a file can't track whether a resource still
-# the same hosts/server/host.nix identity (hostName, hostId).
+# actually exists, and this used to be checked against variables.nix's
-if [[ "$allow_duplicate_host" -eq 0 ]]; then
+# deployedTargets, which drifted stale (it kept naming a VM as "the real
-  deployed_targets_json="$(nix eval --json --no-use-registries --no-accept-flake-config \
+# deployment" well after that VM had been destroyed, blocking its own
-    --file "${repo_root}/variables.nix" deployedTargets)"
+# redeploy) until that list was dropped in favour of this live check. This
-  for dt in $(echo "$deployed_targets_json" | jq -r '.[]'); do
+# only catches guests identified with the default --name (== --host, what
-    dt_hostname="$(nix eval --raw --no-use-registries --no-accept-flake-config \
+# this script itself always uses unless --name is overridden) -- a guest
-      "${repo_root}#nixosConfigurations.${dt}.config.networking.hostName" 2>/dev/null || true)"
+# manually renamed on the node afterwards wouldn't match, but nothing here
-    if [[ "$dt_hostname" == "$host" ]]; then
+# creates guests that way.
-      echo "ERROR: '${host}' already has a real deployment (${dt}, per variables.nix's" >&2
+if [[ "$allow_duplicate_host" -eq 1 ]]; then
-      echo "deployedTargets). Creating ${flake_target} would share its hostName/hostId --" >&2
+  echo
-      echo "refusing by default. Pass --allow-duplicate-host if you really mean to spin" >&2
+  echo "--allow-duplicate-host: skipping the check for an existing '${host}' on ${node}."
-      echo "up a separate test instance of this host (it'll still get its own distinct" >&2
+elif [[ "$dry_run" -eq 1 ]]; then
-      echo "sops key and VMID, never touching ${dt})." >&2
+  echo
-      exit 1
+  echo "[dry-run] would check ${node} for an existing VM/CT identified as '${host}'"
-    fi
+else
-  done
+  echo
+  echo "==> Checking ${node} for an existing VM/CT identified as '${host}'..."
+  ssh_check_status=0
+  existing="$(ssh "$ssh_target" bash -s -- "$host" <<'REMOTE_SCRIPT'
+target="$1"
+for id in $(qm list 2>/dev/null | awk 'NR>1{print $1}'); do
+  n="$(qm config "$id" 2>/dev/null | grep -oP '^name:\s*\K\S+' || true)"
+  [[ "$n" == "$target" ]] && echo "vm ${id} ${n}"
+done
+for id in $(pct list 2>/dev/null | awk 'NR>1{print $1}'); do
+  n="$(pct config "$id" 2>/dev/null | grep -oP '^hostname:\s*\K\S+' || true)"
+  [[ "$n" == "$target" ]] && echo "lxc ${id} ${n}"
+done
+exit 0
+REMOTE_SCRIPT
+  )" || ssh_check_status=$?
+  if [[ "$ssh_check_status" -ne 0 ]]; then
+    echo "ERROR: couldn't reach ${node} (ssh exited ${ssh_check_status}) to check for an" >&2
+    echo "existing '${host}' resource -- refusing to guess. Fix connectivity and retry," >&2
+    echo "or pass --allow-duplicate-host if you're sure none exists (this skips the" >&2
+    echo "check entirely)." >&2
+    exit 1
+  fi
+  if [[ -n "$existing" ]]; then
+    echo "ERROR: '${host}' already exists on ${node}:" >&2
+    echo "$existing" | while read -r kind id n; do
+      echo "  - ${kind} VMID ${id} (${n})" >&2
+    done
+    echo "Refusing to create a second resource sharing this identity. Pass" >&2
+    echo "--allow-duplicate-host to create one anyway (it gets its own distinct" >&2
+    echo "sops key and VMID -- the existing resource(s) above are left untouched)," >&2
+    echo "or use --modify to reconfigure the existing one instead." >&2
+    exit 1
+  fi
 fi
 
 echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '${name}'"
@@ -383,9 +429,19 @@ else
 fi
 
 if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
+  # Mirrors the real build commands' "${NIX_OPTS[@]}" below -- nix_extra_opts
+  # (called earlier, once) has already decided whether nix-cache is in play,
+  # and the dry-run preview needs to reflect that decision instead of always
+  # printing the same command regardless of outcome.
+  nix_opts_display=""
+  if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
+    printf -v nix_opts_display '%q ' "${NIX_OPTS[@]}"
+    nix_opts_display=" ${nix_opts_display% }"
+  fi
   if [[ "$type" == "lxc" ]]; then
     if [[ "$dry_run" -eq 1 ]]; then
       echo "[dry-run] would build: NIXOS_HOST_KEYS_DIR=${repo_root}/host-keys nix build --impure \\"
+      echo "[dry-run]   --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
       echo "[dry-run]   .#nixosConfigurations.${flake_target}.config.system.build.tarball"
       local_image="<built-tarball>"
     else
@@ -399,7 +455,8 @@ if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
     fi
   else
     if [[ "$dry_run" -eq 1 ]]; then
-      echo "[dry-run] would build: nix build .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
+      echo "[dry-run] would build: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
+      echo "[dry-run]   .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
       echo "[dry-run] would run:   sudo ./result-${flake_target} \\"
       echo "[dry-run]                --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\"
       echo "[dry-run]                --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\"

scripts/env.sh

@@ -88,13 +88,60 @@ nix_extra_opts() {
   fi
   export NIX_EXTRA_OPTS_DECIDED=1
   NIX_OPTS=()
-  if ! curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
+
+  # Retry a couple of times, 1s apart, before believing either check --
+  # belt-and-suspenders against a genuine multi-second blip (nix-cache
+  # restarting), on top of the fix below. Worst case (~11s total, host
+  # genuinely gone) is still nowhere near the 15s+ *per lookup* nix's own
+  # substituter retries would cost if this check didn't exist at all.
+  local attempt cache_up=0 builder_up=0
+  for attempt in 1 2 3; do
+    if curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
+      cache_up=1
+      break
+    fi
+    [[ "$attempt" -lt 3 ]] && sleep 1
+  done
+
+  if [[ "$cache_up" -eq 0 ]]; then
     echo "nix-cache (http://${NIX_CACHE_HOST}) is unreachable -- skipping it (substituter + remote builder) for the rest of this run." >&2
     NIX_OPTS=(--option substituters "https://cache.nixos.org/" --builders "")
-  elif ! timeout 3 bash -c "cat < /dev/tcp/${NIX_CACHE_HOST}/22" >/dev/null 2>&1; then
+  else
-    echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
+    for attempt in 1 2 3; do
-    NIX_OPTS=(--builders "")
+      # `exec 3<>/dev/tcp/...` just opens the fd and returns -- it does NOT
+      # read from it. Confirmed live this is load-bearing, not stylistic:
+      # the previous `cat < /dev/tcp/.../22` blocked forever and always hit
+      # the timeout even against a perfectly healthy nix-cache, because
+      # sshd sends its banner and then holds the connection open waiting
+      # for the client to speak next -- `cat` never sees EOF, so this
+      # check reported "unreachable" unconditionally, 100% of the time,
+      # regardless of whether the remote builder was actually up.
+      if timeout 3 bash -c "exec 3<>/dev/tcp/${NIX_CACHE_HOST}/22" 2>/dev/null; then
+        builder_up=1
+        break
+      fi
+      [[ "$attempt" -lt 3 ]] && sleep 1
+    done
+    if [[ "$builder_up" -eq 0 ]]; then
+      echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
+      NIX_OPTS=(--builders "")
+    fi
+  fi
+
+  # `printf '%q '` with a genuinely empty NIX_OPTS still runs one format
+  # pass over a missing argument and yields the literal `'' ` rather than
+  # an empty string (confirmed live) -- a subprocess that later does
+  # `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` (the branch above, for e.g.
+  # sync-host-keys.sh reusing this process's decision) would then rebuild
+  # a 1-element array holding an empty string instead of a 0-element
+  # array, and `nix-shell "${NIX_OPTS[@]}" -p <pkg>` chokes on that stray
+  # element as a bogus positional argument. Guard the empty case
+  # explicitly so nix-cache being reachable (NIX_OPTS legitimately empty)
+  # round-trips as truly empty instead.
+  if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
+    printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
+  else
+    NIX_EXTRA_OPTS=""
   fi
-  printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
   export NIX_EXTRA_OPTS
 }

variables.nix

@@ -19,6 +19,15 @@
 
   remoteBuilderUser = "nixremote"; # remote builder SSH user
 
+  # nix-cache's own SSH host public key (not a secret — the private half
+  # never leaves the host). Wired into every client's
+  # programs.ssh.knownHosts by modules/nix-cache/remote-builder-client.nix
+  # so distributed builds don't hit "Host key verification failed" on a
+  # fresh client that has never manually ssh'd to nix-cache before. Update
+  # this if nix-cache's host key is ever rotated or the host is rebuilt
+  # from scratch.
+  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
+
   # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
   # host (modules/nix-cache/server.nix) — one per client host that's allowed
   # to use it as a distributed builder.
@@ -147,20 +156,4 @@
     keep = 20; # number of rotated logs to retain before deleting the oldest
   };
 
-  # Flake targets with a real, currently-running deployment somewhere —
-  # matches README.md's Hosts table "(real, deployed)" annotations; update
-  # both together. Not consumed by any NixOS module (nothing in the actual
-  # system config should behave differently because of this) — it's read
-  # by scripts/create-proxmox-resource.sh to refuse creating a same-identity
-  # duplicate of an already-deployed host (shared hostName/hostId) unless
-  # you explicitly pass --allow-duplicate-host.
-  deployedTargets = [
-    "linode-minimal"
-    "proxmox-minimal"
-    "proxmox-nix-cache"
-    "proxmox-server"
-    "proxmox-docker"
-    "proxmox-gui"
-    "proxmox-pxe-boot"
-  ];
 }