Stop tracking deployment status in the Hosts table

Same problem as the deployedTargets removal, just in markdown instead of
Nix: which variant of a buildtype is actually deployed is live
infrastructure state, and a committed table can't stay accurate as that
changes -- it already required a manual edit on every migration and had
drifted before. Keep only what doesn't rot: what each target is for, and
stable naming history. Point at the live node / /etc/flake-target instead
for actual deployment status.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

CLAUDE.md

@@ -93,9 +93,10 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
   (`--force-rebuild` to skip that and always rebuild), and probes
   nix-cache's substituter/remote-builder reachability once up front rather
   than letting every `nix build` call retry against it individually.
-  Refuses to create a target whose host identity already has a real
-  deployment elsewhere (`variables.nix`'s `deployedTargets`) unless
-  `--allow-duplicate-host` is passed. `--dry-run` throughout both modes.
+  Refuses to create a target whose host identity already exists live on
+  the node (checked directly via `qm`/`pct`, not any file in this repo)
+  unless `--allow-duplicate-host` is passed. `--dry-run` throughout both
+  modes.
 - `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
   default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
   cross-script config here instead of duplicating it per-script.

README.md

@@ -18,20 +18,23 @@ list:
 
 | Target | Purpose |
 | --- | --- |
-| `linode-minimal` | Minimal NixOS host profile on a Linode VPS (real, deployed) |
-| `proxmox-minimal` | Minimal NixOS host profile on Proxmox (real, deployed — previously the flat `nix-minimal` target) |
+| `linode-minimal` | Minimal NixOS host profile on a Linode VPS |
+| `proxmox-minimal` | Minimal NixOS host profile on Proxmox — previously the flat `nix-minimal` target |
 | `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container |
-| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder (`proxmox-nix-cache` is the real, deployed one — previously the flat `nix-cache` target) |
-| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host (`proxmox-server` is the real, deployed one — previously the flat `server` target) |
-| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack (`proxmox-docker` is the real, deployed one — previously the flat `docker` target) |
-| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation (`proxmox-gui` is the real, deployed one — previously the flat `nixos` target) |
-| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host (`proxmox-pxe-boot` is the real, deployed one — previously the flat `pxe-boot` target) |
-| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node (no deployed target yet; `lxc-tailscale-exit-node` is the one planned for actual use) |
+| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder — previously the flat `nix-cache` target |
+| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host — previously the flat `server` target |
+| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack — previously the flat `docker` target |
+| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target |
+| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host — previously the flat `pxe-boot` target |
+| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node |
 
-The "(real, deployed)" targets above are also tracked machine-readably in
-`variables.nix`'s `deployedTargets` — keep both in sync when a deployment
-changes. `scripts/create-proxmox-resource.sh` reads that list to refuse
-creating a same-identity duplicate of an already-deployed host by accident.
+Which variant of a given buildtype is actually deployed isn't tracked
+anywhere in this repo — that's live infrastructure state, not something a
+committed file can keep accurate, and it changes independently of the code.
+Check the Proxmox node itself, or `/etc/flake-target` on a running host (see
+below), if you need to know what's really out there right now.
+`scripts/create-proxmox-resource.sh`'s duplicate-host guard works the same
+way: it checks the Proxmox node directly rather than any file here.
 
 Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
 (hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay

docs/nix-cache.md

@@ -59,6 +59,15 @@ On `nix-cache`, install the matching public key used by `nixremote` authorized k
 The committed `nixremote` authorized keys are public SSH keys only. Keep the
 matching private keys on client hosts and out of the repository.
 
+nix-cache's own SSH *host* key is trusted declaratively via
+`programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
+sourced from `vars.nixCacheHostKey` (`variables.nix`) — every client rebuild
+picks it up automatically, so distributed builds don't fail with "Host key
+verification failed" on a client that has never manually SSH'd to nix-cache
+before. If nix-cache's host key is ever rotated or the host rebuilt from
+scratch, update `vars.nixCacheHostKey` to match its new
+`/etc/ssh/ssh_host_ed25519_key.pub`.
+
 ## Manual verification
 
 After deployment:

modules/nix-cache/remote-builder-client.nix

@@ -5,6 +5,14 @@
   #   sudo install -d -m 0700 /root/.ssh
   #   sudo install -m 0600 ./nixremote /root/.ssh/nixremote
   #   sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
+  # Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
+  # can connect the first time without a manual ssh-keyscan/known_hosts
+  # step on every new client.
+  programs.ssh.knownHosts.${vars.nixCacheHost} = {
+    hostNames = [ vars.nixCacheHost ];
+    publicKey = vars.nixCacheHostKey;
+  };
+
   nix = {
     distributedBuilds = true;
 

scripts/create-proxmox-resource.sh

@@ -10,7 +10,9 @@
 #
 # SAFETY:
 #   - The default (create) mode only ever creates a NEW resource -- it
-#     refuses to run if the target VMID already exists on the node.
+#     refuses to run if the target VMID already exists on the node, or if
+#     a VM/CT identified as --host already exists under any other VMID
+#     (checked live against the node; --allow-duplicate-host overrides).
 #   - --modify only ever touches a resource you name explicitly via
 #     --vmid, shows exactly what will change first, and (outside
 #     --dry-run) always requires typing that VMID back to confirm before
@@ -56,10 +58,11 @@ Create mode (default):
   --force-rebuild                Skip the "does the node already have this
                                   image" check -- always build fresh and
                                   overwrite what's there.
-  --allow-duplicate-host        Required if --host already has a real
-                                deployment elsewhere (variables.nix's
-                                deployedTargets) -- otherwise refused, since
-                                it'd share that host's hostName/hostId.
+  --allow-duplicate-host        Required if a VM/CT identified as --host
+                                already exists on the node (checked live via
+                                qm/pct, not any file in this repo) --
+                                otherwise refused, since it'd share that
+                                host's hostName/hostId.
 
 Modify mode (reconfigure an EXISTING resource -- requires --modify):
   --modify                Switch to modify mode.
@@ -286,25 +289,57 @@ fi
 # feeds straight into the guest's real hostname) disagree with host.nix.
 [[ -z "$name" ]] && name="$host"
 
-# --- refuse to duplicate a host that's already really deployed ----------
-# Checked by hostName, not exact flake target: proxmox-server being
-# deployed also blocks --type lxc --host server, since both would carry
-# the same hosts/server/host.nix identity (hostName, hostId).
-if [[ "$allow_duplicate_host" -eq 0 ]]; then
-  deployed_targets_json="$(nix eval --json --no-use-registries --no-accept-flake-config \
-    --file "${repo_root}/variables.nix" deployedTargets)"
-  for dt in $(echo "$deployed_targets_json" | jq -r '.[]'); do
-    dt_hostname="$(nix eval --raw --no-use-registries --no-accept-flake-config \
-      "${repo_root}#nixosConfigurations.${dt}.config.networking.hostName" 2>/dev/null || true)"
-    if [[ "$dt_hostname" == "$host" ]]; then
-      echo "ERROR: '${host}' already has a real deployment (${dt}, per variables.nix's" >&2
-      echo "deployedTargets). Creating ${flake_target} would share its hostName/hostId --" >&2
-      echo "refusing by default. Pass --allow-duplicate-host if you really mean to spin" >&2
-      echo "up a separate test instance of this host (it'll still get its own distinct" >&2
-      echo "sops key and VMID, never touching ${dt})." >&2
-      exit 1
-    fi
-  done
+# --- refuse to duplicate a host that's already live on the node ---------
+# Queries the node itself (qm/pct's own name/hostname config), not any
+# static list in this repo -- a file can't track whether a resource still
+# actually exists, and this used to be checked against variables.nix's
+# deployedTargets, which drifted stale (it kept naming a VM as "the real
+# deployment" well after that VM had been destroyed, blocking its own
+# redeploy) until that list was dropped in favour of this live check. This
+# only catches guests identified with the default --name (== --host, what
+# this script itself always uses unless --name is overridden) -- a guest
+# manually renamed on the node afterwards wouldn't match, but nothing here
+# creates guests that way.
+if [[ "$allow_duplicate_host" -eq 1 ]]; then
+  echo
+  echo "--allow-duplicate-host: skipping the check for an existing '${host}' on ${node}."
+elif [[ "$dry_run" -eq 1 ]]; then
+  echo
+  echo "[dry-run] would check ${node} for an existing VM/CT identified as '${host}'"
+else
+  echo
+  echo "==> Checking ${node} for an existing VM/CT identified as '${host}'..."
+  ssh_check_status=0
+  existing="$(ssh "$ssh_target" bash -s -- "$host" <<'REMOTE_SCRIPT'
+target="$1"
+for id in $(qm list 2>/dev/null | awk 'NR>1{print $1}'); do
+  n="$(qm config "$id" 2>/dev/null | grep -oP '^name:\s*\K\S+' || true)"
+  [[ "$n" == "$target" ]] && echo "vm ${id} ${n}"
+done
+for id in $(pct list 2>/dev/null | awk 'NR>1{print $1}'); do
+  n="$(pct config "$id" 2>/dev/null | grep -oP '^hostname:\s*\K\S+' || true)"
+  [[ "$n" == "$target" ]] && echo "lxc ${id} ${n}"
+done
+REMOTE_SCRIPT
+  )" || ssh_check_status=$?
+  if [[ "$ssh_check_status" -ne 0 ]]; then
+    echo "ERROR: couldn't reach ${node} (ssh exited ${ssh_check_status}) to check for an" >&2
+    echo "existing '${host}' resource -- refusing to guess. Fix connectivity and retry," >&2
+    echo "or pass --allow-duplicate-host if you're sure none exists (this skips the" >&2
+    echo "check entirely)." >&2
+    exit 1
+  fi
+  if [[ -n "$existing" ]]; then
+    echo "ERROR: '${host}' already exists on ${node}:" >&2
+    echo "$existing" | while read -r kind id n; do
+      echo "  - ${kind} VMID ${id} (${n})" >&2
+    done
+    echo "Refusing to create a second resource sharing this identity. Pass" >&2
+    echo "--allow-duplicate-host to create one anyway (it gets its own distinct" >&2
+    echo "sops key and VMID -- the existing resource(s) above are left untouched)," >&2
+    echo "or use --modify to reconfigure the existing one instead." >&2
+    exit 1
+  fi
 fi
 
 echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '${name}'"

scripts/env.sh

@@ -88,13 +88,60 @@ nix_extra_opts() {
   fi
   export NIX_EXTRA_OPTS_DECIDED=1
   NIX_OPTS=()
-  if ! curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
+
+  # Retry a couple of times, 1s apart, before believing either check --
+  # belt-and-suspenders against a genuine multi-second blip (nix-cache
+  # restarting), on top of the fix below. Worst case (~11s total, host
+  # genuinely gone) is still nowhere near the 15s+ *per lookup* nix's own
+  # substituter retries would cost if this check didn't exist at all.
+  local attempt cache_up=0 builder_up=0
+  for attempt in 1 2 3; do
+    if curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
+      cache_up=1
+      break
+    fi
+    [[ "$attempt" -lt 3 ]] && sleep 1
+  done
+
+  if [[ "$cache_up" -eq 0 ]]; then
     echo "nix-cache (http://${NIX_CACHE_HOST}) is unreachable -- skipping it (substituter + remote builder) for the rest of this run." >&2
     NIX_OPTS=(--option substituters "https://cache.nixos.org/" --builders "")
-  elif ! timeout 3 bash -c "cat < /dev/tcp/${NIX_CACHE_HOST}/22" >/dev/null 2>&1; then
-    echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
-    NIX_OPTS=(--builders "")
+  else
+    for attempt in 1 2 3; do
+      # `exec 3<>/dev/tcp/...` just opens the fd and returns -- it does NOT
+      # read from it. Confirmed live this is load-bearing, not stylistic:
+      # the previous `cat < /dev/tcp/.../22` blocked forever and always hit
+      # the timeout even against a perfectly healthy nix-cache, because
+      # sshd sends its banner and then holds the connection open waiting
+      # for the client to speak next -- `cat` never sees EOF, so this
+      # check reported "unreachable" unconditionally, 100% of the time,
+      # regardless of whether the remote builder was actually up.
+      if timeout 3 bash -c "exec 3<>/dev/tcp/${NIX_CACHE_HOST}/22" 2>/dev/null; then
+        builder_up=1
+        break
+      fi
+      [[ "$attempt" -lt 3 ]] && sleep 1
+    done
+    if [[ "$builder_up" -eq 0 ]]; then
+      echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
+      NIX_OPTS=(--builders "")
+    fi
+  fi
+
+  # `printf '%q '` with a genuinely empty NIX_OPTS still runs one format
+  # pass over a missing argument and yields the literal `'' ` rather than
+  # an empty string (confirmed live) -- a subprocess that later does
+  # `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` (the branch above, for e.g.
+  # sync-host-keys.sh reusing this process's decision) would then rebuild
+  # a 1-element array holding an empty string instead of a 0-element
+  # array, and `nix-shell "${NIX_OPTS[@]}" -p <pkg>` chokes on that stray
+  # element as a bogus positional argument. Guard the empty case
+  # explicitly so nix-cache being reachable (NIX_OPTS legitimately empty)
+  # round-trips as truly empty instead.
+  if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
+    printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
+  else
+    NIX_EXTRA_OPTS=""
   fi
-  printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
   export NIX_EXTRA_OPTS
 }

variables.nix

@@ -19,6 +19,15 @@
 
   remoteBuilderUser = "nixremote"; # remote builder SSH user
 
+  # nix-cache's own SSH host public key (not a secret — the private half
+  # never leaves the host). Wired into every client's
+  # programs.ssh.knownHosts by modules/nix-cache/remote-builder-client.nix
+  # so distributed builds don't hit "Host key verification failed" on a
+  # fresh client that has never manually ssh'd to nix-cache before. Update
+  # this if nix-cache's host key is ever rotated or the host is rebuilt
+  # from scratch.
+  nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
+
   # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
   # host (modules/nix-cache/server.nix) — one per client host that's allowed
   # to use it as a distributed builder.
@@ -147,20 +156,4 @@
     keep = 20; # number of rotated logs to retain before deleting the oldest
   };
 
-  # Flake targets with a real, currently-running deployment somewhere —
-  # matches README.md's Hosts table "(real, deployed)" annotations; update
-  # both together. Not consumed by any NixOS module (nothing in the actual
-  # system config should behave differently because of this) — it's read
-  # by scripts/create-proxmox-resource.sh to refuse creating a same-identity
-  # duplicate of an already-deployed host (shared hostName/hostId) unless
-  # you explicitly pass --allow-duplicate-host.
-  deployedTargets = [
-    "linode-minimal"
-    "proxmox-minimal"
-    "proxmox-nix-cache"
-    "proxmox-server"
-    "proxmox-docker"
-    "proxmox-gui"
-    "proxmox-pxe-boot"
-  ];
 }