Fix nix-cache retry storms and lxc creation in create-proxmox-resource.sh

Two independent problems found while actually running the script:

1. nix build/nix-shell retry each unreachable substituter/builder up to
   5x with backoff, per store path -- with nix-cache down this compounds
   into minutes of noise. scripts/env.sh gains nix_extra_opts(), which
   probes http://nix-cache and nixremote@nix-cache:22 once via plain
   curl/TCP (bypassing Nix's own retry logic entirely -- confirmed
   nix store ping still retries 5x even with a short connect-timeout)
   and exports the decision so create-proxmox-resource.sh and the
   sync-host-keys.sh subprocess it shells out to both reuse it instead
   of probing independently.

2. The actual failure: "archive contains no configuration file". pct
   restore expects a vzdump backup archive with embedded config;
   config.system.build.tarball is a plain CT template tarball -- wrong
   Proxmox mechanism entirely. Fixed to pct create against it as a vztmpl
   template instead, uploaded to /var/lib/vz/template/cache/ rather than
   /var/lib/vz/dump/. This same wrong claim had propagated into
   docs/auto-installer.md, README.md, and CLAUDE.md from when the script
   was first written -- corrected everywhere.

Also: checks for an already-uploaded image on the node (fixed
<flake_target>.tar.xz/.raw naming) before building, skipping build+upload
entirely if found (--force-rebuild to always rebuild).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot

CLAUDE.md

@@ -86,12 +86,16 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
   has.
 - `scripts/create-proxmox-resource.sh` — builds a `lxc-*`/`proxmox-*`
   target's tarball/disk image and creates it on a real Proxmox node
-  (`pct restore` / `qm create`+`importdisk`), or reconfigures an existing
-  resource's cores/memory/disk size (`--modify`, always requires typing the
-  VMID back to confirm). Refuses to create a target whose host identity
-  already has a real deployment elsewhere (`variables.nix`'s
-  `deployedTargets`) unless `--allow-duplicate-host` is passed.
-  `--dry-run` throughout both modes.
+  (`pct create` against the tarball as a CT template / `qm create`+
+  `importdisk`), or reconfigures an existing resource's cores/memory/disk
+  size (`--modify`, always requires typing the VMID back to confirm).
+  Checks for an already-uploaded image on the node before building
+  (`--force-rebuild` to skip that and always rebuild), and probes
+  nix-cache's substituter/remote-builder reachability once up front rather
+  than letting every `nix build` call retry against it individually.
+  Refuses to create a target whose host identity already has a real
+  deployment elsewhere (`variables.nix`'s `deployedTargets`) unless
+  `--allow-duplicate-host` is passed. `--dry-run` throughout both modes.
 - `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
   default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
   cross-script config here instead of duplicating it per-script.
@@ -153,8 +157,10 @@ removing a host.
   from `flake.nix`. `lxc.nix` has no hardware-configuration counterpart since
   containers share the host kernel; instead it imports nixpkgs' own
   `virtualisation/proxmox-lxc.nix`, which gives every `lxc-*` host a
-  `config.system.build.tarball` output (`pct restore`-ready, no install step
-  — see `docs/auto-installer.md`).
+  `config.system.build.tarball` output — a plain rootfs tarball, used as a
+  `pct create ... vztmpl` CT template (**not** `pct restore`, which expects
+  `vzdump` backup-archive metadata this doesn't have), no install step —
+  see `docs/auto-installer.md`.
 - `modules/build-types/*.nix` — what a system is for:
   minimal/server/docker/gui/pxe-boot/nix-cache.
 - `modules/common/configuration.nix` — base NixOS config imported by every

README.md

@@ -103,15 +103,15 @@ Three different paths depending on target, none of them involving a manual
   section covers how this stays non-destructive for `linode-*`, whose disks
   Linode itself provisions ahead of time).
 - `lxc-*` targets: not installed at all — build a ready-to-run container
-  tarball and `pct restore` it directly. `docs/auto-installer.md` covers why
-  (and the installer's menu excludes them for the same reason).
+  tarball and `pct create` it as a CT template directly. `docs/auto-installer.md`
+  covers why (and the installer's menu excludes them for the same reason).
 - `proxmox-*` targets: can alternatively be built as a standalone `.raw`
   disk image and attached to a new VM with no install step — see
   `docs/proxmox-images.md`.
 
 `scripts/create-proxmox-resource.sh --type lxc|vm --host <name>` automates
 either of the last two end to end (build, host-key registration, upload,
-`pct restore`/`qm create`), with `--dry-run` and a guard against duplicating
+`pct create`/`qm create`), with `--dry-run` and a guard against duplicating
 an already-deployed host's identity. See its `--help`.
 
 ## Security Notes

docs/auto-installer.md

@@ -34,10 +34,26 @@ container image, no install step at all:
 nix build .#nixosConfigurations.lxc-minimal.config.system.build.tarball
 ```
 
-Then, on the Proxmox host, `pct restore` (or the GUI's "Create CT" → upload
-template flow) that tarball directly as a new container. First boot runs
-`boot.postBootCommands` (registers the Nix store DB and system profile) —
-there's no separate activation step to run yourself.
+This is a plain rootfs tarball, not a `vzdump` backup archive — restoring it
+with `pct restore` fails ("archive contains no configuration file"), since
+that command expects backup-archive metadata this tarball doesn't have. Use
+it as a CT *template* instead: drop it under Proxmox's template storage
+(conventionally `/var/lib/vz/template/cache/` for the `local` storage, or
+the GUI's "Create CT" → upload-as-template flow) and create a container
+from it, supplying all config on the command line since a template has none
+of its own:
+
+```sh
+pct create <vmid> local:vztmpl/<file>.tar.xz \
+  --rootfs local-lvm:8 --hostname <name> --cores 2 --memory 2048 \
+  --net0 name=eth0,bridge=vmbr0,ip=dhcp
+pct start <vmid>
+```
+
+First boot runs `boot.postBootCommands` (registers the Nix store DB and
+system profile) — there's no separate activation step to run yourself.
+`scripts/create-proxmox-resource.sh --type lxc --host <name>` automates all
+of this (build, host-key handling, upload, `pct create`) — see its `--help`.
 
 Host keys still need pre-seeding the same way as any other host (see "Host
 keys" below) — the sops-nix activation-vs-first-boot race is identical

scripts/create-proxmox-resource.sh

@@ -34,7 +34,7 @@ Usage: $0 --type lxc|vm --host <name> [options]      (create)
        $0 --modify --vmid <n> [options]                (reconfigure)
 
 Create mode (default):
-  --type lxc|vm        lxc = container, built as a pct-restorable tarball.
+  --type lxc|vm        lxc = container, built as a CT template tarball.
                         vm  = VM, built as a Disko .raw disk image (UEFI/OVMF).
   --host <name>         Which host identity to deploy -- matches
                         config.networking.hostName (server, docker,
@@ -45,10 +45,13 @@ Create mode (default):
   --vmid <n>              Numeric VMID (default: next free, via
                           \`pvesh get /cluster/nextid\` on the node).
                           Refuses to run if this ID already exists.
-  --disk-size <GB>           lxc only, at create time: overrides the
-                              restored rootfs's absolute size.
+  --disk-size <GB>           lxc only: rootfs size for \`pct create\`
+                              (default: \$PROXMOX_DEFAULT_LXC_DISK_GB, ${PROXMOX_DEFAULT_LXC_DISK_GB}).
   --image <path>              Use this local image/tarball instead of
-                              building one from the flake.
+                              checking the node / building one from the flake.
+  --force-rebuild                Skip the "does the node already have this
+                                  image" check -- always build fresh and
+                                  overwrite what's there.
   --allow-duplicate-host        Required if --host already has a real
                                 deployment elsewhere (variables.nix's
                                 deployedTargets) -- otherwise refused, since
@@ -101,6 +104,7 @@ bridge="$PROXMOX_BRIDGE"
 node="$PROXMOX_HOST"
 do_list=0
 allow_duplicate_host=0
+force_rebuild=0
 
 while [[ $# -gt 0 ]]; do
   case "$1" in
@@ -119,6 +123,7 @@ while [[ $# -gt 0 ]]; do
   --node) node="$2"; shift 2 ;;
   --list) do_list=1; shift ;;
   --allow-duplicate-host) allow_duplicate_host=1; shift ;;
+  --force-rebuild) force_rebuild=1; shift ;;
   --modify) modify=1; shift ;;
   --dry-run) dry_run=1; shift ;;
   -h | --help) usage; exit 0 ;;
@@ -288,6 +293,13 @@ fi
 
 echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '${name}'"
 
+# Decide on nix-cache once, here -- this is the earliest point that needs
+# it (sync-host-keys.sh below needs nix-shell packages regardless of
+# whether an image ends up getting built later), and the decision is
+# exported so that subprocess -- and this script's own later build step,
+# if it gets there -- both reuse it instead of probing again.
+nix_extra_opts
+
 # --- make sure this target has a registered host key --------------------
 echo
 echo "==> Ensuring host key exists and is registered..."
@@ -325,16 +337,47 @@ if [[ "$dry_run" -eq 0 ]]; then
   fi
 fi
 
-# --- build (or reuse) the image ------------------------------------------
+# --- resolve the remote path -- fixed naming (not the nix store's own
+# derivation-hash-based filename), so a later run can check for it by name.
+# lxc uploads as a CT *template* (Proxmox's "vztmpl" content type, under
+# iso_storage) -- config.system.build.tarball is a plain rootfs tarball,
+# not a vzdump backup archive, so it's created with `pct create ... vztmpl`,
+# not restored with `pct restore` (that expects backup-archive metadata
+# this tarball doesn't have, and fails with "archive contains no
+# configuration file").
+remote_dir="/var/lib/vz/import"
+remote_filename="${flake_target}.raw"
+if [[ "$type" == "lxc" ]]; then
+  remote_dir="/var/lib/vz/template/cache"
+  remote_filename="${flake_target}.tar.xz"
+fi
+remote_path="${remote_dir}/${remote_filename}"
+
+# --- build (or reuse an image already on the node) ------------------------
 echo
 local_image=""
-remote_dir=""
+image_already_remote=0
+
 if [[ -n "$image" ]]; then
   [[ -f "$image" ]] || { echo "ERROR: --image '${image}' not found." >&2; exit 1; }
   local_image="$image"
   echo "Using provided image: ${local_image}"
-elif [[ "$type" == "lxc" ]]; then
-  remote_dir="/var/lib/vz/dump"
+elif [[ "$force_rebuild" -eq 1 ]]; then
+  echo "--force-rebuild: skipping the existing-image check on ${node}."
+else
+  echo "==> Checking whether ${node} already has ${remote_path}..."
+  if [[ "$dry_run" -eq 1 ]]; then
+    echo "[dry-run] would check: ssh ${ssh_target} -- test -f ${remote_path}"
+  elif ssh "$ssh_target" "test -f '${remote_path}'" 2>/dev/null; then
+    echo "Found it -- reusing, skipping build and upload (use --force-rebuild to override)."
+    image_already_remote=1
+  else
+    echo "Not found -- will build."
+  fi
+fi
+
+if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
+  if [[ "$type" == "lxc" ]]; then
     if [[ "$dry_run" -eq 1 ]]; then
       echo "[dry-run] would build: NIXOS_HOST_KEYS_DIR=${repo_root}/host-keys nix build --impure \\"
       echo "[dry-run]   .#nixosConfigurations.${flake_target}.config.system.build.tarball"
@@ -342,14 +385,13 @@ elif [[ "$type" == "lxc" ]]; then
     else
       echo "==> Building LXC tarball for ${flake_target}..."
       NIXOS_HOST_KEYS_DIR="${repo_root}/host-keys" nix build --impure \
-      --no-use-registries --no-accept-flake-config \
+        --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \
         ".#nixosConfigurations.${flake_target}.config.system.build.tarball" \
         --out-link "${repo_root}/result-${flake_target}"
       local_image="$(find "${repo_root}/result-${flake_target}/tarball" -maxdepth 1 -type f | head -1)"
       echo "Built: ${local_image}"
     fi
-else
-  remote_dir="/var/lib/vz/import"
+  else
     if [[ "$dry_run" -eq 1 ]]; then
       echo "[dry-run] would build: nix build .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
       echo "[dry-run] would run:   sudo ./result-${flake_target} \\"
@@ -359,7 +401,7 @@ else
       local_image="<built-image>.raw"
     else
       echo "==> Building Disko image script for ${flake_target}..."
-    nix build --no-use-registries --no-accept-flake-config \
+      nix build --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \
         ".#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript" \
         --out-link "${repo_root}/result-${flake_target}"
       echo "==> Running it (builds the .raw image in a temporary QEMU VM, needs sudo)..."
@@ -374,12 +416,14 @@ else
       fi
       echo "Built: ${local_image}"
     fi
+  fi
 fi
 
-# --- upload ---------------------------------------------------------------
+# --- upload (skip entirely if reusing an image already on the node) ------
 echo
-remote_path="${remote_dir}/$(basename "${local_image}")"
-if [[ "$dry_run" -eq 1 ]]; then
+if [[ "$image_already_remote" -eq 1 ]]; then
+  : # nothing to upload
+elif [[ "$dry_run" -eq 1 ]]; then
   echo "[dry-run] would upload: scp ${local_image} ${ssh_target}:${remote_path}"
 else
   echo "==> Uploading to ${node}:${remote_path}..."
@@ -391,9 +435,9 @@ fi
 echo
 if [[ "$type" == "lxc" ]]; then
   echo "==> Creating LXC container ${vmid} (${name})..."
-  restore_cmd="pct restore ${vmid} ${remote_path} --storage ${storage} --hostname ${name} --cores ${cores} --memory ${memory} --net0 name=eth0,bridge=${bridge},ip=dhcp"
-  [[ -n "$disk_size" ]] && restore_cmd="${restore_cmd} --rootfs ${storage}:${disk_size}"
-  remote "$restore_cmd"
+  local_disk_size="${disk_size:-$PROXMOX_DEFAULT_LXC_DISK_GB}"
+  create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --net0 name=eth0,bridge=${bridge},ip=dhcp"
+  remote "$create_cmd"
   remote "pct start ${vmid}"
 else
   echo "==> Creating VM ${vmid} (${name})..."

scripts/env.sh

@@ -26,5 +26,57 @@
 : "${PROXMOX_DEFAULT_CORES:=2}"
 : "${PROXMOX_DEFAULT_MEMORY_MB:=2048}"
 
+# `pct create` (unlike `pct restore`) requires an explicit rootfs size --
+# no backup metadata to infer it from. Matches Proxmox's own GUI default.
+: "${PROXMOX_DEFAULT_LXC_DISK_GB:=8}"
+
 export PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE PROXMOX_ISO_STORAGE \
-  PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES PROXMOX_DEFAULT_MEMORY_MB
+  PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES PROXMOX_DEFAULT_MEMORY_MB PROXMOX_DEFAULT_LXC_DISK_GB
+
+# Matches variables.nix's nixCacheHost -- update both if it ever changes.
+: "${NIX_CACHE_HOST:=nix-cache}"
+export NIX_CACHE_HOST
+
+# nix_extra_opts: call as a plain statement (NOT inside $(...)/<(...) --
+# that forks a subshell, and the whole point is exporting a decision back
+# into *this* shell) to populate the global NIX_OPTS array with whatever
+# extra `nix`/`nix-shell` CLI options are needed to avoid nix-cache when
+# it's unreachable:
+#   nix_extra_opts
+#   nix build "${NIX_OPTS[@]}" ...
+#
+# Without this, every single `nix eval`/`nix build` call retries each
+# store path against a dead substituter up to 5 times with backoff
+# (confirmed: ~15s+ per lookup even with a short connect-timeout, because
+# nix's own retry count isn't controllable that way), and separately
+# tries it as a remote builder too -- both fail independently, so both
+# are checked.
+#
+# Checked with a single fast `curl`/TCP probe (bypassing nix's retry logic
+# entirely) the first time this is called in a given process, and the
+# result is exported as NIX_EXTRA_OPTS so a script that shells out to
+# another script in this repo (e.g. create-proxmox-resource.sh calling
+# sync-host-keys.sh) reuses the same decision instead of probing twice.
+declare -a NIX_OPTS=()
+
+nix_extra_opts() {
+  if [[ -n "${NIX_EXTRA_OPTS_DECIDED:-}" ]]; then
+    if [[ -n "${NIX_EXTRA_OPTS:-}" ]]; then
+      eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"
+    else
+      NIX_OPTS=()
+    fi
+    return
+  fi
+  export NIX_EXTRA_OPTS_DECIDED=1
+  NIX_OPTS=()
+  if ! curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
+    echo "nix-cache (http://${NIX_CACHE_HOST}) is unreachable -- skipping it (substituter + remote builder) for the rest of this run." >&2
+    NIX_OPTS=(--option substituters "https://cache.nixos.org/" --builders "")
+  elif ! timeout 3 bash -c "cat < /dev/tcp/${NIX_CACHE_HOST}/22" >/dev/null 2>&1; then
+    echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
+    NIX_OPTS=(--builders "")
+  fi
+  printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
+  export NIX_EXTRA_OPTS
+}

scripts/sync-host-keys.sh

@@ -29,6 +29,9 @@ sops_yaml="${repo_root}/.sops.yaml"
 keydir="${repo_root}/host-keys"
 editor="${repo_root}/scripts/lib/sync-host-keys-edit-sops.py"
 
+# shellcheck source=env.sh
+source "${repo_root}/scripts/env.sh"
+
 mkdir -p "$keydir"
 
 usage() {
@@ -88,9 +91,9 @@ ensure_admin_decrypt_key() {
   echo "No sops age decryption key found (checked \$SOPS_AGE_KEY, \$SOPS_AGE_KEY_FILE, ${key_file})."
   echo "Generating a new one at ${key_file}..."
   mkdir -p "$(dirname "$key_file")"
-  nix-shell -p age --run "age-keygen -o '${key_file}'" 2>&1 | grep -v "^Public key:" || true
+  nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -o '${key_file}'" 2>&1 | grep -v "^Public key:" || true
   local new_pub
-  new_pub="$(nix-shell -p age --run "age-keygen -y '${key_file}'")"
+  new_pub="$(nix-shell "${NIX_OPTS[@]}" -p age --run "age-keygen -y '${key_file}'")"
 
   cat <<EOF
 
@@ -156,7 +159,7 @@ queue_host_sync() {
       echo "[dry-run] ${host}: would generate host key"
     else
       echo "==> ${host}: generating host key"
-      nix-shell -p openssh --run "ssh-keygen -t ed25519 -N '' -C '${host}' -f '${keyfile}'" >/dev/null
+      nix-shell "${NIX_OPTS[@]}" -p openssh --run "ssh-keygen -t ed25519 -N '' -C '${host}' -f '${keyfile}'" >/dev/null
     fi
   else
     echo "==> ${host}: host key already present"
@@ -167,7 +170,7 @@ queue_host_sync() {
     if [[ "$dry_run" -eq 1 ]]; then
       age_pub="dry-run-placeholder-not-a-real-key"
     else
-      age_pub="$(nix-shell -p ssh-to-age --run "ssh-to-age -i '${keyfile}.pub'")"
+      age_pub="$(nix-shell "${NIX_OPTS[@]}" -p ssh-to-age --run "ssh-to-age -i '${keyfile}.pub'")"
     fi
     add_keys_json="$(jq --arg host "$host" --arg key "$age_pub" \
       '. + [{host: $host, age_key: $key}]' <<<"$add_keys_json")"
@@ -203,7 +206,7 @@ apply_edit_plan() {
   fi
 
   local result
-  result="$(echo "$plan" | nix-shell -p python3 --run "python3 '${editor}' '${target}'")"
+  result="$(echo "$plan" | nix-shell "${NIX_OPTS[@]}" -p python3 --run "python3 '${editor}' '${target}'")"
   [[ -n "$tmpfile" ]] && rm -f "$tmpfile"
 
   local added removed changed
@@ -234,7 +237,7 @@ apply_edit_plan() {
       while IFS= read -r basename; do
         [[ -z "$basename" ]] && continue
         echo "==> secrets/${basename}"
-        nix-shell -p sops --run "sops updatekeys --yes '${repo_root}/secrets/${basename}'"
+        nix-shell "${NIX_OPTS[@]}" -p sops --run "sops updatekeys --yes '${repo_root}/secrets/${basename}'"
       done <<<"$changed"
     fi
   fi
@@ -408,6 +411,7 @@ main() {
     echo
   fi
 
+  nix_extra_opts
   ensure_admin_decrypt_key
 
   case "${1:-}" in