Build Proxmox images directly on the node instead of transferring them

create-proxmox-resource.sh no longer builds locally and scp's a
multi-gigabyte image over -- it now clones/pulls this repo onto the
Proxmox node itself (bootstrapping build tooling via the existing
codex-setup.sh on first use) and runs the nix build / disko image
script there, staging the result straight into the node's own import
directory. host-keys/ (gitignored) is copied over separately since a
git pull doesn't carry it. --image still uploads an explicit local
file for the case where you don't want a build at all.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

README.md

@@ -114,9 +114,10 @@ Three different paths depending on target, none of them involving a manual
   `docs/proxmox-images.md`.
 
 `scripts/create-proxmox-resource.sh --type lxc|vm --host <name>` automates
-either of the last two end to end (build, host-key registration, upload,
-`pct create`/`qm create`), with `--dry-run` and a guard against duplicating
-an already-deployed host's identity. See its `--help`.
+either of the last two end to end (host-key registration, building the
+image directly on the Proxmox node itself, `pct create`/`qm create`), with
+`--dry-run` and a guard against duplicating an already-deployed host's
+identity. See its `--help`.
 
 ## Security Notes
 

docs/auto-installer.md

@@ -74,8 +74,8 @@ booting one:
 First boot runs `boot.postBootCommands` (registers the Nix store DB and
 system profile) — there's no separate activation step to run yourself.
 `scripts/create-proxmox-resource.sh --type lxc --host <name>` automates all
-of this (build, host-key handling, upload, `pct create` with the flags
-above) — see its `--help`.
+of this (host-key handling, building the tarball directly on the Proxmox
+node itself, `pct create` with the flags above) — see its `--help`.
 
 Host keys still need pre-seeding the same way as any other host — the
 sops-nix activation-vs-first-boot race is identical regardless of how the

docs/proxmox-images.md

@@ -9,9 +9,11 @@ install, so there's nothing host-specific to write; it's available for every
 
 `scripts/create-proxmox-resource.sh --type vm --host <name>` automates the
 whole walkthrough below (and the equivalent LXC one) end to end, including
-host-key handling and upload — see its `--help`. The steps here are what it
-runs under the hood, useful for doing any of it by hand or understanding
-what it does before you trust it against real infrastructure.
+host-key handling and building the image directly on the Proxmox node
+itself (no local build, no image transfer) — see its `--help`. The steps
+here are what it runs under the hood, useful for doing any of it by hand
+or understanding what it does before you trust it against real
+infrastructure.
 
 ## Building
 

scripts/create-proxmox-resource.sh

@@ -3,6 +3,15 @@
 # existing ones -- the manual workflows in docs/proxmox-images.md (VM) and
 # docs/auto-installer.md's "LXC hosts" section (container), automated.
 #
+# Images are built directly on the Proxmox node (PROXMOX_REMOTE_REPO_DIR /
+# --remote-repo-dir in scripts/env.sh), not on whatever machine runs this
+# script -- there's no multi-gigabyte image to transfer afterward. The first
+# time a node doesn't have that repo path yet, it's bootstrapped: cloned from
+# this checkout's own `origin` remote, then scripts/codex-setup.sh installs
+# the build tooling (Nix, etc.). Every run after that just `git pull`s it and
+# copies over the locally-managed host-keys/ (gitignored, so a git pull
+# alone wouldn't carry it) before building.
+#
 # Usage:
 #   scripts/create-proxmox-resource.sh --type lxc|vm --host <name> [options]
 #   scripts/create-proxmox-resource.sh --type lxc|vm --list
@@ -55,11 +64,15 @@ Create mode (default):
                           Refuses to run if this ID already exists.
   --disk-size <GB>           lxc only: rootfs size for \`pct create\`
                               (default: \$PROXMOX_DEFAULT_LXC_DISK_GB, ${PROXMOX_DEFAULT_LXC_DISK_GB}).
-  --image <path>              Use this local image/tarball instead of
-                              checking the node / building one from the flake.
+  --image <path>              Use this local image/tarball (uploaded to the
+                              node via scp) instead of checking the node /
+                              building one there from the flake.
   --force-rebuild                Skip the "does the node already have this
                                   image" check -- always build fresh and
                                   overwrite what's there.
+  --remote-repo-dir <path>       Where this flake repo lives (or gets
+                                  cloned) on the node, and is built from
+                                  (default: \$PROXMOX_REMOTE_REPO_DIR, ${PROXMOX_REMOTE_REPO_DIR}).
   --allow-duplicate-host        Required if a VM/CT identified as --host
                                 already exists on the node (checked live via
                                 qm/pct, not any file in this repo) --
@@ -116,6 +129,7 @@ storage="$PROXMOX_STORAGE"
 iso_storage="$PROXMOX_ISO_STORAGE"
 bridge="$PROXMOX_BRIDGE"
 node="$PROXMOX_HOST"
+remote_repo_dir="$PROXMOX_REMOTE_REPO_DIR"
 do_list=0
 allow_duplicate_host=0
 force_rebuild=0
@@ -136,6 +150,7 @@ while [[ $# -gt 0 ]]; do
   --iso-storage) iso_storage="$2"; shift 2 ;;
   --bridge) bridge="$2"; shift 2 ;;
   --node) node="$2"; shift 2 ;;
+  --remote-repo-dir) remote_repo_dir="$2"; shift 2 ;;
   --list) do_list=1; shift ;;
   --allow-duplicate-host) allow_duplicate_host=1; shift ;;
   --force-rebuild) force_rebuild=1; shift ;;
@@ -409,6 +424,57 @@ if [[ "$type" == "lxc" ]]; then
 fi
 remote_path="${remote_dir}/${remote_filename}"
 
+# --- ensure the flake repo (+ tooling) exists on the node, and is current --
+# Bootstraps once (git clone from this checkout's own `origin`, then
+# scripts/codex-setup.sh installs Nix + friends) if ${remote_repo_dir}
+# doesn't exist yet on the node; otherwise just `git pull`s it, so the image
+# built there reflects what's actually committed and pushed. Only called
+# right before an actual remote build below -- reusing an image already on
+# the node, or an explicit --image, never touch the node's checkout at all.
+ensure_remote_repo() {
+  echo
+  echo "==> Ensuring ${remote_repo_dir} exists and is current on ${node}..."
+  if [[ "$dry_run" -eq 1 ]]; then
+    echo "[dry-run] would ensure ${remote_repo_dir} exists on ${node} (clone + scripts/codex-setup.sh if missing, git pull if present)"
+    return
+  fi
+
+  if ssh "$ssh_target" "test -d '${remote_repo_dir}/.git'"; then
+    echo "Repo present -- pulling latest..."
+    ssh "$ssh_target" "cd '${remote_repo_dir}' && git pull --ff-only"
+  else
+    local origin_url
+    origin_url="$(git -C "$repo_root" remote get-url origin 2>/dev/null || true)"
+    if [[ -z "$origin_url" ]]; then
+      echo "ERROR: ${remote_repo_dir} doesn't exist on ${node}, and this checkout has no" >&2
+      echo "'origin' remote to clone from. Set one (git remote add origin <url>) or create" >&2
+      echo "${remote_repo_dir} on ${node} yourself (e.g. git clone), then re-run." >&2
+      exit 1
+    fi
+    echo "Not present -- cloning from ${origin_url}..."
+    ssh "$ssh_target" "git clone '${origin_url}' '${remote_repo_dir}'"
+    echo "==> Bootstrapping build tooling on ${node} (scripts/codex-setup.sh)..."
+    ssh "$ssh_target" "cd '${remote_repo_dir}' && bash scripts/codex-setup.sh"
+  fi
+}
+
+# --- sync locally-managed host-keys/ to the node ---------------------------
+# Gitignored (see .gitignore), so `git pull` above never carries it -- both
+# build paths need it present as NIXOS_HOST_KEYS_DIR / --pre-format-files
+# input on the node itself now that the build runs there. scp (not rsync,
+# not already a dependency anywhere else in this repo) mirrors how this
+# script already transfers the --image case below.
+sync_remote_host_keys() {
+  echo
+  echo "==> Syncing host-keys/ to ${node}..."
+  if [[ "$dry_run" -eq 1 ]]; then
+    echo "[dry-run] would copy ${repo_root}/host-keys/ to ${ssh_target}:${remote_repo_dir}/host-keys/"
+    return
+  fi
+  ssh "$ssh_target" "mkdir -p '${remote_repo_dir}/host-keys'"
+  scp -pr "${repo_root}/host-keys/." "${ssh_target}:${remote_repo_dir}/host-keys/"
+}
+
 # --- build (or reuse an image already on the node) ------------------------
 echo
 local_image=""
@@ -425,7 +491,7 @@ else
   if [[ "$dry_run" -eq 1 ]]; then
     echo "[dry-run] would check: ssh ${ssh_target} -- test -f ${remote_path}"
   elif ssh "$ssh_target" "test -f '${remote_path}'" 2>/dev/null; then
-    echo "Found it -- reusing, skipping build and upload (use --force-rebuild to override)."
+    echo "Found it -- reusing, skipping build (use --force-rebuild to override)."
     image_already_remote=1
   else
     echo "Not found -- will build."
@@ -433,70 +499,125 @@ else
 fi
 
 if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
-  # Mirrors the real build commands' "${NIX_OPTS[@]}" below -- nix_extra_opts
-  # (called earlier, once) has already decided whether nix-cache is in play,
-  # and the dry-run preview needs to reflect that decision instead of always
-  # printing the same command regardless of outcome.
+  ensure_remote_repo
+  sync_remote_host_keys
+
+  # Relayed into the remote build below exactly as decided by the local
+  # nix_extra_opts call earlier in this script -- that decision (whether
+  # nix-cache is reachable) is made once, locally, same as it always has
+  # been; only *where* the resulting "${NIX_OPTS[@]}" gets used as a `nix
+  # build` flag moves to the node. NIX_EXTRA_OPTS is already a %q-quoted
+  # string built for exactly this eval-based reconstruction (see env.sh).
   nix_opts_display=""
   if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
     printf -v nix_opts_display '%q ' "${NIX_OPTS[@]}"
     nix_opts_display=" ${nix_opts_display% }"
   fi
+
   if [[ "$type" == "lxc" ]]; then
     if [[ "$dry_run" -eq 1 ]]; then
-      echo "[dry-run] would build: NIXOS_HOST_KEYS_DIR=${repo_root}/host-keys nix build --impure \\"
+      echo "[dry-run] would build on ${node}: NIXOS_HOST_KEYS_DIR=\$(pwd)/host-keys nix build --impure \\"
       echo "[dry-run]   --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
       echo "[dry-run]   .#nixosConfigurations.${flake_target}.config.system.build.tarball"
+      echo "[dry-run] would stage the result at ${remote_path}"
       local_image="<built-tarball>"
     else
-      echo "==> Building LXC tarball for ${flake_target}..."
-      NIXOS_HOST_KEYS_DIR="${repo_root}/host-keys" nix build --impure \
+      echo "==> Building LXC tarball for ${flake_target} on ${node}..."
+      # Built as a single already-%q-quoted command string, not separate ssh
+      # argv elements -- ssh joins remote command args with plain spaces and
+      # hands the result to the remote shell to re-split, which would
+      # otherwise scatter NIX_EXTRA_OPTS (itself several space-separated,
+      # %q-quoted tokens) across the wrong positional parameters below.
+      printf -v remote_cmd 'bash -s -- %q %q %q %q %q' \
+        "$remote_repo_dir" "$flake_target" "$remote_dir" "$remote_filename" "$NIX_EXTRA_OPTS"
+      ssh "$ssh_target" "$remote_cmd" <<'REMOTE_SCRIPT'
+set -euo pipefail
+repo_dir="$1"; target="$2"; dest_dir="$3"; dest_name="$4"; nix_extra_opts_str="$5"
+declare -a NIX_OPTS=()
+[[ -n "$nix_extra_opts_str" ]] && eval "NIX_OPTS=(${nix_extra_opts_str})"
+cd "$repo_dir"
+NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
   --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \
-        ".#nixosConfigurations.${flake_target}.config.system.build.tarball" \
-        --out-link "${repo_root}/result-${flake_target}"
-      local_image="$(find "${repo_root}/result-${flake_target}/tarball" -maxdepth 1 -type f | head -1)"
-      echo "Built: ${local_image}"
+  ".#nixosConfigurations.${target}.config.system.build.tarball" \
+  --out-link "result-${target}"
+built="$(find "result-${target}/tarball" -maxdepth 1 -type f | head -1)"
+if [[ -z "$built" ]]; then
+  echo "ERROR: no tarball found under result-${target}/tarball after build." >&2
+  exit 1
+fi
+mkdir -p "$dest_dir"
+cp "$built" "${dest_dir}/${dest_name}"
+echo "Built and staged: ${dest_dir}/${dest_name}"
+REMOTE_SCRIPT
+      local_image="$remote_path"
+      echo "Built on ${node}: ${remote_path}"
     fi
   else
+    # PROXMOX_SSH_USER defaults to root (env.sh), which needs no sudo and
+    # can't assume it's even installed on a minimal node -- only shell out
+    # through sudo when actually running as a non-root SSH user.
+    sudo_prefix="sudo"
+    sudo_display="sudo "
+    if [[ "$PROXMOX_SSH_USER" == "root" ]]; then
+      sudo_prefix=""
+      sudo_display=""
+    fi
     if [[ "$dry_run" -eq 1 ]]; then
-      echo "[dry-run] would build: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
+      echo "[dry-run] would build on ${node}: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
       echo "[dry-run]   .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
-      echo "[dry-run] would run:   sudo ./result-${flake_target} \\"
+      echo "[dry-run] would run:   ${sudo_display}./result-${flake_target} \\"
       echo "[dry-run]                --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\"
       echo "[dry-run]                --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\"
       echo "[dry-run]                --build-memory 2048"
+      echo "[dry-run] would stage the result at ${remote_path}"
       local_image="<built-image>.raw"
     else
-      echo "==> Building Disko image script for ${flake_target}..."
+      echo "==> Building Disko image for ${flake_target} on ${node}..."
+      # See the LXC branch above for why this is one %q-quoted command
+      # string rather than separate ssh argv elements.
+      printf -v remote_cmd 'bash -s -- %q %q %q %q %q %q' \
+        "$remote_repo_dir" "$flake_target" "$remote_dir" "$remote_filename" "$NIX_EXTRA_OPTS" "$sudo_prefix"
+      ssh "$ssh_target" "$remote_cmd" <<'REMOTE_SCRIPT'
+set -euo pipefail
+repo_dir="$1"; target="$2"; dest_dir="$3"; dest_name="$4"; nix_extra_opts_str="$5"; sudo_prefix="$6"
+declare -a NIX_OPTS=()
+[[ -n "$nix_extra_opts_str" ]] && eval "NIX_OPTS=(${nix_extra_opts_str})"
+cd "$repo_dir"
 nix build --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \
-        ".#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript" \
-        --out-link "${repo_root}/result-${flake_target}"
-      echo "==> Running it (builds the .raw image in a temporary QEMU VM, needs sudo)..."
-      ( cd "$repo_root" && sudo "./result-${flake_target}" \
-          --pre-format-files "host-keys/${flake_target}_ssh_host_ed25519_key" /etc/ssh/ssh_host_ed25519_key \
-          --pre-format-files "host-keys/${flake_target}_ssh_host_ed25519_key.pub" /etc/ssh/ssh_host_ed25519_key.pub \
-          --build-memory 2048 )
-      local_image="$(find "$repo_root" -maxdepth 1 -name "*.raw" -newer "${repo_root}/result-${flake_target}" | head -1)"
-      if [[ -z "$local_image" ]]; then
-        echo "ERROR: expected a .raw image after the build but didn't find one in ${repo_root}." >&2
+  ".#nixosConfigurations.${target}.config.system.build.diskoImagesScript" \
+  --out-link "result-${target}"
+$sudo_prefix "./result-${target}" \
+  --pre-format-files "host-keys/${target}_ssh_host_ed25519_key" /etc/ssh/ssh_host_ed25519_key \
+  --pre-format-files "host-keys/${target}_ssh_host_ed25519_key.pub" /etc/ssh/ssh_host_ed25519_key.pub \
+  --build-memory 2048
+built="$(find . -maxdepth 1 -name '*.raw' -newer "result-${target}" | head -1)"
+if [[ -z "$built" ]]; then
+  echo "ERROR: no .raw image found in ${repo_dir} after build." >&2
   exit 1
 fi
-      echo "Built: ${local_image}"
+mkdir -p "$dest_dir"
+mv "$built" "${dest_dir}/${dest_name}"
+echo "Built and staged: ${dest_dir}/${dest_name}"
+REMOTE_SCRIPT
+      local_image="$remote_path"
+      echo "Built on ${node}: ${remote_path}"
     fi
   fi
 fi
 
-# --- upload (skip entirely if reusing an image already on the node) ------
+# --- upload -- only for an explicit --image; a build above stages its
+# result directly at ${remote_path} on the node already, and reusing an
+# image already on the node needs nothing transferred either. ------------
 echo
-if [[ "$image_already_remote" -eq 1 ]]; then
-  : # nothing to upload
-elif [[ "$dry_run" -eq 1 ]]; then
+if [[ -n "$image" ]]; then
+  if [[ "$dry_run" -eq 1 ]]; then
     echo "[dry-run] would upload: scp ${local_image} ${ssh_target}:${remote_path}"
   else
     echo "==> Uploading to ${node}:${remote_path}..."
     ssh "$ssh_target" "mkdir -p ${remote_dir}"
     scp "$local_image" "${ssh_target}:${remote_path}"
   fi
+fi
 
 # --- create -----------------------------------------------------------------
 echo

scripts/env.sh

@@ -14,6 +14,14 @@
 : "${PROXMOX_HOST:=pve.sweet.home}"
 : "${PROXMOX_SSH_USER:=root}"
 
+# Where this flake repo lives on the Proxmox node itself.
+# scripts/create-proxmox-resource.sh builds images directly on the node
+# instead of transferring them over the network -- it clones the repo here
+# (from this checkout's own `origin` remote) the first time it doesn't
+# find it, installing build tooling via scripts/codex-setup.sh, then
+# `git pull`s it before every subsequent build.
+: "${PROXMOX_REMOTE_REPO_DIR:=/root/nixos}"
+
 # Storage pool names -- Proxmox's own stock-install defaults, but this
 # varies a lot by setup (ZFS pool name, custom LVM-thin volume, etc.).
 # Verify with `pvesm status` on the node and correct these if wrong.
@@ -49,7 +57,8 @@
 
 export PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE PROXMOX_ISO_STORAGE \
   PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES PROXMOX_DEFAULT_MEMORY_MB \
-  PROXMOX_DEFAULT_LXC_DISK_GB PROXMOX_DEFAULT_LXC_FEATURES
+  PROXMOX_DEFAULT_LXC_DISK_GB PROXMOX_DEFAULT_LXC_FEATURES \
+  PROXMOX_REMOTE_REPO_DIR
 
 # Matches variables.nix's nixCacheHost -- update both if it ever changes.
 : "${NIX_CACHE_HOST:=nix-cache}"