From a5990ccf7d93b8e584d760e71be09bf4ef5dfeca Mon Sep 17 00:00:00 2001 From: beatzaplenty Date: Mon, 20 Jul 2026 13:29:45 +0000 Subject: [PATCH] Build Proxmox images directly on the node instead of transferring them create-proxmox-resource.sh no longer builds locally and scp's a multi-gigabyte image over -- it now clones/pulls this repo onto the Proxmox node itself (bootstrapping build tooling via the existing codex-setup.sh on first use) and runs the nix build / disko image script there, staging the result straight into the node's own import directory. host-keys/ (gitignored) is copied over separately since a git pull doesn't carry it. --image still uploads an explicit local file for the case where you don't want a build at all. Co-Authored-By: Claude Sonnet 5 --- README.md | 7 +- docs/auto-installer.md | 4 +- docs/proxmox-images.md | 8 +- scripts/create-proxmox-resource.sh | 203 +++++++++++++++++++++++------ scripts/env.sh | 11 +- 5 files changed, 183 insertions(+), 50 deletions(-) diff --git a/README.md b/README.md index a262d6c..2a4928f 100644 --- a/README.md +++ b/README.md @@ -114,9 +114,10 @@ Three different paths depending on target, none of them involving a manual `docs/proxmox-images.md`. `scripts/create-proxmox-resource.sh --type lxc|vm --host ` automates -either of the last two end to end (build, host-key registration, upload, -`pct create`/`qm create`), with `--dry-run` and a guard against duplicating -an already-deployed host's identity. See its `--help`. +either of the last two end to end (host-key registration, building the +image directly on the Proxmox node itself, `pct create`/`qm create`), with +`--dry-run` and a guard against duplicating an already-deployed host's +identity. See its `--help`. ## Security Notes diff --git a/docs/auto-installer.md b/docs/auto-installer.md index d33ee95..a90392c 100644 --- a/docs/auto-installer.md +++ b/docs/auto-installer.md @@ -74,8 +74,8 @@ booting one: First boot runs `boot.postBootCommands` (registers the Nix store DB and system profile) — there's no separate activation step to run yourself. `scripts/create-proxmox-resource.sh --type lxc --host ` automates all -of this (build, host-key handling, upload, `pct create` with the flags -above) — see its `--help`. +of this (host-key handling, building the tarball directly on the Proxmox +node itself, `pct create` with the flags above) — see its `--help`. Host keys still need pre-seeding the same way as any other host — the sops-nix activation-vs-first-boot race is identical regardless of how the diff --git a/docs/proxmox-images.md b/docs/proxmox-images.md index b362682..5ce94f3 100644 --- a/docs/proxmox-images.md +++ b/docs/proxmox-images.md @@ -9,9 +9,11 @@ install, so there's nothing host-specific to write; it's available for every `scripts/create-proxmox-resource.sh --type vm --host ` automates the whole walkthrough below (and the equivalent LXC one) end to end, including -host-key handling and upload — see its `--help`. The steps here are what it -runs under the hood, useful for doing any of it by hand or understanding -what it does before you trust it against real infrastructure. +host-key handling and building the image directly on the Proxmox node +itself (no local build, no image transfer) — see its `--help`. The steps +here are what it runs under the hood, useful for doing any of it by hand +or understanding what it does before you trust it against real +infrastructure. ## Building diff --git a/scripts/create-proxmox-resource.sh b/scripts/create-proxmox-resource.sh index 1225fbf..757ed22 100755 --- a/scripts/create-proxmox-resource.sh +++ b/scripts/create-proxmox-resource.sh @@ -3,6 +3,15 @@ # existing ones -- the manual workflows in docs/proxmox-images.md (VM) and # docs/auto-installer.md's "LXC hosts" section (container), automated. # +# Images are built directly on the Proxmox node (PROXMOX_REMOTE_REPO_DIR / +# --remote-repo-dir in scripts/env.sh), not on whatever machine runs this +# script -- there's no multi-gigabyte image to transfer afterward. The first +# time a node doesn't have that repo path yet, it's bootstrapped: cloned from +# this checkout's own `origin` remote, then scripts/codex-setup.sh installs +# the build tooling (Nix, etc.). Every run after that just `git pull`s it and +# copies over the locally-managed host-keys/ (gitignored, so a git pull +# alone wouldn't carry it) before building. +# # Usage: # scripts/create-proxmox-resource.sh --type lxc|vm --host [options] # scripts/create-proxmox-resource.sh --type lxc|vm --list @@ -55,11 +64,15 @@ Create mode (default): Refuses to run if this ID already exists. --disk-size lxc only: rootfs size for \`pct create\` (default: \$PROXMOX_DEFAULT_LXC_DISK_GB, ${PROXMOX_DEFAULT_LXC_DISK_GB}). - --image Use this local image/tarball instead of - checking the node / building one from the flake. + --image Use this local image/tarball (uploaded to the + node via scp) instead of checking the node / + building one there from the flake. --force-rebuild Skip the "does the node already have this image" check -- always build fresh and overwrite what's there. + --remote-repo-dir Where this flake repo lives (or gets + cloned) on the node, and is built from + (default: \$PROXMOX_REMOTE_REPO_DIR, ${PROXMOX_REMOTE_REPO_DIR}). --allow-duplicate-host Required if a VM/CT identified as --host already exists on the node (checked live via qm/pct, not any file in this repo) -- @@ -116,6 +129,7 @@ storage="$PROXMOX_STORAGE" iso_storage="$PROXMOX_ISO_STORAGE" bridge="$PROXMOX_BRIDGE" node="$PROXMOX_HOST" +remote_repo_dir="$PROXMOX_REMOTE_REPO_DIR" do_list=0 allow_duplicate_host=0 force_rebuild=0 @@ -136,6 +150,7 @@ while [[ $# -gt 0 ]]; do --iso-storage) iso_storage="$2"; shift 2 ;; --bridge) bridge="$2"; shift 2 ;; --node) node="$2"; shift 2 ;; + --remote-repo-dir) remote_repo_dir="$2"; shift 2 ;; --list) do_list=1; shift ;; --allow-duplicate-host) allow_duplicate_host=1; shift ;; --force-rebuild) force_rebuild=1; shift ;; @@ -409,6 +424,57 @@ if [[ "$type" == "lxc" ]]; then fi remote_path="${remote_dir}/${remote_filename}" +# --- ensure the flake repo (+ tooling) exists on the node, and is current -- +# Bootstraps once (git clone from this checkout's own `origin`, then +# scripts/codex-setup.sh installs Nix + friends) if ${remote_repo_dir} +# doesn't exist yet on the node; otherwise just `git pull`s it, so the image +# built there reflects what's actually committed and pushed. Only called +# right before an actual remote build below -- reusing an image already on +# the node, or an explicit --image, never touch the node's checkout at all. +ensure_remote_repo() { + echo + echo "==> Ensuring ${remote_repo_dir} exists and is current on ${node}..." + if [[ "$dry_run" -eq 1 ]]; then + echo "[dry-run] would ensure ${remote_repo_dir} exists on ${node} (clone + scripts/codex-setup.sh if missing, git pull if present)" + return + fi + + if ssh "$ssh_target" "test -d '${remote_repo_dir}/.git'"; then + echo "Repo present -- pulling latest..." + ssh "$ssh_target" "cd '${remote_repo_dir}' && git pull --ff-only" + else + local origin_url + origin_url="$(git -C "$repo_root" remote get-url origin 2>/dev/null || true)" + if [[ -z "$origin_url" ]]; then + echo "ERROR: ${remote_repo_dir} doesn't exist on ${node}, and this checkout has no" >&2 + echo "'origin' remote to clone from. Set one (git remote add origin ) or create" >&2 + echo "${remote_repo_dir} on ${node} yourself (e.g. git clone), then re-run." >&2 + exit 1 + fi + echo "Not present -- cloning from ${origin_url}..." + ssh "$ssh_target" "git clone '${origin_url}' '${remote_repo_dir}'" + echo "==> Bootstrapping build tooling on ${node} (scripts/codex-setup.sh)..." + ssh "$ssh_target" "cd '${remote_repo_dir}' && bash scripts/codex-setup.sh" + fi +} + +# --- sync locally-managed host-keys/ to the node --------------------------- +# Gitignored (see .gitignore), so `git pull` above never carries it -- both +# build paths need it present as NIXOS_HOST_KEYS_DIR / --pre-format-files +# input on the node itself now that the build runs there. scp (not rsync, +# not already a dependency anywhere else in this repo) mirrors how this +# script already transfers the --image case below. +sync_remote_host_keys() { + echo + echo "==> Syncing host-keys/ to ${node}..." + if [[ "$dry_run" -eq 1 ]]; then + echo "[dry-run] would copy ${repo_root}/host-keys/ to ${ssh_target}:${remote_repo_dir}/host-keys/" + return + fi + ssh "$ssh_target" "mkdir -p '${remote_repo_dir}/host-keys'" + scp -pr "${repo_root}/host-keys/." "${ssh_target}:${remote_repo_dir}/host-keys/" +} + # --- build (or reuse an image already on the node) ------------------------ echo local_image="" @@ -425,7 +491,7 @@ else if [[ "$dry_run" -eq 1 ]]; then echo "[dry-run] would check: ssh ${ssh_target} -- test -f ${remote_path}" elif ssh "$ssh_target" "test -f '${remote_path}'" 2>/dev/null; then - echo "Found it -- reusing, skipping build and upload (use --force-rebuild to override)." + echo "Found it -- reusing, skipping build (use --force-rebuild to override)." image_already_remote=1 else echo "Not found -- will build." @@ -433,69 +499,124 @@ else fi if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then - # Mirrors the real build commands' "${NIX_OPTS[@]}" below -- nix_extra_opts - # (called earlier, once) has already decided whether nix-cache is in play, - # and the dry-run preview needs to reflect that decision instead of always - # printing the same command regardless of outcome. + ensure_remote_repo + sync_remote_host_keys + + # Relayed into the remote build below exactly as decided by the local + # nix_extra_opts call earlier in this script -- that decision (whether + # nix-cache is reachable) is made once, locally, same as it always has + # been; only *where* the resulting "${NIX_OPTS[@]}" gets used as a `nix + # build` flag moves to the node. NIX_EXTRA_OPTS is already a %q-quoted + # string built for exactly this eval-based reconstruction (see env.sh). nix_opts_display="" if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then printf -v nix_opts_display '%q ' "${NIX_OPTS[@]}" nix_opts_display=" ${nix_opts_display% }" fi + if [[ "$type" == "lxc" ]]; then if [[ "$dry_run" -eq 1 ]]; then - echo "[dry-run] would build: NIXOS_HOST_KEYS_DIR=${repo_root}/host-keys nix build --impure \\" + echo "[dry-run] would build on ${node}: NIXOS_HOST_KEYS_DIR=\$(pwd)/host-keys nix build --impure \\" echo "[dry-run] --no-use-registries --no-accept-flake-config${nix_opts_display} \\" echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.tarball" + echo "[dry-run] would stage the result at ${remote_path}" local_image="" else - echo "==> Building LXC tarball for ${flake_target}..." - NIXOS_HOST_KEYS_DIR="${repo_root}/host-keys" nix build --impure \ - --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \ - ".#nixosConfigurations.${flake_target}.config.system.build.tarball" \ - --out-link "${repo_root}/result-${flake_target}" - local_image="$(find "${repo_root}/result-${flake_target}/tarball" -maxdepth 1 -type f | head -1)" - echo "Built: ${local_image}" + echo "==> Building LXC tarball for ${flake_target} on ${node}..." + # Built as a single already-%q-quoted command string, not separate ssh + # argv elements -- ssh joins remote command args with plain spaces and + # hands the result to the remote shell to re-split, which would + # otherwise scatter NIX_EXTRA_OPTS (itself several space-separated, + # %q-quoted tokens) across the wrong positional parameters below. + printf -v remote_cmd 'bash -s -- %q %q %q %q %q' \ + "$remote_repo_dir" "$flake_target" "$remote_dir" "$remote_filename" "$NIX_EXTRA_OPTS" + ssh "$ssh_target" "$remote_cmd" <<'REMOTE_SCRIPT' +set -euo pipefail +repo_dir="$1"; target="$2"; dest_dir="$3"; dest_name="$4"; nix_extra_opts_str="$5" +declare -a NIX_OPTS=() +[[ -n "$nix_extra_opts_str" ]] && eval "NIX_OPTS=(${nix_extra_opts_str})" +cd "$repo_dir" +NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \ + --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \ + ".#nixosConfigurations.${target}.config.system.build.tarball" \ + --out-link "result-${target}" +built="$(find "result-${target}/tarball" -maxdepth 1 -type f | head -1)" +if [[ -z "$built" ]]; then + echo "ERROR: no tarball found under result-${target}/tarball after build." >&2 + exit 1 +fi +mkdir -p "$dest_dir" +cp "$built" "${dest_dir}/${dest_name}" +echo "Built and staged: ${dest_dir}/${dest_name}" +REMOTE_SCRIPT + local_image="$remote_path" + echo "Built on ${node}: ${remote_path}" fi else + # PROXMOX_SSH_USER defaults to root (env.sh), which needs no sudo and + # can't assume it's even installed on a minimal node -- only shell out + # through sudo when actually running as a non-root SSH user. + sudo_prefix="sudo" + sudo_display="sudo " + if [[ "$PROXMOX_SSH_USER" == "root" ]]; then + sudo_prefix="" + sudo_display="" + fi if [[ "$dry_run" -eq 1 ]]; then - echo "[dry-run] would build: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\" + echo "[dry-run] would build on ${node}: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\" echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript" - echo "[dry-run] would run: sudo ./result-${flake_target} \\" + echo "[dry-run] would run: ${sudo_display}./result-${flake_target} \\" echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\" echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\" echo "[dry-run] --build-memory 2048" + echo "[dry-run] would stage the result at ${remote_path}" local_image=".raw" else - echo "==> Building Disko image script for ${flake_target}..." - nix build --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \ - ".#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript" \ - --out-link "${repo_root}/result-${flake_target}" - echo "==> Running it (builds the .raw image in a temporary QEMU VM, needs sudo)..." - ( cd "$repo_root" && sudo "./result-${flake_target}" \ - --pre-format-files "host-keys/${flake_target}_ssh_host_ed25519_key" /etc/ssh/ssh_host_ed25519_key \ - --pre-format-files "host-keys/${flake_target}_ssh_host_ed25519_key.pub" /etc/ssh/ssh_host_ed25519_key.pub \ - --build-memory 2048 ) - local_image="$(find "$repo_root" -maxdepth 1 -name "*.raw" -newer "${repo_root}/result-${flake_target}" | head -1)" - if [[ -z "$local_image" ]]; then - echo "ERROR: expected a .raw image after the build but didn't find one in ${repo_root}." >&2 - exit 1 - fi - echo "Built: ${local_image}" + echo "==> Building Disko image for ${flake_target} on ${node}..." + # See the LXC branch above for why this is one %q-quoted command + # string rather than separate ssh argv elements. + printf -v remote_cmd 'bash -s -- %q %q %q %q %q %q' \ + "$remote_repo_dir" "$flake_target" "$remote_dir" "$remote_filename" "$NIX_EXTRA_OPTS" "$sudo_prefix" + ssh "$ssh_target" "$remote_cmd" <<'REMOTE_SCRIPT' +set -euo pipefail +repo_dir="$1"; target="$2"; dest_dir="$3"; dest_name="$4"; nix_extra_opts_str="$5"; sudo_prefix="$6" +declare -a NIX_OPTS=() +[[ -n "$nix_extra_opts_str" ]] && eval "NIX_OPTS=(${nix_extra_opts_str})" +cd "$repo_dir" +nix build --no-use-registries --no-accept-flake-config "${NIX_OPTS[@]}" \ + ".#nixosConfigurations.${target}.config.system.build.diskoImagesScript" \ + --out-link "result-${target}" +$sudo_prefix "./result-${target}" \ + --pre-format-files "host-keys/${target}_ssh_host_ed25519_key" /etc/ssh/ssh_host_ed25519_key \ + --pre-format-files "host-keys/${target}_ssh_host_ed25519_key.pub" /etc/ssh/ssh_host_ed25519_key.pub \ + --build-memory 2048 +built="$(find . -maxdepth 1 -name '*.raw' -newer "result-${target}" | head -1)" +if [[ -z "$built" ]]; then + echo "ERROR: no .raw image found in ${repo_dir} after build." >&2 + exit 1 +fi +mkdir -p "$dest_dir" +mv "$built" "${dest_dir}/${dest_name}" +echo "Built and staged: ${dest_dir}/${dest_name}" +REMOTE_SCRIPT + local_image="$remote_path" + echo "Built on ${node}: ${remote_path}" fi fi fi -# --- upload (skip entirely if reusing an image already on the node) ------ +# --- upload -- only for an explicit --image; a build above stages its +# result directly at ${remote_path} on the node already, and reusing an +# image already on the node needs nothing transferred either. ------------ echo -if [[ "$image_already_remote" -eq 1 ]]; then - : # nothing to upload -elif [[ "$dry_run" -eq 1 ]]; then - echo "[dry-run] would upload: scp ${local_image} ${ssh_target}:${remote_path}" -else - echo "==> Uploading to ${node}:${remote_path}..." - ssh "$ssh_target" "mkdir -p ${remote_dir}" - scp "$local_image" "${ssh_target}:${remote_path}" +if [[ -n "$image" ]]; then + if [[ "$dry_run" -eq 1 ]]; then + echo "[dry-run] would upload: scp ${local_image} ${ssh_target}:${remote_path}" + else + echo "==> Uploading to ${node}:${remote_path}..." + ssh "$ssh_target" "mkdir -p ${remote_dir}" + scp "$local_image" "${ssh_target}:${remote_path}" + fi fi # --- create ----------------------------------------------------------------- diff --git a/scripts/env.sh b/scripts/env.sh index b0c48c1..cf053fa 100755 --- a/scripts/env.sh +++ b/scripts/env.sh @@ -14,6 +14,14 @@ : "${PROXMOX_HOST:=pve.sweet.home}" : "${PROXMOX_SSH_USER:=root}" +# Where this flake repo lives on the Proxmox node itself. +# scripts/create-proxmox-resource.sh builds images directly on the node +# instead of transferring them over the network -- it clones the repo here +# (from this checkout's own `origin` remote) the first time it doesn't +# find it, installing build tooling via scripts/codex-setup.sh, then +# `git pull`s it before every subsequent build. +: "${PROXMOX_REMOTE_REPO_DIR:=/root/nixos}" + # Storage pool names -- Proxmox's own stock-install defaults, but this # varies a lot by setup (ZFS pool name, custom LVM-thin volume, etc.). # Verify with `pvesm status` on the node and correct these if wrong. @@ -49,7 +57,8 @@ export PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE PROXMOX_ISO_STORAGE \ PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES PROXMOX_DEFAULT_MEMORY_MB \ - PROXMOX_DEFAULT_LXC_DISK_GB PROXMOX_DEFAULT_LXC_FEATURES + PROXMOX_DEFAULT_LXC_DISK_GB PROXMOX_DEFAULT_LXC_FEATURES \ + PROXMOX_REMOTE_REPO_DIR # Matches variables.nix's nixCacheHost -- update both if it ever changes. : "${NIX_CACHE_HOST:=nix-cache}" -- 2.54.0