Merge branch 'main' into worktree-migrate-vm-script

Reuses the clone script's vzdump/relay/restore flow but as a move rather
than a copy: preserves the original MAC by default, adds a stop-source/
start-target cutover step, and an opt-in --remove-source to destroy the
source guest afterward. Not a true qm migrate (nodes aren't clustered),
so the default --mode snapshot has a documented write gap between
snapshot and cutover; --mode stop avoids it at the cost of downtime.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013Lkv8k24Wuua3GRxrfMBLz

CLAUDE.md

@@ -256,6 +256,33 @@ lift the shared part into `scripts/lib/` instead of copying it.
   `/etc/ssh/ssh_known_hosts`. Idempotent (re-running replaces its own
   marked block rather than duplicating it); restarts `nix-daemon` by
   default so the change takes effect immediately.
+- `scripts/proxmox/clone-pve1-to-pve-test.sh --vmid <n> [options]` — ad hoc
+  copy of a single VM/CT from one node to another via `vzdump` +
+  `qmrestore`/`pct restore`, relayed through this machine
+  (`ssh src cat ... | ssh dst cat > ...`, no on-disk staging copy).
+  Defaults `pve1` → `pve-test` (see "Two Proxmox nodes" above); source is
+  left running throughout (`--mode snapshot` default) and untouched
+  beyond the vzdump + a typed-VMID confirm, restored on the target with a
+  fresh MAC (`--unique 1`, always) since the source is expected to still
+  be live on the same LAN. Backup archives are deleted from both nodes
+  afterward unless `--keep-backup`. `--dry-run` throughout.
+- `scripts/proxmox/migrate-vm.sh --vmid <n> [options]` — moves (rather
+  than clones) a single VM/CT between nodes, built on the same
+  vzdump/relay/restore pattern as `clone-pve1-to-pve-test.sh` above, plus
+  a cutover step (stop the guest on the source node, start it on the
+  target) and, optionally, `--remove-source` to destroy it on the source
+  node afterward (its own separate typed confirmation, since that step is
+  irreversible) — the default instead leaves the stopped source guest
+  intact as a safety net. Restores with the *original* MAC (no
+  `--unique`) unless `--unique` is passed, since the source is expected to
+  go away rather than keep running alongside the target. **Not** a true
+  `qm migrate`-style live migration — pve1/pve-test aren't a clustered
+  pair with shared storage, so this is vzdump-based and, with the default
+  `--mode snapshot`, any writes on the source between the snapshot and
+  cutover are lost (use `--mode stop` if that's not acceptable). Generic
+  node-to-node (not pve1/pve-test-specific) via
+  `--source-node`/`--target-node`, defaulting to pve1 → pve-test like the
+  clone script. `--dry-run` throughout.
 
 ### `scripts/lib/`
 

docs/auto-installer.md

@@ -150,7 +150,11 @@ use case.
 
 The `pxe` variant is also built automatically as part of the `pxe-boot` host
 itself (`modules/pxe-boot/stage-installer-artifacts.nix`) and served over
-iPXE — see `docs/pxe-boot.md`.
+iPXE as the menu's "NixOS Auto-Installer" entry — see `docs/pxe-boot.md`.
+That same host also builds and serves `packages.x86_64-linux.pxe-minimal`,
+a vanilla NixOS minimal netboot image with none of this auto-installer's
+wiring, as a separate "NixOS Minimal" menu entry — also documented in
+`docs/pxe-boot.md`, not covered further here since it's not this installer.
 
 ## Host keys
 

docs/pxe-boot.md

@@ -1,9 +1,10 @@
 # pxe-boot
 
-The `pxe-boot` host serves HTTP boot assets for iPXE clients — including a
+The `pxe-boot` host serves HTTP boot assets for iPXE clients — including
-self-staged copy of this flake's own auto-installer netboot image, see
+self-staged copies of both this flake's own auto-installer netboot image
-`docs/auto-installer.md` for what that image actually is and does once
+(see `docs/auto-installer.md` for what that image actually is and does once
-booted.
+booted) and a vanilla, unmodified NixOS minimal netboot image for plain
+rescue/inspection use.
 
 ## Host Role
 
@@ -28,7 +29,8 @@ The host creates these directories with systemd tmpfiles:
 /srv/pxe
 /srv/pxe/http
 /srv/pxe/http/images
-/srv/pxe/http/nixos
+/srv/pxe/http/auto-installer
+/srv/pxe/http/nixos-minimal
 /srv/pxe/http/systemrescue
 /srv/pxe/http/ubuntu
 /srv/pxe/http/rescue
@@ -37,7 +39,7 @@ The host creates these directories with systemd tmpfiles:
 
 Mount shared image storage under `/srv/pxe/http`, preferably
 `/srv/pxe/http/images` unless a menu entry expects files in a specific
-directory such as `/srv/pxe/http/nixos`.
+directory such as `/srv/pxe/http/auto-installer`.
 
 The HTTP iPXE chain is:
 
@@ -50,20 +52,37 @@ undionly.kpxe or ipxe.efi
 
 The generated menu currently exposes entries for:
 
-- NixOS installer
+- NixOS Auto-Installer
+- NixOS Minimal
 - SystemRescue environment
 - iPXE shell
 - Reboot
 
-The NixOS installer entry chain-loads `/srv/pxe/http/nixos/netboot.ipxe`,
+Both NixOS entries chain-load a `netboot.ipxe` staged into their own
-which is nixpkgs' own generated netboot iPXE script (correct `init=`/`initrd=`
+directory (`/srv/pxe/http/auto-installer/netboot.ipxe` and
-kernel parameters included) rather than a hand-rolled boot line — that script
+`/srv/pxe/http/nixos-minimal/netboot.ipxe`), each nixpkgs' own generated
-in turn expects its kernel/initrd siblings in the same directory. All three
+netboot iPXE script (correct `init=`/`initrd=` kernel parameters included)
-files (`bzImage`, `initrd`, `netboot.ipxe`) are built from this flake's own
+rather than a hand-rolled boot line — that script in turn expects its
-`modules/installer/iso.nix` netboot image (the same one `nix build .#pxe`
+kernel/initrd siblings in the same directory. Each directory's three files
-produces) and staged automatically by
+(`bzImage`, `initrd`, `netboot.ipxe`) are built from source and staged
-`modules/pxe-boot/stage-installer-artifacts.nix` via `systemd.tmpfiles.rules`
+automatically by `modules/pxe-boot/stage-installer-artifacts.nix` via
-— no manual operator step required.
+`systemd.tmpfiles.rules` — no manual operator step required:
+
+- `auto-installer` is this flake's own `netbootSystem` (`flake.nix`) — the
+  same auto-installer image `nix build .#pxe` produces. See
+  `docs/auto-installer.md`.
+- `nixos-minimal` is `netbootMinimalSystem` (`flake.nix`) — nixpkgs'
+  `netboot-minimal.nix` composed on its own, with none of this flake's
+  auto-installer wiring (no `common.nix`, no `auto-install.sh`, no baked
+  host keys or custom users). Same `nix build .#pxe-minimal` mechanism as
+  the auto-installer image, just a different module composition. Useful
+  as a plain rescue/inspection shell that doesn't assume anything about
+  this flake.
+
+Both images set `networking.hostName` to match their menu entry/staged
+directory name (`auto-installer` / `nixos-minimal`), so each one's
+generated system name (`nixos-system-<name>-*`) is self-describing rather
+than the nixpkgs default of `nixos-system-nixos-*` for both.
 
 The SystemRescue entry expects the source ISO at:
 

flake.nix

@@ -65,7 +65,7 @@
           # file without a same-option circular dependency (a module
           # contributing to environment.etc can't read the merged
           # environment.etc it's itself contributing to).
-          specialArgs = { inherit inputs vars netbootSystem flakeTarget; };
+          specialArgs = { inherit inputs vars netbootSystem netbootMinimalSystem flakeTarget; };
         };
 
       # Generated platform x build-type matrix. pxe-boot has no linode
@@ -119,19 +119,61 @@
 
       # Same installer environment, built as netboot (kernel + initrd +
       # iPXE script) instead of an ISO — this is what packages.pxe bundles.
+      #
+      # Deliberately imports common.nix directly, NOT ./modules/installer/iso.nix
+      # (which pulls in nixpkgs' installation-cd-minimal.nix) -- confirmed live
+      # that composing the ISO module together with netboot-minimal.nix hangs
+      # every boot waiting for a device that can never exist on a netboot
+      # client ("A start job is running for /dev/disk/by-label/nixos-minimal-...").
+      # Both installation-cd-base.nix and netboot.nix set fileSystems."/" via
+      # the identical lib.mkImageMediaOverride (mkOverride 60) priority --
+      # genuinely conflicting root-filesystem strategies (ISO-by-label vs.
+      # netboot-tmpfs) at the same priority, and the ISO one was winning.
+      # netboot-minimal.nix's own chain (netboot-base.nix) already imports
+      # profiles/installation-device.nix independently, so common.nix's
+      # initialHashedPassword override (which assumes that profile is
+      # present) still applies correctly without iso.nix in the mix.
+      #
+      # networking.hostName is set explicitly (rather than left at nixpkgs'
+      # own "nixos" default) so this image's generated system name
+      # (nixos-system-auto-installer-*) matches its iPXE menu entry —
+      # see modules/build-types/pxe-boot.nix's :auto-installer item — and
+      # its staged directory, /srv/pxe/http/auto-installer.
       netbootSystem = nixpkgs.lib.nixosSystem {
         inherit system;
         modules = [
-          ./modules/installer/iso.nix
+          ./modules/installer/common.nix
           ({ modulesPath, ... }: {
             imports = [
               (modulesPath + "/installer/netboot/netboot-minimal.nix")
             ];
           })
+          { networking.hostName = "auto-installer"; }
         ];
         specialArgs = { inherit vars; };
       };
 
+      # A genuinely vanilla NixOS minimal netboot image: nixpkgs'
+      # netboot-minimal.nix on its own, with none of this flake's
+      # auto-installer wiring (no common.nix — no auto-install.sh, no
+      # baked host keys, no custom users/passwords). Built from source via
+      # the same nixosSystem + netboot-minimal.nix path as netbootSystem
+      # above, so both go through an identical build mechanism; the only
+      # difference is what's composed in. hostName again matches this
+      # image's iPXE menu entry (:nixos-minimal) and staged directory
+      # (/srv/pxe/http/nixos-minimal).
+      netbootMinimalSystem = nixpkgs.lib.nixosSystem {
+        inherit system;
+        modules = [
+          ({ modulesPath, ... }: {
+            imports = [
+              (modulesPath + "/installer/netboot/netboot-minimal.nix")
+            ];
+          })
+          { networking.hostName = "nixos-minimal"; }
+        ];
+      };
+
     in
     {
 
@@ -153,6 +195,15 @@
           { name = "initrd"; path = netbootSystem.config.system.build.netbootRamdisk; }
           { name = "kernel"; path = netbootSystem.config.system.build.kernel; }
         ];
+
+        # Vanilla NixOS minimal netboot bundle — see netbootMinimalSystem
+        # above. Staged onto the pxe-boot host alongside packages.pxe by
+        # modules/pxe-boot/stage-installer-artifacts.nix.
+        pxe-minimal = pkgs.linkFarm "pxe-minimal" [
+          { name = "netboot.ipxe"; path = netbootMinimalSystem.config.system.build.netbootIpxeScript; }
+          { name = "initrd"; path = netbootMinimalSystem.config.system.build.netbootRamdisk; }
+          { name = "kernel"; path = netbootMinimalSystem.config.system.build.kernel; }
+        ];
       };
     };
 }

modules/build-types/pxe-boot.nix

@@ -68,15 +68,19 @@ let
     set base ${pxeBaseUrl}
 
     menu PXE Boot Menu
-    item nixos NixOS Installer
+    item auto-installer NixOS Auto-Installer
+    item nixos-minimal NixOS Minimal
     item rescue Rescue Environment
     item shell iPXE Shell
     item reboot Reboot
 
     choose target && goto ''${target}
 
-    :nixos
+    :auto-installer
-    chain ''${base}/nixos/netboot.ipxe
+    chain ''${base}/auto-installer/netboot.ipxe
+
+    :nixos-minimal
+    chain ''${base}/nixos-minimal/netboot.ipxe
 
     :rescue
     chain ''${base}/systemrescue.ipxe
@@ -129,7 +133,8 @@ in
     "d ${pxeRoot} 0755 root root -"
     "d ${httpRoot} 0755 root root -"
     "d ${httpRoot}/images 0755 root root -"
-    "d ${httpRoot}/nixos 0755 root root -"
+    "d ${httpRoot}/auto-installer 0755 root root -"
+    "d ${httpRoot}/nixos-minimal 0755 root root -"
     "d ${httpRoot}/systemrescue 0755 root root -"
     "d ${httpRoot}/ubuntu 0755 root root -"
     "d ${httpRoot}/rescue 0755 root root -"

modules/pxe-boot/stage-installer-artifacts.nix

@@ -1,23 +1,32 @@
-{ netbootSystem, ... }:
+{ netbootSystem, netbootMinimalSystem, ... }:
 
 let
   # config.system.build.kernel and .netbootRamdisk are directories, not the
   # files themselves — nixpkgs' own system.build.kexecTree does the same
   # ${...}/<file> dereference for the same reason.
-  inherit (netbootSystem.config.system.boot.loader) kernelFile;
+  mkStageRules = { dirName, system }:
+    let
+      inherit (system.config.system.boot.loader) kernelFile;
+      dir = "/srv/pxe/http/${dirName}";
+    in
+    [
+      # Declared here too (not just in build-types/pxe-boot.nix) so this
+      # module's C+ rules don't depend on cross-module list-merge ordering —
+      # tmpfiles' C type needs the target directory to already exist.
+      "d ${dir} 0755 root root -"
+      "C+ ${dir}/${kernelFile} 0644 root root - ${system.config.system.build.kernel}/${kernelFile}"
+      "C+ ${dir}/initrd 0644 root root - ${system.config.system.build.netbootRamdisk}/initrd"
+      "C+ ${dir}/netboot.ipxe 0644 root root - ${system.config.system.build.netbootIpxeScript}/netboot.ipxe"
+    ];
 in
 {
   # Builds this flake's own installer netboot image (the same one
-  # `nix build .#pxe` produces) and stages it where menu.ipxe's :nixos
+  # `nix build .#pxe` produces) plus the vanilla NixOS minimal netboot image
-  # entry expects it, so the pxe-boot host is self-contained — no manual
+  # (`nix build .#pxe-minimal`), and stages both where menu.ipxe's
-  # operator step to populate /srv/pxe/http/nixos after deploy.
+  # :auto-installer / :nixos-minimal entries expect them, so the pxe-boot
-  systemd.tmpfiles.rules = [
+  # host is self-contained — no manual operator step to populate
-    # Declared here too (not just in build-types/pxe-boot.nix) so this
+  # /srv/pxe/http after deploy.
-    # module's C+ rules don't depend on cross-module list-merge ordering —
+  systemd.tmpfiles.rules =
-    # tmpfiles' C type needs the target directory to already exist.
+    mkStageRules { dirName = "auto-installer"; system = netbootSystem; }
-    "d /srv/pxe/http/nixos 0755 root root -"
+    ++ mkStageRules { dirName = "nixos-minimal"; system = netbootMinimalSystem; };
-    "C+ /srv/pxe/http/nixos/${kernelFile} 0644 root root - ${netbootSystem.config.system.build.kernel}/${kernelFile}"
-    "C+ /srv/pxe/http/nixos/initrd 0644 root root - ${netbootSystem.config.system.build.netbootRamdisk}/initrd"
-    "C+ /srv/pxe/http/nixos/netboot.ipxe 0644 root root - ${netbootSystem.config.system.build.netbootIpxeScript}/netboot.ipxe"
-  ];
 }

scripts/proxmox/clone-pve1-to-pve-test.sh

@@ -0,0 +1,221 @@
+#!/usr/bin/env bash
+# Ad hoc clone of a single VM/CT from pve1 (production) to pve-test
+# (sandbox), via vzdump + qmrestore/pct restore -- not a general-purpose
+# backup tool, just a quick "give me a disposable copy of this thing on
+# pve-test" for testing against real-ish data without touching prod.
+#
+# Flow:
+#   1. vzdump the resource on pve1 into its "local" storage (--mode
+#      snapshot by default, so the source keeps running throughout --
+#      see --mode below for when that's not possible).
+#   2. Stream the resulting archive straight from pve1 to pve-test
+#      (ssh pve1 cat ... | ssh pve-test cat > ...) -- this machine is
+#      just the relay, no separate on-disk staging copy here.
+#   3. qmrestore / pct restore it on pve-test under --new-vmid (default:
+#      same VMID as the source -- pve-test is a separate node/cluster, so
+#      no collision unless that VMID is already in use there too).
+#      Always restored with --unique 1 (fresh MAC addresses) since the
+#      source is typically still running on the same LAN -- restoring
+#      with the *same* MAC would put two live guests on the wire with
+#      identical hardware addresses.
+#   4. Delete the vzdump archive from pve1's local storage and the
+#      relayed copy on pve-test, so neither node accumulates ad hoc
+#      backup files from this script. Only the pve1 original is
+#      preserved on any failure after step 1, so a failed
+#      transfer/restore can be retried without re-running the backup.
+#
+# This script's own defaults are pve1 -> pve-test, unlike
+# create-proxmox-resource.sh's --node (which defaults to production) --
+# see CLAUDE.md's "Two Proxmox nodes" section. pve1 is only ever touched
+# here after typing the source VMID back to confirm; pve-test is treated
+# as disposable, matching this repo's usual policy for that node.
+#
+# See --help for the full option list.
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
+# shellcheck source=../env.sh
+source "${repo_root}/scripts/env.sh"
+# shellcheck source=../lib/confirm.sh
+source "${repo_root}/scripts/lib/confirm.sh"
+
+usage() {
+  cat <<EOF
+Usage: $0 --vmid <n> [options]
+
+  --vmid <n>              Required: VMID on the source node to clone.
+                           Kind (qemu VM vs LXC CT) is auto-detected.
+  --new-vmid <n>           VMID to restore as on the target node
+                           (default: same as --vmid).
+  --mode snapshot|suspend|stop
+                           vzdump backup mode (default: snapshot -- the
+                           source resource keeps running throughout;
+                           requires snapshot-capable storage, e.g.
+                           ZFS/LVM-thin/Ceph/qcow2). Fall back to
+                           "suspend" (brief pause) or "stop" (source
+                           goes down for the duration) if the source's
+                           storage doesn't support live snapshots --
+                           vzdump's own error will say so.
+  --source-node <host>      (default: \$PVE1_HOST, ${PVE1_HOST})
+  --target-node <host>       (default: \$PVE_TEST_HOST, ${PVE_TEST_HOST})
+  --source-storage <pool>      Where vzdump writes the backup on the
+                               source node (default: local).
+  --target-storage <pool>       Where the restored disk/rootfs lands on
+                               the target node (default: \$PROXMOX_STORAGE, ${PROXMOX_STORAGE}).
+  --keep-backup                  Don't delete the vzdump archive from
+                                 either node afterward (debugging aid).
+  --yes                            Skip the typed VMID confirmation
+                                   before touching the source node.
+  --dry-run                          Print the full plan and skip every
+                                     mutating step (vzdump, transfer,
+                                     restore, delete) and the confirm
+                                     prompt. Still makes read-only SSH
+                                     calls to look up the source kind
+                                     and check the target VMID is free
+                                     -- harmless on either node.
+  -h, --help
+EOF
+}
+
+vmid=""
+new_vmid=""
+mode="snapshot"
+source_node="$PVE1_HOST"
+target_node="$PVE_TEST_HOST"
+source_storage="local"
+target_storage="$PROXMOX_STORAGE"
+keep_backup=0
+skip_confirm=0
+dry_run=0
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+  --vmid) vmid="$2"; shift 2 ;;
+  --new-vmid) new_vmid="$2"; shift 2 ;;
+  --mode) mode="$2"; shift 2 ;;
+  --source-node) source_node="$2"; shift 2 ;;
+  --target-node) target_node="$2"; shift 2 ;;
+  --source-storage) source_storage="$2"; shift 2 ;;
+  --target-storage) target_storage="$2"; shift 2 ;;
+  --keep-backup) keep_backup=1; shift ;;
+  --yes) skip_confirm=1; shift ;;
+  --dry-run) dry_run=1; shift ;;
+  -h | --help) usage; exit 0 ;;
+  *) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;;
+  esac
+done
+
+if [[ -z "$vmid" ]]; then
+  echo "ERROR: --vmid is required." >&2
+  usage >&2
+  exit 1
+fi
+if [[ "$mode" != "snapshot" && "$mode" != "suspend" && "$mode" != "stop" ]]; then
+  echo "ERROR: --mode must be snapshot, suspend, or stop." >&2
+  exit 1
+fi
+[[ -z "$new_vmid" ]] && new_vmid="$vmid"
+
+source_target="${PROXMOX_SSH_USER}@${source_node}"
+target_target="${PROXMOX_SSH_USER}@${target_node}"
+
+# No dry-run wrapper needed for the calls below: every mutating step
+# (vzdump, transfer, restore, delete) is reached only after the --dry-run
+# early-exit further down, so a plain `ssh` call is never in the dry-run
+# path.
+
+# --- identify the resource kind on the source node -----------------------
+echo "==> Looking up VMID ${vmid} on ${source_node}..."
+kind=""
+if ssh "$source_target" "qm status ${vmid}" >/dev/null 2>&1; then
+  kind="vm"
+elif ssh "$source_target" "pct status ${vmid}" >/dev/null 2>&1; then
+  kind="lxc"
+else
+  echo "ERROR: VMID ${vmid} doesn't exist on ${source_node} as either a VM or CT." >&2
+  exit 1
+fi
+echo "VMID ${vmid} on ${source_node} is a ${kind}."
+
+# --- refuse to clobber an existing resource on the target node -----------
+if ssh "$target_target" "qm status ${new_vmid}" >/dev/null 2>&1 \
+  || ssh "$target_target" "pct status ${new_vmid}" >/dev/null 2>&1; then
+  echo "ERROR: VMID ${new_vmid} already exists on ${target_node}. Pass --new-vmid" >&2
+  echo "with a free ID, or remove the existing resource there first." >&2
+  exit 1
+fi
+
+echo
+echo "Plan:"
+echo "  source: ${kind} VMID ${vmid} on ${source_node} (storage: ${source_storage}, mode: ${mode})"
+echo "  target: VMID ${new_vmid} on ${target_node} (storage: ${target_storage}, fresh MAC via --unique)"
+[[ "$keep_backup" -eq 1 ]] && echo "  backup archives are kept on both nodes afterward (--keep-backup)"
+
+if [[ "$dry_run" -eq 1 ]]; then
+  echo
+  echo "[dry-run] No backup, transfer, restore, or delete was performed."
+  exit 0
+fi
+
+if [[ "$skip_confirm" -ne 1 ]]; then
+  echo
+  if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) to confirm backing it up from ${source_node}: "; then
+    echo "Cancelled -- input didn't match ${vmid}." >&2
+    exit 1
+  fi
+fi
+
+# --- vzdump on the source node --------------------------------------------
+echo
+echo "==> Backing up VMID ${vmid} on ${source_node} (mode=${mode}, storage=${source_storage})..."
+vzdump_log="$(ssh "$source_target" \
+  "vzdump ${vmid} --mode ${mode} --storage ${source_storage} --compress zstd" 2>&1)" \
+  || {
+    echo "$vzdump_log" >&2
+    echo "ERROR: vzdump failed on ${source_node}." >&2
+    exit 1
+  }
+echo "$vzdump_log"
+
+archive="$(echo "$vzdump_log" | grep -oP "creating vzdump archive '\K[^']+" | tail -n1)"
+if [[ -z "$archive" ]]; then
+  echo "ERROR: couldn't find the archive path in vzdump's output above." >&2
+  exit 1
+fi
+archive_basename="$(basename "$archive")"
+target_tmp_archive="/var/tmp/${archive_basename}"
+echo "Archive: ${archive}"
+
+# Always clean up the relayed copy on the target node, success or failure
+# -- it's only ever a working copy, restored or not.
+cleanup_target_tmp() {
+  if [[ "$keep_backup" -ne 1 ]]; then
+    ssh "$target_target" "rm -f '${target_tmp_archive}'" >/dev/null 2>&1 || true
+  fi
+}
+trap cleanup_target_tmp EXIT
+
+# --- relay the archive from source to target ------------------------------
+echo
+echo "==> Transferring archive to ${target_node}..."
+ssh "$source_target" "cat '${archive}'" | ssh "$target_target" "cat > '${target_tmp_archive}'"
+
+# --- restore on the target node --------------------------------------------
+echo
+echo "==> Restoring as VMID ${new_vmid} on ${target_node} (storage=${target_storage})..."
+if [[ "$kind" == "vm" ]]; then
+  ssh "$target_target" "qmrestore '${target_tmp_archive}' ${new_vmid} --storage ${target_storage} --unique 1"
+else
+  ssh "$target_target" "pct restore ${new_vmid} '${target_tmp_archive}' --storage ${target_storage} --unique 1"
+fi
+
+# --- clean up the source backup now that the restore succeeded -----------
+if [[ "$keep_backup" -ne 1 ]]; then
+  echo
+  echo "==> Deleting backup archive from ${source_node}'s ${source_storage} storage..."
+  ssh "$source_target" "rm -f '${archive}' '${archive}.notes' '${archive}.log'" >/dev/null 2>&1 || true
+fi
+
+echo
+echo "Done. VMID ${new_vmid} (${kind}) is now on ${target_node}, cloned from" \
+  "VMID ${vmid} on ${source_node}."

scripts/proxmox/migrate-vm.sh

@@ -0,0 +1,347 @@
+#!/usr/bin/env bash
+# Move a single VM/CT from one Proxmox node to another via vzdump +
+# qmrestore/pct restore -- the same relay pattern as
+# clone-pve1-to-pve-test.sh, but for a *migration* (the guest ends up
+# living on the target node only) rather than a *clone* (a disposable
+# copy, source untouched, fresh MAC).
+#
+# There is no real `qm migrate` here: that command only works between
+# nodes in the same Proxmox cluster with shared/replicated storage, which
+# pve1 and pve-test are not (see CLAUDE.md's "Two Proxmox nodes" section --
+# they're deliberately separate, unclustered nodes). This script is the
+# closest equivalent across two independent nodes, built out of the same
+# primitives as the clone script. It is NOT a true live migration -- read
+# the caveat below before using it for anything where losing a few
+# seconds/minutes of writes is unacceptable.
+#
+# Flow:
+#   1. vzdump the resource on the source node (--mode snapshot by
+#      default, so the source keeps running throughout the bulk copy --
+#      this is the part that's "live": the guest stays reachable while
+#      its disk data is captured and shipped to the target).
+#   2. Stream the archive straight from the source node to the target
+#      node (ssh source cat ... | ssh target cat > ...), same relay this
+#      machine does in the clone script -- no separate on-disk staging
+#      copy here either.
+#   3. qmrestore / pct restore it on the target node under the same VMID
+#      by default (--new-vmid to pick a different one). Restored WITHOUT
+#      --unique (i.e. original MAC preserved) unless --unique is passed
+#      explicitly -- unlike the clone script, the source is being retired,
+#      not left running alongside a copy, so there's no address collision
+#      to avoid. qmrestore/pct restore don't start the guest, so nothing
+#      is live on the target node yet.
+#   4. Cutover: stop the guest on the source node, then (unless --no-start)
+#      start it on the target node. This is the only real downtime window
+#      -- everything before this point runs with the source guest still
+#      up.
+#   5. Unless --remove-source is passed, the now-stopped guest is LEFT ON
+#      the source node as a safety net (config + disk intact, just
+#      powered off) -- easy to start back up if the target copy turns out
+#      to be broken. Pass --remove-source to actually destroy it there
+#      once you've verified the target is good; this step gets its own
+#      typed confirmation since qm destroy/pct destroy is irreversible.
+#   6. Delete the vzdump archive from both nodes' storage (unless
+#      --keep-backup), same as the clone script -- neither node
+#      accumulates ad hoc backup files from this script.
+#
+# --- IMPORTANT CAVEAT: this is not byte-perfect live migration ---
+# vzdump is not incremental. With the default --mode snapshot, the source
+# guest keeps running (and can keep writing to disk) for the entire time
+# between when the snapshot is taken and when this script stops it at
+# cutover. Any writes in that window are NOT captured in the archive and
+# will NOT exist on the target. For a large disk this window can be
+# minutes. If the workload can't tolerate that:
+#   - use --mode stop (or --mode suspend) instead, which makes the source
+#     guest go down *before* vzdump reads its disk, so the archive is
+#     exactly the state being migrated and cutover has nothing left to
+#     lose -- at the cost of the guest being down for the whole backup
+#     duration instead of just the final cutover.
+#   - or don't use this script -- put both nodes in an actual Proxmox
+#     cluster with shared storage and use `qm migrate --online` instead,
+#     which is the real thing this script is only approximating.
+#
+# This script's own defaults are pve1 -> pve-test, matching
+# clone-pve1-to-pve-test.sh and CLAUDE.md's "Two Proxmox nodes" section --
+# but unlike that script, this one is a generic node-to-node mover: pass
+# --source-node/--target-node for any other pair. Regardless of node
+# names, the source resource is only ever touched here after typing the
+# source VMID back to confirm, and destroying it on the source node
+# (--remove-source) requires a second, separate typed confirmation.
+#
+# See --help for the full option list.
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
+# shellcheck source=../env.sh
+source "${repo_root}/scripts/env.sh"
+# shellcheck source=../lib/confirm.sh
+source "${repo_root}/scripts/lib/confirm.sh"
+
+usage() {
+  cat <<EOF
+Usage: $0 --vmid <n> [options]
+
+  --vmid <n>              Required: VMID on the source node to migrate.
+                           Kind (qemu VM vs LXC CT) is auto-detected.
+  --new-vmid <n>           VMID to restore as on the target node
+                           (default: same as --vmid).
+  --mode snapshot|suspend|stop
+                           vzdump backup mode (default: snapshot -- the
+                           source resource keeps running until cutover;
+                           requires snapshot-capable storage, e.g.
+                           ZFS/LVM-thin/Ceph/qcow2). Use "stop" for a
+                           byte-perfect migration with no post-snapshot
+                           write gap (source goes down for the whole
+                           backup duration instead of just cutover) --
+                           see the caveat at the top of this script.
+  --source-node <host>      (default: \$PVE1_HOST, ${PVE1_HOST})
+  --target-node <host>       (default: \$PVE_TEST_HOST, ${PVE_TEST_HOST})
+  --source-storage <pool>      Where vzdump writes the backup on the
+                               source node (default: local).
+  --target-storage <pool>       Where the restored disk/rootfs lands on
+                               the target node (default: \$PROXMOX_STORAGE, ${PROXMOX_STORAGE}).
+  --unique                        Restore with a fresh MAC address
+                                  (--unique 1), as the clone script always
+                                  does. Off by default here since the
+                                  source is being retired, not left
+                                  running alongside the target.
+  --no-start                        Don't start the guest on the target
+                                    node after cutover (default: start
+                                    it).
+  --remove-source                    Destroy the guest on the source
+                                     node (qm destroy/pct destroy) after
+                                     a successful cutover, instead of
+                                     just leaving it stopped there.
+                                     Irreversible -- prompts for its own
+                                     typed confirmation unless --yes.
+  --keep-backup                  Don't delete the vzdump archive from
+                                 either node afterward (debugging aid).
+  --yes                            Skip all typed confirmations (initial
+                                   backup, cutover, and --remove-source
+                                   if passed).
+  --dry-run                          Print the full plan and skip every
+                                     mutating step (vzdump, transfer,
+                                     restore, cutover, destroy, delete)
+                                     and every confirm prompt. Still makes
+                                     read-only SSH calls to look up the
+                                     source kind and check the target
+                                     VMID is free -- harmless on either
+                                     node.
+  -h, --help
+EOF
+}
+
+vmid=""
+new_vmid=""
+mode="snapshot"
+source_node="$PVE1_HOST"
+target_node="$PVE_TEST_HOST"
+source_storage="local"
+target_storage="$PROXMOX_STORAGE"
+unique=0
+start_target=1
+remove_source=0
+keep_backup=0
+skip_confirm=0
+dry_run=0
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+  --vmid) vmid="$2"; shift 2 ;;
+  --new-vmid) new_vmid="$2"; shift 2 ;;
+  --mode) mode="$2"; shift 2 ;;
+  --source-node) source_node="$2"; shift 2 ;;
+  --target-node) target_node="$2"; shift 2 ;;
+  --source-storage) source_storage="$2"; shift 2 ;;
+  --target-storage) target_storage="$2"; shift 2 ;;
+  --unique) unique=1; shift ;;
+  --no-start) start_target=0; shift ;;
+  --remove-source) remove_source=1; shift ;;
+  --keep-backup) keep_backup=1; shift ;;
+  --yes) skip_confirm=1; shift ;;
+  --dry-run) dry_run=1; shift ;;
+  -h | --help) usage; exit 0 ;;
+  *) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;;
+  esac
+done
+
+if [[ -z "$vmid" ]]; then
+  echo "ERROR: --vmid is required." >&2
+  usage >&2
+  exit 1
+fi
+if [[ "$mode" != "snapshot" && "$mode" != "suspend" && "$mode" != "stop" ]]; then
+  echo "ERROR: --mode must be snapshot, suspend, or stop." >&2
+  exit 1
+fi
+[[ -z "$new_vmid" ]] && new_vmid="$vmid"
+
+source_target="${PROXMOX_SSH_USER}@${source_node}"
+target_target="${PROXMOX_SSH_USER}@${target_node}"
+
+# No dry-run wrapper needed for the calls below: every mutating step
+# (vzdump, transfer, restore, cutover, destroy, delete) is reached only
+# after the --dry-run early-exit further down, so a plain `ssh` call is
+# never in the dry-run path.
+
+# --- identify the resource kind on the source node -----------------------
+echo "==> Looking up VMID ${vmid} on ${source_node}..."
+kind=""
+if ssh "$source_target" "qm status ${vmid}" >/dev/null 2>&1; then
+  kind="vm"
+elif ssh "$source_target" "pct status ${vmid}" >/dev/null 2>&1; then
+  kind="lxc"
+else
+  echo "ERROR: VMID ${vmid} doesn't exist on ${source_node} as either a VM or CT." >&2
+  exit 1
+fi
+echo "VMID ${vmid} on ${source_node} is a ${kind}."
+
+# --- refuse to clobber an existing resource on the target node -----------
+if ssh "$target_target" "qm status ${new_vmid}" >/dev/null 2>&1 \
+  || ssh "$target_target" "pct status ${new_vmid}" >/dev/null 2>&1; then
+  echo "ERROR: VMID ${new_vmid} already exists on ${target_node}. Pass --new-vmid" >&2
+  echo "with a free ID, or remove the existing resource there first." >&2
+  exit 1
+fi
+
+echo
+echo "Plan:"
+echo "  source: ${kind} VMID ${vmid} on ${source_node} (storage: ${source_storage}, mode: ${mode})"
+echo "  target: VMID ${new_vmid} on ${target_node} (storage: ${target_storage}," \
+  "$([[ "$unique" -eq 1 ]] && echo "fresh MAC via --unique" || echo "original MAC preserved"))"
+echo "  cutover: stop VMID ${vmid} on ${source_node}," \
+  "$([[ "$start_target" -eq 1 ]] && echo "then start VMID ${new_vmid} on ${target_node}" || echo "target left stopped (--no-start)")"
+if [[ "$remove_source" -eq 1 ]]; then
+  echo "  after cutover: DESTROY VMID ${vmid} on ${source_node} (--remove-source, irreversible)"
+else
+  echo "  after cutover: source VMID ${vmid} left stopped (but intact) on ${source_node}"
+fi
+[[ "$keep_backup" -eq 1 ]] && echo "  backup archives are kept on both nodes afterward (--keep-backup)"
+
+if [[ "$dry_run" -eq 1 ]]; then
+  echo
+  echo "[dry-run] No backup, transfer, restore, cutover, destroy, or delete was performed."
+  exit 0
+fi
+
+if [[ "$skip_confirm" -ne 1 ]]; then
+  echo
+  if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) to confirm backing it up from ${source_node} for migration: "; then
+    echo "Cancelled -- input didn't match ${vmid}." >&2
+    exit 1
+  fi
+fi
+
+# --- vzdump on the source node --------------------------------------------
+echo
+echo "==> Backing up VMID ${vmid} on ${source_node} (mode=${mode}, storage=${source_storage})..."
+vzdump_log="$(ssh "$source_target" \
+  "vzdump ${vmid} --mode ${mode} --storage ${source_storage} --compress zstd" 2>&1)" \
+  || {
+    echo "$vzdump_log" >&2
+    echo "ERROR: vzdump failed on ${source_node}." >&2
+    exit 1
+  }
+echo "$vzdump_log"
+
+archive="$(echo "$vzdump_log" | grep -oP "creating vzdump archive '\K[^']+" | tail -n1)"
+if [[ -z "$archive" ]]; then
+  echo "ERROR: couldn't find the archive path in vzdump's output above." >&2
+  exit 1
+fi
+archive_basename="$(basename "$archive")"
+target_tmp_archive="/var/tmp/${archive_basename}"
+echo "Archive: ${archive}"
+
+# Always clean up the relayed copy on the target node, success or failure
+# -- it's only ever a working copy, restored or not.
+cleanup_target_tmp() {
+  if [[ "$keep_backup" -ne 1 ]]; then
+    ssh "$target_target" "rm -f '${target_tmp_archive}'" >/dev/null 2>&1 || true
+  fi
+}
+trap cleanup_target_tmp EXIT
+
+# --- relay the archive from source to target ------------------------------
+echo
+echo "==> Transferring archive to ${target_node}..."
+ssh "$source_target" "cat '${archive}'" | ssh "$target_target" "cat > '${target_tmp_archive}'"
+
+# --- restore on the target node --------------------------------------------
+echo
+echo "==> Restoring as VMID ${new_vmid} on ${target_node} (storage=${target_storage})..."
+restore_unique_flag=0
+[[ "$unique" -eq 1 ]] && restore_unique_flag=1
+if [[ "$kind" == "vm" ]]; then
+  ssh "$target_target" "qmrestore '${target_tmp_archive}' ${new_vmid} --storage ${target_storage} --unique ${restore_unique_flag}"
+else
+  ssh "$target_target" "pct restore ${new_vmid} '${target_tmp_archive}' --storage ${target_storage} --unique ${restore_unique_flag}"
+fi
+echo "Restored on ${target_node}. Source VMID ${vmid} on ${source_node} is still up --" \
+  "the guest hasn't moved yet."
+
+# --- cutover: stop source, start target -----------------------------------
+if [[ "$skip_confirm" -ne 1 ]]; then
+  echo
+  if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) again to confirm CUTOVER (stop it on ${source_node}$([[ "$start_target" -eq 1 ]] && echo ", start VMID ${new_vmid} on ${target_node}")): "; then
+    echo "Cancelled before cutover -- input didn't match ${vmid}." >&2
+    echo "The restored (but not started) copy remains on ${target_node} as VMID ${new_vmid};" >&2
+    echo "the source on ${source_node} is untouched. Re-run cutover manually, or clean up" >&2
+    echo "the target copy if you no longer want it." >&2
+    exit 1
+  fi
+fi
+
+echo
+echo "==> Stopping VMID ${vmid} on ${source_node}..."
+if [[ "$kind" == "vm" ]]; then
+  ssh "$source_target" "qm stop ${vmid}"
+else
+  ssh "$source_target" "pct stop ${vmid}"
+fi
+
+if [[ "$start_target" -eq 1 ]]; then
+  echo
+  echo "==> Starting VMID ${new_vmid} on ${target_node}..."
+  if [[ "$kind" == "vm" ]]; then
+    ssh "$target_target" "qm start ${new_vmid}"
+  else
+    ssh "$target_target" "pct start ${new_vmid}"
+  fi
+fi
+
+# --- optionally destroy the now-stopped source resource -------------------
+if [[ "$remove_source" -eq 1 ]]; then
+  if [[ "$skip_confirm" -ne 1 ]]; then
+    echo
+    if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) one more time to permanently DESTROY it on ${source_node}: "; then
+      echo "Cancelled -- input didn't match ${vmid}." >&2
+      echo "VMID ${vmid} on ${source_node} is left stopped (not destroyed). VMID ${new_vmid}" >&2
+      echo "on ${target_node} is up and running." >&2
+      exit 1
+    fi
+  fi
+  echo
+  echo "==> Destroying VMID ${vmid} on ${source_node}..."
+  if [[ "$kind" == "vm" ]]; then
+    ssh "$source_target" "qm destroy ${vmid} --purge"
+  else
+    ssh "$source_target" "pct destroy ${vmid} --purge"
+  fi
+fi
+
+# --- clean up backup archives now that the migration succeeded -----------
+if [[ "$keep_backup" -ne 1 ]]; then
+  echo
+  echo "==> Deleting backup archive from ${source_node}'s ${source_storage} storage..."
+  ssh "$source_target" "rm -f '${archive}' '${archive}.notes' '${archive}.log'" >/dev/null 2>&1 || true
+fi
+
+echo
+echo "Done. VMID ${new_vmid} (${kind}) is now on ${target_node}, migrated from" \
+  "VMID ${vmid} on ${source_node}."
+if [[ "$remove_source" -ne 1 ]]; then
+  echo "The source copy is stopped but still present on ${source_node} -- re-run with" \
+    "--remove-source once you've verified the target, or remove it manually."
+fi