diff --git a/CLAUDE.md b/CLAUDE.md index cdc1c79..a341ece 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -256,6 +256,33 @@ lift the shared part into `scripts/lib/` instead of copying it. `/etc/ssh/ssh_known_hosts`. Idempotent (re-running replaces its own marked block rather than duplicating it); restarts `nix-daemon` by default so the change takes effect immediately. +- `scripts/proxmox/clone-pve1-to-pve-test.sh --vmid [options]` — ad hoc + copy of a single VM/CT from one node to another via `vzdump` + + `qmrestore`/`pct restore`, relayed through this machine + (`ssh src cat ... | ssh dst cat > ...`, no on-disk staging copy). + Defaults `pve1` → `pve-test` (see "Two Proxmox nodes" above); source is + left running throughout (`--mode snapshot` default) and untouched + beyond the vzdump + a typed-VMID confirm, restored on the target with a + fresh MAC (`--unique 1`, always) since the source is expected to still + be live on the same LAN. Backup archives are deleted from both nodes + afterward unless `--keep-backup`. `--dry-run` throughout. +- `scripts/proxmox/migrate-vm.sh --vmid [options]` — moves (rather + than clones) a single VM/CT between nodes, built on the same + vzdump/relay/restore pattern as `clone-pve1-to-pve-test.sh` above, plus + a cutover step (stop the guest on the source node, start it on the + target) and, optionally, `--remove-source` to destroy it on the source + node afterward (its own separate typed confirmation, since that step is + irreversible) — the default instead leaves the stopped source guest + intact as a safety net. Restores with the *original* MAC (no + `--unique`) unless `--unique` is passed, since the source is expected to + go away rather than keep running alongside the target. **Not** a true + `qm migrate`-style live migration — pve1/pve-test aren't a clustered + pair with shared storage, so this is vzdump-based and, with the default + `--mode snapshot`, any writes on the source between the snapshot and + cutover are lost (use `--mode stop` if that's not acceptable). Generic + node-to-node (not pve1/pve-test-specific) via + `--source-node`/`--target-node`, defaulting to pve1 → pve-test like the + clone script. `--dry-run` throughout. ### `scripts/lib/` diff --git a/scripts/proxmox/migrate-vm.sh b/scripts/proxmox/migrate-vm.sh new file mode 100755 index 0000000..7c0d26f --- /dev/null +++ b/scripts/proxmox/migrate-vm.sh @@ -0,0 +1,347 @@ +#!/usr/bin/env bash +# Move a single VM/CT from one Proxmox node to another via vzdump + +# qmrestore/pct restore -- the same relay pattern as +# clone-pve1-to-pve-test.sh, but for a *migration* (the guest ends up +# living on the target node only) rather than a *clone* (a disposable +# copy, source untouched, fresh MAC). +# +# There is no real `qm migrate` here: that command only works between +# nodes in the same Proxmox cluster with shared/replicated storage, which +# pve1 and pve-test are not (see CLAUDE.md's "Two Proxmox nodes" section -- +# they're deliberately separate, unclustered nodes). This script is the +# closest equivalent across two independent nodes, built out of the same +# primitives as the clone script. It is NOT a true live migration -- read +# the caveat below before using it for anything where losing a few +# seconds/minutes of writes is unacceptable. +# +# Flow: +# 1. vzdump the resource on the source node (--mode snapshot by +# default, so the source keeps running throughout the bulk copy -- +# this is the part that's "live": the guest stays reachable while +# its disk data is captured and shipped to the target). +# 2. Stream the archive straight from the source node to the target +# node (ssh source cat ... | ssh target cat > ...), same relay this +# machine does in the clone script -- no separate on-disk staging +# copy here either. +# 3. qmrestore / pct restore it on the target node under the same VMID +# by default (--new-vmid to pick a different one). Restored WITHOUT +# --unique (i.e. original MAC preserved) unless --unique is passed +# explicitly -- unlike the clone script, the source is being retired, +# not left running alongside a copy, so there's no address collision +# to avoid. qmrestore/pct restore don't start the guest, so nothing +# is live on the target node yet. +# 4. Cutover: stop the guest on the source node, then (unless --no-start) +# start it on the target node. This is the only real downtime window +# -- everything before this point runs with the source guest still +# up. +# 5. Unless --remove-source is passed, the now-stopped guest is LEFT ON +# the source node as a safety net (config + disk intact, just +# powered off) -- easy to start back up if the target copy turns out +# to be broken. Pass --remove-source to actually destroy it there +# once you've verified the target is good; this step gets its own +# typed confirmation since qm destroy/pct destroy is irreversible. +# 6. Delete the vzdump archive from both nodes' storage (unless +# --keep-backup), same as the clone script -- neither node +# accumulates ad hoc backup files from this script. +# +# --- IMPORTANT CAVEAT: this is not byte-perfect live migration --- +# vzdump is not incremental. With the default --mode snapshot, the source +# guest keeps running (and can keep writing to disk) for the entire time +# between when the snapshot is taken and when this script stops it at +# cutover. Any writes in that window are NOT captured in the archive and +# will NOT exist on the target. For a large disk this window can be +# minutes. If the workload can't tolerate that: +# - use --mode stop (or --mode suspend) instead, which makes the source +# guest go down *before* vzdump reads its disk, so the archive is +# exactly the state being migrated and cutover has nothing left to +# lose -- at the cost of the guest being down for the whole backup +# duration instead of just the final cutover. +# - or don't use this script -- put both nodes in an actual Proxmox +# cluster with shared storage and use `qm migrate --online` instead, +# which is the real thing this script is only approximating. +# +# This script's own defaults are pve1 -> pve-test, matching +# clone-pve1-to-pve-test.sh and CLAUDE.md's "Two Proxmox nodes" section -- +# but unlike that script, this one is a generic node-to-node mover: pass +# --source-node/--target-node for any other pair. Regardless of node +# names, the source resource is only ever touched here after typing the +# source VMID back to confirm, and destroying it on the source node +# (--remove-source) requires a second, separate typed confirmation. +# +# See --help for the full option list. +set -euo pipefail + +repo_root="$(cd "$(dirname "$0")/../.." && pwd)" +# shellcheck source=../env.sh +source "${repo_root}/scripts/env.sh" +# shellcheck source=../lib/confirm.sh +source "${repo_root}/scripts/lib/confirm.sh" + +usage() { + cat < [options] + + --vmid Required: VMID on the source node to migrate. + Kind (qemu VM vs LXC CT) is auto-detected. + --new-vmid VMID to restore as on the target node + (default: same as --vmid). + --mode snapshot|suspend|stop + vzdump backup mode (default: snapshot -- the + source resource keeps running until cutover; + requires snapshot-capable storage, e.g. + ZFS/LVM-thin/Ceph/qcow2). Use "stop" for a + byte-perfect migration with no post-snapshot + write gap (source goes down for the whole + backup duration instead of just cutover) -- + see the caveat at the top of this script. + --source-node (default: \$PVE1_HOST, ${PVE1_HOST}) + --target-node (default: \$PVE_TEST_HOST, ${PVE_TEST_HOST}) + --source-storage Where vzdump writes the backup on the + source node (default: local). + --target-storage Where the restored disk/rootfs lands on + the target node (default: \$PROXMOX_STORAGE, ${PROXMOX_STORAGE}). + --unique Restore with a fresh MAC address + (--unique 1), as the clone script always + does. Off by default here since the + source is being retired, not left + running alongside the target. + --no-start Don't start the guest on the target + node after cutover (default: start + it). + --remove-source Destroy the guest on the source + node (qm destroy/pct destroy) after + a successful cutover, instead of + just leaving it stopped there. + Irreversible -- prompts for its own + typed confirmation unless --yes. + --keep-backup Don't delete the vzdump archive from + either node afterward (debugging aid). + --yes Skip all typed confirmations (initial + backup, cutover, and --remove-source + if passed). + --dry-run Print the full plan and skip every + mutating step (vzdump, transfer, + restore, cutover, destroy, delete) + and every confirm prompt. Still makes + read-only SSH calls to look up the + source kind and check the target + VMID is free -- harmless on either + node. + -h, --help +EOF +} + +vmid="" +new_vmid="" +mode="snapshot" +source_node="$PVE1_HOST" +target_node="$PVE_TEST_HOST" +source_storage="local" +target_storage="$PROXMOX_STORAGE" +unique=0 +start_target=1 +remove_source=0 +keep_backup=0 +skip_confirm=0 +dry_run=0 + +while [[ $# -gt 0 ]]; do + case "$1" in + --vmid) vmid="$2"; shift 2 ;; + --new-vmid) new_vmid="$2"; shift 2 ;; + --mode) mode="$2"; shift 2 ;; + --source-node) source_node="$2"; shift 2 ;; + --target-node) target_node="$2"; shift 2 ;; + --source-storage) source_storage="$2"; shift 2 ;; + --target-storage) target_storage="$2"; shift 2 ;; + --unique) unique=1; shift ;; + --no-start) start_target=0; shift ;; + --remove-source) remove_source=1; shift ;; + --keep-backup) keep_backup=1; shift ;; + --yes) skip_confirm=1; shift ;; + --dry-run) dry_run=1; shift ;; + -h | --help) usage; exit 0 ;; + *) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;; + esac +done + +if [[ -z "$vmid" ]]; then + echo "ERROR: --vmid is required." >&2 + usage >&2 + exit 1 +fi +if [[ "$mode" != "snapshot" && "$mode" != "suspend" && "$mode" != "stop" ]]; then + echo "ERROR: --mode must be snapshot, suspend, or stop." >&2 + exit 1 +fi +[[ -z "$new_vmid" ]] && new_vmid="$vmid" + +source_target="${PROXMOX_SSH_USER}@${source_node}" +target_target="${PROXMOX_SSH_USER}@${target_node}" + +# No dry-run wrapper needed for the calls below: every mutating step +# (vzdump, transfer, restore, cutover, destroy, delete) is reached only +# after the --dry-run early-exit further down, so a plain `ssh` call is +# never in the dry-run path. + +# --- identify the resource kind on the source node ----------------------- +echo "==> Looking up VMID ${vmid} on ${source_node}..." +kind="" +if ssh "$source_target" "qm status ${vmid}" >/dev/null 2>&1; then + kind="vm" +elif ssh "$source_target" "pct status ${vmid}" >/dev/null 2>&1; then + kind="lxc" +else + echo "ERROR: VMID ${vmid} doesn't exist on ${source_node} as either a VM or CT." >&2 + exit 1 +fi +echo "VMID ${vmid} on ${source_node} is a ${kind}." + +# --- refuse to clobber an existing resource on the target node ----------- +if ssh "$target_target" "qm status ${new_vmid}" >/dev/null 2>&1 \ + || ssh "$target_target" "pct status ${new_vmid}" >/dev/null 2>&1; then + echo "ERROR: VMID ${new_vmid} already exists on ${target_node}. Pass --new-vmid" >&2 + echo "with a free ID, or remove the existing resource there first." >&2 + exit 1 +fi + +echo +echo "Plan:" +echo " source: ${kind} VMID ${vmid} on ${source_node} (storage: ${source_storage}, mode: ${mode})" +echo " target: VMID ${new_vmid} on ${target_node} (storage: ${target_storage}," \ + "$([[ "$unique" -eq 1 ]] && echo "fresh MAC via --unique" || echo "original MAC preserved"))" +echo " cutover: stop VMID ${vmid} on ${source_node}," \ + "$([[ "$start_target" -eq 1 ]] && echo "then start VMID ${new_vmid} on ${target_node}" || echo "target left stopped (--no-start)")" +if [[ "$remove_source" -eq 1 ]]; then + echo " after cutover: DESTROY VMID ${vmid} on ${source_node} (--remove-source, irreversible)" +else + echo " after cutover: source VMID ${vmid} left stopped (but intact) on ${source_node}" +fi +[[ "$keep_backup" -eq 1 ]] && echo " backup archives are kept on both nodes afterward (--keep-backup)" + +if [[ "$dry_run" -eq 1 ]]; then + echo + echo "[dry-run] No backup, transfer, restore, cutover, destroy, or delete was performed." + exit 0 +fi + +if [[ "$skip_confirm" -ne 1 ]]; then + echo + if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) to confirm backing it up from ${source_node} for migration: "; then + echo "Cancelled -- input didn't match ${vmid}." >&2 + exit 1 + fi +fi + +# --- vzdump on the source node -------------------------------------------- +echo +echo "==> Backing up VMID ${vmid} on ${source_node} (mode=${mode}, storage=${source_storage})..." +vzdump_log="$(ssh "$source_target" \ + "vzdump ${vmid} --mode ${mode} --storage ${source_storage} --compress zstd" 2>&1)" \ + || { + echo "$vzdump_log" >&2 + echo "ERROR: vzdump failed on ${source_node}." >&2 + exit 1 + } +echo "$vzdump_log" + +archive="$(echo "$vzdump_log" | grep -oP "creating vzdump archive '\K[^']+" | tail -n1)" +if [[ -z "$archive" ]]; then + echo "ERROR: couldn't find the archive path in vzdump's output above." >&2 + exit 1 +fi +archive_basename="$(basename "$archive")" +target_tmp_archive="/var/tmp/${archive_basename}" +echo "Archive: ${archive}" + +# Always clean up the relayed copy on the target node, success or failure +# -- it's only ever a working copy, restored or not. +cleanup_target_tmp() { + if [[ "$keep_backup" -ne 1 ]]; then + ssh "$target_target" "rm -f '${target_tmp_archive}'" >/dev/null 2>&1 || true + fi +} +trap cleanup_target_tmp EXIT + +# --- relay the archive from source to target ------------------------------ +echo +echo "==> Transferring archive to ${target_node}..." +ssh "$source_target" "cat '${archive}'" | ssh "$target_target" "cat > '${target_tmp_archive}'" + +# --- restore on the target node -------------------------------------------- +echo +echo "==> Restoring as VMID ${new_vmid} on ${target_node} (storage=${target_storage})..." +restore_unique_flag=0 +[[ "$unique" -eq 1 ]] && restore_unique_flag=1 +if [[ "$kind" == "vm" ]]; then + ssh "$target_target" "qmrestore '${target_tmp_archive}' ${new_vmid} --storage ${target_storage} --unique ${restore_unique_flag}" +else + ssh "$target_target" "pct restore ${new_vmid} '${target_tmp_archive}' --storage ${target_storage} --unique ${restore_unique_flag}" +fi +echo "Restored on ${target_node}. Source VMID ${vmid} on ${source_node} is still up --" \ + "the guest hasn't moved yet." + +# --- cutover: stop source, start target ----------------------------------- +if [[ "$skip_confirm" -ne 1 ]]; then + echo + if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) again to confirm CUTOVER (stop it on ${source_node}$([[ "$start_target" -eq 1 ]] && echo ", start VMID ${new_vmid} on ${target_node}")): "; then + echo "Cancelled before cutover -- input didn't match ${vmid}." >&2 + echo "The restored (but not started) copy remains on ${target_node} as VMID ${new_vmid};" >&2 + echo "the source on ${source_node} is untouched. Re-run cutover manually, or clean up" >&2 + echo "the target copy if you no longer want it." >&2 + exit 1 + fi +fi + +echo +echo "==> Stopping VMID ${vmid} on ${source_node}..." +if [[ "$kind" == "vm" ]]; then + ssh "$source_target" "qm stop ${vmid}" +else + ssh "$source_target" "pct stop ${vmid}" +fi + +if [[ "$start_target" -eq 1 ]]; then + echo + echo "==> Starting VMID ${new_vmid} on ${target_node}..." + if [[ "$kind" == "vm" ]]; then + ssh "$target_target" "qm start ${new_vmid}" + else + ssh "$target_target" "pct start ${new_vmid}" + fi +fi + +# --- optionally destroy the now-stopped source resource ------------------- +if [[ "$remove_source" -eq 1 ]]; then + if [[ "$skip_confirm" -ne 1 ]]; then + echo + if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) one more time to permanently DESTROY it on ${source_node}: "; then + echo "Cancelled -- input didn't match ${vmid}." >&2 + echo "VMID ${vmid} on ${source_node} is left stopped (not destroyed). VMID ${new_vmid}" >&2 + echo "on ${target_node} is up and running." >&2 + exit 1 + fi + fi + echo + echo "==> Destroying VMID ${vmid} on ${source_node}..." + if [[ "$kind" == "vm" ]]; then + ssh "$source_target" "qm destroy ${vmid} --purge" + else + ssh "$source_target" "pct destroy ${vmid} --purge" + fi +fi + +# --- clean up backup archives now that the migration succeeded ----------- +if [[ "$keep_backup" -ne 1 ]]; then + echo + echo "==> Deleting backup archive from ${source_node}'s ${source_storage} storage..." + ssh "$source_target" "rm -f '${archive}' '${archive}.notes' '${archive}.log'" >/dev/null 2>&1 || true +fi + +echo +echo "Done. VMID ${new_vmid} (${kind}) is now on ${target_node}, migrated from" \ + "VMID ${vmid} on ${source_node}." +if [[ "$remove_source" -ne 1 ]]; then + echo "The source copy is stopped but still present on ${source_node} -- re-run with" \ + "--remove-source once you've verified the target, or remove it manually." +fi