beatzaplenty/nixos
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add pve1->pve-test VM/CT migration script (vzdump + qmrestore pattern) #32

Closed
beatzaplenty wants to merge 2 commits from worktree-migrate-vm-script into main
pull from: worktree-migrate-vm-script
merge into: :main
Conversation 0 Commits 2 Files Changed 2
+374
2 changed files with 374 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+27
View File
@@ -256,6 +256,33 @@ lift the shared part into `scripts/lib/` instead of copying it.
`/etc/ssh/ssh_known_hosts`. Idempotent (re-running replaces its own
marked block rather than duplicating it); restarts `nix-daemon` by
default so the change takes effect immediately.
- `scripts/proxmox/clone-pve1-to-pve-test.sh --vmid <n> [options]` — ad hoc
copy of a single VM/CT from one node to another via `vzdump` +
`qmrestore`/`pct restore`, relayed through this machine
(`ssh src cat ... | ssh dst cat > ...`, no on-disk staging copy).
Defaults `pve1` → `pve-test` (see "Two Proxmox nodes" above); source is
left running throughout (`--mode snapshot` default) and untouched
beyond the vzdump + a typed-VMID confirm, restored on the target with a
fresh MAC (`--unique 1`, always) since the source is expected to still
be live on the same LAN. Backup archives are deleted from both nodes
afterward unless `--keep-backup`. `--dry-run` throughout.
- `scripts/proxmox/migrate-vm.sh --vmid <n> [options]` — moves (rather
than clones) a single VM/CT between nodes, built on the same
vzdump/relay/restore pattern as `clone-pve1-to-pve-test.sh` above, plus
a cutover step (stop the guest on the source node, start it on the
target) and, optionally, `--remove-source` to destroy it on the source
node afterward (its own separate typed confirmation, since that step is
irreversible) — the default instead leaves the stopped source guest
intact as a safety net. Restores with the *original* MAC (no
`--unique`) unless `--unique` is passed, since the source is expected to
go away rather than keep running alongside the target. **Not** a true
`qm migrate`-style live migration — pve1/pve-test aren't a clustered
pair with shared storage, so this is vzdump-based and, with the default
`--mode snapshot`, any writes on the source between the snapshot and
cutover are lost (use `--mode stop` if that's not acceptable). Generic
node-to-node (not pve1/pve-test-specific) via
`--source-node`/`--target-node`, defaulting to pve1 → pve-test like the
clone script. `--dry-run` throughout.
### `scripts/lib/`
scripts/proxmox/migrate-vm.sh
Executable
+347
View File
@@ -0,0 +1,347 @@
#!/usr/bin/env bash
# Move a single VM/CT from one Proxmox node to another via vzdump +
# qmrestore/pct restore -- the same relay pattern as
# clone-pve1-to-pve-test.sh, but for a *migration* (the guest ends up
# living on the target node only) rather than a *clone* (a disposable
# copy, source untouched, fresh MAC).
#
# There is no real `qm migrate` here: that command only works between
# nodes in the same Proxmox cluster with shared/replicated storage, which
# pve1 and pve-test are not (see CLAUDE.md's "Two Proxmox nodes" section --
# they're deliberately separate, unclustered nodes). This script is the
# closest equivalent across two independent nodes, built out of the same
# primitives as the clone script. It is NOT a true live migration -- read
# the caveat below before using it for anything where losing a few
# seconds/minutes of writes is unacceptable.
#
# Flow:
# 1. vzdump the resource on the source node (--mode snapshot by
# default, so the source keeps running throughout the bulk copy --
# this is the part that's "live": the guest stays reachable while
# its disk data is captured and shipped to the target).
# 2. Stream the archive straight from the source node to the target
# node (ssh source cat ... | ssh target cat > ...), same relay this
# machine does in the clone script -- no separate on-disk staging
# copy here either.
# 3. qmrestore / pct restore it on the target node under the same VMID
# by default (--new-vmid to pick a different one). Restored WITHOUT
# --unique (i.e. original MAC preserved) unless --unique is passed
# explicitly -- unlike the clone script, the source is being retired,
# not left running alongside a copy, so there's no address collision
# to avoid. qmrestore/pct restore don't start the guest, so nothing
# is live on the target node yet.
# 4. Cutover: stop the guest on the source node, then (unless --no-start)
# start it on the target node. This is the only real downtime window
# -- everything before this point runs with the source guest still
# up.
# 5. Unless --remove-source is passed, the now-stopped guest is LEFT ON
# the source node as a safety net (config + disk intact, just
# powered off) -- easy to start back up if the target copy turns out
# to be broken. Pass --remove-source to actually destroy it there
# once you've verified the target is good; this step gets its own
# typed confirmation since qm destroy/pct destroy is irreversible.
# 6. Delete the vzdump archive from both nodes' storage (unless
# --keep-backup), same as the clone script -- neither node
# accumulates ad hoc backup files from this script.
#
# --- IMPORTANT CAVEAT: this is not byte-perfect live migration ---
# vzdump is not incremental. With the default --mode snapshot, the source
# guest keeps running (and can keep writing to disk) for the entire time
# between when the snapshot is taken and when this script stops it at
# cutover. Any writes in that window are NOT captured in the archive and
# will NOT exist on the target. For a large disk this window can be
# minutes. If the workload can't tolerate that:
# - use --mode stop (or --mode suspend) instead, which makes the source
# guest go down *before* vzdump reads its disk, so the archive is
# exactly the state being migrated and cutover has nothing left to
# lose -- at the cost of the guest being down for the whole backup
# duration instead of just the final cutover.
# - or don't use this script -- put both nodes in an actual Proxmox
# cluster with shared storage and use `qm migrate --online` instead,
# which is the real thing this script is only approximating.
#
# This script's own defaults are pve1 -> pve-test, matching
# clone-pve1-to-pve-test.sh and CLAUDE.md's "Two Proxmox nodes" section --
# but unlike that script, this one is a generic node-to-node mover: pass
# --source-node/--target-node for any other pair. Regardless of node
# names, the source resource is only ever touched here after typing the
# source VMID back to confirm, and destroying it on the source node
# (--remove-source) requires a second, separate typed confirmation.
#
# See --help for the full option list.
set -euo pipefail
repo_root="$(cd "$(dirname "$0")/../.." && pwd)"
# shellcheck source=../env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=../lib/confirm.sh
source "${repo_root}/scripts/lib/confirm.sh"
usage() {
cat <<EOF
Usage: $0 --vmid <n> [options]
--vmid <n> Required: VMID on the source node to migrate.
Kind (qemu VM vs LXC CT) is auto-detected.
--new-vmid <n> VMID to restore as on the target node
(default: same as --vmid).
--mode snapshot|suspend|stop
vzdump backup mode (default: snapshot -- the
source resource keeps running until cutover;
requires snapshot-capable storage, e.g.
ZFS/LVM-thin/Ceph/qcow2). Use "stop" for a
byte-perfect migration with no post-snapshot
write gap (source goes down for the whole
backup duration instead of just cutover) --
see the caveat at the top of this script.
--source-node <host> (default: \$PVE1_HOST, ${PVE1_HOST})
--target-node <host> (default: \$PVE_TEST_HOST, ${PVE_TEST_HOST})
--source-storage <pool> Where vzdump writes the backup on the
source node (default: local).
--target-storage <pool> Where the restored disk/rootfs lands on
the target node (default: \$PROXMOX_STORAGE, ${PROXMOX_STORAGE}).
--unique Restore with a fresh MAC address
(--unique 1), as the clone script always
does. Off by default here since the
source is being retired, not left
running alongside the target.
--no-start Don't start the guest on the target
node after cutover (default: start
it).
--remove-source Destroy the guest on the source
node (qm destroy/pct destroy) after
a successful cutover, instead of
just leaving it stopped there.
Irreversible -- prompts for its own
typed confirmation unless --yes.
--keep-backup Don't delete the vzdump archive from
either node afterward (debugging aid).
--yes Skip all typed confirmations (initial
backup, cutover, and --remove-source
if passed).
--dry-run Print the full plan and skip every
mutating step (vzdump, transfer,
restore, cutover, destroy, delete)
and every confirm prompt. Still makes
read-only SSH calls to look up the
source kind and check the target
VMID is free -- harmless on either
node.
-h, --help
EOF
}
vmid=""
new_vmid=""
mode="snapshot"
source_node="$PVE1_HOST"
target_node="$PVE_TEST_HOST"
source_storage="local"
target_storage="$PROXMOX_STORAGE"
unique=0
start_target=1
remove_source=0
keep_backup=0
skip_confirm=0
dry_run=0
while [[ $# -gt 0 ]]; do
case "$1" in
--vmid) vmid="$2"; shift 2 ;;
--new-vmid) new_vmid="$2"; shift 2 ;;
--mode) mode="$2"; shift 2 ;;
--source-node) source_node="$2"; shift 2 ;;
--target-node) target_node="$2"; shift 2 ;;
--source-storage) source_storage="$2"; shift 2 ;;
--target-storage) target_storage="$2"; shift 2 ;;
--unique) unique=1; shift ;;
--no-start) start_target=0; shift ;;
--remove-source) remove_source=1; shift ;;
--keep-backup) keep_backup=1; shift ;;
--yes) skip_confirm=1; shift ;;
--dry-run) dry_run=1; shift ;;
-h | --help) usage; exit 0 ;;
*) echo "Unknown option: $1" >&2; usage >&2; exit 1 ;;
esac
done
if [[ -z "$vmid" ]]; then
echo "ERROR: --vmid is required." >&2
usage >&2
exit 1
fi
if [[ "$mode" != "snapshot" && "$mode" != "suspend" && "$mode" != "stop" ]]; then
echo "ERROR: --mode must be snapshot, suspend, or stop." >&2
exit 1
fi
[[ -z "$new_vmid" ]] && new_vmid="$vmid"
source_target="${PROXMOX_SSH_USER}@${source_node}"
target_target="${PROXMOX_SSH_USER}@${target_node}"
# No dry-run wrapper needed for the calls below: every mutating step
# (vzdump, transfer, restore, cutover, destroy, delete) is reached only
# after the --dry-run early-exit further down, so a plain `ssh` call is
# never in the dry-run path.
# --- identify the resource kind on the source node -----------------------
echo "==> Looking up VMID ${vmid} on ${source_node}..."
kind=""
if ssh "$source_target" "qm status ${vmid}" >/dev/null 2>&1; then
kind="vm"
elif ssh "$source_target" "pct status ${vmid}" >/dev/null 2>&1; then
kind="lxc"
else
echo "ERROR: VMID ${vmid} doesn't exist on ${source_node} as either a VM or CT." >&2
exit 1
fi
echo "VMID ${vmid} on ${source_node} is a ${kind}."
# --- refuse to clobber an existing resource on the target node -----------
if ssh "$target_target" "qm status ${new_vmid}" >/dev/null 2>&1 \
|| ssh "$target_target" "pct status ${new_vmid}" >/dev/null 2>&1; then
echo "ERROR: VMID ${new_vmid} already exists on ${target_node}. Pass --new-vmid" >&2
echo "with a free ID, or remove the existing resource there first." >&2
exit 1
fi
echo
echo "Plan:"
echo " source: ${kind} VMID ${vmid} on ${source_node} (storage: ${source_storage}, mode: ${mode})"
echo " target: VMID ${new_vmid} on ${target_node} (storage: ${target_storage}," \
"$([[ "$unique" -eq 1 ]] && echo "fresh MAC via --unique" || echo "original MAC preserved"))"
echo " cutover: stop VMID ${vmid} on ${source_node}," \
"$([[ "$start_target" -eq 1 ]] && echo "then start VMID ${new_vmid} on ${target_node}" || echo "target left stopped (--no-start)")"
if [[ "$remove_source" -eq 1 ]]; then
echo " after cutover: DESTROY VMID ${vmid} on ${source_node} (--remove-source, irreversible)"
else
echo " after cutover: source VMID ${vmid} left stopped (but intact) on ${source_node}"
fi
[[ "$keep_backup" -eq 1 ]] && echo " backup archives are kept on both nodes afterward (--keep-backup)"
if [[ "$dry_run" -eq 1 ]]; then
echo
echo "[dry-run] No backup, transfer, restore, cutover, destroy, or delete was performed."
exit 0
fi
if [[ "$skip_confirm" -ne 1 ]]; then
echo
if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) to confirm backing it up from ${source_node} for migration: "; then
echo "Cancelled -- input didn't match ${vmid}." >&2
exit 1
fi
fi
# --- vzdump on the source node --------------------------------------------
echo
echo "==> Backing up VMID ${vmid} on ${source_node} (mode=${mode}, storage=${source_storage})..."
vzdump_log="$(ssh "$source_target" \
"vzdump ${vmid} --mode ${mode} --storage ${source_storage} --compress zstd" 2>&1)" \
|| {
echo "$vzdump_log" >&2
echo "ERROR: vzdump failed on ${source_node}." >&2
exit 1
}
echo "$vzdump_log"
archive="$(echo "$vzdump_log" | grep -oP "creating vzdump archive '\K[^']+" | tail -n1)"
if [[ -z "$archive" ]]; then
echo "ERROR: couldn't find the archive path in vzdump's output above." >&2
exit 1
fi
archive_basename="$(basename "$archive")"
target_tmp_archive="/var/tmp/${archive_basename}"
echo "Archive: ${archive}"
# Always clean up the relayed copy on the target node, success or failure
# -- it's only ever a working copy, restored or not.
cleanup_target_tmp() {
if [[ "$keep_backup" -ne 1 ]]; then
ssh "$target_target" "rm -f '${target_tmp_archive}'" >/dev/null 2>&1 || true
fi
}
trap cleanup_target_tmp EXIT
# --- relay the archive from source to target ------------------------------
echo
echo "==> Transferring archive to ${target_node}..."
ssh "$source_target" "cat '${archive}'" | ssh "$target_target" "cat > '${target_tmp_archive}'"
# --- restore on the target node --------------------------------------------
echo
echo "==> Restoring as VMID ${new_vmid} on ${target_node} (storage=${target_storage})..."
restore_unique_flag=0
[[ "$unique" -eq 1 ]] && restore_unique_flag=1
if [[ "$kind" == "vm" ]]; then
ssh "$target_target" "qmrestore '${target_tmp_archive}' ${new_vmid} --storage ${target_storage} --unique ${restore_unique_flag}"
else
ssh "$target_target" "pct restore ${new_vmid} '${target_tmp_archive}' --storage ${target_storage} --unique ${restore_unique_flag}"
fi
echo "Restored on ${target_node}. Source VMID ${vmid} on ${source_node} is still up --" \
"the guest hasn't moved yet."
# --- cutover: stop source, start target -----------------------------------
if [[ "$skip_confirm" -ne 1 ]]; then
echo
if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) again to confirm CUTOVER (stop it on ${source_node}$([[ "$start_target" -eq 1 ]] && echo ", start VMID ${new_vmid} on ${target_node}")): "; then
echo "Cancelled before cutover -- input didn't match ${vmid}." >&2
echo "The restored (but not started) copy remains on ${target_node} as VMID ${new_vmid};" >&2
echo "the source on ${source_node} is untouched. Re-run cutover manually, or clean up" >&2
echo "the target copy if you no longer want it." >&2
exit 1
fi
fi
echo
echo "==> Stopping VMID ${vmid} on ${source_node}..."
if [[ "$kind" == "vm" ]]; then
ssh "$source_target" "qm stop ${vmid}"
else
ssh "$source_target" "pct stop ${vmid}"
fi
if [[ "$start_target" -eq 1 ]]; then
echo
echo "==> Starting VMID ${new_vmid} on ${target_node}..."
if [[ "$kind" == "vm" ]]; then
ssh "$target_target" "qm start ${new_vmid}"
else
ssh "$target_target" "pct start ${new_vmid}"
fi
fi
# --- optionally destroy the now-stopped source resource -------------------
if [[ "$remove_source" -eq 1 ]]; then
if [[ "$skip_confirm" -ne 1 ]]; then
echo
if ! confirm_typed "$vmid" "Type the source VMID (${vmid}) one more time to permanently DESTROY it on ${source_node}: "; then
echo "Cancelled -- input didn't match ${vmid}." >&2
echo "VMID ${vmid} on ${source_node} is left stopped (not destroyed). VMID ${new_vmid}" >&2
echo "on ${target_node} is up and running." >&2
exit 1
fi
fi
echo
echo "==> Destroying VMID ${vmid} on ${source_node}..."
if [[ "$kind" == "vm" ]]; then
ssh "$source_target" "qm destroy ${vmid} --purge"
else
ssh "$source_target" "pct destroy ${vmid} --purge"
fi
fi
# --- clean up backup archives now that the migration succeeded -----------
if [[ "$keep_backup" -ne 1 ]]; then
echo
echo "==> Deleting backup archive from ${source_node}'s ${source_storage} storage..."
ssh "$source_target" "rm -f '${archive}' '${archive}.notes' '${archive}.log'" >/dev/null 2>&1 || true
fi
echo
echo "Done. VMID ${new_vmid} (${kind}) is now on ${target_node}, migrated from" \
"VMID ${vmid} on ${source_node}."
if [[ "$remove_source" -ne 1 ]]; then
echo "The source copy is stopped but still present on ${source_node} -- re-run with" \
"--remove-source once you've verified the target, or remove it manually."
fi