Initial planning docs and hardening scripts for HA rebuild

Covers node 1 hardware/network layout, LVM-thin -> ZFS migration path, Ceph as the future HA storage upgrade, and baseline SSH/firewall hardening.
2026-07-20 11:45:54 +10:00
commit a854412117
10 changed files with 414 additions and 0 deletions
@@ -0,0 +1,2 @@
 .claude/settings.local.json
 *.swp
@@ -0,0 +1,29 @@
 # Proxmox Configuration
 Config, planning docs, and scripts for rebuilding my Proxmox VE homelab on new
 hardware, with the end goal of a 3-node HA cluster.
 ## Goals
 - Fresh install on new hardware (old box has a KVM/virtualization bug that
  forced disabling hardware-accelerated virtualization — not expected to
  recur on new hardware; verify VT-x/AMD-V + IOMMU in BIOS before assuming
  otherwise).
 - Move off LVM-thin to ZFS for local storage.
 - Build node 1 so its disk and network layout doesn't need rework when nodes
  2 and 3 are added later.
 - End state: 3-node cluster, quorum via corosync, HA-managed VMs backed by
  ZFS replication and/or Ceph.
 ## Repo layout
 - `docs/` — planning docs: hardware layout, storage migration, networking,
  security hardening. Read `docs/00-overview.md` first.
 - `scripts/` — scripts to apply configuration on a node (SSH hardening,
  repo switch, etc.). Idempotent, safe to re-run.
 - `config/` — reference config files/snippets to drop onto a node (firewall
  rules, sshd config, etc.).
 ## Status
 Planning stage — node 1 not yet built.
@@ -0,0 +1,31 @@
 # Example cluster-wide firewall rules for /etc/pve/firewall/cluster.fw
 #
 # Placeholders to fill in once the network is built:
 #   <MGMT_CIDR>    - management VLAN/subnet, e.g. 192.168.10.0/24
 #   <COROSYNC_CIDR> - corosync VLAN/subnet
 #   <CEPH_CIDR>    - Ceph public + backend VLAN/subnet (once Ceph is live)
 #
 # Copy to /etc/pve/firewall/cluster.fw and edit before enabling.
 [OPTIONS]
 enable: 1
 policy_in: DROP
 policy_out: ACCEPT
 [IPSET mgmt]
 <MGMT_CIDR>
 [RULES]
 # Web UI + SSH only from the management network
 IN ACCEPT -source +mgmt -p tcp -dport 8006 -log nolog
 IN ACCEPT -source +mgmt -p tcp -dport 22 -log nolog
 # Corosync (cluster quorum) - nodes only, restrict source to the
 # corosync subnet once it's provisioned
 IN ACCEPT -source <COROSYNC_CIDR> -p udp -dport 5404:5405 -log nolog
 # Ceph (uncomment once Ceph is live; ports: mon 3300,6789, osd/mgr/mds
 # 6800-7300)
 # IN ACCEPT -source <CEPH_CIDR> -p tcp -dport 3300 -log nolog
 # IN ACCEPT -source <CEPH_CIDR> -p tcp -dport 6789 -log nolog
 # IN ACCEPT -source <CEPH_CIDR> -p tcp -dport 6800:7300 -log nolog
@@ -0,0 +1,57 @@
 # Overview & Roadmap
 ## Background
 Current hardware requires disabling KVM hardware virtualization for VMs to
 start at all (falls back to software emulation — slow). This is a
 host/BIOS-level issue, not a Proxmox limitation, and shouldn't be needed on
 new hardware. Before reusing that workaround on the new box:
 - Confirm VT-x (Intel) / AMD-V (AMD) is enabled in BIOS/UEFI.
 - Confirm IOMMU is enabled if passthrough is planned.
 - Update BIOS/microcode first.
 - Rule out running Proxmox nested inside another hypervisor.
 ## End goal
 3-node Proxmox VE cluster with HA-managed VMs. Two ways to get VM disks
 available on more than one node (see `02-storage-zfs-ceph.md`):
 1. **Ceph** — true distributed shared storage, sync replication, needs 3+
   nodes and a fast dedicated network. Zero/near-zero RPO on failover.
 2. **ZFS + storage replication** — local ZFS pool per node, Proxmox
   replicates VM disks between nodes on a schedule (as often as every
   minute). Lighter weight, async — failover loses whatever changed since
   last replication.
 Starting point: ZFS + replication (lighter, works from node 1 onward).
 Ceph is the upgrade path once 3 nodes exist and/or zero-RPO failover
 matters enough to justify the overhead.
 ## Cluster fundamentals (apply from node 1 onward)
 - 3 nodes minimum for real quorum. If starting with 2, add a QDevice
  (small VM or Raspberry Pi) as tie-breaker.
 - Dedicated network for corosync (cluster/quorum traffic) — never shared
  with VM or storage traffic. Needs low, consistent latency (well under
  5ms); jitter matters more than bandwidth.
 - All nodes on the same PVE version, NTP-synced, SSH reachable between
  nodes.
 - Set VM CPU type to a portable type (e.g. `x86-64-v2-AES` or `kvm64`)
  rather than `host` if nodes will ever have different CPUs — needed for
  clean live migration.
 ## Rollout sequence
 1. Build node 1 per `01-hardware-node1.md` — fresh PVE install on ZFS boot
   mirror, local ZFS "tier 2" pool for VM disks, Ceph-earmarked disks left
   idle or as a temporary ZFS pool.
 2. Migrate VMs off old hardware via `vzdump` → copy backups → `qmrestore`
   onto the new ZFS storage (converts disks to ZVOLs).
 3. Apply hardening (`04-security-hardening.md`) and networking
   (`03-networking.md`) before exposing the node beyond the LAN.
 4. Add nodes 2 and 3 identically (same disk/network layout).
 5. Join cluster, stand up dedicated corosync network, configure
   replication and HA groups.
 6. Optionally migrate the Ceph-earmarked disks from temporary ZFS to real
   Ceph OSDs once 3 nodes are up.
@@ -0,0 +1,69 @@
 # Node 1 Hardware Layout
 Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect
 disks or network when the cluster grows.
 ## Disks — three separate roles, physically separate devices
 1. **Boot/OS pool (`rpool`)** — 2x small SSDs (240-480GB plenty), ZFS
   mirror. Proxmox itself only. Never share with Ceph OSDs or bulk ZFS
   data pools.
 2. **Future Ceph OSD disks** — must end up as raw, unformatted devices —
   no ZFS/RAID/LVM underneath (Ceph does its own replication; anything
   underneath just doubles copy-on-write/checksumming and hurts
   performance). Use enterprise SATA/NVMe SSDs with power-loss protection
   (PLP) — matters far more for Ceph write latency than for general ZFS
   use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks
   either sit idle or run as a temporary local ZFS pool to be wiped and
   handed to Ceph once nodes 2/3 exist.
 3. **Local ZFS "replicated tier" disks** — separate set of disks
   (mirror or small raidz) for VMs kept on local storage + PVE
   replication rather than Ceph (latency-sensitive or non-critical
   workloads). This pool is permanent, not a placeholder.
 If budget only allows one extra disk set right now: prioritize the
 future-Ceph disks, run everything on ZFS locally until nodes 2/3 arrive,
 then split workloads out. Avoid consumer QLC SSDs for either role — Ceph
 punishes it on latency, ZFS on sync writes/scrub.
 ## Networking — cable and provision for the final topology now
 Logically separate networks (ideally separate NICs/VLANs):
 - **Management** — web UI / SSH
 - **Corosync** — cluster quorum traffic, low-latency, unshared
 - **Ceph public** — VM-to-OSD traffic
 - **Ceph cluster/backend** — OSD-to-OSD replication (heaviest load)
 Practical layout: 2x 10/25GbE bonded or split — one pair for Ceph, one
 for mgmt + corosync + VM traffic, with corosync on its own VLAN even if
 sharing a physical NIC. Get switch/cabling right on node 1 so nodes 2/3
 are identical drops.
 ## CPU / RAM sizing
 Size for the end state, not day one — RAM is the hardest thing to
 retrofit. Budget covers:
 - OS + ZFS ARC (ZFS wants RAM, not just disk)
 - Ceph OSD daemons — realistically 3-5GB per OSD once running
 - Actual VM workloads
 Roughly a core per OSD on top of what VMs need. If OSDs won't be active
 for a while, that's headway, but buy for 3 nodes' worth of eventual OSD
 load.
 ## Backup target (PBS)
 Keep it off the Ceph/compute nodes if possible — its failure domain
 should be independent of the cluster. Modest separate machine or NAS:
 ZFS mirror or raidz2, ECC RAM if possible, capacity for retention policy.
 If it has to run as a VM inside the cluster short-term, that's a known
 compromise, not the end state.
 ## Node 1 install sequence
 1. Install Proxmox VE fresh onto the ZFS boot mirror.
 2. Create the local ZFS "tier 2" pool for VM disks.
 3. Leave Ceph-earmarked disks idle, or provision as a temporary ZFS pool
   (to be wiped once Ceph goes live at 3 nodes).
@@ -0,0 +1,46 @@
 # Storage: LVM-thin → ZFS, and the path to Ceph
 ## Why move off LVM-thin
 Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs
 a VM's disk reachable from more than one node so it can restart elsewhere
 on host failure. LVM-thin has no answer for that. ZFS does, via
 replication; Ceph does natively.
 ## Two paths to HA-capable storage
 | | ZFS + replication | Ceph |
 |---|---|---|
 | Nodes required | 1+ (replication needs 2+ targets) | 3+ |
 | Consistency | Async — snapshot-based, as often as every minute | Sync — real shared storage |
 | Data loss on failover | Whatever changed since last replication cycle | ~None |
 | Network needs | Normal cluster link | Fast dedicated network (see `03-networking.md`) |
 | Overhead | Low | Higher RAM/CPU/disk |
 Starting point: **ZFS + replication**. Revisit Ceph once 3 nodes exist or
 zero-RPO failover is worth the overhead.
 ## Migration plan (old hardware → new hardware)
 Don't convert the old LVM-thin box in place. Rebuild fresh on new
 hardware with ZFS from the installer (mirror if 2+ disks), then move VMs:
 1. On the old host: `vzdump` each VM to a backup file (external drive,
   NFS share, or PBS if available).
 2. Copy backups to the new host.
 3. `qmrestore` onto the new ZFS storage — disks land as ZVOLs.
 Alternative if both hosts can see each other on the network: temporarily
 cluster them and use the GUI "Migrate" with a storage move (offline only
 — live migration doesn't cross storage types).
 ## Later: handing Ceph-earmarked disks over
 Once nodes 2 and 3 are up and the Ceph-earmarked disks (see
 `01-hardware-node1.md`) can be pooled 3-node minimum:
 1. Wipe any temporary ZFS pool on those disks.
 2. Initialize Ceph across the 3 nodes.
 3. Create OSDs directly on the raw disks (no ZFS/RAID underneath).
 4. Migrate VMs that need zero-RPO failover from the ZFS-replicated tier
   onto Ceph-backed storage.
@@ -0,0 +1,42 @@
 # Networking
 ## Required separation
 Keep these on logically separate networks/VLANs, ideally separate NICs:
 - **Management** — web UI (8006), SSH
 - **Corosync** — cluster quorum. Low, *consistent* latency (well under
  5ms) matters more than bandwidth. Never share with VM/storage traffic.
 - **Ceph public** — VM-to-OSD traffic (once Ceph is live)
 - **Ceph cluster/backend** — OSD-to-OSD replication, heaviest load
 ## Practical layout
 2x 10/25GbE bonded or split:
 - Link pair A → Ceph (public + backend, or split further if 4 NICs
  available)
 - Link pair B → management + corosync + VM traffic, with corosync on its
  own VLAN even when sharing a physical NIC with the rest
 ## Cluster join requirements
 - All nodes reachable to each other on SSH (22) and the corosync network
 - Same PVE version across nodes
 - NTP-synced clocks
 ## Firewall
 Proxmox's built-in firewall operates at datacenter and node level.
 Default-deny, then whitelist:
 - SSH from the management network/VLAN only
 - Web UI (8006) from the management network/VLAN only
 - Corosync ports between cluster nodes
 - Ceph ports between cluster nodes (once Ceph is live)
 Enforce the network separation above at the firewall — corosync and Ceph
 traffic shouldn't be reachable from the VM network even if they end up
 sharing a physical link.
 See `config/pve-firewall/` for a starting rule set.
@@ -0,0 +1,54 @@
 # Security Hardening
 Proxmox has no `sudo` out of the box — everything defaults to root. That's
 the install default, not the recommended end state. Two layers to harden
 separately.
 ## Linux/SSH layer
 - `PermitRootLogin prohibit-password` in `sshd_config` — root can only
  log in via SSH key, never password. Kills most brute-force attempts.
  See `scripts/harden-ssh.sh`.
 - fail2ban jail for SSH on top of that.
 - Restrict SSH to the management VLAN/trusted IPs via the Proxmox
  firewall (see `03-networking.md`) rather than exposing broadly.
 - A separate Linux sudo user isn't strictly required for day-to-day PVE
  admin (the PVE permission system below governs that), but worth adding
  if multiple people SSH into the box directly, for accountability.
 ## PVE/web layer (the one that actually matters day-to-day)
 - Keep `root@pam` for emergencies only.
 - Create a named user (e.g. `wayne@pve`) with the Administrator role for
  routine cluster management: Datacenter → Permissions → Users.
 - Enable 2FA (TOTP or hardware key) on both that account and `root@pam`:
  Datacenter → Permissions → Realms/Users.
 - For API integrations (monitoring, automation, Terraform, etc.), issue
  scoped API tokens with least-privilege roles (e.g. `PVEAuditor` or a
  custom role) — never hand out root credentials.
 ## Firewall
 Default-deny at datacenter/node level, whitelist only what's needed (see
 `03-networking.md` for the specifics). Config templates in
 `config/pve-firewall/`.
 ## Repos and updates
 Fresh installs point at the enterprise repo, which fails on `apt update`
 without a subscription. Switch to the no-subscription repo (or pay for
 enterprise). See `scripts/switch-to-no-subscription-repo.sh`. Keep the
 host patched — hypervisor CVEs are high-value targets.
 ## Misc
 - Management interface on a network you trust, not the same broadcast
  domain as guest VM traffic.
 - If the web UI is ever needed outside the LAN, put it behind a VPN —
  don't port-forward 8006 directly.
 ## Further reading
 - CIS Benchmark for Proxmox VE
 - Community PVE hardening guides (kernel parameters, audit logging,
  storage encryption)
@@ -0,0 +1,51 @@
 #!/bin/bash
 # Apply baseline SSH hardening to a Proxmox VE node: key-only root login
 # + fail2ban. Idempotent - safe to re-run. Run as root on the PVE host.
 set -euo pipefail
 DROPIN_DIR="/etc/ssh/sshd_config.d"
 DROPIN_FILE="${DROPIN_DIR}/99-hardening.conf"
 if [ "$(id -u)" -ne 0 ]; then
    echo "Must run as root." >&2
    exit 1
 fi
 mkdir -p "$DROPIN_DIR"
 cat > "$DROPIN_FILE" <<'EOF'
 PermitRootLogin prohibit-password
 PasswordAuthentication no
 EOF
 echo "Wrote $DROPIN_FILE"
 if ! authorized_keys_present=$(find /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys -type f 2>/dev/null | head -n1); then
    authorized_keys_present=""
 fi
 if [ -z "$authorized_keys_present" ]; then
    echo "WARNING: no authorized_keys found for any user yet." >&2
    echo "Add your SSH public key before disconnecting, or you'll lock yourself out." >&2
 fi
 sshd -t
 systemctl reload sshd
 echo "sshd reloaded with key-only root login."
 if ! dpkg -s fail2ban >/dev/null 2>&1; then
    apt-get update
    apt-get install -y fail2ban
 fi
 mkdir -p /etc/fail2ban/jail.d
 cat > /etc/fail2ban/jail.d/sshd.local <<'EOF'
 [sshd]
 enabled = true
 port = ssh
 backend = systemd
 maxretry = 5
 bantime = 1h
 findtime = 10m
 EOF
 systemctl enable --now fail2ban
 systemctl restart fail2ban
 echo "fail2ban enabled for sshd."
@@ -0,0 +1,33 @@
 #!/bin/bash
 # Switch a fresh Proxmox VE install from the enterprise repo (which fails
 # on apt update without a paid subscription) to the no-subscription repo.
 # Idempotent - safe to re-run. Run as root on the PVE host.
 set -euo pipefail
 if [ "$(id -u)" -ne 0 ]; then
    echo "Must run as root." >&2
    exit 1
 fi
 CODENAME="$(. /etc/os-release && echo "$VERSION_CODENAME")"
 ENTERPRISE_LIST="/etc/apt/sources.list.d/pve-enterprise.list"
 if [ -f "$ENTERPRISE_LIST" ]; then
    sed -i 's/^deb/#deb/' "$ENTERPRISE_LIST"
    echo "Disabled $ENTERPRISE_LIST"
 fi
 CEPH_ENTERPRISE_LIST="/etc/apt/sources.list.d/ceph.list"
 if [ -f "$CEPH_ENTERPRISE_LIST" ] && grep -q enterprise "$CEPH_ENTERPRISE_LIST" 2>/dev/null; then
    sed -i 's/^deb/#deb/' "$CEPH_ENTERPRISE_LIST"
    echo "Disabled $CEPH_ENTERPRISE_LIST"
 fi
 NOSUB_LIST="/etc/apt/sources.list.d/pve-no-subscription.list"
 cat > "$NOSUB_LIST" <<EOF
 deb http://download.proxmox.com/debian/pve ${CODENAME} pve-no-subscription
 EOF
 echo "Wrote $NOSUB_LIST for codename '${CODENAME}'."
 apt-get update
 echo "Repo switched. Review 'apt list --upgradable' before upgrading."