Initial planning docs and hardening scripts for HA rebuild

Covers node 1 hardware/network layout, LVM-thin -> ZFS migration path,
Ceph as the future HA storage upgrade, and baseline SSH/firewall
hardening.

.gitignore

@@ -0,0 +1,2 @@
+.claude/settings.local.json
+*.swp

README.md

@@ -0,0 +1,29 @@
+# Proxmox Configuration
+
+Config, planning docs, and scripts for rebuilding my Proxmox VE homelab on new
+hardware, with the end goal of a 3-node HA cluster.
+
+## Goals
+
+- Fresh install on new hardware (old box has a KVM/virtualization bug that
+  forced disabling hardware-accelerated virtualization — not expected to
+  recur on new hardware; verify VT-x/AMD-V + IOMMU in BIOS before assuming
+  otherwise).
+- Move off LVM-thin to ZFS for local storage.
+- Build node 1 so its disk and network layout doesn't need rework when nodes
+  2 and 3 are added later.
+- End state: 3-node cluster, quorum via corosync, HA-managed VMs backed by
+  ZFS replication and/or Ceph.
+
+## Repo layout
+
+- `docs/` — planning docs: hardware layout, storage migration, networking,
+  security hardening. Read `docs/00-overview.md` first.
+- `scripts/` — scripts to apply configuration on a node (SSH hardening,
+  repo switch, etc.). Idempotent, safe to re-run.
+- `config/` — reference config files/snippets to drop onto a node (firewall
+  rules, sshd config, etc.).
+
+## Status
+
+Planning stage — node 1 not yet built.

config/pve-firewall/cluster.fw.example

@@ -0,0 +1,31 @@
+# Example cluster-wide firewall rules for /etc/pve/firewall/cluster.fw
+#
+# Placeholders to fill in once the network is built:
+#   <MGMT_CIDR>    - management VLAN/subnet, e.g. 192.168.10.0/24
+#   <COROSYNC_CIDR> - corosync VLAN/subnet
+#   <CEPH_CIDR>    - Ceph public + backend VLAN/subnet (once Ceph is live)
+#
+# Copy to /etc/pve/firewall/cluster.fw and edit before enabling.
+
+[OPTIONS]
+enable: 1
+policy_in: DROP
+policy_out: ACCEPT
+
+[IPSET mgmt]
+<MGMT_CIDR>
+
+[RULES]
+# Web UI + SSH only from the management network
+IN ACCEPT -source +mgmt -p tcp -dport 8006 -log nolog
+IN ACCEPT -source +mgmt -p tcp -dport 22 -log nolog
+
+# Corosync (cluster quorum) - nodes only, restrict source to the
+# corosync subnet once it's provisioned
+IN ACCEPT -source <COROSYNC_CIDR> -p udp -dport 5404:5405 -log nolog
+
+# Ceph (uncomment once Ceph is live; ports: mon 3300,6789, osd/mgr/mds
+# 6800-7300)
+# IN ACCEPT -source <CEPH_CIDR> -p tcp -dport 3300 -log nolog
+# IN ACCEPT -source <CEPH_CIDR> -p tcp -dport 6789 -log nolog
+# IN ACCEPT -source <CEPH_CIDR> -p tcp -dport 6800:7300 -log nolog

docs/00-overview.md

@@ -0,0 +1,57 @@
+# Overview & Roadmap
+
+## Background
+
+Current hardware requires disabling KVM hardware virtualization for VMs to
+start at all (falls back to software emulation — slow). This is a
+host/BIOS-level issue, not a Proxmox limitation, and shouldn't be needed on
+new hardware. Before reusing that workaround on the new box:
+
+- Confirm VT-x (Intel) / AMD-V (AMD) is enabled in BIOS/UEFI.
+- Confirm IOMMU is enabled if passthrough is planned.
+- Update BIOS/microcode first.
+- Rule out running Proxmox nested inside another hypervisor.
+
+## End goal
+
+3-node Proxmox VE cluster with HA-managed VMs. Two ways to get VM disks
+available on more than one node (see `02-storage-zfs-ceph.md`):
+
+1. **Ceph** — true distributed shared storage, sync replication, needs 3+
+   nodes and a fast dedicated network. Zero/near-zero RPO on failover.
+2. **ZFS + storage replication** — local ZFS pool per node, Proxmox
+   replicates VM disks between nodes on a schedule (as often as every
+   minute). Lighter weight, async — failover loses whatever changed since
+   last replication.
+
+Starting point: ZFS + replication (lighter, works from node 1 onward).
+Ceph is the upgrade path once 3 nodes exist and/or zero-RPO failover
+matters enough to justify the overhead.
+
+## Cluster fundamentals (apply from node 1 onward)
+
+- 3 nodes minimum for real quorum. If starting with 2, add a QDevice
+  (small VM or Raspberry Pi) as tie-breaker.
+- Dedicated network for corosync (cluster/quorum traffic) — never shared
+  with VM or storage traffic. Needs low, consistent latency (well under
+  5ms); jitter matters more than bandwidth.
+- All nodes on the same PVE version, NTP-synced, SSH reachable between
+  nodes.
+- Set VM CPU type to a portable type (e.g. `x86-64-v2-AES` or `kvm64`)
+  rather than `host` if nodes will ever have different CPUs — needed for
+  clean live migration.
+
+## Rollout sequence
+
+1. Build node 1 per `01-hardware-node1.md` — fresh PVE install on ZFS boot
+   mirror, local ZFS "tier 2" pool for VM disks, Ceph-earmarked disks left
+   idle or as a temporary ZFS pool.
+2. Migrate VMs off old hardware via `vzdump` → copy backups → `qmrestore`
+   onto the new ZFS storage (converts disks to ZVOLs).
+3. Apply hardening (`04-security-hardening.md`) and networking
+   (`03-networking.md`) before exposing the node beyond the LAN.
+4. Add nodes 2 and 3 identically (same disk/network layout).
+5. Join cluster, stand up dedicated corosync network, configure
+   replication and HA groups.
+6. Optionally migrate the Ceph-earmarked disks from temporary ZFS to real
+   Ceph OSDs once 3 nodes are up.

docs/01-hardware-node1.md

@@ -0,0 +1,69 @@
+# Node 1 Hardware Layout
+
+Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect
+disks or network when the cluster grows.
+
+## Disks — three separate roles, physically separate devices
+
+1. **Boot/OS pool (`rpool`)** — 2x small SSDs (240-480GB plenty), ZFS
+   mirror. Proxmox itself only. Never share with Ceph OSDs or bulk ZFS
+   data pools.
+2. **Future Ceph OSD disks** — must end up as raw, unformatted devices —
+   no ZFS/RAID/LVM underneath (Ceph does its own replication; anything
+   underneath just doubles copy-on-write/checksumming and hurts
+   performance). Use enterprise SATA/NVMe SSDs with power-loss protection
+   (PLP) — matters far more for Ceph write latency than for general ZFS
+   use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks
+   either sit idle or run as a temporary local ZFS pool to be wiped and
+   handed to Ceph once nodes 2/3 exist.
+3. **Local ZFS "replicated tier" disks** — separate set of disks
+   (mirror or small raidz) for VMs kept on local storage + PVE
+   replication rather than Ceph (latency-sensitive or non-critical
+   workloads). This pool is permanent, not a placeholder.
+
+If budget only allows one extra disk set right now: prioritize the
+future-Ceph disks, run everything on ZFS locally until nodes 2/3 arrive,
+then split workloads out. Avoid consumer QLC SSDs for either role — Ceph
+punishes it on latency, ZFS on sync writes/scrub.
+
+## Networking — cable and provision for the final topology now
+
+Logically separate networks (ideally separate NICs/VLANs):
+
+- **Management** — web UI / SSH
+- **Corosync** — cluster quorum traffic, low-latency, unshared
+- **Ceph public** — VM-to-OSD traffic
+- **Ceph cluster/backend** — OSD-to-OSD replication (heaviest load)
+
+Practical layout: 2x 10/25GbE bonded or split — one pair for Ceph, one
+for mgmt + corosync + VM traffic, with corosync on its own VLAN even if
+sharing a physical NIC. Get switch/cabling right on node 1 so nodes 2/3
+are identical drops.
+
+## CPU / RAM sizing
+
+Size for the end state, not day one — RAM is the hardest thing to
+retrofit. Budget covers:
+
+- OS + ZFS ARC (ZFS wants RAM, not just disk)
+- Ceph OSD daemons — realistically 3-5GB per OSD once running
+- Actual VM workloads
+
+Roughly a core per OSD on top of what VMs need. If OSDs won't be active
+for a while, that's headway, but buy for 3 nodes' worth of eventual OSD
+load.
+
+## Backup target (PBS)
+
+Keep it off the Ceph/compute nodes if possible — its failure domain
+should be independent of the cluster. Modest separate machine or NAS:
+ZFS mirror or raidz2, ECC RAM if possible, capacity for retention policy.
+If it has to run as a VM inside the cluster short-term, that's a known
+compromise, not the end state.
+
+## Node 1 install sequence
+
+1. Install Proxmox VE fresh onto the ZFS boot mirror.
+2. Create the local ZFS "tier 2" pool for VM disks.
+3. Leave Ceph-earmarked disks idle, or provision as a temporary ZFS pool
+   (to be wiped once Ceph goes live at 3 nodes).

docs/02-storage-zfs-ceph.md

@@ -0,0 +1,46 @@
+# Storage: LVM-thin → ZFS, and the path to Ceph
+
+## Why move off LVM-thin
+
+Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs
+a VM's disk reachable from more than one node so it can restart elsewhere
+on host failure. LVM-thin has no answer for that. ZFS does, via
+replication; Ceph does natively.
+
+## Two paths to HA-capable storage
+
+| | ZFS + replication | Ceph |
+|---|---|---|
+| Nodes required | 1+ (replication needs 2+ targets) | 3+ |
+| Consistency | Async — snapshot-based, as often as every minute | Sync — real shared storage |
+| Data loss on failover | Whatever changed since last replication cycle | ~None |
+| Network needs | Normal cluster link | Fast dedicated network (see `03-networking.md`) |
+| Overhead | Low | Higher RAM/CPU/disk |
+
+Starting point: **ZFS + replication**. Revisit Ceph once 3 nodes exist or
+zero-RPO failover is worth the overhead.
+
+## Migration plan (old hardware → new hardware)
+
+Don't convert the old LVM-thin box in place. Rebuild fresh on new
+hardware with ZFS from the installer (mirror if 2+ disks), then move VMs:
+
+1. On the old host: `vzdump` each VM to a backup file (external drive,
+   NFS share, or PBS if available).
+2. Copy backups to the new host.
+3. `qmrestore` onto the new ZFS storage — disks land as ZVOLs.
+
+Alternative if both hosts can see each other on the network: temporarily
+cluster them and use the GUI "Migrate" with a storage move (offline only
+— live migration doesn't cross storage types).
+
+## Later: handing Ceph-earmarked disks over
+
+Once nodes 2 and 3 are up and the Ceph-earmarked disks (see
+`01-hardware-node1.md`) can be pooled 3-node minimum:
+
+1. Wipe any temporary ZFS pool on those disks.
+2. Initialize Ceph across the 3 nodes.
+3. Create OSDs directly on the raw disks (no ZFS/RAID underneath).
+4. Migrate VMs that need zero-RPO failover from the ZFS-replicated tier
+   onto Ceph-backed storage.

docs/03-networking.md

@@ -0,0 +1,42 @@
+# Networking
+
+## Required separation
+
+Keep these on logically separate networks/VLANs, ideally separate NICs:
+
+- **Management** — web UI (8006), SSH
+- **Corosync** — cluster quorum. Low, *consistent* latency (well under
+  5ms) matters more than bandwidth. Never share with VM/storage traffic.
+- **Ceph public** — VM-to-OSD traffic (once Ceph is live)
+- **Ceph cluster/backend** — OSD-to-OSD replication, heaviest load
+
+## Practical layout
+
+2x 10/25GbE bonded or split:
+
+- Link pair A → Ceph (public + backend, or split further if 4 NICs
+  available)
+- Link pair B → management + corosync + VM traffic, with corosync on its
+  own VLAN even when sharing a physical NIC with the rest
+
+## Cluster join requirements
+
+- All nodes reachable to each other on SSH (22) and the corosync network
+- Same PVE version across nodes
+- NTP-synced clocks
+
+## Firewall
+
+Proxmox's built-in firewall operates at datacenter and node level.
+Default-deny, then whitelist:
+
+- SSH from the management network/VLAN only
+- Web UI (8006) from the management network/VLAN only
+- Corosync ports between cluster nodes
+- Ceph ports between cluster nodes (once Ceph is live)
+
+Enforce the network separation above at the firewall — corosync and Ceph
+traffic shouldn't be reachable from the VM network even if they end up
+sharing a physical link.
+
+See `config/pve-firewall/` for a starting rule set.

docs/04-security-hardening.md

@@ -0,0 +1,54 @@
+# Security Hardening
+
+Proxmox has no `sudo` out of the box — everything defaults to root. That's
+the install default, not the recommended end state. Two layers to harden
+separately.
+
+## Linux/SSH layer
+
+- `PermitRootLogin prohibit-password` in `sshd_config` — root can only
+  log in via SSH key, never password. Kills most brute-force attempts.
+  See `scripts/harden-ssh.sh`.
+- fail2ban jail for SSH on top of that.
+- Restrict SSH to the management VLAN/trusted IPs via the Proxmox
+  firewall (see `03-networking.md`) rather than exposing broadly.
+- A separate Linux sudo user isn't strictly required for day-to-day PVE
+  admin (the PVE permission system below governs that), but worth adding
+  if multiple people SSH into the box directly, for accountability.
+
+## PVE/web layer (the one that actually matters day-to-day)
+
+- Keep `root@pam` for emergencies only.
+- Create a named user (e.g. `wayne@pve`) with the Administrator role for
+  routine cluster management: Datacenter → Permissions → Users.
+- Enable 2FA (TOTP or hardware key) on both that account and `root@pam`:
+  Datacenter → Permissions → Realms/Users.
+- For API integrations (monitoring, automation, Terraform, etc.), issue
+  scoped API tokens with least-privilege roles (e.g. `PVEAuditor` or a
+  custom role) — never hand out root credentials.
+
+## Firewall
+
+Default-deny at datacenter/node level, whitelist only what's needed (see
+`03-networking.md` for the specifics). Config templates in
+`config/pve-firewall/`.
+
+## Repos and updates
+
+Fresh installs point at the enterprise repo, which fails on `apt update`
+without a subscription. Switch to the no-subscription repo (or pay for
+enterprise). See `scripts/switch-to-no-subscription-repo.sh`. Keep the
+host patched — hypervisor CVEs are high-value targets.
+
+## Misc
+
+- Management interface on a network you trust, not the same broadcast
+  domain as guest VM traffic.
+- If the web UI is ever needed outside the LAN, put it behind a VPN —
+  don't port-forward 8006 directly.
+
+## Further reading
+
+- CIS Benchmark for Proxmox VE
+- Community PVE hardening guides (kernel parameters, audit logging,
+  storage encryption)

scripts/harden-ssh.sh

@@ -0,0 +1,51 @@
+#!/bin/bash
+# Apply baseline SSH hardening to a Proxmox VE node: key-only root login
+# + fail2ban. Idempotent - safe to re-run. Run as root on the PVE host.
+set -euo pipefail
+
+DROPIN_DIR="/etc/ssh/sshd_config.d"
+DROPIN_FILE="${DROPIN_DIR}/99-hardening.conf"
+
+if [ "$(id -u)" -ne 0 ]; then
+    echo "Must run as root." >&2
+    exit 1
+fi
+
+mkdir -p "$DROPIN_DIR"
+cat > "$DROPIN_FILE" <<'EOF'
+PermitRootLogin prohibit-password
+PasswordAuthentication no
+EOF
+echo "Wrote $DROPIN_FILE"
+
+if ! authorized_keys_present=$(find /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys -type f 2>/dev/null | head -n1); then
+    authorized_keys_present=""
+fi
+if [ -z "$authorized_keys_present" ]; then
+    echo "WARNING: no authorized_keys found for any user yet." >&2
+    echo "Add your SSH public key before disconnecting, or you'll lock yourself out." >&2
+fi
+
+sshd -t
+systemctl reload sshd
+echo "sshd reloaded with key-only root login."
+
+if ! dpkg -s fail2ban >/dev/null 2>&1; then
+    apt-get update
+    apt-get install -y fail2ban
+fi
+
+mkdir -p /etc/fail2ban/jail.d
+cat > /etc/fail2ban/jail.d/sshd.local <<'EOF'
+[sshd]
+enabled = true
+port = ssh
+backend = systemd
+maxretry = 5
+bantime = 1h
+findtime = 10m
+EOF
+
+systemctl enable --now fail2ban
+systemctl restart fail2ban
+echo "fail2ban enabled for sshd."

scripts/switch-to-no-subscription-repo.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+# Switch a fresh Proxmox VE install from the enterprise repo (which fails
+# on apt update without a paid subscription) to the no-subscription repo.
+# Idempotent - safe to re-run. Run as root on the PVE host.
+set -euo pipefail
+
+if [ "$(id -u)" -ne 0 ]; then
+    echo "Must run as root." >&2
+    exit 1
+fi
+
+CODENAME="$(. /etc/os-release && echo "$VERSION_CODENAME")"
+
+ENTERPRISE_LIST="/etc/apt/sources.list.d/pve-enterprise.list"
+if [ -f "$ENTERPRISE_LIST" ]; then
+    sed -i 's/^deb/#deb/' "$ENTERPRISE_LIST"
+    echo "Disabled $ENTERPRISE_LIST"
+fi
+
+CEPH_ENTERPRISE_LIST="/etc/apt/sources.list.d/ceph.list"
+if [ -f "$CEPH_ENTERPRISE_LIST" ] && grep -q enterprise "$CEPH_ENTERPRISE_LIST" 2>/dev/null; then
+    sed -i 's/^deb/#deb/' "$CEPH_ENTERPRISE_LIST"
+    echo "Disabled $CEPH_ENTERPRISE_LIST"
+fi
+
+NOSUB_LIST="/etc/apt/sources.list.d/pve-no-subscription.list"
+cat > "$NOSUB_LIST" <<EOF
+deb http://download.proxmox.com/debian/pve ${CODENAME} pve-no-subscription
+EOF
+echo "Wrote $NOSUB_LIST for codename '${CODENAME}'."
+
+apt-get update
+echo "Repo switched. Review 'apt list --upgradable' before upgrading."