Initial planning docs and hardening scripts for HA rebuild

Covers node 1 hardware/network layout, LVM-thin -> ZFS migration path,
Ceph as the future HA storage upgrade, and baseline SSH/firewall
hardening.

docs/00-overview.md

@@ -0,0 +1,57 @@
+# Overview & Roadmap
+
+## Background
+
+Current hardware requires disabling KVM hardware virtualization for VMs to
+start at all (falls back to software emulation — slow). This is a
+host/BIOS-level issue, not a Proxmox limitation, and shouldn't be needed on
+new hardware. Before reusing that workaround on the new box:
+
+- Confirm VT-x (Intel) / AMD-V (AMD) is enabled in BIOS/UEFI.
+- Confirm IOMMU is enabled if passthrough is planned.
+- Update BIOS/microcode first.
+- Rule out running Proxmox nested inside another hypervisor.
+
+## End goal
+
+3-node Proxmox VE cluster with HA-managed VMs. Two ways to get VM disks
+available on more than one node (see `02-storage-zfs-ceph.md`):
+
+1. **Ceph** — true distributed shared storage, sync replication, needs 3+
+   nodes and a fast dedicated network. Zero/near-zero RPO on failover.
+2. **ZFS + storage replication** — local ZFS pool per node, Proxmox
+   replicates VM disks between nodes on a schedule (as often as every
+   minute). Lighter weight, async — failover loses whatever changed since
+   last replication.
+
+Starting point: ZFS + replication (lighter, works from node 1 onward).
+Ceph is the upgrade path once 3 nodes exist and/or zero-RPO failover
+matters enough to justify the overhead.
+
+## Cluster fundamentals (apply from node 1 onward)
+
+- 3 nodes minimum for real quorum. If starting with 2, add a QDevice
+  (small VM or Raspberry Pi) as tie-breaker.
+- Dedicated network for corosync (cluster/quorum traffic) — never shared
+  with VM or storage traffic. Needs low, consistent latency (well under
+  5ms); jitter matters more than bandwidth.
+- All nodes on the same PVE version, NTP-synced, SSH reachable between
+  nodes.
+- Set VM CPU type to a portable type (e.g. `x86-64-v2-AES` or `kvm64`)
+  rather than `host` if nodes will ever have different CPUs — needed for
+  clean live migration.
+
+## Rollout sequence
+
+1. Build node 1 per `01-hardware-node1.md` — fresh PVE install on ZFS boot
+   mirror, local ZFS "tier 2" pool for VM disks, Ceph-earmarked disks left
+   idle or as a temporary ZFS pool.
+2. Migrate VMs off old hardware via `vzdump` → copy backups → `qmrestore`
+   onto the new ZFS storage (converts disks to ZVOLs).
+3. Apply hardening (`04-security-hardening.md`) and networking
+   (`03-networking.md`) before exposing the node beyond the LAN.
+4. Add nodes 2 and 3 identically (same disk/network layout).
+5. Join cluster, stand up dedicated corosync network, configure
+   replication and HA groups.
+6. Optionally migrate the Ceph-earmarked disks from temporary ZFS to real
+   Ceph OSDs once 3 nodes are up.

docs/01-hardware-node1.md

@@ -0,0 +1,69 @@
+# Node 1 Hardware Layout
+
+Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect
+disks or network when the cluster grows.
+
+## Disks — three separate roles, physically separate devices
+
+1. **Boot/OS pool (`rpool`)** — 2x small SSDs (240-480GB plenty), ZFS
+   mirror. Proxmox itself only. Never share with Ceph OSDs or bulk ZFS
+   data pools.
+2. **Future Ceph OSD disks** — must end up as raw, unformatted devices —
+   no ZFS/RAID/LVM underneath (Ceph does its own replication; anything
+   underneath just doubles copy-on-write/checksumming and hurts
+   performance). Use enterprise SATA/NVMe SSDs with power-loss protection
+   (PLP) — matters far more for Ceph write latency than for general ZFS
+   use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks
+   either sit idle or run as a temporary local ZFS pool to be wiped and
+   handed to Ceph once nodes 2/3 exist.
+3. **Local ZFS "replicated tier" disks** — separate set of disks
+   (mirror or small raidz) for VMs kept on local storage + PVE
+   replication rather than Ceph (latency-sensitive or non-critical
+   workloads). This pool is permanent, not a placeholder.
+
+If budget only allows one extra disk set right now: prioritize the
+future-Ceph disks, run everything on ZFS locally until nodes 2/3 arrive,
+then split workloads out. Avoid consumer QLC SSDs for either role — Ceph
+punishes it on latency, ZFS on sync writes/scrub.
+
+## Networking — cable and provision for the final topology now
+
+Logically separate networks (ideally separate NICs/VLANs):
+
+- **Management** — web UI / SSH
+- **Corosync** — cluster quorum traffic, low-latency, unshared
+- **Ceph public** — VM-to-OSD traffic
+- **Ceph cluster/backend** — OSD-to-OSD replication (heaviest load)
+
+Practical layout: 2x 10/25GbE bonded or split — one pair for Ceph, one
+for mgmt + corosync + VM traffic, with corosync on its own VLAN even if
+sharing a physical NIC. Get switch/cabling right on node 1 so nodes 2/3
+are identical drops.
+
+## CPU / RAM sizing
+
+Size for the end state, not day one — RAM is the hardest thing to
+retrofit. Budget covers:
+
+- OS + ZFS ARC (ZFS wants RAM, not just disk)
+- Ceph OSD daemons — realistically 3-5GB per OSD once running
+- Actual VM workloads
+
+Roughly a core per OSD on top of what VMs need. If OSDs won't be active
+for a while, that's headway, but buy for 3 nodes' worth of eventual OSD
+load.
+
+## Backup target (PBS)
+
+Keep it off the Ceph/compute nodes if possible — its failure domain
+should be independent of the cluster. Modest separate machine or NAS:
+ZFS mirror or raidz2, ECC RAM if possible, capacity for retention policy.
+If it has to run as a VM inside the cluster short-term, that's a known
+compromise, not the end state.
+
+## Node 1 install sequence
+
+1. Install Proxmox VE fresh onto the ZFS boot mirror.
+2. Create the local ZFS "tier 2" pool for VM disks.
+3. Leave Ceph-earmarked disks idle, or provision as a temporary ZFS pool
+   (to be wiped once Ceph goes live at 3 nodes).

docs/02-storage-zfs-ceph.md

@@ -0,0 +1,46 @@
+# Storage: LVM-thin → ZFS, and the path to Ceph
+
+## Why move off LVM-thin
+
+Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs
+a VM's disk reachable from more than one node so it can restart elsewhere
+on host failure. LVM-thin has no answer for that. ZFS does, via
+replication; Ceph does natively.
+
+## Two paths to HA-capable storage
+
+| | ZFS + replication | Ceph |
+|---|---|---|
+| Nodes required | 1+ (replication needs 2+ targets) | 3+ |
+| Consistency | Async — snapshot-based, as often as every minute | Sync — real shared storage |
+| Data loss on failover | Whatever changed since last replication cycle | ~None |
+| Network needs | Normal cluster link | Fast dedicated network (see `03-networking.md`) |
+| Overhead | Low | Higher RAM/CPU/disk |
+
+Starting point: **ZFS + replication**. Revisit Ceph once 3 nodes exist or
+zero-RPO failover is worth the overhead.
+
+## Migration plan (old hardware → new hardware)
+
+Don't convert the old LVM-thin box in place. Rebuild fresh on new
+hardware with ZFS from the installer (mirror if 2+ disks), then move VMs:
+
+1. On the old host: `vzdump` each VM to a backup file (external drive,
+   NFS share, or PBS if available).
+2. Copy backups to the new host.
+3. `qmrestore` onto the new ZFS storage — disks land as ZVOLs.
+
+Alternative if both hosts can see each other on the network: temporarily
+cluster them and use the GUI "Migrate" with a storage move (offline only
+— live migration doesn't cross storage types).
+
+## Later: handing Ceph-earmarked disks over
+
+Once nodes 2 and 3 are up and the Ceph-earmarked disks (see
+`01-hardware-node1.md`) can be pooled 3-node minimum:
+
+1. Wipe any temporary ZFS pool on those disks.
+2. Initialize Ceph across the 3 nodes.
+3. Create OSDs directly on the raw disks (no ZFS/RAID underneath).
+4. Migrate VMs that need zero-RPO failover from the ZFS-replicated tier
+   onto Ceph-backed storage.

docs/03-networking.md

@@ -0,0 +1,42 @@
+# Networking
+
+## Required separation
+
+Keep these on logically separate networks/VLANs, ideally separate NICs:
+
+- **Management** — web UI (8006), SSH
+- **Corosync** — cluster quorum. Low, *consistent* latency (well under
+  5ms) matters more than bandwidth. Never share with VM/storage traffic.
+- **Ceph public** — VM-to-OSD traffic (once Ceph is live)
+- **Ceph cluster/backend** — OSD-to-OSD replication, heaviest load
+
+## Practical layout
+
+2x 10/25GbE bonded or split:
+
+- Link pair A → Ceph (public + backend, or split further if 4 NICs
+  available)
+- Link pair B → management + corosync + VM traffic, with corosync on its
+  own VLAN even when sharing a physical NIC with the rest
+
+## Cluster join requirements
+
+- All nodes reachable to each other on SSH (22) and the corosync network
+- Same PVE version across nodes
+- NTP-synced clocks
+
+## Firewall
+
+Proxmox's built-in firewall operates at datacenter and node level.
+Default-deny, then whitelist:
+
+- SSH from the management network/VLAN only
+- Web UI (8006) from the management network/VLAN only
+- Corosync ports between cluster nodes
+- Ceph ports between cluster nodes (once Ceph is live)
+
+Enforce the network separation above at the firewall — corosync and Ceph
+traffic shouldn't be reachable from the VM network even if they end up
+sharing a physical link.
+
+See `config/pve-firewall/` for a starting rule set.

docs/04-security-hardening.md

@@ -0,0 +1,54 @@
+# Security Hardening
+
+Proxmox has no `sudo` out of the box — everything defaults to root. That's
+the install default, not the recommended end state. Two layers to harden
+separately.
+
+## Linux/SSH layer
+
+- `PermitRootLogin prohibit-password` in `sshd_config` — root can only
+  log in via SSH key, never password. Kills most brute-force attempts.
+  See `scripts/harden-ssh.sh`.
+- fail2ban jail for SSH on top of that.
+- Restrict SSH to the management VLAN/trusted IPs via the Proxmox
+  firewall (see `03-networking.md`) rather than exposing broadly.
+- A separate Linux sudo user isn't strictly required for day-to-day PVE
+  admin (the PVE permission system below governs that), but worth adding
+  if multiple people SSH into the box directly, for accountability.
+
+## PVE/web layer (the one that actually matters day-to-day)
+
+- Keep `root@pam` for emergencies only.
+- Create a named user (e.g. `wayne@pve`) with the Administrator role for
+  routine cluster management: Datacenter → Permissions → Users.
+- Enable 2FA (TOTP or hardware key) on both that account and `root@pam`:
+  Datacenter → Permissions → Realms/Users.
+- For API integrations (monitoring, automation, Terraform, etc.), issue
+  scoped API tokens with least-privilege roles (e.g. `PVEAuditor` or a
+  custom role) — never hand out root credentials.
+
+## Firewall
+
+Default-deny at datacenter/node level, whitelist only what's needed (see
+`03-networking.md` for the specifics). Config templates in
+`config/pve-firewall/`.
+
+## Repos and updates
+
+Fresh installs point at the enterprise repo, which fails on `apt update`
+without a subscription. Switch to the no-subscription repo (or pay for
+enterprise). See `scripts/switch-to-no-subscription-repo.sh`. Keep the
+host patched — hypervisor CVEs are high-value targets.
+
+## Misc
+
+- Management interface on a network you trust, not the same broadcast
+  domain as guest VM traffic.
+- If the web UI is ever needed outside the LAN, put it behind a VPN —
+  don't port-forward 8006 directly.
+
+## Further reading
+
+- CIS Benchmark for Proxmox VE
+- Community PVE hardening guides (kernel parameters, audit logging,
+  storage encryption)