commit a85441211794c2a7b733b109f3df1bb06fc073fc Author: beatzaplenty Date: Mon Jul 20 11:45:54 2026 +1000 Initial planning docs and hardening scripts for HA rebuild Covers node 1 hardware/network layout, LVM-thin -> ZFS migration path, Ceph as the future HA storage upgrade, and baseline SSH/firewall hardening. diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2437a07 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +.claude/settings.local.json +*.swp diff --git a/README.md b/README.md new file mode 100644 index 0000000..eab9916 --- /dev/null +++ b/README.md @@ -0,0 +1,29 @@ +# Proxmox Configuration + +Config, planning docs, and scripts for rebuilding my Proxmox VE homelab on new +hardware, with the end goal of a 3-node HA cluster. + +## Goals + +- Fresh install on new hardware (old box has a KVM/virtualization bug that + forced disabling hardware-accelerated virtualization — not expected to + recur on new hardware; verify VT-x/AMD-V + IOMMU in BIOS before assuming + otherwise). +- Move off LVM-thin to ZFS for local storage. +- Build node 1 so its disk and network layout doesn't need rework when nodes + 2 and 3 are added later. +- End state: 3-node cluster, quorum via corosync, HA-managed VMs backed by + ZFS replication and/or Ceph. + +## Repo layout + +- `docs/` — planning docs: hardware layout, storage migration, networking, + security hardening. Read `docs/00-overview.md` first. +- `scripts/` — scripts to apply configuration on a node (SSH hardening, + repo switch, etc.). Idempotent, safe to re-run. +- `config/` — reference config files/snippets to drop onto a node (firewall + rules, sshd config, etc.). + +## Status + +Planning stage — node 1 not yet built. diff --git a/config/pve-firewall/cluster.fw.example b/config/pve-firewall/cluster.fw.example new file mode 100644 index 0000000..db795a2 --- /dev/null +++ b/config/pve-firewall/cluster.fw.example @@ -0,0 +1,31 @@ +# Example cluster-wide firewall rules for /etc/pve/firewall/cluster.fw +# +# Placeholders to fill in once the network is built: +# - management VLAN/subnet, e.g. 192.168.10.0/24 +# - corosync VLAN/subnet +# - Ceph public + backend VLAN/subnet (once Ceph is live) +# +# Copy to /etc/pve/firewall/cluster.fw and edit before enabling. + +[OPTIONS] +enable: 1 +policy_in: DROP +policy_out: ACCEPT + +[IPSET mgmt] + + +[RULES] +# Web UI + SSH only from the management network +IN ACCEPT -source +mgmt -p tcp -dport 8006 -log nolog +IN ACCEPT -source +mgmt -p tcp -dport 22 -log nolog + +# Corosync (cluster quorum) - nodes only, restrict source to the +# corosync subnet once it's provisioned +IN ACCEPT -source -p udp -dport 5404:5405 -log nolog + +# Ceph (uncomment once Ceph is live; ports: mon 3300,6789, osd/mgr/mds +# 6800-7300) +# IN ACCEPT -source -p tcp -dport 3300 -log nolog +# IN ACCEPT -source -p tcp -dport 6789 -log nolog +# IN ACCEPT -source -p tcp -dport 6800:7300 -log nolog diff --git a/docs/00-overview.md b/docs/00-overview.md new file mode 100644 index 0000000..32d763a --- /dev/null +++ b/docs/00-overview.md @@ -0,0 +1,57 @@ +# Overview & Roadmap + +## Background + +Current hardware requires disabling KVM hardware virtualization for VMs to +start at all (falls back to software emulation — slow). This is a +host/BIOS-level issue, not a Proxmox limitation, and shouldn't be needed on +new hardware. Before reusing that workaround on the new box: + +- Confirm VT-x (Intel) / AMD-V (AMD) is enabled in BIOS/UEFI. +- Confirm IOMMU is enabled if passthrough is planned. +- Update BIOS/microcode first. +- Rule out running Proxmox nested inside another hypervisor. + +## End goal + +3-node Proxmox VE cluster with HA-managed VMs. Two ways to get VM disks +available on more than one node (see `02-storage-zfs-ceph.md`): + +1. **Ceph** — true distributed shared storage, sync replication, needs 3+ + nodes and a fast dedicated network. Zero/near-zero RPO on failover. +2. **ZFS + storage replication** — local ZFS pool per node, Proxmox + replicates VM disks between nodes on a schedule (as often as every + minute). Lighter weight, async — failover loses whatever changed since + last replication. + +Starting point: ZFS + replication (lighter, works from node 1 onward). +Ceph is the upgrade path once 3 nodes exist and/or zero-RPO failover +matters enough to justify the overhead. + +## Cluster fundamentals (apply from node 1 onward) + +- 3 nodes minimum for real quorum. If starting with 2, add a QDevice + (small VM or Raspberry Pi) as tie-breaker. +- Dedicated network for corosync (cluster/quorum traffic) — never shared + with VM or storage traffic. Needs low, consistent latency (well under + 5ms); jitter matters more than bandwidth. +- All nodes on the same PVE version, NTP-synced, SSH reachable between + nodes. +- Set VM CPU type to a portable type (e.g. `x86-64-v2-AES` or `kvm64`) + rather than `host` if nodes will ever have different CPUs — needed for + clean live migration. + +## Rollout sequence + +1. Build node 1 per `01-hardware-node1.md` — fresh PVE install on ZFS boot + mirror, local ZFS "tier 2" pool for VM disks, Ceph-earmarked disks left + idle or as a temporary ZFS pool. +2. Migrate VMs off old hardware via `vzdump` → copy backups → `qmrestore` + onto the new ZFS storage (converts disks to ZVOLs). +3. Apply hardening (`04-security-hardening.md`) and networking + (`03-networking.md`) before exposing the node beyond the LAN. +4. Add nodes 2 and 3 identically (same disk/network layout). +5. Join cluster, stand up dedicated corosync network, configure + replication and HA groups. +6. Optionally migrate the Ceph-earmarked disks from temporary ZFS to real + Ceph OSDs once 3 nodes are up. diff --git a/docs/01-hardware-node1.md b/docs/01-hardware-node1.md new file mode 100644 index 0000000..c2f2d16 --- /dev/null +++ b/docs/01-hardware-node1.md @@ -0,0 +1,69 @@ +# Node 1 Hardware Layout + +Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect +disks or network when the cluster grows. + +## Disks — three separate roles, physically separate devices + +1. **Boot/OS pool (`rpool`)** — 2x small SSDs (240-480GB plenty), ZFS + mirror. Proxmox itself only. Never share with Ceph OSDs or bulk ZFS + data pools. +2. **Future Ceph OSD disks** — must end up as raw, unformatted devices — + no ZFS/RAID/LVM underneath (Ceph does its own replication; anything + underneath just doubles copy-on-write/checksumming and hurts + performance). Use enterprise SATA/NVMe SSDs with power-loss protection + (PLP) — matters far more for Ceph write latency than for general ZFS + use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks + either sit idle or run as a temporary local ZFS pool to be wiped and + handed to Ceph once nodes 2/3 exist. +3. **Local ZFS "replicated tier" disks** — separate set of disks + (mirror or small raidz) for VMs kept on local storage + PVE + replication rather than Ceph (latency-sensitive or non-critical + workloads). This pool is permanent, not a placeholder. + +If budget only allows one extra disk set right now: prioritize the +future-Ceph disks, run everything on ZFS locally until nodes 2/3 arrive, +then split workloads out. Avoid consumer QLC SSDs for either role — Ceph +punishes it on latency, ZFS on sync writes/scrub. + +## Networking — cable and provision for the final topology now + +Logically separate networks (ideally separate NICs/VLANs): + +- **Management** — web UI / SSH +- **Corosync** — cluster quorum traffic, low-latency, unshared +- **Ceph public** — VM-to-OSD traffic +- **Ceph cluster/backend** — OSD-to-OSD replication (heaviest load) + +Practical layout: 2x 10/25GbE bonded or split — one pair for Ceph, one +for mgmt + corosync + VM traffic, with corosync on its own VLAN even if +sharing a physical NIC. Get switch/cabling right on node 1 so nodes 2/3 +are identical drops. + +## CPU / RAM sizing + +Size for the end state, not day one — RAM is the hardest thing to +retrofit. Budget covers: + +- OS + ZFS ARC (ZFS wants RAM, not just disk) +- Ceph OSD daemons — realistically 3-5GB per OSD once running +- Actual VM workloads + +Roughly a core per OSD on top of what VMs need. If OSDs won't be active +for a while, that's headway, but buy for 3 nodes' worth of eventual OSD +load. + +## Backup target (PBS) + +Keep it off the Ceph/compute nodes if possible — its failure domain +should be independent of the cluster. Modest separate machine or NAS: +ZFS mirror or raidz2, ECC RAM if possible, capacity for retention policy. +If it has to run as a VM inside the cluster short-term, that's a known +compromise, not the end state. + +## Node 1 install sequence + +1. Install Proxmox VE fresh onto the ZFS boot mirror. +2. Create the local ZFS "tier 2" pool for VM disks. +3. Leave Ceph-earmarked disks idle, or provision as a temporary ZFS pool + (to be wiped once Ceph goes live at 3 nodes). diff --git a/docs/02-storage-zfs-ceph.md b/docs/02-storage-zfs-ceph.md new file mode 100644 index 0000000..cac009f --- /dev/null +++ b/docs/02-storage-zfs-ceph.md @@ -0,0 +1,46 @@ +# Storage: LVM-thin → ZFS, and the path to Ceph + +## Why move off LVM-thin + +Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs +a VM's disk reachable from more than one node so it can restart elsewhere +on host failure. LVM-thin has no answer for that. ZFS does, via +replication; Ceph does natively. + +## Two paths to HA-capable storage + +| | ZFS + replication | Ceph | +|---|---|---| +| Nodes required | 1+ (replication needs 2+ targets) | 3+ | +| Consistency | Async — snapshot-based, as often as every minute | Sync — real shared storage | +| Data loss on failover | Whatever changed since last replication cycle | ~None | +| Network needs | Normal cluster link | Fast dedicated network (see `03-networking.md`) | +| Overhead | Low | Higher RAM/CPU/disk | + +Starting point: **ZFS + replication**. Revisit Ceph once 3 nodes exist or +zero-RPO failover is worth the overhead. + +## Migration plan (old hardware → new hardware) + +Don't convert the old LVM-thin box in place. Rebuild fresh on new +hardware with ZFS from the installer (mirror if 2+ disks), then move VMs: + +1. On the old host: `vzdump` each VM to a backup file (external drive, + NFS share, or PBS if available). +2. Copy backups to the new host. +3. `qmrestore` onto the new ZFS storage — disks land as ZVOLs. + +Alternative if both hosts can see each other on the network: temporarily +cluster them and use the GUI "Migrate" with a storage move (offline only +— live migration doesn't cross storage types). + +## Later: handing Ceph-earmarked disks over + +Once nodes 2 and 3 are up and the Ceph-earmarked disks (see +`01-hardware-node1.md`) can be pooled 3-node minimum: + +1. Wipe any temporary ZFS pool on those disks. +2. Initialize Ceph across the 3 nodes. +3. Create OSDs directly on the raw disks (no ZFS/RAID underneath). +4. Migrate VMs that need zero-RPO failover from the ZFS-replicated tier + onto Ceph-backed storage. diff --git a/docs/03-networking.md b/docs/03-networking.md new file mode 100644 index 0000000..20dee50 --- /dev/null +++ b/docs/03-networking.md @@ -0,0 +1,42 @@ +# Networking + +## Required separation + +Keep these on logically separate networks/VLANs, ideally separate NICs: + +- **Management** — web UI (8006), SSH +- **Corosync** — cluster quorum. Low, *consistent* latency (well under + 5ms) matters more than bandwidth. Never share with VM/storage traffic. +- **Ceph public** — VM-to-OSD traffic (once Ceph is live) +- **Ceph cluster/backend** — OSD-to-OSD replication, heaviest load + +## Practical layout + +2x 10/25GbE bonded or split: + +- Link pair A → Ceph (public + backend, or split further if 4 NICs + available) +- Link pair B → management + corosync + VM traffic, with corosync on its + own VLAN even when sharing a physical NIC with the rest + +## Cluster join requirements + +- All nodes reachable to each other on SSH (22) and the corosync network +- Same PVE version across nodes +- NTP-synced clocks + +## Firewall + +Proxmox's built-in firewall operates at datacenter and node level. +Default-deny, then whitelist: + +- SSH from the management network/VLAN only +- Web UI (8006) from the management network/VLAN only +- Corosync ports between cluster nodes +- Ceph ports between cluster nodes (once Ceph is live) + +Enforce the network separation above at the firewall — corosync and Ceph +traffic shouldn't be reachable from the VM network even if they end up +sharing a physical link. + +See `config/pve-firewall/` for a starting rule set. diff --git a/docs/04-security-hardening.md b/docs/04-security-hardening.md new file mode 100644 index 0000000..e341114 --- /dev/null +++ b/docs/04-security-hardening.md @@ -0,0 +1,54 @@ +# Security Hardening + +Proxmox has no `sudo` out of the box — everything defaults to root. That's +the install default, not the recommended end state. Two layers to harden +separately. + +## Linux/SSH layer + +- `PermitRootLogin prohibit-password` in `sshd_config` — root can only + log in via SSH key, never password. Kills most brute-force attempts. + See `scripts/harden-ssh.sh`. +- fail2ban jail for SSH on top of that. +- Restrict SSH to the management VLAN/trusted IPs via the Proxmox + firewall (see `03-networking.md`) rather than exposing broadly. +- A separate Linux sudo user isn't strictly required for day-to-day PVE + admin (the PVE permission system below governs that), but worth adding + if multiple people SSH into the box directly, for accountability. + +## PVE/web layer (the one that actually matters day-to-day) + +- Keep `root@pam` for emergencies only. +- Create a named user (e.g. `wayne@pve`) with the Administrator role for + routine cluster management: Datacenter → Permissions → Users. +- Enable 2FA (TOTP or hardware key) on both that account and `root@pam`: + Datacenter → Permissions → Realms/Users. +- For API integrations (monitoring, automation, Terraform, etc.), issue + scoped API tokens with least-privilege roles (e.g. `PVEAuditor` or a + custom role) — never hand out root credentials. + +## Firewall + +Default-deny at datacenter/node level, whitelist only what's needed (see +`03-networking.md` for the specifics). Config templates in +`config/pve-firewall/`. + +## Repos and updates + +Fresh installs point at the enterprise repo, which fails on `apt update` +without a subscription. Switch to the no-subscription repo (or pay for +enterprise). See `scripts/switch-to-no-subscription-repo.sh`. Keep the +host patched — hypervisor CVEs are high-value targets. + +## Misc + +- Management interface on a network you trust, not the same broadcast + domain as guest VM traffic. +- If the web UI is ever needed outside the LAN, put it behind a VPN — + don't port-forward 8006 directly. + +## Further reading + +- CIS Benchmark for Proxmox VE +- Community PVE hardening guides (kernel parameters, audit logging, + storage encryption) diff --git a/scripts/harden-ssh.sh b/scripts/harden-ssh.sh new file mode 100755 index 0000000..1b0c2c1 --- /dev/null +++ b/scripts/harden-ssh.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# Apply baseline SSH hardening to a Proxmox VE node: key-only root login +# + fail2ban. Idempotent - safe to re-run. Run as root on the PVE host. +set -euo pipefail + +DROPIN_DIR="/etc/ssh/sshd_config.d" +DROPIN_FILE="${DROPIN_DIR}/99-hardening.conf" + +if [ "$(id -u)" -ne 0 ]; then + echo "Must run as root." >&2 + exit 1 +fi + +mkdir -p "$DROPIN_DIR" +cat > "$DROPIN_FILE" <<'EOF' +PermitRootLogin prohibit-password +PasswordAuthentication no +EOF +echo "Wrote $DROPIN_FILE" + +if ! authorized_keys_present=$(find /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys -type f 2>/dev/null | head -n1); then + authorized_keys_present="" +fi +if [ -z "$authorized_keys_present" ]; then + echo "WARNING: no authorized_keys found for any user yet." >&2 + echo "Add your SSH public key before disconnecting, or you'll lock yourself out." >&2 +fi + +sshd -t +systemctl reload sshd +echo "sshd reloaded with key-only root login." + +if ! dpkg -s fail2ban >/dev/null 2>&1; then + apt-get update + apt-get install -y fail2ban +fi + +mkdir -p /etc/fail2ban/jail.d +cat > /etc/fail2ban/jail.d/sshd.local <<'EOF' +[sshd] +enabled = true +port = ssh +backend = systemd +maxretry = 5 +bantime = 1h +findtime = 10m +EOF + +systemctl enable --now fail2ban +systemctl restart fail2ban +echo "fail2ban enabled for sshd." diff --git a/scripts/switch-to-no-subscription-repo.sh b/scripts/switch-to-no-subscription-repo.sh new file mode 100755 index 0000000..3ccf8eb --- /dev/null +++ b/scripts/switch-to-no-subscription-repo.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# Switch a fresh Proxmox VE install from the enterprise repo (which fails +# on apt update without a paid subscription) to the no-subscription repo. +# Idempotent - safe to re-run. Run as root on the PVE host. +set -euo pipefail + +if [ "$(id -u)" -ne 0 ]; then + echo "Must run as root." >&2 + exit 1 +fi + +CODENAME="$(. /etc/os-release && echo "$VERSION_CODENAME")" + +ENTERPRISE_LIST="/etc/apt/sources.list.d/pve-enterprise.list" +if [ -f "$ENTERPRISE_LIST" ]; then + sed -i 's/^deb/#deb/' "$ENTERPRISE_LIST" + echo "Disabled $ENTERPRISE_LIST" +fi + +CEPH_ENTERPRISE_LIST="/etc/apt/sources.list.d/ceph.list" +if [ -f "$CEPH_ENTERPRISE_LIST" ] && grep -q enterprise "$CEPH_ENTERPRISE_LIST" 2>/dev/null; then + sed -i 's/^deb/#deb/' "$CEPH_ENTERPRISE_LIST" + echo "Disabled $CEPH_ENTERPRISE_LIST" +fi + +NOSUB_LIST="/etc/apt/sources.list.d/pve-no-subscription.list" +cat > "$NOSUB_LIST" <