From 24a0047fa8c0dcdc21216b8c9ac861af50f82f37 Mon Sep 17 00:00:00 2001 From: beatzaplenty Date: Mon, 20 Jul 2026 11:54:17 +1000 Subject: [PATCH] Drop ZFS-replication as an interim HA step; jump straight to Ceph Simplifies the storage model to one long-term tier (Ceph) instead of two. Node 1 runs local ZFS only until nodes 2/3 join, at which point Ceph goes live and VMs migrate onto it directly. --- README.md | 5 ++-- docs/00-overview.md | 37 ++++++++++++------------ docs/01-hardware-node1.md | 29 +++++++++---------- docs/02-storage-zfs-ceph.md | 57 +++++++++++++++++++++++-------------- 4 files changed, 70 insertions(+), 58 deletions(-) diff --git a/README.md b/README.md index eab9916..e3ad05a 100644 --- a/README.md +++ b/README.md @@ -9,11 +9,12 @@ hardware, with the end goal of a 3-node HA cluster. forced disabling hardware-accelerated virtualization — not expected to recur on new hardware; verify VT-x/AMD-V + IOMMU in BIOS before assuming otherwise). -- Move off LVM-thin to ZFS for local storage. +- Move off LVM-thin to ZFS for local (boot + pre-cluster) storage. - Build node 1 so its disk and network layout doesn't need rework when nodes 2 and 3 are added later. - End state: 3-node cluster, quorum via corosync, HA-managed VMs backed by - ZFS replication and/or Ceph. + Ceph. No intermediate ZFS-replication HA step — Ceph goes live as soon as + node 2/3 join. ## Repo layout diff --git a/docs/00-overview.md b/docs/00-overview.md index 32d763a..aaf2a73 100644 --- a/docs/00-overview.md +++ b/docs/00-overview.md @@ -14,19 +14,16 @@ new hardware. Before reusing that workaround on the new box: ## End goal -3-node Proxmox VE cluster with HA-managed VMs. Two ways to get VM disks -available on more than one node (see `02-storage-zfs-ceph.md`): +3-node Proxmox VE cluster with HA-managed VMs backed by **Ceph** — true +distributed shared storage, sync replication, near-zero RPO on failover +(see `02-storage-zfs-ceph.md`). Ceph needs 3+ nodes and a fast dedicated +network, so it can't exist until nodes 2 and 3 are up. -1. **Ceph** — true distributed shared storage, sync replication, needs 3+ - nodes and a fast dedicated network. Zero/near-zero RPO on failover. -2. **ZFS + storage replication** — local ZFS pool per node, Proxmox - replicates VM disks between nodes on a schedule (as often as every - minute). Lighter weight, async — failover loses whatever changed since - last replication. - -Starting point: ZFS + replication (lighter, works from node 1 onward). -Ceph is the upgrade path once 3 nodes exist and/or zero-RPO failover -matters enough to justify the overhead. +Deliberately no intermediate "ZFS + storage replication" HA step. Node 1 +runs local ZFS for boot + VM storage with no cluster-wide HA until Ceph +goes live — as soon as nodes 2/3 join, VMs move onto Ceph rather than +adopting ZFS replication as a stopgap. Simpler end state, one storage +model to operate instead of two. ## Cluster fundamentals (apply from node 1 onward) @@ -44,14 +41,16 @@ matters enough to justify the overhead. ## Rollout sequence 1. Build node 1 per `01-hardware-node1.md` — fresh PVE install on ZFS boot - mirror, local ZFS "tier 2" pool for VM disks, Ceph-earmarked disks left - idle or as a temporary ZFS pool. + mirror, Ceph-earmarked disks left idle (or as a temporary local ZFS + pool, to be wiped later — see `02-storage-zfs-ceph.md`). 2. Migrate VMs off old hardware via `vzdump` → copy backups → `qmrestore` - onto the new ZFS storage (converts disks to ZVOLs). + onto the new ZFS storage (converts disks to ZVOLs). No HA yet — single + node. 3. Apply hardening (`04-security-hardening.md`) and networking (`03-networking.md`) before exposing the node beyond the LAN. 4. Add nodes 2 and 3 identically (same disk/network layout). -5. Join cluster, stand up dedicated corosync network, configure - replication and HA groups. -6. Optionally migrate the Ceph-earmarked disks from temporary ZFS to real - Ceph OSDs once 3 nodes are up. +5. Join cluster, stand up dedicated corosync network. +6. Wipe the Ceph-earmarked disks (if used as temporary ZFS) and + initialize Ceph across all 3 nodes. +7. Migrate VMs from local ZFS onto Ceph-backed storage, then configure HA + groups. diff --git a/docs/01-hardware-node1.md b/docs/01-hardware-node1.md index c2f2d16..63ece02 100644 --- a/docs/01-hardware-node1.md +++ b/docs/01-hardware-node1.md @@ -3,28 +3,24 @@ Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect disks or network when the cluster grows. -## Disks — three separate roles, physically separate devices +## Disks — two roles, physically separate devices 1. **Boot/OS pool (`rpool`)** — 2x small SSDs (240-480GB plenty), ZFS - mirror. Proxmox itself only. Never share with Ceph OSDs or bulk ZFS - data pools. + mirror. Proxmox itself only. Never share with Ceph OSDs. 2. **Future Ceph OSD disks** — must end up as raw, unformatted devices — no ZFS/RAID/LVM underneath (Ceph does its own replication; anything underneath just doubles copy-on-write/checksumming and hurts performance). Use enterprise SATA/NVMe SSDs with power-loss protection (PLP) — matters far more for Ceph write latency than for general ZFS use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks - either sit idle or run as a temporary local ZFS pool to be wiped and - handed to Ceph once nodes 2/3 exist. -3. **Local ZFS "replicated tier" disks** — separate set of disks - (mirror or small raidz) for VMs kept on local storage + PVE - replication rather than Ceph (latency-sensitive or non-critical - workloads). This pool is permanent, not a placeholder. + either sit idle or run as a temporary local ZFS pool (all VMs live + here until nodes 2/3 exist), to be wiped and handed to Ceph once the + cluster can actually run it. See `02-storage-zfs-ceph.md`. -If budget only allows one extra disk set right now: prioritize the -future-Ceph disks, run everything on ZFS locally until nodes 2/3 arrive, -then split workloads out. Avoid consumer QLC SSDs for either role — Ceph -punishes it on latency, ZFS on sync writes/scrub. +No permanent local-ZFS "replicated tier" — once Ceph is live, it's the +only HA storage; local ZFS is boot pool + this temporary pre-Ceph staging +role, not an ongoing parallel tier. Avoid consumer QLC SSDs for either +role — Ceph punishes it on latency, ZFS on sync writes/scrub. ## Networking — cable and provision for the final topology now @@ -64,6 +60,7 @@ compromise, not the end state. ## Node 1 install sequence 1. Install Proxmox VE fresh onto the ZFS boot mirror. -2. Create the local ZFS "tier 2" pool for VM disks. -3. Leave Ceph-earmarked disks idle, or provision as a temporary ZFS pool - (to be wiped once Ceph goes live at 3 nodes). +2. Provision the Ceph-earmarked disks as a temporary local ZFS pool and + run all VMs from it (or leave idle if VMs aren't moving over yet). +3. Once nodes 2/3 join and Ceph goes live: wipe this pool, hand the disks + to Ceph, migrate VMs onto Ceph-backed storage. diff --git a/docs/02-storage-zfs-ceph.md b/docs/02-storage-zfs-ceph.md index cac009f..2cb5d6d 100644 --- a/docs/02-storage-zfs-ceph.md +++ b/docs/02-storage-zfs-ceph.md @@ -1,24 +1,22 @@ -# Storage: LVM-thin → ZFS, and the path to Ceph +# Storage: LVM-thin → ZFS → Ceph ## Why move off LVM-thin Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs a VM's disk reachable from more than one node so it can restart elsewhere -on host failure. LVM-thin has no answer for that. ZFS does, via -replication; Ceph does natively. +on host failure. LVM-thin has no answer for that. Ceph does, natively. -## Two paths to HA-capable storage +## Storage model: single node → 3-node cluster -| | ZFS + replication | Ceph | -|---|---|---| -| Nodes required | 1+ (replication needs 2+ targets) | 3+ | -| Consistency | Async — snapshot-based, as often as every minute | Sync — real shared storage | -| Data loss on failover | Whatever changed since last replication cycle | ~None | -| Network needs | Normal cluster link | Fast dedicated network (see `03-networking.md`) | -| Overhead | Low | Higher RAM/CPU/disk | +No intermediate "ZFS + replication" HA step. The plan is deliberately a +single storage model at the end (Ceph), not two to operate long-term: -Starting point: **ZFS + replication**. Revisit Ceph once 3 nodes exist or -zero-RPO failover is worth the overhead. +- **Single node (node 1 only)**: local ZFS pool, no cluster-wide HA. This + is a temporary state, not a design to build tooling around. +- **3 nodes with Ceph live**: VMs run on Ceph-backed storage — true + distributed, synchronous storage across all nodes, near-zero RPO on + failover. Needs 3+ nodes and a fast dedicated network (see + `03-networking.md`), which is exactly why it can't exist before then. ## Migration plan (old hardware → new hardware) @@ -34,13 +32,30 @@ Alternative if both hosts can see each other on the network: temporarily cluster them and use the GUI "Migrate" with a storage move (offline only — live migration doesn't cross storage types). -## Later: handing Ceph-earmarked disks over +## Standing up Ceph once nodes 2 and 3 exist -Once nodes 2 and 3 are up and the Ceph-earmarked disks (see -`01-hardware-node1.md`) can be pooled 3-node minimum: +1. Wipe the temporary local ZFS pool on the Ceph-earmarked disks (see + `01-hardware-node1.md`) on all 3 nodes — they need to end up raw, + unformatted. +2. Install the Ceph packages on all 3 nodes (`pveceph install`) and + initialize the cluster (`pveceph init`), using the dedicated Ceph + network from `03-networking.md`. +3. Create Ceph monitors and managers (3 mons for quorum, matching node + count). +4. Create OSDs directly on the raw disks on each node — no ZFS/RAID + underneath. +5. Create a Ceph pool sized for your VM storage needs (replica count, + typically 3 for full redundancy across 3 nodes). +6. Add the pool as PVE storage (RBD), then migrate VMs from local ZFS + onto it — offline migration if crossing storage types, or storage + migration via the GUI. +7. Configure HA groups once VMs are on Ceph-backed storage. -1. Wipe any temporary ZFS pool on those disks. -2. Initialize Ceph across the 3 nodes. -3. Create OSDs directly on the raw disks (no ZFS/RAID underneath). -4. Migrate VMs that need zero-RPO failover from the ZFS-replicated tier - onto Ceph-backed storage. +## Notes + +- Ceph RAM/CPU overhead is real — budget per `01-hardware-node1.md` + (roughly 3-5GB RAM and a core per OSD, on top of VM workloads). +- Enterprise SSDs with power-loss protection (PLP) matter far more here + than for plain ZFS — Ceph write latency is sensitive to it. +- Once Ceph is live, local ZFS remains only for each node's boot pool — + it's not a fallback tier for VM storage going forward.