Drop ZFS-replication as an interim HA step; jump straight to Ceph

Simplifies the storage model to one long-term tier (Ceph) instead of
two. Node 1 runs local ZFS only until nodes 2/3 join, at which point
Ceph goes live and VMs migrate onto it directly.
This commit is contained in:
2026-07-20 11:54:17 +10:00
parent a854412117
commit 24a0047fa8
4 changed files with 70 additions and 58 deletions
+13 -16
View File
@@ -3,28 +3,24 @@
Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect
disks or network when the cluster grows.
## Disks — three separate roles, physically separate devices
## Disks — two roles, physically separate devices
1. **Boot/OS pool (`rpool`)** — 2x small SSDs (240-480GB plenty), ZFS
mirror. Proxmox itself only. Never share with Ceph OSDs or bulk ZFS
data pools.
mirror. Proxmox itself only. Never share with Ceph OSDs.
2. **Future Ceph OSD disks** — must end up as raw, unformatted devices —
no ZFS/RAID/LVM underneath (Ceph does its own replication; anything
underneath just doubles copy-on-write/checksumming and hurts
performance). Use enterprise SATA/NVMe SSDs with power-loss protection
(PLP) — matters far more for Ceph write latency than for general ZFS
use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks
either sit idle or run as a temporary local ZFS pool to be wiped and
handed to Ceph once nodes 2/3 exist.
3. **Local ZFS "replicated tier" disks** — separate set of disks
(mirror or small raidz) for VMs kept on local storage + PVE
replication rather than Ceph (latency-sensitive or non-critical
workloads). This pool is permanent, not a placeholder.
either sit idle or run as a temporary local ZFS pool (all VMs live
here until nodes 2/3 exist), to be wiped and handed to Ceph once the
cluster can actually run it. See `02-storage-zfs-ceph.md`.
If budget only allows one extra disk set right now: prioritize the
future-Ceph disks, run everything on ZFS locally until nodes 2/3 arrive,
then split workloads out. Avoid consumer QLC SSDs for either role — Ceph
punishes it on latency, ZFS on sync writes/scrub.
No permanent local-ZFS "replicated tier" — once Ceph is live, it's the
only HA storage; local ZFS is boot pool + this temporary pre-Ceph staging
role, not an ongoing parallel tier. Avoid consumer QLC SSDs for either
role — Ceph punishes it on latency, ZFS on sync writes/scrub.
## Networking — cable and provision for the final topology now
@@ -64,6 +60,7 @@ compromise, not the end state.
## Node 1 install sequence
1. Install Proxmox VE fresh onto the ZFS boot mirror.
2. Create the local ZFS "tier 2" pool for VM disks.
3. Leave Ceph-earmarked disks idle, or provision as a temporary ZFS pool
(to be wiped once Ceph goes live at 3 nodes).
2. Provision the Ceph-earmarked disks as a temporary local ZFS pool and
run all VMs from it (or leave idle if VMs aren't moving over yet).
3. Once nodes 2/3 join and Ceph goes live: wipe this pool, hand the disks
to Ceph, migrate VMs onto Ceph-backed storage.