Archived
Drop ZFS-replication as an interim HA step; jump straight to Ceph
Simplifies the storage model to one long-term tier (Ceph) instead of two. Node 1 runs local ZFS only until nodes 2/3 join, at which point Ceph goes live and VMs migrate onto it directly.
This commit is contained in:
+13
-16
@@ -3,28 +3,24 @@
|
||||
Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect
|
||||
disks or network when the cluster grows.
|
||||
|
||||
## Disks — three separate roles, physically separate devices
|
||||
## Disks — two roles, physically separate devices
|
||||
|
||||
1. **Boot/OS pool (`rpool`)** — 2x small SSDs (240-480GB plenty), ZFS
|
||||
mirror. Proxmox itself only. Never share with Ceph OSDs or bulk ZFS
|
||||
data pools.
|
||||
mirror. Proxmox itself only. Never share with Ceph OSDs.
|
||||
2. **Future Ceph OSD disks** — must end up as raw, unformatted devices —
|
||||
no ZFS/RAID/LVM underneath (Ceph does its own replication; anything
|
||||
underneath just doubles copy-on-write/checksumming and hurts
|
||||
performance). Use enterprise SATA/NVMe SSDs with power-loss protection
|
||||
(PLP) — matters far more for Ceph write latency than for general ZFS
|
||||
use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks
|
||||
either sit idle or run as a temporary local ZFS pool to be wiped and
|
||||
handed to Ceph once nodes 2/3 exist.
|
||||
3. **Local ZFS "replicated tier" disks** — separate set of disks
|
||||
(mirror or small raidz) for VMs kept on local storage + PVE
|
||||
replication rather than Ceph (latency-sensitive or non-critical
|
||||
workloads). This pool is permanent, not a placeholder.
|
||||
either sit idle or run as a temporary local ZFS pool (all VMs live
|
||||
here until nodes 2/3 exist), to be wiped and handed to Ceph once the
|
||||
cluster can actually run it. See `02-storage-zfs-ceph.md`.
|
||||
|
||||
If budget only allows one extra disk set right now: prioritize the
|
||||
future-Ceph disks, run everything on ZFS locally until nodes 2/3 arrive,
|
||||
then split workloads out. Avoid consumer QLC SSDs for either role — Ceph
|
||||
punishes it on latency, ZFS on sync writes/scrub.
|
||||
No permanent local-ZFS "replicated tier" — once Ceph is live, it's the
|
||||
only HA storage; local ZFS is boot pool + this temporary pre-Ceph staging
|
||||
role, not an ongoing parallel tier. Avoid consumer QLC SSDs for either
|
||||
role — Ceph punishes it on latency, ZFS on sync writes/scrub.
|
||||
|
||||
## Networking — cable and provision for the final topology now
|
||||
|
||||
@@ -64,6 +60,7 @@ compromise, not the end state.
|
||||
## Node 1 install sequence
|
||||
|
||||
1. Install Proxmox VE fresh onto the ZFS boot mirror.
|
||||
2. Create the local ZFS "tier 2" pool for VM disks.
|
||||
3. Leave Ceph-earmarked disks idle, or provision as a temporary ZFS pool
|
||||
(to be wiped once Ceph goes live at 3 nodes).
|
||||
2. Provision the Ceph-earmarked disks as a temporary local ZFS pool and
|
||||
run all VMs from it (or leave idle if VMs aren't moving over yet).
|
||||
3. Once nodes 2/3 join and Ceph goes live: wipe this pool, hand the disks
|
||||
to Ceph, migrate VMs onto Ceph-backed storage.
|
||||
|
||||
Reference in New Issue
Block a user