Archived

This repository has been archived on 2026-08-17. You can view files and clone it. You cannot open issues or pull requests or push a commit.

Files

T

beatzaplenty 24a0047fa8 Drop ZFS-replication as an interim HA step; jump straight to Ceph

Simplifies the storage model to one long-term tier (Ceph) instead of
two. Node 1 runs local ZFS only until nodes 2/3 join, at which point
Ceph goes live and VMs migrate onto it directly.

2026-07-20 11:54:17 +10:00

2.8 KiB

Raw Blame History

Node 1 Hardware Layout

Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect disks or network when the cluster grows.

Disks — two roles, physically separate devices

Boot/OS pool (rpool) — 2x small SSDs (240-480GB plenty), ZFS mirror. Proxmox itself only. Never share with Ceph OSDs.
Future Ceph OSD disks — must end up as raw, unformatted devices — no ZFS/RAID/LVM underneath (Ceph does its own replication; anything underneath just doubles copy-on-write/checksumming and hurts performance). Use enterprise SATA/NVMe SSDs with power-loss protection (PLP) — matters far more for Ceph write latency than for general ZFS use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks either sit idle or run as a temporary local ZFS pool (all VMs live here until nodes 2/3 exist), to be wiped and handed to Ceph once the cluster can actually run it. See 02-storage-zfs-ceph.md.

No permanent local-ZFS "replicated tier" — once Ceph is live, it's the only HA storage; local ZFS is boot pool + this temporary pre-Ceph staging role, not an ongoing parallel tier. Avoid consumer QLC SSDs for either role — Ceph punishes it on latency, ZFS on sync writes/scrub.

Networking — cable and provision for the final topology now

Logically separate networks (ideally separate NICs/VLANs):

Management — web UI / SSH
Corosync — cluster quorum traffic, low-latency, unshared
Ceph public — VM-to-OSD traffic
Ceph cluster/backend — OSD-to-OSD replication (heaviest load)

Practical layout: 2x 10/25GbE bonded or split — one pair for Ceph, one for mgmt + corosync + VM traffic, with corosync on its own VLAN even if sharing a physical NIC. Get switch/cabling right on node 1 so nodes 2/3 are identical drops.

CPU / RAM sizing

Size for the end state, not day one — RAM is the hardest thing to retrofit. Budget covers:

OS + ZFS ARC (ZFS wants RAM, not just disk)
Ceph OSD daemons — realistically 3-5GB per OSD once running
Actual VM workloads

Roughly a core per OSD on top of what VMs need. If OSDs won't be active for a while, that's headway, but buy for 3 nodes' worth of eventual OSD load.

Backup target (PBS)

Keep it off the Ceph/compute nodes if possible — its failure domain should be independent of the cluster. Modest separate machine or NAS: ZFS mirror or raidz2, ECC RAM if possible, capacity for retention policy. If it has to run as a VM inside the cluster short-term, that's a known compromise, not the end state.

Node 1 install sequence

Install Proxmox VE fresh onto the ZFS boot mirror.
Provision the Ceph-earmarked disks as a temporary local ZFS pool and run all VMs from it (or leave idle if VMs aren't moving over yet).
Once nodes 2/3 join and Ceph goes live: wipe this pool, hand the disks to Ceph, migrate VMs onto Ceph-backed storage.

2.8 KiB Raw Blame History