Drop ZFS-replication as an interim HA step; jump straight to Ceph

Simplifies the storage model to one long-term tier (Ceph) instead of
two. Node 1 runs local ZFS only until nodes 2/3 join, at which point
Ceph goes live and VMs migrate onto it directly.

README.md

@@ -9,11 +9,12 @@ hardware, with the end goal of a 3-node HA cluster.
   forced disabling hardware-accelerated virtualization — not expected to
   recur on new hardware; verify VT-x/AMD-V + IOMMU in BIOS before assuming
   otherwise).
-- Move off LVM-thin to ZFS for local storage.
+- Move off LVM-thin to ZFS for local (boot + pre-cluster) storage.
 - Build node 1 so its disk and network layout doesn't need rework when nodes
   2 and 3 are added later.
 - End state: 3-node cluster, quorum via corosync, HA-managed VMs backed by
-  ZFS replication and/or Ceph.
+  Ceph. No intermediate ZFS-replication HA step — Ceph goes live as soon as
+  node 2/3 join.
 
 ## Repo layout
 

docs/00-overview.md

@@ -14,19 +14,16 @@ new hardware. Before reusing that workaround on the new box:
 
 ## End goal
 
-3-node Proxmox VE cluster with HA-managed VMs. Two ways to get VM disks
+3-node Proxmox VE cluster with HA-managed VMs backed by **Ceph** — true
-available on more than one node (see `02-storage-zfs-ceph.md`):
+distributed shared storage, sync replication, near-zero RPO on failover
+(see `02-storage-zfs-ceph.md`). Ceph needs 3+ nodes and a fast dedicated
+network, so it can't exist until nodes 2 and 3 are up.
 
-1. **Ceph** — true distributed shared storage, sync replication, needs 3+
+Deliberately no intermediate "ZFS + storage replication" HA step. Node 1
-   nodes and a fast dedicated network. Zero/near-zero RPO on failover.
+runs local ZFS for boot + VM storage with no cluster-wide HA until Ceph
-2. **ZFS + storage replication** — local ZFS pool per node, Proxmox
+goes live — as soon as nodes 2/3 join, VMs move onto Ceph rather than
-   replicates VM disks between nodes on a schedule (as often as every
+adopting ZFS replication as a stopgap. Simpler end state, one storage
-   minute). Lighter weight, async — failover loses whatever changed since
+model to operate instead of two.
-   last replication.
-
-Starting point: ZFS + replication (lighter, works from node 1 onward).
-Ceph is the upgrade path once 3 nodes exist and/or zero-RPO failover
-matters enough to justify the overhead.
 
 ## Cluster fundamentals (apply from node 1 onward)
 
@@ -44,14 +41,16 @@ matters enough to justify the overhead.
 ## Rollout sequence
 
 1. Build node 1 per `01-hardware-node1.md` — fresh PVE install on ZFS boot
-   mirror, local ZFS "tier 2" pool for VM disks, Ceph-earmarked disks left
+   mirror, Ceph-earmarked disks left idle (or as a temporary local ZFS
-   idle or as a temporary ZFS pool.
+   pool, to be wiped later — see `02-storage-zfs-ceph.md`).
 2. Migrate VMs off old hardware via `vzdump` → copy backups → `qmrestore`
-   onto the new ZFS storage (converts disks to ZVOLs).
+   onto the new ZFS storage (converts disks to ZVOLs). No HA yet — single
+   node.
 3. Apply hardening (`04-security-hardening.md`) and networking
    (`03-networking.md`) before exposing the node beyond the LAN.
 4. Add nodes 2 and 3 identically (same disk/network layout).
-5. Join cluster, stand up dedicated corosync network, configure
+5. Join cluster, stand up dedicated corosync network.
-   replication and HA groups.
+6. Wipe the Ceph-earmarked disks (if used as temporary ZFS) and
-6. Optionally migrate the Ceph-earmarked disks from temporary ZFS to real
+   initialize Ceph across all 3 nodes.
-   Ceph OSDs once 3 nodes are up.
+7. Migrate VMs from local ZFS onto Ceph-backed storage, then configure HA
+   groups.

docs/01-hardware-node1.md

@@ -3,28 +3,24 @@
 Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect
 disks or network when the cluster grows.
 
-## Disks — three separate roles, physically separate devices
+## Disks — two roles, physically separate devices
 
 1. **Boot/OS pool (`rpool`)** — 2x small SSDs (240-480GB plenty), ZFS
-   mirror. Proxmox itself only. Never share with Ceph OSDs or bulk ZFS
+   mirror. Proxmox itself only. Never share with Ceph OSDs.
-   data pools.
 2. **Future Ceph OSD disks** — must end up as raw, unformatted devices —
    no ZFS/RAID/LVM underneath (Ceph does its own replication; anything
    underneath just doubles copy-on-write/checksumming and hurts
    performance). Use enterprise SATA/NVMe SSDs with power-loss protection
    (PLP) — matters far more for Ceph write latency than for general ZFS
    use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks
-   either sit idle or run as a temporary local ZFS pool to be wiped and
+   either sit idle or run as a temporary local ZFS pool (all VMs live
-   handed to Ceph once nodes 2/3 exist.
+   here until nodes 2/3 exist), to be wiped and handed to Ceph once the
-3. **Local ZFS "replicated tier" disks** — separate set of disks
+   cluster can actually run it. See `02-storage-zfs-ceph.md`.
-   (mirror or small raidz) for VMs kept on local storage + PVE
-   replication rather than Ceph (latency-sensitive or non-critical
-   workloads). This pool is permanent, not a placeholder.
 
-If budget only allows one extra disk set right now: prioritize the
+No permanent local-ZFS "replicated tier" — once Ceph is live, it's the
-future-Ceph disks, run everything on ZFS locally until nodes 2/3 arrive,
+only HA storage; local ZFS is boot pool + this temporary pre-Ceph staging
-then split workloads out. Avoid consumer QLC SSDs for either role — Ceph
+role, not an ongoing parallel tier. Avoid consumer QLC SSDs for either
-punishes it on latency, ZFS on sync writes/scrub.
+role — Ceph punishes it on latency, ZFS on sync writes/scrub.
 
 ## Networking — cable and provision for the final topology now
 
@@ -64,6 +60,7 @@ compromise, not the end state.
 ## Node 1 install sequence
 
 1. Install Proxmox VE fresh onto the ZFS boot mirror.
-2. Create the local ZFS "tier 2" pool for VM disks.
+2. Provision the Ceph-earmarked disks as a temporary local ZFS pool and
-3. Leave Ceph-earmarked disks idle, or provision as a temporary ZFS pool
+   run all VMs from it (or leave idle if VMs aren't moving over yet).
-   (to be wiped once Ceph goes live at 3 nodes).
+3. Once nodes 2/3 join and Ceph goes live: wipe this pool, hand the disks
+   to Ceph, migrate VMs onto Ceph-backed storage.

docs/02-storage-zfs-ceph.md

@@ -1,24 +1,22 @@
-# Storage: LVM-thin → ZFS, and the path to Ceph
+# Storage: LVM-thin → ZFS → Ceph
 
 ## Why move off LVM-thin
 
 Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs
 a VM's disk reachable from more than one node so it can restart elsewhere
-on host failure. LVM-thin has no answer for that. ZFS does, via
+on host failure. LVM-thin has no answer for that. Ceph does, natively.
-replication; Ceph does natively.
 
-## Two paths to HA-capable storage
+## Storage model: single node → 3-node cluster
 
-| | ZFS + replication | Ceph |
+No intermediate "ZFS + replication" HA step. The plan is deliberately a
-|---|---|---|
+single storage model at the end (Ceph), not two to operate long-term:
-| Nodes required | 1+ (replication needs 2+ targets) | 3+ |
-| Consistency | Async — snapshot-based, as often as every minute | Sync — real shared storage |
-| Data loss on failover | Whatever changed since last replication cycle | ~None |
-| Network needs | Normal cluster link | Fast dedicated network (see `03-networking.md`) |
-| Overhead | Low | Higher RAM/CPU/disk |
 
-Starting point: **ZFS + replication**. Revisit Ceph once 3 nodes exist or
+- **Single node (node 1 only)**: local ZFS pool, no cluster-wide HA. This
-zero-RPO failover is worth the overhead.
+  is a temporary state, not a design to build tooling around.
+- **3 nodes with Ceph live**: VMs run on Ceph-backed storage — true
+  distributed, synchronous storage across all nodes, near-zero RPO on
+  failover. Needs 3+ nodes and a fast dedicated network (see
+  `03-networking.md`), which is exactly why it can't exist before then.
 
 ## Migration plan (old hardware → new hardware)
 
@@ -34,13 +32,30 @@ Alternative if both hosts can see each other on the network: temporarily
 cluster them and use the GUI "Migrate" with a storage move (offline only
 — live migration doesn't cross storage types).
 
-## Later: handing Ceph-earmarked disks over
+## Standing up Ceph once nodes 2 and 3 exist
 
-Once nodes 2 and 3 are up and the Ceph-earmarked disks (see
+1. Wipe the temporary local ZFS pool on the Ceph-earmarked disks (see
-`01-hardware-node1.md`) can be pooled 3-node minimum:
+   `01-hardware-node1.md`) on all 3 nodes — they need to end up raw,
+   unformatted.
+2. Install the Ceph packages on all 3 nodes (`pveceph install`) and
+   initialize the cluster (`pveceph init`), using the dedicated Ceph
+   network from `03-networking.md`.
+3. Create Ceph monitors and managers (3 mons for quorum, matching node
+   count).
+4. Create OSDs directly on the raw disks on each node — no ZFS/RAID
+   underneath.
+5. Create a Ceph pool sized for your VM storage needs (replica count,
+   typically 3 for full redundancy across 3 nodes).
+6. Add the pool as PVE storage (RBD), then migrate VMs from local ZFS
+   onto it — offline migration if crossing storage types, or storage
+   migration via the GUI.
+7. Configure HA groups once VMs are on Ceph-backed storage.
 
-1. Wipe any temporary ZFS pool on those disks.
+## Notes
-2. Initialize Ceph across the 3 nodes.
+
-3. Create OSDs directly on the raw disks (no ZFS/RAID underneath).
+- Ceph RAM/CPU overhead is real — budget per `01-hardware-node1.md`
-4. Migrate VMs that need zero-RPO failover from the ZFS-replicated tier
+  (roughly 3-5GB RAM and a core per OSD, on top of VM workloads).
-   onto Ceph-backed storage.
+- Enterprise SSDs with power-loss protection (PLP) matter far more here
+  than for plain ZFS — Ceph write latency is sensitive to it.
+- Once Ceph is live, local ZFS remains only for each node's boot pool —
+  it's not a fallback tier for VM storage going forward.