Drop ZFS-replication as an interim HA step; jump straight to Ceph

Simplifies the storage model to one long-term tier (Ceph) instead of
two. Node 1 runs local ZFS only until nodes 2/3 join, at which point
Ceph goes live and VMs migrate onto it directly.

docs/02-storage-zfs-ceph.md

@@ -1,24 +1,22 @@
-# Storage: LVM-thin → ZFS, and the path to Ceph
+# Storage: LVM-thin → ZFS → Ceph
 
 ## Why move off LVM-thin
 
 Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs
 a VM's disk reachable from more than one node so it can restart elsewhere
-on host failure. LVM-thin has no answer for that. ZFS does, via
-replication; Ceph does natively.
+on host failure. LVM-thin has no answer for that. Ceph does, natively.
 
-## Two paths to HA-capable storage
+## Storage model: single node → 3-node cluster
 
-| | ZFS + replication | Ceph |
-|---|---|---|
-| Nodes required | 1+ (replication needs 2+ targets) | 3+ |
-| Consistency | Async — snapshot-based, as often as every minute | Sync — real shared storage |
-| Data loss on failover | Whatever changed since last replication cycle | ~None |
-| Network needs | Normal cluster link | Fast dedicated network (see `03-networking.md`) |
-| Overhead | Low | Higher RAM/CPU/disk |
+No intermediate "ZFS + replication" HA step. The plan is deliberately a
+single storage model at the end (Ceph), not two to operate long-term:
 
-Starting point: **ZFS + replication**. Revisit Ceph once 3 nodes exist or
-zero-RPO failover is worth the overhead.
+- **Single node (node 1 only)**: local ZFS pool, no cluster-wide HA. This
+  is a temporary state, not a design to build tooling around.
+- **3 nodes with Ceph live**: VMs run on Ceph-backed storage — true
+  distributed, synchronous storage across all nodes, near-zero RPO on
+  failover. Needs 3+ nodes and a fast dedicated network (see
+  `03-networking.md`), which is exactly why it can't exist before then.
 
 ## Migration plan (old hardware → new hardware)
 
@@ -34,13 +32,30 @@ Alternative if both hosts can see each other on the network: temporarily
 cluster them and use the GUI "Migrate" with a storage move (offline only
 — live migration doesn't cross storage types).
 
-## Later: handing Ceph-earmarked disks over
+## Standing up Ceph once nodes 2 and 3 exist
 
-Once nodes 2 and 3 are up and the Ceph-earmarked disks (see
-`01-hardware-node1.md`) can be pooled 3-node minimum:
+1. Wipe the temporary local ZFS pool on the Ceph-earmarked disks (see
+   `01-hardware-node1.md`) on all 3 nodes — they need to end up raw,
+   unformatted.
+2. Install the Ceph packages on all 3 nodes (`pveceph install`) and
+   initialize the cluster (`pveceph init`), using the dedicated Ceph
+   network from `03-networking.md`.
+3. Create Ceph monitors and managers (3 mons for quorum, matching node
+   count).
+4. Create OSDs directly on the raw disks on each node — no ZFS/RAID
+   underneath.
+5. Create a Ceph pool sized for your VM storage needs (replica count,
+   typically 3 for full redundancy across 3 nodes).
+6. Add the pool as PVE storage (RBD), then migrate VMs from local ZFS
+   onto it — offline migration if crossing storage types, or storage
+   migration via the GUI.
+7. Configure HA groups once VMs are on Ceph-backed storage.
 
-1. Wipe any temporary ZFS pool on those disks.
-2. Initialize Ceph across the 3 nodes.
-3. Create OSDs directly on the raw disks (no ZFS/RAID underneath).
-4. Migrate VMs that need zero-RPO failover from the ZFS-replicated tier
-   onto Ceph-backed storage.
+## Notes
+
+- Ceph RAM/CPU overhead is real — budget per `01-hardware-node1.md`
+  (roughly 3-5GB RAM and a core per OSD, on top of VM workloads).
+- Enterprise SSDs with power-loss protection (PLP) matter far more here
+  than for plain ZFS — Ceph write latency is sensitive to it.
+- Once Ceph is live, local ZFS remains only for each node's boot pool —
+  it's not a fallback tier for VM storage going forward.