# Storage: LVM-thin → ZFS → Ceph

## Why move off LVM-thin

Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs
a VM's disk reachable from more than one node so it can restart elsewhere
on host failure. LVM-thin has no answer for that. Ceph does, natively.

## Storage model: single node → 3-node cluster

No intermediate "ZFS + replication" HA step. The plan is deliberately a
single storage model at the end (Ceph), not two to operate long-term:

- **Single node (node 1 only)**: local ZFS pool, no cluster-wide HA. This
  is a temporary state, not a design to build tooling around.
- **3 nodes with Ceph live**: VMs run on Ceph-backed storage — true
  distributed, synchronous storage across all nodes, near-zero RPO on
  failover. Needs 3+ nodes and a fast dedicated network (see
  `03-networking.md`), which is exactly why it can't exist before then.

## Migration plan (old hardware → new hardware)

Don't convert the old LVM-thin box in place. Rebuild fresh on new
hardware with ZFS from the installer (mirror if 2+ disks), then move VMs:

1. On the old host: `vzdump` each VM to a backup file (external drive,
   NFS share, or PBS if available).
2. Copy backups to the new host.
3. `qmrestore` onto the new ZFS storage — disks land as ZVOLs.

Alternative if both hosts can see each other on the network: temporarily
cluster them and use the GUI "Migrate" with a storage move (offline only
— live migration doesn't cross storage types).

## Standing up Ceph once nodes 2 and 3 exist

1. Wipe the temporary local ZFS pool on the Ceph-earmarked disks (see
   `01-hardware-node1.md`) on all 3 nodes — they need to end up raw,
   unformatted.
2. Install the Ceph packages on all 3 nodes (`pveceph install`) and
   initialize the cluster (`pveceph init`), using the dedicated Ceph
   network from `03-networking.md`.
3. Create Ceph monitors and managers (3 mons for quorum, matching node
   count).
4. Create OSDs directly on the raw disks on each node — no ZFS/RAID
   underneath.
5. Create a Ceph pool sized for your VM storage needs (replica count,
   typically 3 for full redundancy across 3 nodes).
6. Add the pool as PVE storage (RBD), then migrate VMs from local ZFS
   onto it — offline migration if crossing storage types, or storage
   migration via the GUI.
7. Configure HA groups once VMs are on Ceph-backed storage.

## Notes

- Ceph RAM/CPU overhead is real — budget per `01-hardware-node1.md`
  (roughly 3-5GB RAM and a core per OSD, on top of VM workloads).
- Enterprise SSDs with power-loss protection (PLP) matter far more here
  than for plain ZFS — Ceph write latency is sensitive to it.
- Once Ceph is live, local ZFS remains only for each node's boot pool —
  it's not a fallback tier for VM storage going forward.