This repository has been archived on 2026-08-17. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
debian-configuration/proxmox/docs/02-storage-zfs-ceph.md
beatzaplentyandClaude Sonnet 4.6 2909db5d04 restructure: move proxmox/ into subfolder, add pihole/ section
All existing content moved from repo root into proxmox/ to make room
for other Debian machine configs. Adds pihole/ with:

- config/pihole.toml — snapshot of current Pi-hole v6 config
- config/dnsmasq.d/99-ipxe-chainload.conf — custom PXE DHCP rules
  (EFI/BIOS iPXE chainload, fixed tag-specificity bug for UEFI boot)
- pull-config.sh <source-host> <dest-dir> — pull live config to disk
- apply-config.sh <source-dir> <dest-host> — push config to a Pi-hole

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XRzqNDrbnYR22ZgZj1Bg3s
2026-07-23 12:16:35 +10:00

3.0 KiB

Storage: LVM-thin → ZFS → Ceph

Stage 2 (future). Describes the target storage model once dedicated cluster hardware exists. pve1's current single ZFS mirror (boot + VM storage combined, see 01-hardware-node1.md) is the Stage 1 end state for now, not an intermediate step being actively migrated from.

Why move off LVM-thin

Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs a VM's disk reachable from more than one node so it can restart elsewhere on host failure. LVM-thin has no answer for that. Ceph does, natively.

Storage model: single node → 3-node cluster

No intermediate "ZFS + replication" HA step. The plan is deliberately a single storage model at the end (Ceph), not two to operate long-term:

  • Single node (node 1 only): local ZFS pool, no cluster-wide HA. This is a temporary state, not a design to build tooling around.
  • 3 nodes with Ceph live: VMs run on Ceph-backed storage — true distributed, synchronous storage across all nodes, near-zero RPO on failover. Needs 3+ nodes and a fast dedicated network (see 03-networking.md), which is exactly why it can't exist before then.

Migration plan (old hardware → new hardware)

Don't convert the old LVM-thin box in place. Rebuild fresh on new hardware with ZFS from the installer (mirror if 2+ disks), then move VMs:

  1. On the old host: vzdump each VM to a backup file (external drive, NFS share, or PBS if available).
  2. Copy backups to the new host.
  3. qmrestore onto the new ZFS storage — disks land as ZVOLs.

Alternative if both hosts can see each other on the network: temporarily cluster them and use the GUI "Migrate" with a storage move (offline only — live migration doesn't cross storage types).

Standing up Ceph once nodes 2 and 3 exist

  1. Wipe the temporary local ZFS pool on the Ceph-earmarked disks (see 01-hardware-node1.md) on all 3 nodes — they need to end up raw, unformatted.
  2. Install the Ceph packages on all 3 nodes (pveceph install) and initialize the cluster (pveceph init), using the dedicated Ceph network from 03-networking.md.
  3. Create Ceph monitors and managers (3 mons for quorum, matching node count).
  4. Create OSDs directly on the raw disks on each node — no ZFS/RAID underneath.
  5. Create a Ceph pool sized for your VM storage needs (replica count, typically 3 for full redundancy across 3 nodes).
  6. Add the pool as PVE storage (RBD), then migrate VMs from local ZFS onto it — offline migration if crossing storage types, or storage migration via the GUI.
  7. Configure HA groups once VMs are on Ceph-backed storage.

Notes

  • Ceph RAM/CPU overhead is real — budget per 01-hardware-node1.md (roughly 3-5GB RAM and a core per OSD, on top of VM workloads).
  • Enterprise SSDs with power-loss protection (PLP) matter far more here than for plain ZFS — Ceph write latency is sensitive to it.
  • Once Ceph is live, local ZFS remains only for each node's boot pool — it's not a fallback tier for VM storage going forward.