beatzaplenty
2cca6a7dc0
Splits the repo into Stage 1 (base host config/hardening, applies to any node) and Stage 2 (future HA/Ceph cluster, deferred - pve1's mini-PC hardware can't support the assumed split-disk/multi-NIC layout). Adds the Stage 1 toolset: firewall deploy, named admin user creation, unattended security upgrades, subscription-nag removal (with an apt hook so the patch survives package updates), and a read-only audit script. Fixes switch-to-no-subscription-repo.sh, which only handled the legacy .list format and silently no-op'd against PVE 9's deb822 .sources files; it now removes enterprise sources outright rather than commenting them out. Shared logic (root check, idempotent file writes, backups) factored into scripts/lib/common.sh. Ran the full sequence against pve1 via scripts/bootstrap.sh + create-admin-user.sh; scripts/audit.sh confirms all checks pass.
3.0 KiB
Storage: LVM-thin → ZFS → Ceph
Stage 2 (future). Describes the target storage model once dedicated
cluster hardware exists. pve1's current single ZFS mirror (boot + VM
storage combined, see 01-hardware-node1.md) is the Stage 1 end state for
now, not an intermediate step being actively migrated from.
Why move off LVM-thin
Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs a VM's disk reachable from more than one node so it can restart elsewhere on host failure. LVM-thin has no answer for that. Ceph does, natively.
Storage model: single node → 3-node cluster
No intermediate "ZFS + replication" HA step. The plan is deliberately a single storage model at the end (Ceph), not two to operate long-term:
- Single node (node 1 only): local ZFS pool, no cluster-wide HA. This is a temporary state, not a design to build tooling around.
- 3 nodes with Ceph live: VMs run on Ceph-backed storage — true
distributed, synchronous storage across all nodes, near-zero RPO on
failover. Needs 3+ nodes and a fast dedicated network (see
03-networking.md), which is exactly why it can't exist before then.
Migration plan (old hardware → new hardware)
Don't convert the old LVM-thin box in place. Rebuild fresh on new hardware with ZFS from the installer (mirror if 2+ disks), then move VMs:
- On the old host:
vzdumpeach VM to a backup file (external drive, NFS share, or PBS if available). - Copy backups to the new host.
qmrestoreonto the new ZFS storage — disks land as ZVOLs.
Alternative if both hosts can see each other on the network: temporarily cluster them and use the GUI "Migrate" with a storage move (offline only — live migration doesn't cross storage types).
Standing up Ceph once nodes 2 and 3 exist
- Wipe the temporary local ZFS pool on the Ceph-earmarked disks (see
01-hardware-node1.md) on all 3 nodes — they need to end up raw, unformatted. - Install the Ceph packages on all 3 nodes (
pveceph install) and initialize the cluster (pveceph init), using the dedicated Ceph network from03-networking.md. - Create Ceph monitors and managers (3 mons for quorum, matching node count).
- Create OSDs directly on the raw disks on each node — no ZFS/RAID underneath.
- Create a Ceph pool sized for your VM storage needs (replica count, typically 3 for full redundancy across 3 nodes).
- Add the pool as PVE storage (RBD), then migrate VMs from local ZFS onto it — offline migration if crossing storage types, or storage migration via the GUI.
- Configure HA groups once VMs are on Ceph-backed storage.
Notes
- Ceph RAM/CPU overhead is real — budget per
01-hardware-node1.md(roughly 3-5GB RAM and a core per OSD, on top of VM workloads). - Enterprise SSDs with power-loss protection (PLP) matter far more here than for plain ZFS — Ceph write latency is sensitive to it.
- Once Ceph is live, local ZFS remains only for each node's boot pool — it's not a fallback tier for VM storage going forward.