Archived
93 lines
4.1 KiB
Markdown
93 lines
4.1 KiB
Markdown
# Node 1 Hardware Layout
|
|
|
|
## pve1 as built (current reality)
|
|
|
|
`pve1` is an ASUS PN53 mini PC (Ryzen 7 7735HS, 32GB RAM), not the
|
|
dedicated-server hardware the target design below assumes:
|
|
|
|
- **Disks**: 2x 2TB NVMe (Crucial CT2000E100SSD8), both in a single ZFS
|
|
mirror (`rpool`) that serves as both the boot pool and VM storage
|
|
(`local-zfs` = `rpool/data`). No spare disks to earmark for Ceph — the
|
|
chassis only has 2 NVMe slots.
|
|
- **Network**: one physical NIC (Realtek RTL8125, 2.5GbE), bridged as
|
|
`vmbr0`. No second NIC for a dedicated corosync/Ceph link. (An unused
|
|
`nic1` stanza in `/etc/network/interfaces` is a leftover from the
|
|
installer template — there is no second NIC on this hardware.)
|
|
|
|
This is sufficient and correct for **Stage 1** (see `00-overview.md`) —
|
|
base config and hardening don't need split disks or multiple NICs. It is
|
|
*not* sufficient for **Stage 2** (Ceph/HA) as designed below without
|
|
either different hardware or a materially different plan (e.g.
|
|
USB/Thunderbolt-attached OSD storage, which trades away the
|
|
enterprise-SSD/PLP guidance below — not recommended, revisit when
|
|
actually provisioning nodes 2/3). Treat everything from here down as the
|
|
Stage 2 target design for purpose-built hardware, not a description of
|
|
`pve1`.
|
|
|
|
## Target design (Stage 2, future dedicated hardware)
|
|
|
|
Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect
|
|
disks or network when the cluster grows.
|
|
|
|
## Disks — two roles, physically separate devices
|
|
|
|
1. **Boot/OS pool (`rpool`)** — 2x small SSDs (240-480GB plenty), ZFS
|
|
mirror. Proxmox itself only. Never share with Ceph OSDs.
|
|
2. **Future Ceph OSD disks** — must end up as raw, unformatted devices —
|
|
no ZFS/RAID/LVM underneath (Ceph does its own replication; anything
|
|
underneath just doubles copy-on-write/checksumming and hurts
|
|
performance). Use enterprise SATA/NVMe SSDs with power-loss protection
|
|
(PLP) — matters far more for Ceph write latency than for general ZFS
|
|
use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks
|
|
either sit idle or run as a temporary local ZFS pool (all VMs live
|
|
here until nodes 2/3 exist), to be wiped and handed to Ceph once the
|
|
cluster can actually run it. See `02-storage-zfs-ceph.md`.
|
|
|
|
No permanent local-ZFS "replicated tier" — once Ceph is live, it's the
|
|
only HA storage; local ZFS is boot pool + this temporary pre-Ceph staging
|
|
role, not an ongoing parallel tier. Avoid consumer QLC SSDs for either
|
|
role — Ceph punishes it on latency, ZFS on sync writes/scrub.
|
|
|
|
## Networking — cable and provision for the final topology now
|
|
|
|
Logically separate networks (ideally separate NICs/VLANs):
|
|
|
|
- **Management** — web UI / SSH
|
|
- **Corosync** — cluster quorum traffic, low-latency, unshared
|
|
- **Ceph public** — VM-to-OSD traffic
|
|
- **Ceph cluster/backend** — OSD-to-OSD replication (heaviest load)
|
|
|
|
Practical layout: 2x 10/25GbE bonded or split — one pair for Ceph, one
|
|
for mgmt + corosync + VM traffic, with corosync on its own VLAN even if
|
|
sharing a physical NIC. Get switch/cabling right on node 1 so nodes 2/3
|
|
are identical drops.
|
|
|
|
## CPU / RAM sizing
|
|
|
|
Size for the end state, not day one — RAM is the hardest thing to
|
|
retrofit. Budget covers:
|
|
|
|
- OS + ZFS ARC (ZFS wants RAM, not just disk)
|
|
- Ceph OSD daemons — realistically 3-5GB per OSD once running
|
|
- Actual VM workloads
|
|
|
|
Roughly a core per OSD on top of what VMs need. If OSDs won't be active
|
|
for a while, that's headway, but buy for 3 nodes' worth of eventual OSD
|
|
load.
|
|
|
|
## Backup target (PBS)
|
|
|
|
Keep it off the Ceph/compute nodes if possible — its failure domain
|
|
should be independent of the cluster. Modest separate machine or NAS:
|
|
ZFS mirror or raidz2, ECC RAM if possible, capacity for retention policy.
|
|
If it has to run as a VM inside the cluster short-term, that's a known
|
|
compromise, not the end state.
|
|
|
|
## Node 1 install sequence
|
|
|
|
1. Install Proxmox VE fresh onto the ZFS boot mirror.
|
|
2. Provision the Ceph-earmarked disks as a temporary local ZFS pool and
|
|
run all VMs from it (or leave idle if VMs aren't moving over yet).
|
|
3. Once nodes 2/3 join and Ceph goes live: wipe this pool, hand the disks
|
|
to Ceph, migrate VMs onto Ceph-backed storage.
|