Archived
beatzaplenty
ba73420350
pve-test was briefly clustered with pve1 then deliberately de-clustered so it could move to wifi-primary networking (4addr bridge mode, bonded with a wired LAN backup) - a change not achievable while clustered given corosync's latency requirements. Captures that as a reproducible script plus docs: cluster separation procedure, the wifi network design and the live-cutover pitfalls hit along the way, and node-role/history context. Also adds CLAUDE.md guardrails for pve1 (production) vs pve-test (sandbox) - this repo had none before, despite scripts here being able to make real changes to both. Separately: both nodes' mgmt firewalls were dropping ICMP by default (TCP 22/8006 only), which looked like an outage mid-troubleshooting even though SSH/web UI were fine. Added an explicit ping-allow rule to the firewall template, applied it live on both nodes, and added an audit.sh check so it stays enforced.
67 lines
2.5 KiB
Markdown
67 lines
2.5 KiB
Markdown
# Networking
|
|
|
|
## pve1 as built (Stage 1, current)
|
|
|
|
Single NIC (2.5GbE), single bridge `vmbr0` on the flat LAN
|
|
(`192.168.2.0/24`), no VLANs. There is no corosync or Ceph traffic to
|
|
separate yet — this node isn't clustered. Segmentation for Stage 1 is
|
|
done at the firewall, not the network: `scripts/deploy-firewall.sh`
|
|
restricts SSH (22) and the web UI (8006) to the management CIDR via the
|
|
PVE datacenter firewall (default-deny inbound otherwise). That's
|
|
sufficient until Stage 2 needs actual separate physical/VLAN paths for
|
|
corosync and Ceph traffic — see below.
|
|
|
|
## pve-test as built (sandbox, current)
|
|
|
|
Different node, different design, not a Stage 1/2 example to generalize
|
|
from: `pve-test` runs `vmbr0` bridged over a wifi NIC in 4addr client mode
|
|
(active-backup bonded with a wired NIC as an automatic fallback). Full
|
|
detail, including why this is normally impossible and how it was
|
|
validated before trusting it with the management IP, in
|
|
`06-pve-test-wifi-network.md`. This is intentionally a one-off for a
|
|
standalone sandbox box — never extend it to a node that's clustered or
|
|
Ceph-connected (see the Stage 2 note in `00-overview.md`).
|
|
|
|
## Target design (Stage 2, future cluster)
|
|
|
|
## Required separation
|
|
|
|
Keep these on logically separate networks/VLANs, ideally separate NICs:
|
|
|
|
- **Management** — web UI (8006), SSH
|
|
- **Corosync** — cluster quorum. Low, *consistent* latency (well under
|
|
5ms) matters more than bandwidth. Never share with VM/storage traffic.
|
|
- **Ceph public** — VM-to-OSD traffic (once Ceph is live)
|
|
- **Ceph cluster/backend** — OSD-to-OSD replication, heaviest load
|
|
|
|
## Practical layout
|
|
|
|
2x 10/25GbE bonded or split:
|
|
|
|
- Link pair A → Ceph (public + backend, or split further if 4 NICs
|
|
available)
|
|
- Link pair B → management + corosync + VM traffic, with corosync on its
|
|
own VLAN even when sharing a physical NIC with the rest
|
|
|
|
## Cluster join requirements
|
|
|
|
- All nodes reachable to each other on SSH (22) and the corosync network
|
|
- Same PVE version across nodes
|
|
- NTP-synced clocks
|
|
|
|
## Firewall
|
|
|
|
Proxmox's built-in firewall operates at datacenter and node level.
|
|
Default-deny, then whitelist:
|
|
|
|
- SSH from the management network/VLAN only
|
|
- Web UI (8006) from the management network/VLAN only
|
|
- Corosync ports between cluster nodes
|
|
- Ceph ports between cluster nodes (once Ceph is live)
|
|
|
|
Enforce the network separation above at the firewall — corosync and Ceph
|
|
traffic shouldn't be reachable from the VM network even if they end up
|
|
sharing a physical link.
|
|
|
|
See `config/pve-firewall/` for a starting rule set.
|