Archived
Document pve-test's wifi-primary network and add Claude node guardrails
pve-test was briefly clustered with pve1 then deliberately de-clustered so it could move to wifi-primary networking (4addr bridge mode, bonded with a wired LAN backup) - a change not achievable while clustered given corosync's latency requirements. Captures that as a reproducible script plus docs: cluster separation procedure, the wifi network design and the live-cutover pitfalls hit along the way, and node-role/history context. Also adds CLAUDE.md guardrails for pve1 (production) vs pve-test (sandbox) - this repo had none before, despite scripts here being able to make real changes to both. Separately: both nodes' mgmt firewalls were dropping ICMP by default (TCP 22/8006 only), which looked like an outage mid-troubleshooting even though SSH/web UI were fine. Added an explicit ping-allow rule to the firewall template, applied it live on both nodes, and added an audit.sh check so it stays enforced.
This commit is contained in:
@@ -59,6 +59,16 @@ require or assume the split-disk/multi-NIC layout Stage 2 wants.
|
||||
Everything below this point is the target design for when nodes 2 and 3
|
||||
are actually being provisioned. Not applicable to `pve1` as it stands.
|
||||
|
||||
**`pve-test` existing does not mean node 2 exists.** A second physical
|
||||
node (`pve-test`) does run alongside `pve1` — see `05-node-roles.md` —
|
||||
but it's a sandbox/test box, not built to this Stage 2 design, and as of
|
||||
this writing runs on **wifi** networking
|
||||
(`06-pve-test-wifi-network.md`), which is directly incompatible with
|
||||
corosync's latency/jitter requirements below. `pve-test` and `pve1` were
|
||||
briefly clustered and then deliberately de-clustered for exactly this
|
||||
reason. Don't treat `pve-test` as progress toward Stage 2 without a
|
||||
deliberate decision to rebuild its networking first.
|
||||
|
||||
### End goal
|
||||
|
||||
3-node Proxmox VE cluster with HA-managed VMs backed by **Ceph** — true
|
||||
|
||||
@@ -11,6 +11,17 @@ PVE datacenter firewall (default-deny inbound otherwise). That's
|
||||
sufficient until Stage 2 needs actual separate physical/VLAN paths for
|
||||
corosync and Ceph traffic — see below.
|
||||
|
||||
## pve-test as built (sandbox, current)
|
||||
|
||||
Different node, different design, not a Stage 1/2 example to generalize
|
||||
from: `pve-test` runs `vmbr0` bridged over a wifi NIC in 4addr client mode
|
||||
(active-backup bonded with a wired NIC as an automatic fallback). Full
|
||||
detail, including why this is normally impossible and how it was
|
||||
validated before trusting it with the management IP, in
|
||||
`06-pve-test-wifi-network.md`. This is intentionally a one-off for a
|
||||
standalone sandbox box — never extend it to a node that's clustered or
|
||||
Ceph-connected (see the Stage 2 note in `00-overview.md`).
|
||||
|
||||
## Target design (Stage 2, future cluster)
|
||||
|
||||
## Required separation
|
||||
|
||||
@@ -54,6 +54,23 @@ Default-deny at datacenter/node level, whitelist only what's needed (see
|
||||
`config/pve-firewall/cluster.fw.example`, applied by
|
||||
`scripts/deploy-firewall.sh`.
|
||||
|
||||
**ICMP echo (ping) is explicitly allowed from the management network**,
|
||||
alongside SSH/8006 — not required for anything to function, but without
|
||||
it `policy_in: DROP` silently eats ping while SSH/web UI keep working.
|
||||
That split (ping dead, everything else fine) reads exactly like a real
|
||||
outage mid-troubleshooting; see `06-pve-test-wifi-network.md` for a case
|
||||
this caused genuine confusion after a network change. If you ever debug
|
||||
"can't ping but can SSH", check this firewall before assuming the
|
||||
network itself is broken.
|
||||
|
||||
**This file is cluster-wide, not per-node**: `cluster.fw` lives in
|
||||
`/etc/pve/firewall/` — shared via pmxcfs across every node in a cluster.
|
||||
A node that joins a cluster inherits whatever's already there, and (per
|
||||
`docs/05-node-roles.md`) keeps its own local copy after leaving. Don't
|
||||
assume a node's firewall state matches what `deploy-firewall.sh` was
|
||||
last run with directly against it — check `pve-firewall status` /
|
||||
`/etc/pve/firewall/cluster.fw` on the actual node.
|
||||
|
||||
## Repos and updates
|
||||
|
||||
Fresh installs point at the enterprise repo, which fails on `apt update`
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
# Node Roles: pve1 and pve-test
|
||||
|
||||
Two Proxmox VE hosts exist on the LAN. They are **not interchangeable** —
|
||||
different purpose, different risk tolerance, different guardrails. See
|
||||
`CLAUDE.md` for the actual rules Claude follows; this doc is the
|
||||
background/context those rules assume.
|
||||
|
||||
## pve1 (production)
|
||||
|
||||
ASUS PN53 mini PC (Ryzen 7 7735HS, 32GB RAM) — see `01-hardware-node1.md`
|
||||
for the full hardware layout. Stage 1 base hardening applied and verified
|
||||
(`04-security-hardening.md`).
|
||||
|
||||
Runs real production workloads:
|
||||
|
||||
- `server-nixos` (VM) and LXC containers `pihole`, `claude`, `nix-cache`,
|
||||
`docker`, `pdm`, `pxe-boot` — DNS, this Claude Code environment itself,
|
||||
the LAN's Nix binary cache, etc.
|
||||
|
||||
Off-limits to Claude by default — see `CLAUDE.md`.
|
||||
|
||||
## pve-test (sandbox)
|
||||
|
||||
A separate physical node kept for testing. Has a capable GPU (earmarked
|
||||
for future GPU workloads — transcoding, local inference, etc. — nothing
|
||||
built on it yet as of this writing). No production guests run here.
|
||||
|
||||
Safe for Claude to create/destroy scratch VMs and containers on without
|
||||
asking — see `CLAUDE.md` for the exact boundary (node-level config is
|
||||
still not free-for-all even here).
|
||||
|
||||
### History: briefly clustered with pve1, then deliberately separated
|
||||
|
||||
On 2026-07-21/22, pve-test was joined to pve1 in a 2-node Proxmox cluster
|
||||
(corosync + shared `/etc/pve`) for a period, then the cluster was
|
||||
**intentionally destroyed** at the operator's explicit request — not a
|
||||
failure or accident. Rationale: the operator wanted pve-test moved onto
|
||||
wifi networking (see `06-pve-test-wifi-network.md`), and wifi is
|
||||
fundamentally unsuitable for corosync's latency/jitter requirements (token
|
||||
timeouts, flapping membership). Rather than fight that, the cluster was
|
||||
torn down first, then wifi was configured on the now-standalone node.
|
||||
|
||||
Teardown procedure used (safe to repeat if pve-test or pve1 is ever
|
||||
reclustered and needs separating again):
|
||||
|
||||
1. Un-manage any HA-assigned resources first if HA is in use
|
||||
(`ha-manager remove <sid>` per resource) — **critical**: if a node has
|
||||
HA-managed guests and its watchdog is still armed when it loses
|
||||
quorum, HA's self-fence behavior can force-reboot it. Removing the
|
||||
resource assignments first (not stopping the guests — just
|
||||
un-managing them) prevents this.
|
||||
2. Stop `pve-ha-lrm`/`pve-ha-crm` on all nodes once no resources remain
|
||||
assigned, confirm via `ha-manager status` that fencing is no longer
|
||||
armed.
|
||||
3. From a node that will remain in the cluster: `pvecm delnode <name>`
|
||||
for the node being removed. In a 2-node cluster with no QDevice, this
|
||||
can itself cause the *remaining* node to transiently lose quorum
|
||||
(killing a node's live corosync membership drops total votes below
|
||||
expected votes before the nodelist file can be rewritten to match) —
|
||||
this is expected, not a failure; proceed to step 4 regardless.
|
||||
4. On **each** node being separated (including one you just ran `delnode`
|
||||
from, if it's also being decoupled): `systemctl stop pve-cluster
|
||||
corosync`, `pmxcfs -l` (local mode, bypasses the quorum requirement
|
||||
for local writes), remove `/etc/pve/corosync.conf` and
|
||||
`/etc/corosync/*`, `killall pmxcfs`, `systemctl start pve-cluster`.
|
||||
This is the official "separate a node without reinstalling" procedure
|
||||
and doesn't touch any running VM/CT — they're independent OS
|
||||
processes, unaffected by corosync/pmxcfs state either way.
|
||||
5. Optional cosmetic cleanup: the node(s) staying up may retain a stale
|
||||
`/etc/pve/nodes/<removed-node>/` directory (cached VM/CT configs from
|
||||
when they were clustered) — safe to `rm -rf` once separation is
|
||||
confirmed; it's dead data, not a live reference to anything.
|
||||
|
||||
Guests were never at risk during this procedure (confirmed: all 7 of
|
||||
pve1's guests ran continuously throughout) — the only real hazard is the
|
||||
HA/watchdog self-fence path in step 1, and losing corosync-provided quorum
|
||||
temporarily blocking `/etc/pve` writes (not guest execution) in step 3.
|
||||
|
||||
### Current state: standalone, wifi-primary networking
|
||||
|
||||
pve-test now runs entirely independently of pve1, on the network design
|
||||
documented in `06-pve-test-wifi-network.md`. **Do not re-cluster
|
||||
pve-test while it's on wifi** — see that doc for why.
|
||||
@@ -0,0 +1,191 @@
|
||||
# pve-test's Wifi-Primary Network
|
||||
|
||||
Non-standard, deliberately chosen against normal Proxmox guidance (a
|
||||
hypervisor's management network is not supposed to be wireless). Applies
|
||||
**only** to `pve-test` as a standalone sandbox node — see
|
||||
`05-node-roles.md` for why this and cluster membership don't mix, and
|
||||
never apply this pattern to a node that's joined (or might join) a
|
||||
cluster.
|
||||
|
||||
## Why this works at all
|
||||
|
||||
Wifi (802.11) normally cannot be bridged the way Ethernet can: an access
|
||||
point only accepts frames whose source MAC matches the MAC that
|
||||
associated with it. A Linux bridge with a wifi port as a member sends
|
||||
frames tagged with whatever MAC actually generated them (the bridge's own
|
||||
MAC, a VM's MAC, etc.) — none of which match the wifi card's hardware
|
||||
MAC, so the AP silently drops them. This isn't a Linux/Proxmox limitation,
|
||||
it's how WiFi association works.
|
||||
|
||||
**4addr (WDS) client mode** is the exception: if both the wifi driver and
|
||||
the AP support it, the client tags frames with a 4th address field
|
||||
carrying the original MAC, and the AP forwards them like a real Ethernet
|
||||
segment. Most consumer routers do **not** support this (it's common on
|
||||
OpenWrt/DD-WRT/enterprise APs, rare on stock ISP hardware) — it has to be
|
||||
confirmed empirically, per-AP, before trusting it with a management IP.
|
||||
|
||||
pve-test's card (Intel AX200, `iwlwifi` driver, interface `wlp3s0`)
|
||||
supports 4addr client-side generically (this is a `mac80211` core
|
||||
feature, not driver-specific). The home AP it associates to
|
||||
(`nbn-fttp-net-5G`) was confirmed to honor it — see validation method
|
||||
below.
|
||||
|
||||
## Validate before ever touching a live management IP
|
||||
|
||||
Don't add the wifi NIC straight into your real bridge to "see if it
|
||||
works" — if 4addr isn't actually honored, you lose access to the box with
|
||||
no diagnostic trail (association succeeds; only the *data plane* silently
|
||||
fails). Test in an isolated scratch namespace first:
|
||||
|
||||
```bash
|
||||
# assumes wlp3s0 already associated with 4addr on (see systemd unit below)
|
||||
ip link add testbr0 type bridge && ip link set testbr0 up
|
||||
ip link set wlp3s0 master testbr0 && ip link set wlp3s0 up
|
||||
|
||||
ip netns add wifitest
|
||||
ip link add veth-host type veth peer name veth-ns
|
||||
ip link set veth-host master testbr0 && ip link set veth-host up
|
||||
ip link set veth-ns netns wifitest
|
||||
ip netns exec wifitest ip link set veth-ns address 02:11:22:33:44:55 # deliberately NOT the wifi card's MAC
|
||||
ip netns exec wifitest ip link set veth-ns up
|
||||
ip netns exec wifitest ip link set lo up
|
||||
|
||||
ip netns exec wifitest dhclient -v -1 veth-ns # real DHCP lease = AP forwards foreign MACs = 4addr genuinely works
|
||||
```
|
||||
|
||||
If that gets a lease (or at minimum a ping reply from the gateway) from a
|
||||
MAC that was never the one that associated, the AP is forwarding
|
||||
arbitrary-MAC frames — real bridge mode will work. If not, don't proceed
|
||||
to bridge mode; fall back to a routed/NAT design instead (wifi gets its
|
||||
own IP via DHCP, VMs NAT out through it — never covered here since it
|
||||
wasn't needed, but keep this fallback in mind if reproducing on different
|
||||
hardware/AP).
|
||||
|
||||
Tear the test namespace/bridge down afterwards (`ip netns del wifitest;
|
||||
ip link del veth-host; ip link set wlp3s0 nomaster; ip link del testbr0`)
|
||||
— it's scratch, not part of the real config.
|
||||
|
||||
## Final architecture
|
||||
|
||||
```
|
||||
wlp3s0 (wifi, 4addr client mode) ─┐
|
||||
├─ bond0 (active-backup, wlp3s0 primary) ─ vmbr0 (192.168.2.251/24)
|
||||
nic0 (wired, LAN backup) ─┘
|
||||
```
|
||||
|
||||
- **`wpa-4addr-<iface>.service`** (systemd unit, `Before=network-pre.target`) —
|
||||
sets 4addr mode and starts `wpa_supplicant` *before* ifupdown2 processes
|
||||
`/etc/network/interfaces`. Deliberately not using `wpasupplicant`'s own
|
||||
`wpa-conf` ifupdown2 hook integration — its ordering relative to a
|
||||
custom `pre-up iw ... set 4addr on` line is implementation-specific and
|
||||
not worth gambling on for something this hard to debug if it's wrong.
|
||||
The systemd unit gives full, deterministic control over sequencing
|
||||
instead.
|
||||
- **`bond0`** — active-backup bonding, `wlp3s0` as `bond-primary`,
|
||||
`nic0` as backup, `bond-miimon 100`. Bonding a wifi interface works here
|
||||
because `mac80211`/`iwlwifi` properly reports carrier state
|
||||
(`netif_carrier_on/off`) on association/disassociation, which is what
|
||||
bonding's `miimon` (with the default `use_carrier=1`) actually watches
|
||||
— it doesn't require a "real" MII-capable NIC.
|
||||
- **`vmbr0`** — unchanged IP (`192.168.2.251/24`), bridged over `bond0`
|
||||
instead of directly over a physical NIC.
|
||||
|
||||
Reproduce with `scripts/setup-wifi-bond-network.sh` — see its header
|
||||
comment for usage and required env vars (SSID/passphrase, management
|
||||
address/gateway). It stages `/etc/network/interfaces` and the systemd
|
||||
unit but does **not** auto-apply (`ifreload -a`) — see the next section
|
||||
for why that step needs care, not automation.
|
||||
|
||||
## Applying a live bridge-port change: what went wrong once, and what worked
|
||||
|
||||
Moving `vmbr0`'s underlying port live — while SSH'd in over the very
|
||||
address that lives on that bridge — is inherently risky: if the new port
|
||||
doesn't actually pass traffic, you lose the connection you're using to
|
||||
fix it.
|
||||
|
||||
**What failed**: doing the port swap via raw `ip link set nic0 nomaster`
|
||||
+ `ip link set nic0 down` (with `wlp3s0` already added as a second bridge
|
||||
member) caused an extended outage that needed a physical power-cycle to
|
||||
recover — `nic0` never came back up on its own even well past a 60s
|
||||
auto-revert watchdog's deadline. Importantly, the *host itself* never
|
||||
crashed — `journalctl -b -1` showed completely normal operation (a
|
||||
`pvestatd` polling loop, no gaps, no panic/OOM/watchdog trigger) right up
|
||||
until a keyboard was plugged in and it was manually reset. So the failure
|
||||
was specifically in the live network transition, not the OS — root cause
|
||||
unconfirmed, but plausibly bridge FDB/MAC-identity handling when a port
|
||||
carrying an already-live IP is swapped out via raw `ip link` rather than
|
||||
a coordinated reconfiguration.
|
||||
|
||||
**What worked**: writing the final state into `/etc/network/interfaces`
|
||||
and applying with `ifreload -a` (Proxmox's own supported hot-reload path)
|
||||
instead. Same end state, same risk window, but ifupdown2 evidently
|
||||
sequences the transition in a way raw `ip link` surgery didn't. This
|
||||
succeeded on the first retry using this method.
|
||||
|
||||
**Either way, expect ~30-60 seconds of apparent breakage even on a
|
||||
successful change** — the upstream switch/AP needs to relearn which port
|
||||
`192.168.2.251`'s MAC now lives behind. Don't judge success/failure
|
||||
before that window passes; a `ping`/SSH failure at the 5-10s mark is not
|
||||
yet a sign anything is wrong.
|
||||
|
||||
**Recommended safety net for any future live change here**: arm a
|
||||
backgrounded auto-revert-on-timeout before applying, e.g.:
|
||||
|
||||
```bash
|
||||
cp /etc/network/interfaces /etc/network/interfaces.bak
|
||||
cat > /root/revert.sh <<'EOF'
|
||||
#!/bin/bash
|
||||
cp /etc/network/interfaces.bak /etc/network/interfaces
|
||||
ifreload -a
|
||||
EOF
|
||||
chmod +x /root/revert.sh
|
||||
setsid nohup bash -c "sleep 45 && /root/revert.sh" >/root/revert.log 2>&1 < /dev/null &
|
||||
disown
|
||||
# now apply the real change, e.g.: ifreload -a
|
||||
# once confirmed working: pkill -f "sleep 45 && /root/revert.sh"
|
||||
```
|
||||
|
||||
This only self-heals if the box stays responsive enough for the
|
||||
backgrounded job to run to completion — it is not a substitute for
|
||||
physical/console access being available, just a way to avoid needing it
|
||||
for the common case.
|
||||
|
||||
## Troubleshooting: "can't ping it, web UI won't load" isn't necessarily an outage
|
||||
|
||||
Two independent things can make `pve-test` look dead when it isn't:
|
||||
|
||||
1. **Stale ARP after any change to `vmbr0`'s active port.** `vmbr0`'s MAC
|
||||
address follows whichever interface is currently active in `bond0`
|
||||
(`wlp3s0`'s MAC while wifi is primary, `nic0`'s if it fails over) — it
|
||||
is *not* fixed. Other devices on the LAN (including your own
|
||||
workstation) that cached the old MAC before a change won't notice
|
||||
until their ARP entry naturally expires or gets flushed
|
||||
(`ip neigh flush <ip>` / `arp -d <ip>`). Symptom: intermittent or
|
||||
totally dead connectivity from one specific device while others (or
|
||||
the host itself) are fine.
|
||||
2. **ICMP ping is blocked by the management firewall, unrelated to
|
||||
network health.** See `04-security-hardening.md`'s firewall section —
|
||||
`cluster.fw`'s `policy_in: DROP` only explicitly allows TCP 22/8006
|
||||
(now also ICMP echo-request, after this was hit) from the mgmt
|
||||
network. Before that rule was added, `ping` failed 100% *even with a
|
||||
perfectly correct, fresh ARP entry and full SSH/web UI access* —
|
||||
nothing to do with wifi, bonding, or the network being down. This is
|
||||
exactly what happened once: alarming "can't ping, web UI not loading"
|
||||
turned out to be a stale-ARP moment (which resolved itself) plus a
|
||||
pre-existing firewall policy (ping was never going to work,
|
||||
regardless of wifi).
|
||||
|
||||
**When `pve-test` seems unreachable, check in this order before assuming
|
||||
a real outage**: SSH (`ssh root@pve-test.sweet.home`) → web UI via `curl
|
||||
-sk -o /dev/null -w '%{http_code}' https://pve-test.sweet.home:8006/` →
|
||||
only then worry about `ping` specifically, and check
|
||||
`/etc/pve/firewall/cluster.fw` before blaming the network.
|
||||
|
||||
## Known limitation history
|
||||
|
||||
`nic0` (the bond's backup slave) initially had no cable physically
|
||||
connected when this was first set up — the bond was correctly configured
|
||||
but inert until a cable was plugged in. As of this writing a cable is
|
||||
connected and the backup slave is live (confirm with `cat
|
||||
/proc/net/bonding/bond0`) — if reproducing this setup, verify the same
|
||||
before assuming LAN failover will actually work.
|
||||
Reference in New Issue
Block a user