Updated environment

docs/06-pve-test-wifi-network.md

@@ -0,0 +1,191 @@
+# pve-test's Wifi-Primary Network
+
+Non-standard, deliberately chosen against normal Proxmox guidance (a
+hypervisor's management network is not supposed to be wireless). Applies
+**only** to `pve-test` as a standalone sandbox node — see
+`05-node-roles.md` for why this and cluster membership don't mix, and
+never apply this pattern to a node that's joined (or might join) a
+cluster.
+
+## Why this works at all
+
+Wifi (802.11) normally cannot be bridged the way Ethernet can: an access
+point only accepts frames whose source MAC matches the MAC that
+associated with it. A Linux bridge with a wifi port as a member sends
+frames tagged with whatever MAC actually generated them (the bridge's own
+MAC, a VM's MAC, etc.) — none of which match the wifi card's hardware
+MAC, so the AP silently drops them. This isn't a Linux/Proxmox limitation,
+it's how WiFi association works.
+
+**4addr (WDS) client mode** is the exception: if both the wifi driver and
+the AP support it, the client tags frames with a 4th address field
+carrying the original MAC, and the AP forwards them like a real Ethernet
+segment. Most consumer routers do **not** support this (it's common on
+OpenWrt/DD-WRT/enterprise APs, rare on stock ISP hardware) — it has to be
+confirmed empirically, per-AP, before trusting it with a management IP.
+
+pve-test's card (Intel AX200, `iwlwifi` driver, interface `wlp3s0`)
+supports 4addr client-side generically (this is a `mac80211` core
+feature, not driver-specific). The home AP it associates to
+(`nbn-fttp-net-5G`) was confirmed to honor it — see validation method
+below.
+
+## Validate before ever touching a live management IP
+
+Don't add the wifi NIC straight into your real bridge to "see if it
+works" — if 4addr isn't actually honored, you lose access to the box with
+no diagnostic trail (association succeeds; only the *data plane* silently
+fails). Test in an isolated scratch namespace first:
+
+```bash
+# assumes wlp3s0 already associated with 4addr on (see systemd unit below)
+ip link add testbr0 type bridge && ip link set testbr0 up
+ip link set wlp3s0 master testbr0 && ip link set wlp3s0 up
+
+ip netns add wifitest
+ip link add veth-host type veth peer name veth-ns
+ip link set veth-host master testbr0 && ip link set veth-host up
+ip link set veth-ns netns wifitest
+ip netns exec wifitest ip link set veth-ns address 02:11:22:33:44:55  # deliberately NOT the wifi card's MAC
+ip netns exec wifitest ip link set veth-ns up
+ip netns exec wifitest ip link set lo up
+
+ip netns exec wifitest dhclient -v -1 veth-ns   # real DHCP lease = AP forwards foreign MACs = 4addr genuinely works
+```
+
+If that gets a lease (or at minimum a ping reply from the gateway) from a
+MAC that was never the one that associated, the AP is forwarding
+arbitrary-MAC frames — real bridge mode will work. If not, don't proceed
+to bridge mode; fall back to a routed/NAT design instead (wifi gets its
+own IP via DHCP, VMs NAT out through it — never covered here since it
+wasn't needed, but keep this fallback in mind if reproducing on different
+hardware/AP).
+
+Tear the test namespace/bridge down afterwards (`ip netns del wifitest;
+ip link del veth-host; ip link set wlp3s0 nomaster; ip link del testbr0`)
+— it's scratch, not part of the real config.
+
+## Final architecture
+
+```
+wlp3s0 (wifi, 4addr client mode) ─┐
+                                    ├─ bond0 (active-backup, wlp3s0 primary) ─ vmbr0 (192.168.2.251/24)
+nic0 (wired, LAN backup)          ─┘
+```
+
+- **`wpa-4addr-<iface>.service`** (systemd unit, `Before=network-pre.target`) —
+  sets 4addr mode and starts `wpa_supplicant` *before* ifupdown2 processes
+  `/etc/network/interfaces`. Deliberately not using `wpasupplicant`'s own
+  `wpa-conf` ifupdown2 hook integration — its ordering relative to a
+  custom `pre-up iw ... set 4addr on` line is implementation-specific and
+  not worth gambling on for something this hard to debug if it's wrong.
+  The systemd unit gives full, deterministic control over sequencing
+  instead.
+- **`bond0`** — active-backup bonding, `wlp3s0` as `bond-primary`,
+  `nic0` as backup, `bond-miimon 100`. Bonding a wifi interface works here
+  because `mac80211`/`iwlwifi` properly reports carrier state
+  (`netif_carrier_on/off`) on association/disassociation, which is what
+  bonding's `miimon` (with the default `use_carrier=1`) actually watches
+  — it doesn't require a "real" MII-capable NIC.
+- **`vmbr0`** — unchanged IP (`192.168.2.251/24`), bridged over `bond0`
+  instead of directly over a physical NIC.
+
+Reproduce with `scripts/setup-wifi-bond-network.sh` — see its header
+comment for usage and required env vars (SSID/passphrase, management
+address/gateway). It stages `/etc/network/interfaces` and the systemd
+unit but does **not** auto-apply (`ifreload -a`) — see the next section
+for why that step needs care, not automation.
+
+## Applying a live bridge-port change: what went wrong once, and what worked
+
+Moving `vmbr0`'s underlying port live — while SSH'd in over the very
+address that lives on that bridge — is inherently risky: if the new port
+doesn't actually pass traffic, you lose the connection you're using to
+fix it.
+
+**What failed**: doing the port swap via raw `ip link set nic0 nomaster`
++ `ip link set nic0 down` (with `wlp3s0` already added as a second bridge
+member) caused an extended outage that needed a physical power-cycle to
+recover — `nic0` never came back up on its own even well past a 60s
+auto-revert watchdog's deadline. Importantly, the *host itself* never
+crashed — `journalctl -b -1` showed completely normal operation (a
+`pvestatd` polling loop, no gaps, no panic/OOM/watchdog trigger) right up
+until a keyboard was plugged in and it was manually reset. So the failure
+was specifically in the live network transition, not the OS — root cause
+unconfirmed, but plausibly bridge FDB/MAC-identity handling when a port
+carrying an already-live IP is swapped out via raw `ip link` rather than
+a coordinated reconfiguration.
+
+**What worked**: writing the final state into `/etc/network/interfaces`
+and applying with `ifreload -a` (Proxmox's own supported hot-reload path)
+instead. Same end state, same risk window, but ifupdown2 evidently
+sequences the transition in a way raw `ip link` surgery didn't. This
+succeeded on the first retry using this method.
+
+**Either way, expect ~30-60 seconds of apparent breakage even on a
+successful change** — the upstream switch/AP needs to relearn which port
+`192.168.2.251`'s MAC now lives behind. Don't judge success/failure
+before that window passes; a `ping`/SSH failure at the 5-10s mark is not
+yet a sign anything is wrong.
+
+**Recommended safety net for any future live change here**: arm a
+backgrounded auto-revert-on-timeout before applying, e.g.:
+
+```bash
+cp /etc/network/interfaces /etc/network/interfaces.bak
+cat > /root/revert.sh <<'EOF'
+#!/bin/bash
+cp /etc/network/interfaces.bak /etc/network/interfaces
+ifreload -a
+EOF
+chmod +x /root/revert.sh
+setsid nohup bash -c "sleep 45 && /root/revert.sh" >/root/revert.log 2>&1 < /dev/null &
+disown
+# now apply the real change, e.g.: ifreload -a
+# once confirmed working: pkill -f "sleep 45 && /root/revert.sh"
+```
+
+This only self-heals if the box stays responsive enough for the
+backgrounded job to run to completion — it is not a substitute for
+physical/console access being available, just a way to avoid needing it
+for the common case.
+
+## Troubleshooting: "can't ping it, web UI won't load" isn't necessarily an outage
+
+Two independent things can make `pve-test` look dead when it isn't:
+
+1. **Stale ARP after any change to `vmbr0`'s active port.** `vmbr0`'s MAC
+   address follows whichever interface is currently active in `bond0`
+   (`wlp3s0`'s MAC while wifi is primary, `nic0`'s if it fails over) — it
+   is *not* fixed. Other devices on the LAN (including your own
+   workstation) that cached the old MAC before a change won't notice
+   until their ARP entry naturally expires or gets flushed
+   (`ip neigh flush <ip>` / `arp -d <ip>`). Symptom: intermittent or
+   totally dead connectivity from one specific device while others (or
+   the host itself) are fine.
+2. **ICMP ping is blocked by the management firewall, unrelated to
+   network health.** See `04-security-hardening.md`'s firewall section —
+   `cluster.fw`'s `policy_in: DROP` only explicitly allows TCP 22/8006
+   (now also ICMP echo-request, after this was hit) from the mgmt
+   network. Before that rule was added, `ping` failed 100% *even with a
+   perfectly correct, fresh ARP entry and full SSH/web UI access* —
+   nothing to do with wifi, bonding, or the network being down. This is
+   exactly what happened once: alarming "can't ping, web UI not loading"
+   turned out to be a stale-ARP moment (which resolved itself) plus a
+   pre-existing firewall policy (ping was never going to work,
+   regardless of wifi).
+
+**When `pve-test` seems unreachable, check in this order before assuming
+a real outage**: SSH (`ssh root@pve-test.sweet.home`) → web UI via `curl
+-sk -o /dev/null -w '%{http_code}' https://pve-test.sweet.home:8006/` →
+only then worry about `ping` specifically, and check
+`/etc/pve/firewall/cluster.fw` before blaming the network.
+
+## Known limitation history
+
+`nic0` (the bond's backup slave) initially had no cable physically
+connected when this was first set up — the bond was correctly configured
+but inert until a cable was plugged in. As of this writing a cable is
+connected and the backup slave is live (confirm with `cat
+/proc/net/bonding/bond0`) — if reproducing this setup, verify the same
+before assuming LAN failover will actually work.