Author SHA1 Message Date
beatzaplenty 143eebcbf3 Correct AP-isolation guess: confirmed as a 4addr forwarding gap instead
Tested live: AP isolation was already off on the TP-Link BE9300, and a
second wireless device (phone) failed identically to the laptop - rules
out isolation entirely. Real cause is the router's 4addr/WDS handling
not forwarding wireless-client broadcast/ARP traffic to pve-test's WDS
association, even though wired forwarding to it works fine. Since ARP
never resolves for wireless peers, this affects every protocol (web UI
included), not just ICMP - corrects an earlier draft of this doc that
speculated TCP might still work.
2026-07-21 21:35:08 +00:00
beatzaplenty dc4eff97fa Document AP/client-isolation gotcha for wifi clients reaching pve-test
Found live: a laptop on the same wifi network as pve-test couldn't reach
it at all (no ARP entry, not stale) despite every wired device working
fine and the mgmt firewall already allowing ICMP - the router's AP/client
isolation was blocking wireless-to-wireless traffic. Router-side fix
only; nothing on pve-test to change for this symptom.
2026-07-21 21:26:26 +00:00
beatzaplenty ba73420350 Document pve-test's wifi-primary network and add Claude node guardrails
pve-test was briefly clustered with pve1 then deliberately de-clustered
so it could move to wifi-primary networking (4addr bridge mode, bonded
with a wired LAN backup) - a change not achievable while clustered given
corosync's latency requirements. Captures that as a reproducible script
plus docs: cluster separation procedure, the wifi network design and the
live-cutover pitfalls hit along the way, and node-role/history context.

Also adds CLAUDE.md guardrails for pve1 (production) vs pve-test
(sandbox) - this repo had none before, despite scripts here being able
to make real changes to both.

Separately: both nodes' mgmt firewalls were dropping ICMP by default
(TCP 22/8006 only), which looked like an outage mid-troubleshooting even
though SSH/web UI were fine. Added an explicit ping-allow rule to the
firewall template, applied it live on both nodes, and added an audit.sh
check so it stays enforced.
2026-07-21 21:20:27 +00:00
10 changed files with 592 additions and 5 deletions
+84
View File
@@ -0,0 +1,84 @@
# CLAUDE.md
Guidance for Claude Code working in this repo. IMPORTANT: these
instructions OVERRIDE any default behavior and must be followed exactly
as written.
## Repo purpose
Base configuration/hardening toolset and planning docs for Proxmox VE
hosts (`scripts/`, `config/`, `docs/`) — see `README.md` and
`docs/00-overview.md`. Scripts in this repo are meant to be run **on**
the target Proxmox host itself (as root), not orchestrated remotely.
## Two Proxmox nodes: `pve1` (production) and `pve-test` (sandbox)
Two SSH-reachable Proxmox nodes exist on the LAN. They are **not
interchangeable** — see `docs/05-node-roles.md` for full background on
what each one is and why.
### `pve1` (production — off-limits to Claude by default)
A real, live Proxmox node hosting production VMs/containers (see
`docs/05-node-roles.md` for the current guest list) — not a sandbox, and
not Claude's to touch by default.
- **Off-limits at all times unless the operator has given explicit,
same-session instructions to act on this specific host.** That
authorization is scoped to the task it was given for — don't carry it
forward to unrelated later work in the same conversation, and never
assume it from a previous session.
- **Read-only for existing state is always fine, authorization or not.**
SSH in (or use `pvesm`, `qm list`, `pct list`, `qm config`, `pct
config`, the Proxmox API, etc.) to inspect config, storage, and any
existing VM/container freely.
- **Never** modify, stop, restart, delete, reconfigure, or create
anything on this node (`qm set`, `pct set`, `qm destroy`, `pct
destroy`, `qm stop`, `pct stop`, `qm create`, `pct create`, snapshot
operations, storage changes, running any script in this repo against
it, etc.) without that explicit go-ahead. Use `pve-test` for anything
exploratory instead.
- If a guest on `pve1` is HA-managed, be aware of the self-fence hazard
described in `docs/05-node-roles.md`'s cluster-teardown section before
doing anything that could cost the node quorum.
### `pve-test` (sandbox — Claude's default target)
A separate node set aside for testing — safe to create, interrogate, and
destroy scratch VMs/containers on without asking first.
- **Test VMs/containers are allowed, but must be torn down.** Anything
created this way must be destroyed again in the same session, before
ending the task. Use an obviously-scratch VMID/name.
- **Node-level config is still not yours to change by default.**
Creating/destroying your own scratch guests is fine; Proxmox host
config, storage pools, and networking on `pve-test` itself need the
operator's explicit go-ahead too, same as on `pve1` — the "sandbox"
status covers guest-level experimentation, not the host's own
identity. (`pve-test`'s current network config, `docs/06-pve-test-wifi-network.md`,
was applied under exactly that kind of explicit, same-session
authorization — it's not a standing invitation to keep changing it
further without asking again.)
- **Never re-cluster `pve-test` with `pve1`** without the operator
explicitly asking for it and being aware of the wifi/corosync
incompatibility in `docs/06-pve-test-wifi-network.md` — the two were
deliberately de-clustered for this reason once already.
## Safety rules
- Any script in `scripts/` that isn't `audit.sh` (read-only) makes real
changes when run for real. Don't run one against `pve1`, or against
`pve-test`'s node-level config, without the same-session go-ahead
described above. Running a script *against pve-test's own guests* — a
scratch VM/CT you created for this task — doesn't need separate
permission.
- Do not commit secrets: SSH private keys, wifi passphrases, sops/age
keys, or PVE credentials. `scripts/setup-wifi-bond-network.sh` takes
the SSID/passphrase via environment variables for exactly this reason
— never hardcode them into the script or a committed config file.
- Live changes to a node's own management network (the interface/bridge
carrying the SSH session you're using) can strand the box — see the
"what went wrong once" section in `docs/06-pve-test-wifi-network.md`
before touching `pve-test`'s networking again. Prefer applying via
`ifreload -a` over raw `ip link` surgery, and arm an auto-revert
watchdog first when acting without someone physically at the console.
+12 -5
View File
@@ -15,15 +15,22 @@ deferred).
backed by Ceph. Needs dedicated hardware node 1 (`pve1`, an ASUS PN53 mini
PC) doesn't have — see `docs/01-hardware-node1.md`.
See `CLAUDE.md` for the guardrails Claude Code follows when working
against these hosts (`pve1` is production and off-limits by default;
`pve-test` is the sandbox).
## Repo layout
- `docs/` — planning docs: hardware layout, storage migration, networking,
security hardening. Read `docs/00-overview.md` first.
security hardening, node roles. Read `docs/00-overview.md` first, then
`docs/05-node-roles.md` for what `pve1`/`pve-test` actually are.
- `scripts/` — scripts to apply configuration on a node (SSH hardening, repo
switch, firewall, updates, etc). Idempotent, safe to re-run.
`scripts/bootstrap.sh` runs the full Stage 1 sequence end to end;
`scripts/audit.sh` verifies it (read-only). `scripts/lib/` holds shared
helpers (`common.sh`) sourced by the other scripts.
switch, firewall, updates, wifi/bond networking, etc). Idempotent, safe
to re-run. `scripts/bootstrap.sh` runs the full Stage 1 sequence end to
end; `scripts/audit.sh` verifies it (read-only).
`scripts/setup-wifi-bond-network.sh` reproduces `pve-test`'s wifi
network (see `docs/06-pve-test-wifi-network.md`). `scripts/lib/` holds
shared helpers (`common.sh`) sourced by the other scripts.
- `config/` — reference config files/snippets to drop onto a node (firewall
rules, sshd config, etc.).
+7
View File
@@ -26,6 +26,13 @@ policy_out: ACCEPT
IN ACCEPT -source +mgmt -p tcp -dport 8006 -log nolog
IN ACCEPT -source +mgmt -p tcp -dport 22 -log nolog
# ICMP echo (ping) from the management network - diagnostic convenience
# only, nothing else depends on it. Without this, policy_in DROP silently
# eats ping while SSH/web UI keep working - looks like an outage during
# troubleshooting when the host is actually fine. See
# docs/06-pve-test-wifi-network.md for a case this caused real confusion.
IN ACCEPT -source +mgmt -p icmp -icmp-type echo-request -log nolog
# Stage 2: Corosync (cluster quorum) - uncomment once node 2/3 join and
# the corosync network/VLAN exists.
# IN ACCEPT -source <COROSYNC_CIDR> -p udp -dport 5404:5405 -log nolog
+10
View File
@@ -59,6 +59,16 @@ require or assume the split-disk/multi-NIC layout Stage 2 wants.
Everything below this point is the target design for when nodes 2 and 3
are actually being provisioned. Not applicable to `pve1` as it stands.
**`pve-test` existing does not mean node 2 exists.** A second physical
node (`pve-test`) does run alongside `pve1` — see `05-node-roles.md`
but it's a sandbox/test box, not built to this Stage 2 design, and as of
this writing runs on **wifi** networking
(`06-pve-test-wifi-network.md`), which is directly incompatible with
corosync's latency/jitter requirements below. `pve-test` and `pve1` were
briefly clustered and then deliberately de-clustered for exactly this
reason. Don't treat `pve-test` as progress toward Stage 2 without a
deliberate decision to rebuild its networking first.
### End goal
3-node Proxmox VE cluster with HA-managed VMs backed by **Ceph** — true
+11
View File
@@ -11,6 +11,17 @@ PVE datacenter firewall (default-deny inbound otherwise). That's
sufficient until Stage 2 needs actual separate physical/VLAN paths for
corosync and Ceph traffic — see below.
## pve-test as built (sandbox, current)
Different node, different design, not a Stage 1/2 example to generalize
from: `pve-test` runs `vmbr0` bridged over a wifi NIC in 4addr client mode
(active-backup bonded with a wired NIC as an automatic fallback). Full
detail, including why this is normally impossible and how it was
validated before trusting it with the management IP, in
`06-pve-test-wifi-network.md`. This is intentionally a one-off for a
standalone sandbox box — never extend it to a node that's clustered or
Ceph-connected (see the Stage 2 note in `00-overview.md`).
## Target design (Stage 2, future cluster)
## Required separation
+17
View File
@@ -54,6 +54,23 @@ Default-deny at datacenter/node level, whitelist only what's needed (see
`config/pve-firewall/cluster.fw.example`, applied by
`scripts/deploy-firewall.sh`.
**ICMP echo (ping) is explicitly allowed from the management network**,
alongside SSH/8006 — not required for anything to function, but without
it `policy_in: DROP` silently eats ping while SSH/web UI keep working.
That split (ping dead, everything else fine) reads exactly like a real
outage mid-troubleshooting; see `06-pve-test-wifi-network.md` for a case
this caused genuine confusion after a network change. If you ever debug
"can't ping but can SSH", check this firewall before assuming the
network itself is broken.
**This file is cluster-wide, not per-node**: `cluster.fw` lives in
`/etc/pve/firewall/` — shared via pmxcfs across every node in a cluster.
A node that joins a cluster inherits whatever's already there, and (per
`docs/05-node-roles.md`) keeps its own local copy after leaving. Don't
assume a node's firewall state matches what `deploy-firewall.sh` was
last run with directly against it — check `pve-firewall status` /
`/etc/pve/firewall/cluster.fw` on the actual node.
## Repos and updates
Fresh installs point at the enterprise repo, which fails on `apt update`
+83
View File
@@ -0,0 +1,83 @@
# Node Roles: pve1 and pve-test
Two Proxmox VE hosts exist on the LAN. They are **not interchangeable**
different purpose, different risk tolerance, different guardrails. See
`CLAUDE.md` for the actual rules Claude follows; this doc is the
background/context those rules assume.
## pve1 (production)
ASUS PN53 mini PC (Ryzen 7 7735HS, 32GB RAM) — see `01-hardware-node1.md`
for the full hardware layout. Stage 1 base hardening applied and verified
(`04-security-hardening.md`).
Runs real production workloads:
- `server-nixos` (VM) and LXC containers `pihole`, `claude`, `nix-cache`,
`docker`, `pdm`, `pxe-boot` — DNS, this Claude Code environment itself,
the LAN's Nix binary cache, etc.
Off-limits to Claude by default — see `CLAUDE.md`.
## pve-test (sandbox)
A separate physical node kept for testing. Has a capable GPU (earmarked
for future GPU workloads — transcoding, local inference, etc. — nothing
built on it yet as of this writing). No production guests run here.
Safe for Claude to create/destroy scratch VMs and containers on without
asking — see `CLAUDE.md` for the exact boundary (node-level config is
still not free-for-all even here).
### History: briefly clustered with pve1, then deliberately separated
On 2026-07-21/22, pve-test was joined to pve1 in a 2-node Proxmox cluster
(corosync + shared `/etc/pve`) for a period, then the cluster was
**intentionally destroyed** at the operator's explicit request — not a
failure or accident. Rationale: the operator wanted pve-test moved onto
wifi networking (see `06-pve-test-wifi-network.md`), and wifi is
fundamentally unsuitable for corosync's latency/jitter requirements (token
timeouts, flapping membership). Rather than fight that, the cluster was
torn down first, then wifi was configured on the now-standalone node.
Teardown procedure used (safe to repeat if pve-test or pve1 is ever
reclustered and needs separating again):
1. Un-manage any HA-assigned resources first if HA is in use
(`ha-manager remove <sid>` per resource) — **critical**: if a node has
HA-managed guests and its watchdog is still armed when it loses
quorum, HA's self-fence behavior can force-reboot it. Removing the
resource assignments first (not stopping the guests — just
un-managing them) prevents this.
2. Stop `pve-ha-lrm`/`pve-ha-crm` on all nodes once no resources remain
assigned, confirm via `ha-manager status` that fencing is no longer
armed.
3. From a node that will remain in the cluster: `pvecm delnode <name>`
for the node being removed. In a 2-node cluster with no QDevice, this
can itself cause the *remaining* node to transiently lose quorum
(killing a node's live corosync membership drops total votes below
expected votes before the nodelist file can be rewritten to match) —
this is expected, not a failure; proceed to step 4 regardless.
4. On **each** node being separated (including one you just ran `delnode`
from, if it's also being decoupled): `systemctl stop pve-cluster
corosync`, `pmxcfs -l` (local mode, bypasses the quorum requirement
for local writes), remove `/etc/pve/corosync.conf` and
`/etc/corosync/*`, `killall pmxcfs`, `systemctl start pve-cluster`.
This is the official "separate a node without reinstalling" procedure
and doesn't touch any running VM/CT — they're independent OS
processes, unaffected by corosync/pmxcfs state either way.
5. Optional cosmetic cleanup: the node(s) staying up may retain a stale
`/etc/pve/nodes/<removed-node>/` directory (cached VM/CT configs from
when they were clustered) — safe to `rm -rf` once separation is
confirmed; it's dead data, not a live reference to anything.
Guests were never at risk during this procedure (confirmed: all 7 of
pve1's guests ran continuously throughout) — the only real hazard is the
HA/watchdog self-fence path in step 1, and losing corosync-provided quorum
temporarily blocking `/etc/pve` writes (not guest execution) in step 3.
### Current state: standalone, wifi-primary networking
pve-test now runs entirely independently of pve1, on the network design
documented in `06-pve-test-wifi-network.md`. **Do not re-cluster
pve-test while it's on wifi** — see that doc for why.
+230
View File
@@ -0,0 +1,230 @@
# pve-test's Wifi-Primary Network
Non-standard, deliberately chosen against normal Proxmox guidance (a
hypervisor's management network is not supposed to be wireless). Applies
**only** to `pve-test` as a standalone sandbox node — see
`05-node-roles.md` for why this and cluster membership don't mix, and
never apply this pattern to a node that's joined (or might join) a
cluster.
## Why this works at all
Wifi (802.11) normally cannot be bridged the way Ethernet can: an access
point only accepts frames whose source MAC matches the MAC that
associated with it. A Linux bridge with a wifi port as a member sends
frames tagged with whatever MAC actually generated them (the bridge's own
MAC, a VM's MAC, etc.) — none of which match the wifi card's hardware
MAC, so the AP silently drops them. This isn't a Linux/Proxmox limitation,
it's how WiFi association works.
**4addr (WDS) client mode** is the exception: if both the wifi driver and
the AP support it, the client tags frames with a 4th address field
carrying the original MAC, and the AP forwards them like a real Ethernet
segment. Most consumer routers do **not** support this (it's common on
OpenWrt/DD-WRT/enterprise APs, rare on stock ISP hardware) — it has to be
confirmed empirically, per-AP, before trusting it with a management IP.
pve-test's card (Intel AX200, `iwlwifi` driver, interface `wlp3s0`)
supports 4addr client-side generically (this is a `mac80211` core
feature, not driver-specific). The home AP it associates to
(`nbn-fttp-net-5G`) was confirmed to honor it — see validation method
below.
## Validate before ever touching a live management IP
Don't add the wifi NIC straight into your real bridge to "see if it
works" — if 4addr isn't actually honored, you lose access to the box with
no diagnostic trail (association succeeds; only the *data plane* silently
fails). Test in an isolated scratch namespace first:
```bash
# assumes wlp3s0 already associated with 4addr on (see systemd unit below)
ip link add testbr0 type bridge && ip link set testbr0 up
ip link set wlp3s0 master testbr0 && ip link set wlp3s0 up
ip netns add wifitest
ip link add veth-host type veth peer name veth-ns
ip link set veth-host master testbr0 && ip link set veth-host up
ip link set veth-ns netns wifitest
ip netns exec wifitest ip link set veth-ns address 02:11:22:33:44:55 # deliberately NOT the wifi card's MAC
ip netns exec wifitest ip link set veth-ns up
ip netns exec wifitest ip link set lo up
ip netns exec wifitest dhclient -v -1 veth-ns # real DHCP lease = AP forwards foreign MACs = 4addr genuinely works
```
If that gets a lease (or at minimum a ping reply from the gateway) from a
MAC that was never the one that associated, the AP is forwarding
arbitrary-MAC frames — real bridge mode will work. If not, don't proceed
to bridge mode; fall back to a routed/NAT design instead (wifi gets its
own IP via DHCP, VMs NAT out through it — never covered here since it
wasn't needed, but keep this fallback in mind if reproducing on different
hardware/AP).
Tear the test namespace/bridge down afterwards (`ip netns del wifitest;
ip link del veth-host; ip link set wlp3s0 nomaster; ip link del testbr0`)
— it's scratch, not part of the real config.
## Final architecture
```
wlp3s0 (wifi, 4addr client mode) ─┐
├─ bond0 (active-backup, wlp3s0 primary) ─ vmbr0 (192.168.2.251/24)
nic0 (wired, LAN backup) ─┘
```
- **`wpa-4addr-<iface>.service`** (systemd unit, `Before=network-pre.target`) —
sets 4addr mode and starts `wpa_supplicant` *before* ifupdown2 processes
`/etc/network/interfaces`. Deliberately not using `wpasupplicant`'s own
`wpa-conf` ifupdown2 hook integration — its ordering relative to a
custom `pre-up iw ... set 4addr on` line is implementation-specific and
not worth gambling on for something this hard to debug if it's wrong.
The systemd unit gives full, deterministic control over sequencing
instead.
- **`bond0`** — active-backup bonding, `wlp3s0` as `bond-primary`,
`nic0` as backup, `bond-miimon 100`. Bonding a wifi interface works here
because `mac80211`/`iwlwifi` properly reports carrier state
(`netif_carrier_on/off`) on association/disassociation, which is what
bonding's `miimon` (with the default `use_carrier=1`) actually watches
— it doesn't require a "real" MII-capable NIC.
- **`vmbr0`** — unchanged IP (`192.168.2.251/24`), bridged over `bond0`
instead of directly over a physical NIC.
Reproduce with `scripts/setup-wifi-bond-network.sh` — see its header
comment for usage and required env vars (SSID/passphrase, management
address/gateway). It stages `/etc/network/interfaces` and the systemd
unit but does **not** auto-apply (`ifreload -a`) — see the next section
for why that step needs care, not automation.
## Applying a live bridge-port change: what went wrong once, and what worked
Moving `vmbr0`'s underlying port live — while SSH'd in over the very
address that lives on that bridge — is inherently risky: if the new port
doesn't actually pass traffic, you lose the connection you're using to
fix it.
**What failed**: doing the port swap via raw `ip link set nic0 nomaster`
+ `ip link set nic0 down` (with `wlp3s0` already added as a second bridge
member) caused an extended outage that needed a physical power-cycle to
recover — `nic0` never came back up on its own even well past a 60s
auto-revert watchdog's deadline. Importantly, the *host itself* never
crashed — `journalctl -b -1` showed completely normal operation (a
`pvestatd` polling loop, no gaps, no panic/OOM/watchdog trigger) right up
until a keyboard was plugged in and it was manually reset. So the failure
was specifically in the live network transition, not the OS — root cause
unconfirmed, but plausibly bridge FDB/MAC-identity handling when a port
carrying an already-live IP is swapped out via raw `ip link` rather than
a coordinated reconfiguration.
**What worked**: writing the final state into `/etc/network/interfaces`
and applying with `ifreload -a` (Proxmox's own supported hot-reload path)
instead. Same end state, same risk window, but ifupdown2 evidently
sequences the transition in a way raw `ip link` surgery didn't. This
succeeded on the first retry using this method.
**Either way, expect ~30-60 seconds of apparent breakage even on a
successful change** — the upstream switch/AP needs to relearn which port
`192.168.2.251`'s MAC now lives behind. Don't judge success/failure
before that window passes; a `ping`/SSH failure at the 5-10s mark is not
yet a sign anything is wrong.
**Recommended safety net for any future live change here**: arm a
backgrounded auto-revert-on-timeout before applying, e.g.:
```bash
cp /etc/network/interfaces /etc/network/interfaces.bak
cat > /root/revert.sh <<'EOF'
#!/bin/bash
cp /etc/network/interfaces.bak /etc/network/interfaces
ifreload -a
EOF
chmod +x /root/revert.sh
setsid nohup bash -c "sleep 45 && /root/revert.sh" >/root/revert.log 2>&1 < /dev/null &
disown
# now apply the real change, e.g.: ifreload -a
# once confirmed working: pkill -f "sleep 45 && /root/revert.sh"
```
This only self-heals if the box stays responsive enough for the
backgrounded job to run to completion — it is not a substitute for
physical/console access being available, just a way to avoid needing it
for the common case.
## Troubleshooting: "can't ping it, web UI won't load" isn't necessarily an outage
Two independent things can make `pve-test` look dead when it isn't:
1. **Stale ARP after any change to `vmbr0`'s active port.** `vmbr0`'s MAC
address follows whichever interface is currently active in `bond0`
(`wlp3s0`'s MAC while wifi is primary, `nic0`'s if it fails over) — it
is *not* fixed. Other devices on the LAN (including your own
workstation) that cached the old MAC before a change won't notice
until their ARP entry naturally expires or gets flushed
(`ip neigh flush <ip>` / `arp -d <ip>`). Symptom: intermittent or
totally dead connectivity from one specific device while others (or
the host itself) are fine.
2. **ICMP ping is blocked by the management firewall, unrelated to
network health.** See `04-security-hardening.md`'s firewall section —
`cluster.fw`'s `policy_in: DROP` only explicitly allows TCP 22/8006
(now also ICMP echo-request, after this was hit) from the mgmt
network. Before that rule was added, `ping` failed 100% *even with a
perfectly correct, fresh ARP entry and full SSH/web UI access* —
nothing to do with wifi, bonding, or the network being down. This is
exactly what happened once: alarming "can't ping, web UI not loading"
turned out to be a stale-ARP moment (which resolved itself) plus a
pre-existing firewall policy (ping was never going to work,
regardless of wifi).
**When `pve-test` seems unreachable, check in this order before assuming
a real outage**: SSH (`ssh root@pve-test.sweet.home`) → web UI via `curl
-sk -o /dev/null -w '%{http_code}' https://pve-test.sweet.home:8006/`
only then worry about `ping` specifically, and check
`/etc/pve/firewall/cluster.fw` before blaming the network.
## Confirmed limitation: no wireless client on the same AP can reach pve-test
**This is not an AP-isolation toggle** — tested and ruled out (router:
TP-Link BE9300, AP Isolation confirmed unchecked). Two independent
wireless devices (a laptop and a phone) on the same SSID pve-test uses
(`nbn-fttp-net-5G`) were both completely unable to reach it — not
intermittent, not slow, no ARP entry ever resolves (`arp -a` on Windows /
`ip neigh` on Linux/Mac shows nothing at all, not a stale entry). Every
wired device tested (this repo's own Claude session, `pve1`) reaches it
fine, every time.
Root cause: pve-test's wifi connection uses 4-address (WDS) framing to
make bridging possible at all (see above) — that's what lets the AP
forward frames from arbitrary MACs (VM traffic, the bridge's own
identity) instead of only the wifi card's own hardware MAC. This
router's firmware evidently handles wired↔pve-test forwarding for that
4addr peer correctly, but doesn't correctly forward *wireless-client*
broadcast/ARP traffic to it — an asymmetric gap in the AP's own 4addr
handling, not a policy setting. Likely not fixable short of a firmware
update from TP-Link (if one ever improves 4addr/WDS handling), possibly
not fixable on this hardware at all.
**Practical implication**: anything on `vmbr0` — pve-test itself, or any
VM/CT created on it — is unreachable from a wireless client on this AP,
full stop, for *any* protocol, not just ICMP. ARP resolution is a
prerequisite for sending any IP packet at all (TCP included); since ARP
itself never resolves for a wireless peer here, the web UI (8006) fails
identically to ping, not just ping — confirmed live, not theoretical.
This includes VMs on pve-test too, since they ride the same bridge/wifi
uplink as the host. **Wired access is the only reliable path to
pve-test and anything running on it** — including reaching a VM's
console through the Proxmox web UI, which still requires the accessing
device to reach pve-test's own IP on port 8006 first, so it inherits the
same requirement.
Don't waste time on ARP flushes, firewall rules, or bonding config for
this specific symptom (no ARP entry from a wireless peer, wired fine) —
none of those are the cause.
## Known limitation history
`nic0` (the bond's backup slave) initially had no cable physically
connected when this was first set up — the bond was correctly configured
but inert until a cable was plugged in. As of this writing a cable is
connected and the backup slave is live (confirm with `cat
/proc/net/bonding/bond0`) — if reproducing this setup, verify the same
before assuming LAN failover will actually work.
+5
View File
@@ -58,6 +58,11 @@ if [ -f /etc/pve/firewall/cluster.fw ] && grep -qi '^policy_in:\s*DROP' /etc/pve
else
audit_fail "cluster.fw missing or does not default-deny inbound"
fi
if [ -f /etc/pve/firewall/cluster.fw ] && grep -qi 'icmp-type echo-request' /etc/pve/firewall/cluster.fw 2>/dev/null; then
audit_pass "cluster.fw allows ICMP echo-request from mgmt (ping works)"
else
audit_fail "cluster.fw does not allow ping from mgmt - see docs/04-security-hardening.md firewall section"
fi
# --- unattended-upgrades ---
if dpkg -s unattended-upgrades >/dev/null 2>&1 && systemctl is-enabled --quiet unattended-upgrades 2>/dev/null; then
+133
View File
@@ -0,0 +1,133 @@
#!/bin/bash
# Reproduces pve-test's wifi-primary networking: vmbr0 bridged over a wifi
# NIC in 4addr (WDS) client-bridge mode, active-backup bonded with a wired
# NIC as an automatic LAN fallback. See docs/06-pve-test-wifi-network.md for
# why this exists and how it was validated. Idempotent - safe to re-run.
#
# Requires: a wifi NIC whose driver/AP both support 4addr mode (verify with
# docs/06-pve-test-wifi-network.md's isolated-namespace test *before*
# trusting this against a live management IP - a wifi NIC or AP that
# doesn't support 4addr will associate fine but silently drop bridged
# frames from any MAC other than the card's own).
#
# Usage (run as root on the target PVE host):
# WIFI_SSID="..." WIFI_PASSPHRASE="..." \
# MGMT_ADDR=192.168.2.251/24 MGMT_GATEWAY=192.168.2.254 \
# ./setup-wifi-bond-network.sh
#
# Optional overrides: WIFI_IFACE (default wlp3s0), LAN_IFACE (default nic0)
set -euo pipefail
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
# shellcheck source=lib/common.sh
source "${SCRIPT_DIR}/lib/common.sh"
require_root
WIFI_IFACE="${WIFI_IFACE:-wlp3s0}"
LAN_IFACE="${LAN_IFACE:-nic0}"
for var in WIFI_SSID WIFI_PASSPHRASE MGMT_ADDR MGMT_GATEWAY; do
if [ -z "${!var:-}" ]; then
echo "$var is not set. See usage in this script's header." >&2
exit 1
fi
done
if ! ip link show "$WIFI_IFACE" >/dev/null 2>&1; then
echo "No interface named $WIFI_IFACE on this host. Run 'ip -br link' and set WIFI_IFACE=..." >&2
exit 1
fi
if ! ip link show "$LAN_IFACE" >/dev/null 2>&1; then
echo "No interface named $LAN_IFACE on this host. Run 'ip -br link' and set LAN_IFACE=..." >&2
exit 1
fi
echo "=== 1/4: wifi tooling ==="
apt-get install -y iw wpasupplicant >/dev/null
echo "installed iw, wpasupplicant"
echo
echo "=== 2/4: wpa_supplicant config (SSID: $WIFI_SSID, iface: $WIFI_IFACE) ==="
WPA_CONF="/etc/wpa_supplicant/wpa_supplicant-${WIFI_IFACE}.conf"
wpa_passphrase "$WIFI_SSID" "$WIFI_PASSPHRASE" > "$WPA_CONF"
sed -i '/^\s*#psk=/d' "$WPA_CONF"
chmod 600 "$WPA_CONF"
echo "wrote $WPA_CONF (passphrase hashed, not stored in plaintext)"
echo
echo "=== 3/4: systemd unit to set 4addr mode + start wpa_supplicant ==="
UNIT="/etc/systemd/system/wpa-4addr-${WIFI_IFACE}.service"
write_if_changed "$UNIT" "[Unit]
Description=wpa_supplicant on ${WIFI_IFACE} with 4addr mode enabled
Before=network-pre.target
Wants=network-pre.target
[Service]
Type=simple
ExecStartPre=/sbin/ip link set ${WIFI_IFACE} down
ExecStartPre=/sbin/iw dev ${WIFI_IFACE} set 4addr on
ExecStartPre=/sbin/ip link set ${WIFI_IFACE} up
ExecStart=/sbin/wpa_supplicant -i ${WIFI_IFACE} -c ${WPA_CONF}
Restart=on-failure
RestartSec=3
[Install]
WantedBy=multi-user.target"
systemctl daemon-reload
systemctl enable --now "wpa-4addr-${WIFI_IFACE}.service"
sleep 5
if ! iw dev "$WIFI_IFACE" link | grep -q "^Connected"; then
echo "WARNING: ${WIFI_IFACE} did not associate to '$WIFI_SSID' within 5s - check:" >&2
echo " systemctl status wpa-4addr-${WIFI_IFACE}.service" >&2
echo " journalctl -u wpa-4addr-${WIFI_IFACE}.service" >&2
exit 1
fi
echo "associated: $(iw dev "$WIFI_IFACE" link | grep '^Connected')"
echo
echo "=== 4/4: /etc/network/interfaces (bond0 active-backup: ${WIFI_IFACE} primary, ${LAN_IFACE} backup) ==="
IFACES_FILE="/etc/network/interfaces"
backup_file "$IFACES_FILE"
cat > "$IFACES_FILE" <<EOF
auto lo
iface lo inet loopback
iface ${LAN_IFACE} inet manual
iface ${WIFI_IFACE} inet manual
auto bond0
iface bond0 inet manual
bond-slaves ${WIFI_IFACE} ${LAN_IFACE}
bond-mode active-backup
bond-miimon 100
bond-primary ${WIFI_IFACE}
bond-updelay 200
bond-downdelay 200
auto vmbr0
iface vmbr0 inet static
address ${MGMT_ADDR}
gateway ${MGMT_GATEWAY}
bridge-ports bond0
bridge-stp off
bridge-fd 0
source /etc/network/interfaces.d/*
EOF
echo "wrote $IFACES_FILE"
echo
echo "Config staged but NOT applied yet - applying it live can drop your"
echo "current management connection for ~30-60s while the switch/AP"
echo "relearns MAC locations (expected, self-resolves; see"
echo "docs/06-pve-test-wifi-network.md). Recommended: run this from the"
echo "physical console, or arm a revert-on-timeout watchdog first, e.g.:"
echo
echo " cp ${IFACES_FILE}.bak.* /tmp/interfaces.orig # pick the backup just made"
echo " (sleep 45 && cp /tmp/interfaces.orig ${IFACES_FILE} && ifreload -a) &"
echo " ifreload -a"
echo " # then kill the backgrounded revert job once you confirm connectivity"
echo
echo "Apply now with: ifreload -a"
echo "Verify after with: cat /proc/net/bonding/bond0 ; ip -4 -br addr show vmbr0"