Archived
Document pve-test's wifi network, add Claude guardrails, fix ping firewall #2
@@ -0,0 +1,84 @@
|
|||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
Guidance for Claude Code working in this repo. IMPORTANT: these
|
||||||
|
instructions OVERRIDE any default behavior and must be followed exactly
|
||||||
|
as written.
|
||||||
|
|
||||||
|
## Repo purpose
|
||||||
|
|
||||||
|
Base configuration/hardening toolset and planning docs for Proxmox VE
|
||||||
|
hosts (`scripts/`, `config/`, `docs/`) — see `README.md` and
|
||||||
|
`docs/00-overview.md`. Scripts in this repo are meant to be run **on**
|
||||||
|
the target Proxmox host itself (as root), not orchestrated remotely.
|
||||||
|
|
||||||
|
## Two Proxmox nodes: `pve1` (production) and `pve-test` (sandbox)
|
||||||
|
|
||||||
|
Two SSH-reachable Proxmox nodes exist on the LAN. They are **not
|
||||||
|
interchangeable** — see `docs/05-node-roles.md` for full background on
|
||||||
|
what each one is and why.
|
||||||
|
|
||||||
|
### `pve1` (production — off-limits to Claude by default)
|
||||||
|
|
||||||
|
A real, live Proxmox node hosting production VMs/containers (see
|
||||||
|
`docs/05-node-roles.md` for the current guest list) — not a sandbox, and
|
||||||
|
not Claude's to touch by default.
|
||||||
|
|
||||||
|
- **Off-limits at all times unless the operator has given explicit,
|
||||||
|
same-session instructions to act on this specific host.** That
|
||||||
|
authorization is scoped to the task it was given for — don't carry it
|
||||||
|
forward to unrelated later work in the same conversation, and never
|
||||||
|
assume it from a previous session.
|
||||||
|
- **Read-only for existing state is always fine, authorization or not.**
|
||||||
|
SSH in (or use `pvesm`, `qm list`, `pct list`, `qm config`, `pct
|
||||||
|
config`, the Proxmox API, etc.) to inspect config, storage, and any
|
||||||
|
existing VM/container freely.
|
||||||
|
- **Never** modify, stop, restart, delete, reconfigure, or create
|
||||||
|
anything on this node (`qm set`, `pct set`, `qm destroy`, `pct
|
||||||
|
destroy`, `qm stop`, `pct stop`, `qm create`, `pct create`, snapshot
|
||||||
|
operations, storage changes, running any script in this repo against
|
||||||
|
it, etc.) without that explicit go-ahead. Use `pve-test` for anything
|
||||||
|
exploratory instead.
|
||||||
|
- If a guest on `pve1` is HA-managed, be aware of the self-fence hazard
|
||||||
|
described in `docs/05-node-roles.md`'s cluster-teardown section before
|
||||||
|
doing anything that could cost the node quorum.
|
||||||
|
|
||||||
|
### `pve-test` (sandbox — Claude's default target)
|
||||||
|
|
||||||
|
A separate node set aside for testing — safe to create, interrogate, and
|
||||||
|
destroy scratch VMs/containers on without asking first.
|
||||||
|
|
||||||
|
- **Test VMs/containers are allowed, but must be torn down.** Anything
|
||||||
|
created this way must be destroyed again in the same session, before
|
||||||
|
ending the task. Use an obviously-scratch VMID/name.
|
||||||
|
- **Node-level config is still not yours to change by default.**
|
||||||
|
Creating/destroying your own scratch guests is fine; Proxmox host
|
||||||
|
config, storage pools, and networking on `pve-test` itself need the
|
||||||
|
operator's explicit go-ahead too, same as on `pve1` — the "sandbox"
|
||||||
|
status covers guest-level experimentation, not the host's own
|
||||||
|
identity. (`pve-test`'s current network config, `docs/06-pve-test-wifi-network.md`,
|
||||||
|
was applied under exactly that kind of explicit, same-session
|
||||||
|
authorization — it's not a standing invitation to keep changing it
|
||||||
|
further without asking again.)
|
||||||
|
- **Never re-cluster `pve-test` with `pve1`** without the operator
|
||||||
|
explicitly asking for it and being aware of the wifi/corosync
|
||||||
|
incompatibility in `docs/06-pve-test-wifi-network.md` — the two were
|
||||||
|
deliberately de-clustered for this reason once already.
|
||||||
|
|
||||||
|
## Safety rules
|
||||||
|
|
||||||
|
- Any script in `scripts/` that isn't `audit.sh` (read-only) makes real
|
||||||
|
changes when run for real. Don't run one against `pve1`, or against
|
||||||
|
`pve-test`'s node-level config, without the same-session go-ahead
|
||||||
|
described above. Running a script *against pve-test's own guests* — a
|
||||||
|
scratch VM/CT you created for this task — doesn't need separate
|
||||||
|
permission.
|
||||||
|
- Do not commit secrets: SSH private keys, wifi passphrases, sops/age
|
||||||
|
keys, or PVE credentials. `scripts/setup-wifi-bond-network.sh` takes
|
||||||
|
the SSID/passphrase via environment variables for exactly this reason
|
||||||
|
— never hardcode them into the script or a committed config file.
|
||||||
|
- Live changes to a node's own management network (the interface/bridge
|
||||||
|
carrying the SSH session you're using) can strand the box — see the
|
||||||
|
"what went wrong once" section in `docs/06-pve-test-wifi-network.md`
|
||||||
|
before touching `pve-test`'s networking again. Prefer applying via
|
||||||
|
`ifreload -a` over raw `ip link` surgery, and arm an auto-revert
|
||||||
|
watchdog first when acting without someone physically at the console.
|
||||||
@@ -15,15 +15,22 @@ deferred).
|
|||||||
backed by Ceph. Needs dedicated hardware node 1 (`pve1`, an ASUS PN53 mini
|
backed by Ceph. Needs dedicated hardware node 1 (`pve1`, an ASUS PN53 mini
|
||||||
PC) doesn't have — see `docs/01-hardware-node1.md`.
|
PC) doesn't have — see `docs/01-hardware-node1.md`.
|
||||||
|
|
||||||
|
See `CLAUDE.md` for the guardrails Claude Code follows when working
|
||||||
|
against these hosts (`pve1` is production and off-limits by default;
|
||||||
|
`pve-test` is the sandbox).
|
||||||
|
|
||||||
## Repo layout
|
## Repo layout
|
||||||
|
|
||||||
- `docs/` — planning docs: hardware layout, storage migration, networking,
|
- `docs/` — planning docs: hardware layout, storage migration, networking,
|
||||||
security hardening. Read `docs/00-overview.md` first.
|
security hardening, node roles. Read `docs/00-overview.md` first, then
|
||||||
|
`docs/05-node-roles.md` for what `pve1`/`pve-test` actually are.
|
||||||
- `scripts/` — scripts to apply configuration on a node (SSH hardening, repo
|
- `scripts/` — scripts to apply configuration on a node (SSH hardening, repo
|
||||||
switch, firewall, updates, etc). Idempotent, safe to re-run.
|
switch, firewall, updates, wifi/bond networking, etc). Idempotent, safe
|
||||||
`scripts/bootstrap.sh` runs the full Stage 1 sequence end to end;
|
to re-run. `scripts/bootstrap.sh` runs the full Stage 1 sequence end to
|
||||||
`scripts/audit.sh` verifies it (read-only). `scripts/lib/` holds shared
|
end; `scripts/audit.sh` verifies it (read-only).
|
||||||
helpers (`common.sh`) sourced by the other scripts.
|
`scripts/setup-wifi-bond-network.sh` reproduces `pve-test`'s wifi
|
||||||
|
network (see `docs/06-pve-test-wifi-network.md`). `scripts/lib/` holds
|
||||||
|
shared helpers (`common.sh`) sourced by the other scripts.
|
||||||
- `config/` — reference config files/snippets to drop onto a node (firewall
|
- `config/` — reference config files/snippets to drop onto a node (firewall
|
||||||
rules, sshd config, etc.).
|
rules, sshd config, etc.).
|
||||||
|
|
||||||
|
|||||||
@@ -26,6 +26,13 @@ policy_out: ACCEPT
|
|||||||
IN ACCEPT -source +mgmt -p tcp -dport 8006 -log nolog
|
IN ACCEPT -source +mgmt -p tcp -dport 8006 -log nolog
|
||||||
IN ACCEPT -source +mgmt -p tcp -dport 22 -log nolog
|
IN ACCEPT -source +mgmt -p tcp -dport 22 -log nolog
|
||||||
|
|
||||||
|
# ICMP echo (ping) from the management network - diagnostic convenience
|
||||||
|
# only, nothing else depends on it. Without this, policy_in DROP silently
|
||||||
|
# eats ping while SSH/web UI keep working - looks like an outage during
|
||||||
|
# troubleshooting when the host is actually fine. See
|
||||||
|
# docs/06-pve-test-wifi-network.md for a case this caused real confusion.
|
||||||
|
IN ACCEPT -source +mgmt -p icmp -icmp-type echo-request -log nolog
|
||||||
|
|
||||||
# Stage 2: Corosync (cluster quorum) - uncomment once node 2/3 join and
|
# Stage 2: Corosync (cluster quorum) - uncomment once node 2/3 join and
|
||||||
# the corosync network/VLAN exists.
|
# the corosync network/VLAN exists.
|
||||||
# IN ACCEPT -source <COROSYNC_CIDR> -p udp -dport 5404:5405 -log nolog
|
# IN ACCEPT -source <COROSYNC_CIDR> -p udp -dport 5404:5405 -log nolog
|
||||||
|
|||||||
@@ -59,6 +59,16 @@ require or assume the split-disk/multi-NIC layout Stage 2 wants.
|
|||||||
Everything below this point is the target design for when nodes 2 and 3
|
Everything below this point is the target design for when nodes 2 and 3
|
||||||
are actually being provisioned. Not applicable to `pve1` as it stands.
|
are actually being provisioned. Not applicable to `pve1` as it stands.
|
||||||
|
|
||||||
|
**`pve-test` existing does not mean node 2 exists.** A second physical
|
||||||
|
node (`pve-test`) does run alongside `pve1` — see `05-node-roles.md` —
|
||||||
|
but it's a sandbox/test box, not built to this Stage 2 design, and as of
|
||||||
|
this writing runs on **wifi** networking
|
||||||
|
(`06-pve-test-wifi-network.md`), which is directly incompatible with
|
||||||
|
corosync's latency/jitter requirements below. `pve-test` and `pve1` were
|
||||||
|
briefly clustered and then deliberately de-clustered for exactly this
|
||||||
|
reason. Don't treat `pve-test` as progress toward Stage 2 without a
|
||||||
|
deliberate decision to rebuild its networking first.
|
||||||
|
|
||||||
### End goal
|
### End goal
|
||||||
|
|
||||||
3-node Proxmox VE cluster with HA-managed VMs backed by **Ceph** — true
|
3-node Proxmox VE cluster with HA-managed VMs backed by **Ceph** — true
|
||||||
|
|||||||
@@ -11,6 +11,17 @@ PVE datacenter firewall (default-deny inbound otherwise). That's
|
|||||||
sufficient until Stage 2 needs actual separate physical/VLAN paths for
|
sufficient until Stage 2 needs actual separate physical/VLAN paths for
|
||||||
corosync and Ceph traffic — see below.
|
corosync and Ceph traffic — see below.
|
||||||
|
|
||||||
|
## pve-test as built (sandbox, current)
|
||||||
|
|
||||||
|
Different node, different design, not a Stage 1/2 example to generalize
|
||||||
|
from: `pve-test` runs `vmbr0` bridged over a wifi NIC in 4addr client mode
|
||||||
|
(active-backup bonded with a wired NIC as an automatic fallback). Full
|
||||||
|
detail, including why this is normally impossible and how it was
|
||||||
|
validated before trusting it with the management IP, in
|
||||||
|
`06-pve-test-wifi-network.md`. This is intentionally a one-off for a
|
||||||
|
standalone sandbox box — never extend it to a node that's clustered or
|
||||||
|
Ceph-connected (see the Stage 2 note in `00-overview.md`).
|
||||||
|
|
||||||
## Target design (Stage 2, future cluster)
|
## Target design (Stage 2, future cluster)
|
||||||
|
|
||||||
## Required separation
|
## Required separation
|
||||||
|
|||||||
@@ -54,6 +54,23 @@ Default-deny at datacenter/node level, whitelist only what's needed (see
|
|||||||
`config/pve-firewall/cluster.fw.example`, applied by
|
`config/pve-firewall/cluster.fw.example`, applied by
|
||||||
`scripts/deploy-firewall.sh`.
|
`scripts/deploy-firewall.sh`.
|
||||||
|
|
||||||
|
**ICMP echo (ping) is explicitly allowed from the management network**,
|
||||||
|
alongside SSH/8006 — not required for anything to function, but without
|
||||||
|
it `policy_in: DROP` silently eats ping while SSH/web UI keep working.
|
||||||
|
That split (ping dead, everything else fine) reads exactly like a real
|
||||||
|
outage mid-troubleshooting; see `06-pve-test-wifi-network.md` for a case
|
||||||
|
this caused genuine confusion after a network change. If you ever debug
|
||||||
|
"can't ping but can SSH", check this firewall before assuming the
|
||||||
|
network itself is broken.
|
||||||
|
|
||||||
|
**This file is cluster-wide, not per-node**: `cluster.fw` lives in
|
||||||
|
`/etc/pve/firewall/` — shared via pmxcfs across every node in a cluster.
|
||||||
|
A node that joins a cluster inherits whatever's already there, and (per
|
||||||
|
`docs/05-node-roles.md`) keeps its own local copy after leaving. Don't
|
||||||
|
assume a node's firewall state matches what `deploy-firewall.sh` was
|
||||||
|
last run with directly against it — check `pve-firewall status` /
|
||||||
|
`/etc/pve/firewall/cluster.fw` on the actual node.
|
||||||
|
|
||||||
## Repos and updates
|
## Repos and updates
|
||||||
|
|
||||||
Fresh installs point at the enterprise repo, which fails on `apt update`
|
Fresh installs point at the enterprise repo, which fails on `apt update`
|
||||||
|
|||||||
@@ -0,0 +1,83 @@
|
|||||||
|
# Node Roles: pve1 and pve-test
|
||||||
|
|
||||||
|
Two Proxmox VE hosts exist on the LAN. They are **not interchangeable** —
|
||||||
|
different purpose, different risk tolerance, different guardrails. See
|
||||||
|
`CLAUDE.md` for the actual rules Claude follows; this doc is the
|
||||||
|
background/context those rules assume.
|
||||||
|
|
||||||
|
## pve1 (production)
|
||||||
|
|
||||||
|
ASUS PN53 mini PC (Ryzen 7 7735HS, 32GB RAM) — see `01-hardware-node1.md`
|
||||||
|
for the full hardware layout. Stage 1 base hardening applied and verified
|
||||||
|
(`04-security-hardening.md`).
|
||||||
|
|
||||||
|
Runs real production workloads:
|
||||||
|
|
||||||
|
- `server-nixos` (VM) and LXC containers `pihole`, `claude`, `nix-cache`,
|
||||||
|
`docker`, `pdm`, `pxe-boot` — DNS, this Claude Code environment itself,
|
||||||
|
the LAN's Nix binary cache, etc.
|
||||||
|
|
||||||
|
Off-limits to Claude by default — see `CLAUDE.md`.
|
||||||
|
|
||||||
|
## pve-test (sandbox)
|
||||||
|
|
||||||
|
A separate physical node kept for testing. Has a capable GPU (earmarked
|
||||||
|
for future GPU workloads — transcoding, local inference, etc. — nothing
|
||||||
|
built on it yet as of this writing). No production guests run here.
|
||||||
|
|
||||||
|
Safe for Claude to create/destroy scratch VMs and containers on without
|
||||||
|
asking — see `CLAUDE.md` for the exact boundary (node-level config is
|
||||||
|
still not free-for-all even here).
|
||||||
|
|
||||||
|
### History: briefly clustered with pve1, then deliberately separated
|
||||||
|
|
||||||
|
On 2026-07-21/22, pve-test was joined to pve1 in a 2-node Proxmox cluster
|
||||||
|
(corosync + shared `/etc/pve`) for a period, then the cluster was
|
||||||
|
**intentionally destroyed** at the operator's explicit request — not a
|
||||||
|
failure or accident. Rationale: the operator wanted pve-test moved onto
|
||||||
|
wifi networking (see `06-pve-test-wifi-network.md`), and wifi is
|
||||||
|
fundamentally unsuitable for corosync's latency/jitter requirements (token
|
||||||
|
timeouts, flapping membership). Rather than fight that, the cluster was
|
||||||
|
torn down first, then wifi was configured on the now-standalone node.
|
||||||
|
|
||||||
|
Teardown procedure used (safe to repeat if pve-test or pve1 is ever
|
||||||
|
reclustered and needs separating again):
|
||||||
|
|
||||||
|
1. Un-manage any HA-assigned resources first if HA is in use
|
||||||
|
(`ha-manager remove <sid>` per resource) — **critical**: if a node has
|
||||||
|
HA-managed guests and its watchdog is still armed when it loses
|
||||||
|
quorum, HA's self-fence behavior can force-reboot it. Removing the
|
||||||
|
resource assignments first (not stopping the guests — just
|
||||||
|
un-managing them) prevents this.
|
||||||
|
2. Stop `pve-ha-lrm`/`pve-ha-crm` on all nodes once no resources remain
|
||||||
|
assigned, confirm via `ha-manager status` that fencing is no longer
|
||||||
|
armed.
|
||||||
|
3. From a node that will remain in the cluster: `pvecm delnode <name>`
|
||||||
|
for the node being removed. In a 2-node cluster with no QDevice, this
|
||||||
|
can itself cause the *remaining* node to transiently lose quorum
|
||||||
|
(killing a node's live corosync membership drops total votes below
|
||||||
|
expected votes before the nodelist file can be rewritten to match) —
|
||||||
|
this is expected, not a failure; proceed to step 4 regardless.
|
||||||
|
4. On **each** node being separated (including one you just ran `delnode`
|
||||||
|
from, if it's also being decoupled): `systemctl stop pve-cluster
|
||||||
|
corosync`, `pmxcfs -l` (local mode, bypasses the quorum requirement
|
||||||
|
for local writes), remove `/etc/pve/corosync.conf` and
|
||||||
|
`/etc/corosync/*`, `killall pmxcfs`, `systemctl start pve-cluster`.
|
||||||
|
This is the official "separate a node without reinstalling" procedure
|
||||||
|
and doesn't touch any running VM/CT — they're independent OS
|
||||||
|
processes, unaffected by corosync/pmxcfs state either way.
|
||||||
|
5. Optional cosmetic cleanup: the node(s) staying up may retain a stale
|
||||||
|
`/etc/pve/nodes/<removed-node>/` directory (cached VM/CT configs from
|
||||||
|
when they were clustered) — safe to `rm -rf` once separation is
|
||||||
|
confirmed; it's dead data, not a live reference to anything.
|
||||||
|
|
||||||
|
Guests were never at risk during this procedure (confirmed: all 7 of
|
||||||
|
pve1's guests ran continuously throughout) — the only real hazard is the
|
||||||
|
HA/watchdog self-fence path in step 1, and losing corosync-provided quorum
|
||||||
|
temporarily blocking `/etc/pve` writes (not guest execution) in step 3.
|
||||||
|
|
||||||
|
### Current state: standalone, wifi-primary networking
|
||||||
|
|
||||||
|
pve-test now runs entirely independently of pve1, on the network design
|
||||||
|
documented in `06-pve-test-wifi-network.md`. **Do not re-cluster
|
||||||
|
pve-test while it's on wifi** — see that doc for why.
|
||||||
@@ -0,0 +1,191 @@
|
|||||||
|
# pve-test's Wifi-Primary Network
|
||||||
|
|
||||||
|
Non-standard, deliberately chosen against normal Proxmox guidance (a
|
||||||
|
hypervisor's management network is not supposed to be wireless). Applies
|
||||||
|
**only** to `pve-test` as a standalone sandbox node — see
|
||||||
|
`05-node-roles.md` for why this and cluster membership don't mix, and
|
||||||
|
never apply this pattern to a node that's joined (or might join) a
|
||||||
|
cluster.
|
||||||
|
|
||||||
|
## Why this works at all
|
||||||
|
|
||||||
|
Wifi (802.11) normally cannot be bridged the way Ethernet can: an access
|
||||||
|
point only accepts frames whose source MAC matches the MAC that
|
||||||
|
associated with it. A Linux bridge with a wifi port as a member sends
|
||||||
|
frames tagged with whatever MAC actually generated them (the bridge's own
|
||||||
|
MAC, a VM's MAC, etc.) — none of which match the wifi card's hardware
|
||||||
|
MAC, so the AP silently drops them. This isn't a Linux/Proxmox limitation,
|
||||||
|
it's how WiFi association works.
|
||||||
|
|
||||||
|
**4addr (WDS) client mode** is the exception: if both the wifi driver and
|
||||||
|
the AP support it, the client tags frames with a 4th address field
|
||||||
|
carrying the original MAC, and the AP forwards them like a real Ethernet
|
||||||
|
segment. Most consumer routers do **not** support this (it's common on
|
||||||
|
OpenWrt/DD-WRT/enterprise APs, rare on stock ISP hardware) — it has to be
|
||||||
|
confirmed empirically, per-AP, before trusting it with a management IP.
|
||||||
|
|
||||||
|
pve-test's card (Intel AX200, `iwlwifi` driver, interface `wlp3s0`)
|
||||||
|
supports 4addr client-side generically (this is a `mac80211` core
|
||||||
|
feature, not driver-specific). The home AP it associates to
|
||||||
|
(`nbn-fttp-net-5G`) was confirmed to honor it — see validation method
|
||||||
|
below.
|
||||||
|
|
||||||
|
## Validate before ever touching a live management IP
|
||||||
|
|
||||||
|
Don't add the wifi NIC straight into your real bridge to "see if it
|
||||||
|
works" — if 4addr isn't actually honored, you lose access to the box with
|
||||||
|
no diagnostic trail (association succeeds; only the *data plane* silently
|
||||||
|
fails). Test in an isolated scratch namespace first:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# assumes wlp3s0 already associated with 4addr on (see systemd unit below)
|
||||||
|
ip link add testbr0 type bridge && ip link set testbr0 up
|
||||||
|
ip link set wlp3s0 master testbr0 && ip link set wlp3s0 up
|
||||||
|
|
||||||
|
ip netns add wifitest
|
||||||
|
ip link add veth-host type veth peer name veth-ns
|
||||||
|
ip link set veth-host master testbr0 && ip link set veth-host up
|
||||||
|
ip link set veth-ns netns wifitest
|
||||||
|
ip netns exec wifitest ip link set veth-ns address 02:11:22:33:44:55 # deliberately NOT the wifi card's MAC
|
||||||
|
ip netns exec wifitest ip link set veth-ns up
|
||||||
|
ip netns exec wifitest ip link set lo up
|
||||||
|
|
||||||
|
ip netns exec wifitest dhclient -v -1 veth-ns # real DHCP lease = AP forwards foreign MACs = 4addr genuinely works
|
||||||
|
```
|
||||||
|
|
||||||
|
If that gets a lease (or at minimum a ping reply from the gateway) from a
|
||||||
|
MAC that was never the one that associated, the AP is forwarding
|
||||||
|
arbitrary-MAC frames — real bridge mode will work. If not, don't proceed
|
||||||
|
to bridge mode; fall back to a routed/NAT design instead (wifi gets its
|
||||||
|
own IP via DHCP, VMs NAT out through it — never covered here since it
|
||||||
|
wasn't needed, but keep this fallback in mind if reproducing on different
|
||||||
|
hardware/AP).
|
||||||
|
|
||||||
|
Tear the test namespace/bridge down afterwards (`ip netns del wifitest;
|
||||||
|
ip link del veth-host; ip link set wlp3s0 nomaster; ip link del testbr0`)
|
||||||
|
— it's scratch, not part of the real config.
|
||||||
|
|
||||||
|
## Final architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
wlp3s0 (wifi, 4addr client mode) ─┐
|
||||||
|
├─ bond0 (active-backup, wlp3s0 primary) ─ vmbr0 (192.168.2.251/24)
|
||||||
|
nic0 (wired, LAN backup) ─┘
|
||||||
|
```
|
||||||
|
|
||||||
|
- **`wpa-4addr-<iface>.service`** (systemd unit, `Before=network-pre.target`) —
|
||||||
|
sets 4addr mode and starts `wpa_supplicant` *before* ifupdown2 processes
|
||||||
|
`/etc/network/interfaces`. Deliberately not using `wpasupplicant`'s own
|
||||||
|
`wpa-conf` ifupdown2 hook integration — its ordering relative to a
|
||||||
|
custom `pre-up iw ... set 4addr on` line is implementation-specific and
|
||||||
|
not worth gambling on for something this hard to debug if it's wrong.
|
||||||
|
The systemd unit gives full, deterministic control over sequencing
|
||||||
|
instead.
|
||||||
|
- **`bond0`** — active-backup bonding, `wlp3s0` as `bond-primary`,
|
||||||
|
`nic0` as backup, `bond-miimon 100`. Bonding a wifi interface works here
|
||||||
|
because `mac80211`/`iwlwifi` properly reports carrier state
|
||||||
|
(`netif_carrier_on/off`) on association/disassociation, which is what
|
||||||
|
bonding's `miimon` (with the default `use_carrier=1`) actually watches
|
||||||
|
— it doesn't require a "real" MII-capable NIC.
|
||||||
|
- **`vmbr0`** — unchanged IP (`192.168.2.251/24`), bridged over `bond0`
|
||||||
|
instead of directly over a physical NIC.
|
||||||
|
|
||||||
|
Reproduce with `scripts/setup-wifi-bond-network.sh` — see its header
|
||||||
|
comment for usage and required env vars (SSID/passphrase, management
|
||||||
|
address/gateway). It stages `/etc/network/interfaces` and the systemd
|
||||||
|
unit but does **not** auto-apply (`ifreload -a`) — see the next section
|
||||||
|
for why that step needs care, not automation.
|
||||||
|
|
||||||
|
## Applying a live bridge-port change: what went wrong once, and what worked
|
||||||
|
|
||||||
|
Moving `vmbr0`'s underlying port live — while SSH'd in over the very
|
||||||
|
address that lives on that bridge — is inherently risky: if the new port
|
||||||
|
doesn't actually pass traffic, you lose the connection you're using to
|
||||||
|
fix it.
|
||||||
|
|
||||||
|
**What failed**: doing the port swap via raw `ip link set nic0 nomaster`
|
||||||
|
+ `ip link set nic0 down` (with `wlp3s0` already added as a second bridge
|
||||||
|
member) caused an extended outage that needed a physical power-cycle to
|
||||||
|
recover — `nic0` never came back up on its own even well past a 60s
|
||||||
|
auto-revert watchdog's deadline. Importantly, the *host itself* never
|
||||||
|
crashed — `journalctl -b -1` showed completely normal operation (a
|
||||||
|
`pvestatd` polling loop, no gaps, no panic/OOM/watchdog trigger) right up
|
||||||
|
until a keyboard was plugged in and it was manually reset. So the failure
|
||||||
|
was specifically in the live network transition, not the OS — root cause
|
||||||
|
unconfirmed, but plausibly bridge FDB/MAC-identity handling when a port
|
||||||
|
carrying an already-live IP is swapped out via raw `ip link` rather than
|
||||||
|
a coordinated reconfiguration.
|
||||||
|
|
||||||
|
**What worked**: writing the final state into `/etc/network/interfaces`
|
||||||
|
and applying with `ifreload -a` (Proxmox's own supported hot-reload path)
|
||||||
|
instead. Same end state, same risk window, but ifupdown2 evidently
|
||||||
|
sequences the transition in a way raw `ip link` surgery didn't. This
|
||||||
|
succeeded on the first retry using this method.
|
||||||
|
|
||||||
|
**Either way, expect ~30-60 seconds of apparent breakage even on a
|
||||||
|
successful change** — the upstream switch/AP needs to relearn which port
|
||||||
|
`192.168.2.251`'s MAC now lives behind. Don't judge success/failure
|
||||||
|
before that window passes; a `ping`/SSH failure at the 5-10s mark is not
|
||||||
|
yet a sign anything is wrong.
|
||||||
|
|
||||||
|
**Recommended safety net for any future live change here**: arm a
|
||||||
|
backgrounded auto-revert-on-timeout before applying, e.g.:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp /etc/network/interfaces /etc/network/interfaces.bak
|
||||||
|
cat > /root/revert.sh <<'EOF'
|
||||||
|
#!/bin/bash
|
||||||
|
cp /etc/network/interfaces.bak /etc/network/interfaces
|
||||||
|
ifreload -a
|
||||||
|
EOF
|
||||||
|
chmod +x /root/revert.sh
|
||||||
|
setsid nohup bash -c "sleep 45 && /root/revert.sh" >/root/revert.log 2>&1 < /dev/null &
|
||||||
|
disown
|
||||||
|
# now apply the real change, e.g.: ifreload -a
|
||||||
|
# once confirmed working: pkill -f "sleep 45 && /root/revert.sh"
|
||||||
|
```
|
||||||
|
|
||||||
|
This only self-heals if the box stays responsive enough for the
|
||||||
|
backgrounded job to run to completion — it is not a substitute for
|
||||||
|
physical/console access being available, just a way to avoid needing it
|
||||||
|
for the common case.
|
||||||
|
|
||||||
|
## Troubleshooting: "can't ping it, web UI won't load" isn't necessarily an outage
|
||||||
|
|
||||||
|
Two independent things can make `pve-test` look dead when it isn't:
|
||||||
|
|
||||||
|
1. **Stale ARP after any change to `vmbr0`'s active port.** `vmbr0`'s MAC
|
||||||
|
address follows whichever interface is currently active in `bond0`
|
||||||
|
(`wlp3s0`'s MAC while wifi is primary, `nic0`'s if it fails over) — it
|
||||||
|
is *not* fixed. Other devices on the LAN (including your own
|
||||||
|
workstation) that cached the old MAC before a change won't notice
|
||||||
|
until their ARP entry naturally expires or gets flushed
|
||||||
|
(`ip neigh flush <ip>` / `arp -d <ip>`). Symptom: intermittent or
|
||||||
|
totally dead connectivity from one specific device while others (or
|
||||||
|
the host itself) are fine.
|
||||||
|
2. **ICMP ping is blocked by the management firewall, unrelated to
|
||||||
|
network health.** See `04-security-hardening.md`'s firewall section —
|
||||||
|
`cluster.fw`'s `policy_in: DROP` only explicitly allows TCP 22/8006
|
||||||
|
(now also ICMP echo-request, after this was hit) from the mgmt
|
||||||
|
network. Before that rule was added, `ping` failed 100% *even with a
|
||||||
|
perfectly correct, fresh ARP entry and full SSH/web UI access* —
|
||||||
|
nothing to do with wifi, bonding, or the network being down. This is
|
||||||
|
exactly what happened once: alarming "can't ping, web UI not loading"
|
||||||
|
turned out to be a stale-ARP moment (which resolved itself) plus a
|
||||||
|
pre-existing firewall policy (ping was never going to work,
|
||||||
|
regardless of wifi).
|
||||||
|
|
||||||
|
**When `pve-test` seems unreachable, check in this order before assuming
|
||||||
|
a real outage**: SSH (`ssh root@pve-test.sweet.home`) → web UI via `curl
|
||||||
|
-sk -o /dev/null -w '%{http_code}' https://pve-test.sweet.home:8006/` →
|
||||||
|
only then worry about `ping` specifically, and check
|
||||||
|
`/etc/pve/firewall/cluster.fw` before blaming the network.
|
||||||
|
|
||||||
|
## Known limitation history
|
||||||
|
|
||||||
|
`nic0` (the bond's backup slave) initially had no cable physically
|
||||||
|
connected when this was first set up — the bond was correctly configured
|
||||||
|
but inert until a cable was plugged in. As of this writing a cable is
|
||||||
|
connected and the backup slave is live (confirm with `cat
|
||||||
|
/proc/net/bonding/bond0`) — if reproducing this setup, verify the same
|
||||||
|
before assuming LAN failover will actually work.
|
||||||
@@ -58,6 +58,11 @@ if [ -f /etc/pve/firewall/cluster.fw ] && grep -qi '^policy_in:\s*DROP' /etc/pve
|
|||||||
else
|
else
|
||||||
audit_fail "cluster.fw missing or does not default-deny inbound"
|
audit_fail "cluster.fw missing or does not default-deny inbound"
|
||||||
fi
|
fi
|
||||||
|
if [ -f /etc/pve/firewall/cluster.fw ] && grep -qi 'icmp-type echo-request' /etc/pve/firewall/cluster.fw 2>/dev/null; then
|
||||||
|
audit_pass "cluster.fw allows ICMP echo-request from mgmt (ping works)"
|
||||||
|
else
|
||||||
|
audit_fail "cluster.fw does not allow ping from mgmt - see docs/04-security-hardening.md firewall section"
|
||||||
|
fi
|
||||||
|
|
||||||
# --- unattended-upgrades ---
|
# --- unattended-upgrades ---
|
||||||
if dpkg -s unattended-upgrades >/dev/null 2>&1 && systemctl is-enabled --quiet unattended-upgrades 2>/dev/null; then
|
if dpkg -s unattended-upgrades >/dev/null 2>&1 && systemctl is-enabled --quiet unattended-upgrades 2>/dev/null; then
|
||||||
|
|||||||
Executable
+133
@@ -0,0 +1,133 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Reproduces pve-test's wifi-primary networking: vmbr0 bridged over a wifi
|
||||||
|
# NIC in 4addr (WDS) client-bridge mode, active-backup bonded with a wired
|
||||||
|
# NIC as an automatic LAN fallback. See docs/06-pve-test-wifi-network.md for
|
||||||
|
# why this exists and how it was validated. Idempotent - safe to re-run.
|
||||||
|
#
|
||||||
|
# Requires: a wifi NIC whose driver/AP both support 4addr mode (verify with
|
||||||
|
# docs/06-pve-test-wifi-network.md's isolated-namespace test *before*
|
||||||
|
# trusting this against a live management IP - a wifi NIC or AP that
|
||||||
|
# doesn't support 4addr will associate fine but silently drop bridged
|
||||||
|
# frames from any MAC other than the card's own).
|
||||||
|
#
|
||||||
|
# Usage (run as root on the target PVE host):
|
||||||
|
# WIFI_SSID="..." WIFI_PASSPHRASE="..." \
|
||||||
|
# MGMT_ADDR=192.168.2.251/24 MGMT_GATEWAY=192.168.2.254 \
|
||||||
|
# ./setup-wifi-bond-network.sh
|
||||||
|
#
|
||||||
|
# Optional overrides: WIFI_IFACE (default wlp3s0), LAN_IFACE (default nic0)
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
# shellcheck source=lib/common.sh
|
||||||
|
source "${SCRIPT_DIR}/lib/common.sh"
|
||||||
|
require_root
|
||||||
|
|
||||||
|
WIFI_IFACE="${WIFI_IFACE:-wlp3s0}"
|
||||||
|
LAN_IFACE="${LAN_IFACE:-nic0}"
|
||||||
|
|
||||||
|
for var in WIFI_SSID WIFI_PASSPHRASE MGMT_ADDR MGMT_GATEWAY; do
|
||||||
|
if [ -z "${!var:-}" ]; then
|
||||||
|
echo "$var is not set. See usage in this script's header." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if ! ip link show "$WIFI_IFACE" >/dev/null 2>&1; then
|
||||||
|
echo "No interface named $WIFI_IFACE on this host. Run 'ip -br link' and set WIFI_IFACE=..." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
if ! ip link show "$LAN_IFACE" >/dev/null 2>&1; then
|
||||||
|
echo "No interface named $LAN_IFACE on this host. Run 'ip -br link' and set LAN_IFACE=..." >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "=== 1/4: wifi tooling ==="
|
||||||
|
apt-get install -y iw wpasupplicant >/dev/null
|
||||||
|
echo "installed iw, wpasupplicant"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 2/4: wpa_supplicant config (SSID: $WIFI_SSID, iface: $WIFI_IFACE) ==="
|
||||||
|
WPA_CONF="/etc/wpa_supplicant/wpa_supplicant-${WIFI_IFACE}.conf"
|
||||||
|
wpa_passphrase "$WIFI_SSID" "$WIFI_PASSPHRASE" > "$WPA_CONF"
|
||||||
|
sed -i '/^\s*#psk=/d' "$WPA_CONF"
|
||||||
|
chmod 600 "$WPA_CONF"
|
||||||
|
echo "wrote $WPA_CONF (passphrase hashed, not stored in plaintext)"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 3/4: systemd unit to set 4addr mode + start wpa_supplicant ==="
|
||||||
|
UNIT="/etc/systemd/system/wpa-4addr-${WIFI_IFACE}.service"
|
||||||
|
write_if_changed "$UNIT" "[Unit]
|
||||||
|
Description=wpa_supplicant on ${WIFI_IFACE} with 4addr mode enabled
|
||||||
|
Before=network-pre.target
|
||||||
|
Wants=network-pre.target
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=simple
|
||||||
|
ExecStartPre=/sbin/ip link set ${WIFI_IFACE} down
|
||||||
|
ExecStartPre=/sbin/iw dev ${WIFI_IFACE} set 4addr on
|
||||||
|
ExecStartPre=/sbin/ip link set ${WIFI_IFACE} up
|
||||||
|
ExecStart=/sbin/wpa_supplicant -i ${WIFI_IFACE} -c ${WPA_CONF}
|
||||||
|
Restart=on-failure
|
||||||
|
RestartSec=3
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=multi-user.target"
|
||||||
|
systemctl daemon-reload
|
||||||
|
systemctl enable --now "wpa-4addr-${WIFI_IFACE}.service"
|
||||||
|
sleep 5
|
||||||
|
if ! iw dev "$WIFI_IFACE" link | grep -q "^Connected"; then
|
||||||
|
echo "WARNING: ${WIFI_IFACE} did not associate to '$WIFI_SSID' within 5s - check:" >&2
|
||||||
|
echo " systemctl status wpa-4addr-${WIFI_IFACE}.service" >&2
|
||||||
|
echo " journalctl -u wpa-4addr-${WIFI_IFACE}.service" >&2
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
echo "associated: $(iw dev "$WIFI_IFACE" link | grep '^Connected')"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "=== 4/4: /etc/network/interfaces (bond0 active-backup: ${WIFI_IFACE} primary, ${LAN_IFACE} backup) ==="
|
||||||
|
IFACES_FILE="/etc/network/interfaces"
|
||||||
|
backup_file "$IFACES_FILE"
|
||||||
|
cat > "$IFACES_FILE" <<EOF
|
||||||
|
auto lo
|
||||||
|
iface lo inet loopback
|
||||||
|
|
||||||
|
iface ${LAN_IFACE} inet manual
|
||||||
|
|
||||||
|
iface ${WIFI_IFACE} inet manual
|
||||||
|
|
||||||
|
auto bond0
|
||||||
|
iface bond0 inet manual
|
||||||
|
bond-slaves ${WIFI_IFACE} ${LAN_IFACE}
|
||||||
|
bond-mode active-backup
|
||||||
|
bond-miimon 100
|
||||||
|
bond-primary ${WIFI_IFACE}
|
||||||
|
bond-updelay 200
|
||||||
|
bond-downdelay 200
|
||||||
|
|
||||||
|
auto vmbr0
|
||||||
|
iface vmbr0 inet static
|
||||||
|
address ${MGMT_ADDR}
|
||||||
|
gateway ${MGMT_GATEWAY}
|
||||||
|
bridge-ports bond0
|
||||||
|
bridge-stp off
|
||||||
|
bridge-fd 0
|
||||||
|
|
||||||
|
source /etc/network/interfaces.d/*
|
||||||
|
EOF
|
||||||
|
echo "wrote $IFACES_FILE"
|
||||||
|
|
||||||
|
echo
|
||||||
|
echo "Config staged but NOT applied yet - applying it live can drop your"
|
||||||
|
echo "current management connection for ~30-60s while the switch/AP"
|
||||||
|
echo "relearns MAC locations (expected, self-resolves; see"
|
||||||
|
echo "docs/06-pve-test-wifi-network.md). Recommended: run this from the"
|
||||||
|
echo "physical console, or arm a revert-on-timeout watchdog first, e.g.:"
|
||||||
|
echo
|
||||||
|
echo " cp ${IFACES_FILE}.bak.* /tmp/interfaces.orig # pick the backup just made"
|
||||||
|
echo " (sleep 45 && cp /tmp/interfaces.orig ${IFACES_FILE} && ifreload -a) &"
|
||||||
|
echo " ifreload -a"
|
||||||
|
echo " # then kill the backgrounded revert job once you confirm connectivity"
|
||||||
|
echo
|
||||||
|
echo "Apply now with: ifreload -a"
|
||||||
|
echo "Verify after with: cat /proc/net/bonding/bond0 ; ip -4 -br addr show vmbr0"
|
||||||
Reference in New Issue
Block a user