diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..033ce36 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,84 @@ +# CLAUDE.md + +Guidance for Claude Code working in this repo. IMPORTANT: these +instructions OVERRIDE any default behavior and must be followed exactly +as written. + +## Repo purpose + +Base configuration/hardening toolset and planning docs for Proxmox VE +hosts (`scripts/`, `config/`, `docs/`) — see `README.md` and +`docs/00-overview.md`. Scripts in this repo are meant to be run **on** +the target Proxmox host itself (as root), not orchestrated remotely. + +## Two Proxmox nodes: `pve1` (production) and `pve-test` (sandbox) + +Two SSH-reachable Proxmox nodes exist on the LAN. They are **not +interchangeable** — see `docs/05-node-roles.md` for full background on +what each one is and why. + +### `pve1` (production — off-limits to Claude by default) + +A real, live Proxmox node hosting production VMs/containers (see +`docs/05-node-roles.md` for the current guest list) — not a sandbox, and +not Claude's to touch by default. + +- **Off-limits at all times unless the operator has given explicit, + same-session instructions to act on this specific host.** That + authorization is scoped to the task it was given for — don't carry it + forward to unrelated later work in the same conversation, and never + assume it from a previous session. +- **Read-only for existing state is always fine, authorization or not.** + SSH in (or use `pvesm`, `qm list`, `pct list`, `qm config`, `pct + config`, the Proxmox API, etc.) to inspect config, storage, and any + existing VM/container freely. +- **Never** modify, stop, restart, delete, reconfigure, or create + anything on this node (`qm set`, `pct set`, `qm destroy`, `pct + destroy`, `qm stop`, `pct stop`, `qm create`, `pct create`, snapshot + operations, storage changes, running any script in this repo against + it, etc.) without that explicit go-ahead. Use `pve-test` for anything + exploratory instead. +- If a guest on `pve1` is HA-managed, be aware of the self-fence hazard + described in `docs/05-node-roles.md`'s cluster-teardown section before + doing anything that could cost the node quorum. + +### `pve-test` (sandbox — Claude's default target) + +A separate node set aside for testing — safe to create, interrogate, and +destroy scratch VMs/containers on without asking first. + +- **Test VMs/containers are allowed, but must be torn down.** Anything + created this way must be destroyed again in the same session, before + ending the task. Use an obviously-scratch VMID/name. +- **Node-level config is still not yours to change by default.** + Creating/destroying your own scratch guests is fine; Proxmox host + config, storage pools, and networking on `pve-test` itself need the + operator's explicit go-ahead too, same as on `pve1` — the "sandbox" + status covers guest-level experimentation, not the host's own + identity. (`pve-test`'s current network config, `docs/06-pve-test-wifi-network.md`, + was applied under exactly that kind of explicit, same-session + authorization — it's not a standing invitation to keep changing it + further without asking again.) +- **Never re-cluster `pve-test` with `pve1`** without the operator + explicitly asking for it and being aware of the wifi/corosync + incompatibility in `docs/06-pve-test-wifi-network.md` — the two were + deliberately de-clustered for this reason once already. + +## Safety rules + +- Any script in `scripts/` that isn't `audit.sh` (read-only) makes real + changes when run for real. Don't run one against `pve1`, or against + `pve-test`'s node-level config, without the same-session go-ahead + described above. Running a script *against pve-test's own guests* — a + scratch VM/CT you created for this task — doesn't need separate + permission. +- Do not commit secrets: SSH private keys, wifi passphrases, sops/age + keys, or PVE credentials. `scripts/setup-wifi-bond-network.sh` takes + the SSID/passphrase via environment variables for exactly this reason + — never hardcode them into the script or a committed config file. +- Live changes to a node's own management network (the interface/bridge + carrying the SSH session you're using) can strand the box — see the + "what went wrong once" section in `docs/06-pve-test-wifi-network.md` + before touching `pve-test`'s networking again. Prefer applying via + `ifreload -a` over raw `ip link` surgery, and arm an auto-revert + watchdog first when acting without someone physically at the console. diff --git a/README.md b/README.md index b147304..79bc503 100644 --- a/README.md +++ b/README.md @@ -15,15 +15,22 @@ deferred). backed by Ceph. Needs dedicated hardware node 1 (`pve1`, an ASUS PN53 mini PC) doesn't have — see `docs/01-hardware-node1.md`. +See `CLAUDE.md` for the guardrails Claude Code follows when working +against these hosts (`pve1` is production and off-limits by default; +`pve-test` is the sandbox). + ## Repo layout - `docs/` — planning docs: hardware layout, storage migration, networking, - security hardening. Read `docs/00-overview.md` first. + security hardening, node roles. Read `docs/00-overview.md` first, then + `docs/05-node-roles.md` for what `pve1`/`pve-test` actually are. - `scripts/` — scripts to apply configuration on a node (SSH hardening, repo - switch, firewall, updates, etc). Idempotent, safe to re-run. - `scripts/bootstrap.sh` runs the full Stage 1 sequence end to end; - `scripts/audit.sh` verifies it (read-only). `scripts/lib/` holds shared - helpers (`common.sh`) sourced by the other scripts. + switch, firewall, updates, wifi/bond networking, etc). Idempotent, safe + to re-run. `scripts/bootstrap.sh` runs the full Stage 1 sequence end to + end; `scripts/audit.sh` verifies it (read-only). + `scripts/setup-wifi-bond-network.sh` reproduces `pve-test`'s wifi + network (see `docs/06-pve-test-wifi-network.md`). `scripts/lib/` holds + shared helpers (`common.sh`) sourced by the other scripts. - `config/` — reference config files/snippets to drop onto a node (firewall rules, sshd config, etc.). diff --git a/config/pve-firewall/cluster.fw.example b/config/pve-firewall/cluster.fw.example index 16049c7..f25ae41 100644 --- a/config/pve-firewall/cluster.fw.example +++ b/config/pve-firewall/cluster.fw.example @@ -26,6 +26,13 @@ policy_out: ACCEPT IN ACCEPT -source +mgmt -p tcp -dport 8006 -log nolog IN ACCEPT -source +mgmt -p tcp -dport 22 -log nolog +# ICMP echo (ping) from the management network - diagnostic convenience +# only, nothing else depends on it. Without this, policy_in DROP silently +# eats ping while SSH/web UI keep working - looks like an outage during +# troubleshooting when the host is actually fine. See +# docs/06-pve-test-wifi-network.md for a case this caused real confusion. +IN ACCEPT -source +mgmt -p icmp -icmp-type echo-request -log nolog + # Stage 2: Corosync (cluster quorum) - uncomment once node 2/3 join and # the corosync network/VLAN exists. # IN ACCEPT -source -p udp -dport 5404:5405 -log nolog diff --git a/docs/00-overview.md b/docs/00-overview.md index 2e14db1..297c9ed 100644 --- a/docs/00-overview.md +++ b/docs/00-overview.md @@ -59,6 +59,16 @@ require or assume the split-disk/multi-NIC layout Stage 2 wants. Everything below this point is the target design for when nodes 2 and 3 are actually being provisioned. Not applicable to `pve1` as it stands. +**`pve-test` existing does not mean node 2 exists.** A second physical +node (`pve-test`) does run alongside `pve1` — see `05-node-roles.md` — +but it's a sandbox/test box, not built to this Stage 2 design, and as of +this writing runs on **wifi** networking +(`06-pve-test-wifi-network.md`), which is directly incompatible with +corosync's latency/jitter requirements below. `pve-test` and `pve1` were +briefly clustered and then deliberately de-clustered for exactly this +reason. Don't treat `pve-test` as progress toward Stage 2 without a +deliberate decision to rebuild its networking first. + ### End goal 3-node Proxmox VE cluster with HA-managed VMs backed by **Ceph** — true diff --git a/docs/03-networking.md b/docs/03-networking.md index 07ac6f0..c03a4a1 100644 --- a/docs/03-networking.md +++ b/docs/03-networking.md @@ -11,6 +11,17 @@ PVE datacenter firewall (default-deny inbound otherwise). That's sufficient until Stage 2 needs actual separate physical/VLAN paths for corosync and Ceph traffic — see below. +## pve-test as built (sandbox, current) + +Different node, different design, not a Stage 1/2 example to generalize +from: `pve-test` runs `vmbr0` bridged over a wifi NIC in 4addr client mode +(active-backup bonded with a wired NIC as an automatic fallback). Full +detail, including why this is normally impossible and how it was +validated before trusting it with the management IP, in +`06-pve-test-wifi-network.md`. This is intentionally a one-off for a +standalone sandbox box — never extend it to a node that's clustered or +Ceph-connected (see the Stage 2 note in `00-overview.md`). + ## Target design (Stage 2, future cluster) ## Required separation diff --git a/docs/04-security-hardening.md b/docs/04-security-hardening.md index 2d9287a..0b6c355 100644 --- a/docs/04-security-hardening.md +++ b/docs/04-security-hardening.md @@ -54,6 +54,23 @@ Default-deny at datacenter/node level, whitelist only what's needed (see `config/pve-firewall/cluster.fw.example`, applied by `scripts/deploy-firewall.sh`. +**ICMP echo (ping) is explicitly allowed from the management network**, +alongside SSH/8006 — not required for anything to function, but without +it `policy_in: DROP` silently eats ping while SSH/web UI keep working. +That split (ping dead, everything else fine) reads exactly like a real +outage mid-troubleshooting; see `06-pve-test-wifi-network.md` for a case +this caused genuine confusion after a network change. If you ever debug +"can't ping but can SSH", check this firewall before assuming the +network itself is broken. + +**This file is cluster-wide, not per-node**: `cluster.fw` lives in +`/etc/pve/firewall/` — shared via pmxcfs across every node in a cluster. +A node that joins a cluster inherits whatever's already there, and (per +`docs/05-node-roles.md`) keeps its own local copy after leaving. Don't +assume a node's firewall state matches what `deploy-firewall.sh` was +last run with directly against it — check `pve-firewall status` / +`/etc/pve/firewall/cluster.fw` on the actual node. + ## Repos and updates Fresh installs point at the enterprise repo, which fails on `apt update` diff --git a/docs/05-node-roles.md b/docs/05-node-roles.md new file mode 100644 index 0000000..7d16298 --- /dev/null +++ b/docs/05-node-roles.md @@ -0,0 +1,83 @@ +# Node Roles: pve1 and pve-test + +Two Proxmox VE hosts exist on the LAN. They are **not interchangeable** — +different purpose, different risk tolerance, different guardrails. See +`CLAUDE.md` for the actual rules Claude follows; this doc is the +background/context those rules assume. + +## pve1 (production) + +ASUS PN53 mini PC (Ryzen 7 7735HS, 32GB RAM) — see `01-hardware-node1.md` +for the full hardware layout. Stage 1 base hardening applied and verified +(`04-security-hardening.md`). + +Runs real production workloads: + +- `server-nixos` (VM) and LXC containers `pihole`, `claude`, `nix-cache`, + `docker`, `pdm`, `pxe-boot` — DNS, this Claude Code environment itself, + the LAN's Nix binary cache, etc. + +Off-limits to Claude by default — see `CLAUDE.md`. + +## pve-test (sandbox) + +A separate physical node kept for testing. Has a capable GPU (earmarked +for future GPU workloads — transcoding, local inference, etc. — nothing +built on it yet as of this writing). No production guests run here. + +Safe for Claude to create/destroy scratch VMs and containers on without +asking — see `CLAUDE.md` for the exact boundary (node-level config is +still not free-for-all even here). + +### History: briefly clustered with pve1, then deliberately separated + +On 2026-07-21/22, pve-test was joined to pve1 in a 2-node Proxmox cluster +(corosync + shared `/etc/pve`) for a period, then the cluster was +**intentionally destroyed** at the operator's explicit request — not a +failure or accident. Rationale: the operator wanted pve-test moved onto +wifi networking (see `06-pve-test-wifi-network.md`), and wifi is +fundamentally unsuitable for corosync's latency/jitter requirements (token +timeouts, flapping membership). Rather than fight that, the cluster was +torn down first, then wifi was configured on the now-standalone node. + +Teardown procedure used (safe to repeat if pve-test or pve1 is ever +reclustered and needs separating again): + +1. Un-manage any HA-assigned resources first if HA is in use + (`ha-manager remove ` per resource) — **critical**: if a node has + HA-managed guests and its watchdog is still armed when it loses + quorum, HA's self-fence behavior can force-reboot it. Removing the + resource assignments first (not stopping the guests — just + un-managing them) prevents this. +2. Stop `pve-ha-lrm`/`pve-ha-crm` on all nodes once no resources remain + assigned, confirm via `ha-manager status` that fencing is no longer + armed. +3. From a node that will remain in the cluster: `pvecm delnode ` + for the node being removed. In a 2-node cluster with no QDevice, this + can itself cause the *remaining* node to transiently lose quorum + (killing a node's live corosync membership drops total votes below + expected votes before the nodelist file can be rewritten to match) — + this is expected, not a failure; proceed to step 4 regardless. +4. On **each** node being separated (including one you just ran `delnode` + from, if it's also being decoupled): `systemctl stop pve-cluster + corosync`, `pmxcfs -l` (local mode, bypasses the quorum requirement + for local writes), remove `/etc/pve/corosync.conf` and + `/etc/corosync/*`, `killall pmxcfs`, `systemctl start pve-cluster`. + This is the official "separate a node without reinstalling" procedure + and doesn't touch any running VM/CT — they're independent OS + processes, unaffected by corosync/pmxcfs state either way. +5. Optional cosmetic cleanup: the node(s) staying up may retain a stale + `/etc/pve/nodes//` directory (cached VM/CT configs from + when they were clustered) — safe to `rm -rf` once separation is + confirmed; it's dead data, not a live reference to anything. + +Guests were never at risk during this procedure (confirmed: all 7 of +pve1's guests ran continuously throughout) — the only real hazard is the +HA/watchdog self-fence path in step 1, and losing corosync-provided quorum +temporarily blocking `/etc/pve` writes (not guest execution) in step 3. + +### Current state: standalone, wifi-primary networking + +pve-test now runs entirely independently of pve1, on the network design +documented in `06-pve-test-wifi-network.md`. **Do not re-cluster +pve-test while it's on wifi** — see that doc for why. diff --git a/docs/06-pve-test-wifi-network.md b/docs/06-pve-test-wifi-network.md new file mode 100644 index 0000000..12f2872 --- /dev/null +++ b/docs/06-pve-test-wifi-network.md @@ -0,0 +1,230 @@ +# pve-test's Wifi-Primary Network + +Non-standard, deliberately chosen against normal Proxmox guidance (a +hypervisor's management network is not supposed to be wireless). Applies +**only** to `pve-test` as a standalone sandbox node — see +`05-node-roles.md` for why this and cluster membership don't mix, and +never apply this pattern to a node that's joined (or might join) a +cluster. + +## Why this works at all + +Wifi (802.11) normally cannot be bridged the way Ethernet can: an access +point only accepts frames whose source MAC matches the MAC that +associated with it. A Linux bridge with a wifi port as a member sends +frames tagged with whatever MAC actually generated them (the bridge's own +MAC, a VM's MAC, etc.) — none of which match the wifi card's hardware +MAC, so the AP silently drops them. This isn't a Linux/Proxmox limitation, +it's how WiFi association works. + +**4addr (WDS) client mode** is the exception: if both the wifi driver and +the AP support it, the client tags frames with a 4th address field +carrying the original MAC, and the AP forwards them like a real Ethernet +segment. Most consumer routers do **not** support this (it's common on +OpenWrt/DD-WRT/enterprise APs, rare on stock ISP hardware) — it has to be +confirmed empirically, per-AP, before trusting it with a management IP. + +pve-test's card (Intel AX200, `iwlwifi` driver, interface `wlp3s0`) +supports 4addr client-side generically (this is a `mac80211` core +feature, not driver-specific). The home AP it associates to +(`nbn-fttp-net-5G`) was confirmed to honor it — see validation method +below. + +## Validate before ever touching a live management IP + +Don't add the wifi NIC straight into your real bridge to "see if it +works" — if 4addr isn't actually honored, you lose access to the box with +no diagnostic trail (association succeeds; only the *data plane* silently +fails). Test in an isolated scratch namespace first: + +```bash +# assumes wlp3s0 already associated with 4addr on (see systemd unit below) +ip link add testbr0 type bridge && ip link set testbr0 up +ip link set wlp3s0 master testbr0 && ip link set wlp3s0 up + +ip netns add wifitest +ip link add veth-host type veth peer name veth-ns +ip link set veth-host master testbr0 && ip link set veth-host up +ip link set veth-ns netns wifitest +ip netns exec wifitest ip link set veth-ns address 02:11:22:33:44:55 # deliberately NOT the wifi card's MAC +ip netns exec wifitest ip link set veth-ns up +ip netns exec wifitest ip link set lo up + +ip netns exec wifitest dhclient -v -1 veth-ns # real DHCP lease = AP forwards foreign MACs = 4addr genuinely works +``` + +If that gets a lease (or at minimum a ping reply from the gateway) from a +MAC that was never the one that associated, the AP is forwarding +arbitrary-MAC frames — real bridge mode will work. If not, don't proceed +to bridge mode; fall back to a routed/NAT design instead (wifi gets its +own IP via DHCP, VMs NAT out through it — never covered here since it +wasn't needed, but keep this fallback in mind if reproducing on different +hardware/AP). + +Tear the test namespace/bridge down afterwards (`ip netns del wifitest; +ip link del veth-host; ip link set wlp3s0 nomaster; ip link del testbr0`) +— it's scratch, not part of the real config. + +## Final architecture + +``` +wlp3s0 (wifi, 4addr client mode) ─┐ + ├─ bond0 (active-backup, wlp3s0 primary) ─ vmbr0 (192.168.2.251/24) +nic0 (wired, LAN backup) ─┘ +``` + +- **`wpa-4addr-.service`** (systemd unit, `Before=network-pre.target`) — + sets 4addr mode and starts `wpa_supplicant` *before* ifupdown2 processes + `/etc/network/interfaces`. Deliberately not using `wpasupplicant`'s own + `wpa-conf` ifupdown2 hook integration — its ordering relative to a + custom `pre-up iw ... set 4addr on` line is implementation-specific and + not worth gambling on for something this hard to debug if it's wrong. + The systemd unit gives full, deterministic control over sequencing + instead. +- **`bond0`** — active-backup bonding, `wlp3s0` as `bond-primary`, + `nic0` as backup, `bond-miimon 100`. Bonding a wifi interface works here + because `mac80211`/`iwlwifi` properly reports carrier state + (`netif_carrier_on/off`) on association/disassociation, which is what + bonding's `miimon` (with the default `use_carrier=1`) actually watches + — it doesn't require a "real" MII-capable NIC. +- **`vmbr0`** — unchanged IP (`192.168.2.251/24`), bridged over `bond0` + instead of directly over a physical NIC. + +Reproduce with `scripts/setup-wifi-bond-network.sh` — see its header +comment for usage and required env vars (SSID/passphrase, management +address/gateway). It stages `/etc/network/interfaces` and the systemd +unit but does **not** auto-apply (`ifreload -a`) — see the next section +for why that step needs care, not automation. + +## Applying a live bridge-port change: what went wrong once, and what worked + +Moving `vmbr0`'s underlying port live — while SSH'd in over the very +address that lives on that bridge — is inherently risky: if the new port +doesn't actually pass traffic, you lose the connection you're using to +fix it. + +**What failed**: doing the port swap via raw `ip link set nic0 nomaster` ++ `ip link set nic0 down` (with `wlp3s0` already added as a second bridge +member) caused an extended outage that needed a physical power-cycle to +recover — `nic0` never came back up on its own even well past a 60s +auto-revert watchdog's deadline. Importantly, the *host itself* never +crashed — `journalctl -b -1` showed completely normal operation (a +`pvestatd` polling loop, no gaps, no panic/OOM/watchdog trigger) right up +until a keyboard was plugged in and it was manually reset. So the failure +was specifically in the live network transition, not the OS — root cause +unconfirmed, but plausibly bridge FDB/MAC-identity handling when a port +carrying an already-live IP is swapped out via raw `ip link` rather than +a coordinated reconfiguration. + +**What worked**: writing the final state into `/etc/network/interfaces` +and applying with `ifreload -a` (Proxmox's own supported hot-reload path) +instead. Same end state, same risk window, but ifupdown2 evidently +sequences the transition in a way raw `ip link` surgery didn't. This +succeeded on the first retry using this method. + +**Either way, expect ~30-60 seconds of apparent breakage even on a +successful change** — the upstream switch/AP needs to relearn which port +`192.168.2.251`'s MAC now lives behind. Don't judge success/failure +before that window passes; a `ping`/SSH failure at the 5-10s mark is not +yet a sign anything is wrong. + +**Recommended safety net for any future live change here**: arm a +backgrounded auto-revert-on-timeout before applying, e.g.: + +```bash +cp /etc/network/interfaces /etc/network/interfaces.bak +cat > /root/revert.sh <<'EOF' +#!/bin/bash +cp /etc/network/interfaces.bak /etc/network/interfaces +ifreload -a +EOF +chmod +x /root/revert.sh +setsid nohup bash -c "sleep 45 && /root/revert.sh" >/root/revert.log 2>&1 < /dev/null & +disown +# now apply the real change, e.g.: ifreload -a +# once confirmed working: pkill -f "sleep 45 && /root/revert.sh" +``` + +This only self-heals if the box stays responsive enough for the +backgrounded job to run to completion — it is not a substitute for +physical/console access being available, just a way to avoid needing it +for the common case. + +## Troubleshooting: "can't ping it, web UI won't load" isn't necessarily an outage + +Two independent things can make `pve-test` look dead when it isn't: + +1. **Stale ARP after any change to `vmbr0`'s active port.** `vmbr0`'s MAC + address follows whichever interface is currently active in `bond0` + (`wlp3s0`'s MAC while wifi is primary, `nic0`'s if it fails over) — it + is *not* fixed. Other devices on the LAN (including your own + workstation) that cached the old MAC before a change won't notice + until their ARP entry naturally expires or gets flushed + (`ip neigh flush ` / `arp -d `). Symptom: intermittent or + totally dead connectivity from one specific device while others (or + the host itself) are fine. +2. **ICMP ping is blocked by the management firewall, unrelated to + network health.** See `04-security-hardening.md`'s firewall section — + `cluster.fw`'s `policy_in: DROP` only explicitly allows TCP 22/8006 + (now also ICMP echo-request, after this was hit) from the mgmt + network. Before that rule was added, `ping` failed 100% *even with a + perfectly correct, fresh ARP entry and full SSH/web UI access* — + nothing to do with wifi, bonding, or the network being down. This is + exactly what happened once: alarming "can't ping, web UI not loading" + turned out to be a stale-ARP moment (which resolved itself) plus a + pre-existing firewall policy (ping was never going to work, + regardless of wifi). + +**When `pve-test` seems unreachable, check in this order before assuming +a real outage**: SSH (`ssh root@pve-test.sweet.home`) → web UI via `curl +-sk -o /dev/null -w '%{http_code}' https://pve-test.sweet.home:8006/` → +only then worry about `ping` specifically, and check +`/etc/pve/firewall/cluster.fw` before blaming the network. + +## Confirmed limitation: no wireless client on the same AP can reach pve-test + +**This is not an AP-isolation toggle** — tested and ruled out (router: +TP-Link BE9300, AP Isolation confirmed unchecked). Two independent +wireless devices (a laptop and a phone) on the same SSID pve-test uses +(`nbn-fttp-net-5G`) were both completely unable to reach it — not +intermittent, not slow, no ARP entry ever resolves (`arp -a` on Windows / +`ip neigh` on Linux/Mac shows nothing at all, not a stale entry). Every +wired device tested (this repo's own Claude session, `pve1`) reaches it +fine, every time. + +Root cause: pve-test's wifi connection uses 4-address (WDS) framing to +make bridging possible at all (see above) — that's what lets the AP +forward frames from arbitrary MACs (VM traffic, the bridge's own +identity) instead of only the wifi card's own hardware MAC. This +router's firmware evidently handles wired↔pve-test forwarding for that +4addr peer correctly, but doesn't correctly forward *wireless-client* +broadcast/ARP traffic to it — an asymmetric gap in the AP's own 4addr +handling, not a policy setting. Likely not fixable short of a firmware +update from TP-Link (if one ever improves 4addr/WDS handling), possibly +not fixable on this hardware at all. + +**Practical implication**: anything on `vmbr0` — pve-test itself, or any +VM/CT created on it — is unreachable from a wireless client on this AP, +full stop, for *any* protocol, not just ICMP. ARP resolution is a +prerequisite for sending any IP packet at all (TCP included); since ARP +itself never resolves for a wireless peer here, the web UI (8006) fails +identically to ping, not just ping — confirmed live, not theoretical. +This includes VMs on pve-test too, since they ride the same bridge/wifi +uplink as the host. **Wired access is the only reliable path to +pve-test and anything running on it** — including reaching a VM's +console through the Proxmox web UI, which still requires the accessing +device to reach pve-test's own IP on port 8006 first, so it inherits the +same requirement. + +Don't waste time on ARP flushes, firewall rules, or bonding config for +this specific symptom (no ARP entry from a wireless peer, wired fine) — +none of those are the cause. + +## Known limitation history + +`nic0` (the bond's backup slave) initially had no cable physically +connected when this was first set up — the bond was correctly configured +but inert until a cable was plugged in. As of this writing a cable is +connected and the backup slave is live (confirm with `cat +/proc/net/bonding/bond0`) — if reproducing this setup, verify the same +before assuming LAN failover will actually work. diff --git a/scripts/audit.sh b/scripts/audit.sh index 0317644..3a4edee 100755 --- a/scripts/audit.sh +++ b/scripts/audit.sh @@ -58,6 +58,11 @@ if [ -f /etc/pve/firewall/cluster.fw ] && grep -qi '^policy_in:\s*DROP' /etc/pve else audit_fail "cluster.fw missing or does not default-deny inbound" fi +if [ -f /etc/pve/firewall/cluster.fw ] && grep -qi 'icmp-type echo-request' /etc/pve/firewall/cluster.fw 2>/dev/null; then + audit_pass "cluster.fw allows ICMP echo-request from mgmt (ping works)" +else + audit_fail "cluster.fw does not allow ping from mgmt - see docs/04-security-hardening.md firewall section" +fi # --- unattended-upgrades --- if dpkg -s unattended-upgrades >/dev/null 2>&1 && systemctl is-enabled --quiet unattended-upgrades 2>/dev/null; then diff --git a/scripts/setup-wifi-bond-network.sh b/scripts/setup-wifi-bond-network.sh new file mode 100755 index 0000000..86f0fe7 --- /dev/null +++ b/scripts/setup-wifi-bond-network.sh @@ -0,0 +1,133 @@ +#!/bin/bash +# Reproduces pve-test's wifi-primary networking: vmbr0 bridged over a wifi +# NIC in 4addr (WDS) client-bridge mode, active-backup bonded with a wired +# NIC as an automatic LAN fallback. See docs/06-pve-test-wifi-network.md for +# why this exists and how it was validated. Idempotent - safe to re-run. +# +# Requires: a wifi NIC whose driver/AP both support 4addr mode (verify with +# docs/06-pve-test-wifi-network.md's isolated-namespace test *before* +# trusting this against a live management IP - a wifi NIC or AP that +# doesn't support 4addr will associate fine but silently drop bridged +# frames from any MAC other than the card's own). +# +# Usage (run as root on the target PVE host): +# WIFI_SSID="..." WIFI_PASSPHRASE="..." \ +# MGMT_ADDR=192.168.2.251/24 MGMT_GATEWAY=192.168.2.254 \ +# ./setup-wifi-bond-network.sh +# +# Optional overrides: WIFI_IFACE (default wlp3s0), LAN_IFACE (default nic0) +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +# shellcheck source=lib/common.sh +source "${SCRIPT_DIR}/lib/common.sh" +require_root + +WIFI_IFACE="${WIFI_IFACE:-wlp3s0}" +LAN_IFACE="${LAN_IFACE:-nic0}" + +for var in WIFI_SSID WIFI_PASSPHRASE MGMT_ADDR MGMT_GATEWAY; do + if [ -z "${!var:-}" ]; then + echo "$var is not set. See usage in this script's header." >&2 + exit 1 + fi +done + +if ! ip link show "$WIFI_IFACE" >/dev/null 2>&1; then + echo "No interface named $WIFI_IFACE on this host. Run 'ip -br link' and set WIFI_IFACE=..." >&2 + exit 1 +fi +if ! ip link show "$LAN_IFACE" >/dev/null 2>&1; then + echo "No interface named $LAN_IFACE on this host. Run 'ip -br link' and set LAN_IFACE=..." >&2 + exit 1 +fi + +echo "=== 1/4: wifi tooling ===" +apt-get install -y iw wpasupplicant >/dev/null +echo "installed iw, wpasupplicant" + +echo +echo "=== 2/4: wpa_supplicant config (SSID: $WIFI_SSID, iface: $WIFI_IFACE) ===" +WPA_CONF="/etc/wpa_supplicant/wpa_supplicant-${WIFI_IFACE}.conf" +wpa_passphrase "$WIFI_SSID" "$WIFI_PASSPHRASE" > "$WPA_CONF" +sed -i '/^\s*#psk=/d' "$WPA_CONF" +chmod 600 "$WPA_CONF" +echo "wrote $WPA_CONF (passphrase hashed, not stored in plaintext)" + +echo +echo "=== 3/4: systemd unit to set 4addr mode + start wpa_supplicant ===" +UNIT="/etc/systemd/system/wpa-4addr-${WIFI_IFACE}.service" +write_if_changed "$UNIT" "[Unit] +Description=wpa_supplicant on ${WIFI_IFACE} with 4addr mode enabled +Before=network-pre.target +Wants=network-pre.target + +[Service] +Type=simple +ExecStartPre=/sbin/ip link set ${WIFI_IFACE} down +ExecStartPre=/sbin/iw dev ${WIFI_IFACE} set 4addr on +ExecStartPre=/sbin/ip link set ${WIFI_IFACE} up +ExecStart=/sbin/wpa_supplicant -i ${WIFI_IFACE} -c ${WPA_CONF} +Restart=on-failure +RestartSec=3 + +[Install] +WantedBy=multi-user.target" +systemctl daemon-reload +systemctl enable --now "wpa-4addr-${WIFI_IFACE}.service" +sleep 5 +if ! iw dev "$WIFI_IFACE" link | grep -q "^Connected"; then + echo "WARNING: ${WIFI_IFACE} did not associate to '$WIFI_SSID' within 5s - check:" >&2 + echo " systemctl status wpa-4addr-${WIFI_IFACE}.service" >&2 + echo " journalctl -u wpa-4addr-${WIFI_IFACE}.service" >&2 + exit 1 +fi +echo "associated: $(iw dev "$WIFI_IFACE" link | grep '^Connected')" + +echo +echo "=== 4/4: /etc/network/interfaces (bond0 active-backup: ${WIFI_IFACE} primary, ${LAN_IFACE} backup) ===" +IFACES_FILE="/etc/network/interfaces" +backup_file "$IFACES_FILE" +cat > "$IFACES_FILE" <