updated structure

.claude/worktrees/debian-restructure/proxmox/docs/00-overview.md

@@ -0,0 +1,116 @@
+# Overview & Roadmap
+
+## Staging
+
+This repo now targets two distinct stages, in order:
+
+- **Stage 1 (active)** — base configuration and hardening for a single
+  Proxmox host, applicable to *any* node regardless of eventual cluster
+  plans: repo/updates, SSH, firewall, PVE user/access hardening. This is
+  what `scripts/bootstrap.sh`, `scripts/audit.sh`, and
+  `04-security-hardening.md` cover, and what's being built out against
+  node 1 (`pve1`, an ASUS PN53 mini PC) right now.
+- **Stage 2 (future)** — the multi-node HA/Ceph cluster described below
+  and in `01-hardware-node1.md` / `02-storage-zfs-ceph.md` /
+  `03-networking.md`. Deliberately deferred: `pve1`'s hardware (2 NVMe
+  already merged into one ZFS mirror used for both boot and VM storage, a
+  single 2.5GbE NIC) can't support the separate boot/Ceph disks or
+  bonded/segregated networking those docs assume. Revisit once dedicated
+  cluster hardware (nodes 2/3) is actually being bought and provisioned;
+  until then treat the content below as a target design, not a
+  description of `pve1`.
+
+## Background
+
+Old hardware required disabling KVM hardware virtualization for VMs to
+start at all (falls back to software emulation — slow). This is a
+host/BIOS-level issue, not a Proxmox limitation. Confirmed not present on
+`pve1` (ASUS PN53, Ryzen 7 7735HS): AMD-V and IOMMU both show enabled at
+boot (`dmesg | grep -i iommu`), no workaround needed. If this hardware line
+is reused for nodes 2/3, this should hold there too, but re-verify per
+node before assuming it.
+
+## Stage 1: base config & hardening (active)
+
+Applies to `pve1` now, and to every future node regardless of whether it
+ever joins the Stage 2 cluster. Covered by `04-security-hardening.md` and
+`scripts/bootstrap.sh` / `scripts/audit.sh`:
+
+1. Fresh PVE install; confirm VT-x/AMD-V + IOMMU per the note above.
+2. Switch off the enterprise repos, onto no-subscription
+   (`scripts/switch-to-no-subscription-repo.sh`).
+3. SSH hardening: key-only root login + fail2ban
+   (`scripts/harden-ssh.sh`).
+4. Unattended security upgrades, no auto-reboot
+   (`scripts/setup-unattended-upgrades.sh`).
+5. PVE datacenter firewall, default-deny, mgmt-only SSH/8006
+   (`scripts/deploy-firewall.sh`).
+6. Named PVE admin user (Administrator role) + 2FA, `root@pam` reserved
+   for emergencies (`scripts/create-admin-user.sh`, then manual TOTP
+   enrollment via the web UI).
+7. Verify with `scripts/audit.sh`.
+
+`pve1`'s actual disk/network layout (single ZFS mirror for boot + VMs, one
+2.5GbE NIC) is documented as-is in `01-hardware-node1.md` — Stage 1 doesn't
+require or assume the split-disk/multi-NIC layout Stage 2 wants.
+
+## Stage 2: HA cluster + Ceph (future, deferred)
+
+Everything below this point is the target design for when nodes 2 and 3
+are actually being provisioned. Not applicable to `pve1` as it stands.
+
+**`pve-test` existing does not mean node 2 exists.** A second physical
+node (`pve-test`) does run alongside `pve1` — see `05-node-roles.md` —
+but it's a sandbox/test box, not built to this Stage 2 design, and as of
+this writing runs on **wifi** networking
+(`06-pve-test-wifi-network.md`), which is directly incompatible with
+corosync's latency/jitter requirements below. `pve-test` and `pve1` were
+briefly clustered and then deliberately de-clustered for exactly this
+reason. Don't treat `pve-test` as progress toward Stage 2 without a
+deliberate decision to rebuild its networking first.
+
+### End goal
+
+3-node Proxmox VE cluster with HA-managed VMs backed by **Ceph** — true
+distributed shared storage, sync replication, near-zero RPO on failover
+(see `02-storage-zfs-ceph.md`). Ceph needs 3+ nodes and a fast dedicated
+network, so it can't exist until nodes 2 and 3 are up.
+
+Deliberately no intermediate "ZFS + storage replication" HA step. Node 1
+runs local ZFS for boot + VM storage with no cluster-wide HA until Ceph
+goes live — as soon as nodes 2/3 join, VMs move onto Ceph rather than
+adopting ZFS replication as a stopgap. Simpler end state, one storage
+model to operate instead of two.
+
+### Cluster fundamentals (apply from node 1 onward)
+
+- 3 nodes minimum for real quorum. If starting with 2, add a QDevice
+  (small VM or Raspberry Pi) as tie-breaker.
+- Dedicated network for corosync (cluster/quorum traffic) — never shared
+  with VM or storage traffic. Needs low, consistent latency (well under
+  5ms); jitter matters more than bandwidth.
+- All nodes on the same PVE version, NTP-synced, SSH reachable between
+  nodes.
+- Set VM CPU type to a portable type (e.g. `x86-64-v2-AES` or `kvm64`)
+  rather than `host` if nodes will ever have different CPUs — needed for
+  clean live migration.
+
+### Rollout sequence (Stage 2, once dedicated cluster hardware exists)
+
+1. Build node 1 per `01-hardware-node1.md`'s target design — fresh PVE
+   install on a dedicated ZFS boot mirror, Ceph-earmarked disks left idle
+   (or as a temporary local ZFS pool, to be wiped later — see
+   `02-storage-zfs-ceph.md`). Note: this assumes hardware with enough
+   disks/NICs to separate boot, Ceph, and network roles — `pve1` does not
+   have this and stays on Stage 1 only unless rebuilt on different
+   hardware.
+2. Migrate VMs onto the new node via `vzdump` → copy backups →
+   `qmrestore` (converts disks to ZVOLs). No HA yet — single node.
+3. Stage 1 base hardening already applied; layer on Stage 2 networking
+   (`03-networking.md`) before joining a cluster.
+4. Add nodes 2 and 3 identically (same disk/network layout).
+5. Join cluster, stand up dedicated corosync network.
+6. Wipe the Ceph-earmarked disks (if used as temporary ZFS) and
+   initialize Ceph across all 3 nodes.
+7. Migrate VMs from local ZFS onto Ceph-backed storage, then configure HA
+   groups.

.claude/worktrees/debian-restructure/proxmox/docs/01-hardware-node1.md

@@ -0,0 +1,92 @@
+# Node 1 Hardware Layout
+
+## pve1 as built (current reality)
+
+`pve1` is an ASUS PN53 mini PC (Ryzen 7 7735HS, 32GB RAM), not the
+dedicated-server hardware the target design below assumes:
+
+- **Disks**: 2x 2TB NVMe (Crucial CT2000E100SSD8), both in a single ZFS
+  mirror (`rpool`) that serves as both the boot pool and VM storage
+  (`local-zfs` = `rpool/data`). No spare disks to earmark for Ceph — the
+  chassis only has 2 NVMe slots.
+- **Network**: one physical NIC (Realtek RTL8125, 2.5GbE), bridged as
+  `vmbr0`. No second NIC for a dedicated corosync/Ceph link. (An unused
+  `nic1` stanza in `/etc/network/interfaces` is a leftover from the
+  installer template — there is no second NIC on this hardware.)
+
+This is sufficient and correct for **Stage 1** (see `00-overview.md`) —
+base config and hardening don't need split disks or multiple NICs. It is
+*not* sufficient for **Stage 2** (Ceph/HA) as designed below without
+either different hardware or a materially different plan (e.g.
+USB/Thunderbolt-attached OSD storage, which trades away the
+enterprise-SSD/PLP guidance below — not recommended, revisit when
+actually provisioning nodes 2/3). Treat everything from here down as the
+Stage 2 target design for purpose-built hardware, not a description of
+`pve1`.
+
+## Target design (Stage 2, future dedicated hardware)
+
+Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect
+disks or network when the cluster grows.
+
+## Disks — two roles, physically separate devices
+
+1. **Boot/OS pool (`rpool`)** — 2x small SSDs (240-480GB plenty), ZFS
+   mirror. Proxmox itself only. Never share with Ceph OSDs.
+2. **Future Ceph OSD disks** — must end up as raw, unformatted devices —
+   no ZFS/RAID/LVM underneath (Ceph does its own replication; anything
+   underneath just doubles copy-on-write/checksumming and hurts
+   performance). Use enterprise SATA/NVMe SSDs with power-loss protection
+   (PLP) — matters far more for Ceph write latency than for general ZFS
+   use. Ceph needs 3 nodes minimum to go live, so on node 1 these disks
+   either sit idle or run as a temporary local ZFS pool (all VMs live
+   here until nodes 2/3 exist), to be wiped and handed to Ceph once the
+   cluster can actually run it. See `02-storage-zfs-ceph.md`.
+
+No permanent local-ZFS "replicated tier" — once Ceph is live, it's the
+only HA storage; local ZFS is boot pool + this temporary pre-Ceph staging
+role, not an ongoing parallel tier. Avoid consumer QLC SSDs for either
+role — Ceph punishes it on latency, ZFS on sync writes/scrub.
+
+## Networking — cable and provision for the final topology now
+
+Logically separate networks (ideally separate NICs/VLANs):
+
+- **Management** — web UI / SSH
+- **Corosync** — cluster quorum traffic, low-latency, unshared
+- **Ceph public** — VM-to-OSD traffic
+- **Ceph cluster/backend** — OSD-to-OSD replication (heaviest load)
+
+Practical layout: 2x 10/25GbE bonded or split — one pair for Ceph, one
+for mgmt + corosync + VM traffic, with corosync on its own VLAN even if
+sharing a physical NIC. Get switch/cabling right on node 1 so nodes 2/3
+are identical drops.
+
+## CPU / RAM sizing
+
+Size for the end state, not day one — RAM is the hardest thing to
+retrofit. Budget covers:
+
+- OS + ZFS ARC (ZFS wants RAM, not just disk)
+- Ceph OSD daemons — realistically 3-5GB per OSD once running
+- Actual VM workloads
+
+Roughly a core per OSD on top of what VMs need. If OSDs won't be active
+for a while, that's headway, but buy for 3 nodes' worth of eventual OSD
+load.
+
+## Backup target (PBS)
+
+Keep it off the Ceph/compute nodes if possible — its failure domain
+should be independent of the cluster. Modest separate machine or NAS:
+ZFS mirror or raidz2, ECC RAM if possible, capacity for retention policy.
+If it has to run as a VM inside the cluster short-term, that's a known
+compromise, not the end state.
+
+## Node 1 install sequence
+
+1. Install Proxmox VE fresh onto the ZFS boot mirror.
+2. Provision the Ceph-earmarked disks as a temporary local ZFS pool and
+   run all VMs from it (or leave idle if VMs aren't moving over yet).
+3. Once nodes 2/3 join and Ceph goes live: wipe this pool, hand the disks
+   to Ceph, migrate VMs onto Ceph-backed storage.

.claude/worktrees/debian-restructure/proxmox/docs/02-storage-zfs-ceph.md

@@ -0,0 +1,66 @@
+# Storage: LVM-thin → ZFS → Ceph
+
+**Stage 2 (future).** Describes the target storage model once dedicated
+cluster hardware exists. `pve1`'s current single ZFS mirror (boot + VM
+storage combined, see `01-hardware-node1.md`) is the Stage 1 end state for
+now, not an intermediate step being actively migrated from.
+
+## Why move off LVM-thin
+
+Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs
+a VM's disk reachable from more than one node so it can restart elsewhere
+on host failure. LVM-thin has no answer for that. Ceph does, natively.
+
+## Storage model: single node → 3-node cluster
+
+No intermediate "ZFS + replication" HA step. The plan is deliberately a
+single storage model at the end (Ceph), not two to operate long-term:
+
+- **Single node (node 1 only)**: local ZFS pool, no cluster-wide HA. This
+  is a temporary state, not a design to build tooling around.
+- **3 nodes with Ceph live**: VMs run on Ceph-backed storage — true
+  distributed, synchronous storage across all nodes, near-zero RPO on
+  failover. Needs 3+ nodes and a fast dedicated network (see
+  `03-networking.md`), which is exactly why it can't exist before then.
+
+## Migration plan (old hardware → new hardware)
+
+Don't convert the old LVM-thin box in place. Rebuild fresh on new
+hardware with ZFS from the installer (mirror if 2+ disks), then move VMs:
+
+1. On the old host: `vzdump` each VM to a backup file (external drive,
+   NFS share, or PBS if available).
+2. Copy backups to the new host.
+3. `qmrestore` onto the new ZFS storage — disks land as ZVOLs.
+
+Alternative if both hosts can see each other on the network: temporarily
+cluster them and use the GUI "Migrate" with a storage move (offline only
+— live migration doesn't cross storage types).
+
+## Standing up Ceph once nodes 2 and 3 exist
+
+1. Wipe the temporary local ZFS pool on the Ceph-earmarked disks (see
+   `01-hardware-node1.md`) on all 3 nodes — they need to end up raw,
+   unformatted.
+2. Install the Ceph packages on all 3 nodes (`pveceph install`) and
+   initialize the cluster (`pveceph init`), using the dedicated Ceph
+   network from `03-networking.md`.
+3. Create Ceph monitors and managers (3 mons for quorum, matching node
+   count).
+4. Create OSDs directly on the raw disks on each node — no ZFS/RAID
+   underneath.
+5. Create a Ceph pool sized for your VM storage needs (replica count,
+   typically 3 for full redundancy across 3 nodes).
+6. Add the pool as PVE storage (RBD), then migrate VMs from local ZFS
+   onto it — offline migration if crossing storage types, or storage
+   migration via the GUI.
+7. Configure HA groups once VMs are on Ceph-backed storage.
+
+## Notes
+
+- Ceph RAM/CPU overhead is real — budget per `01-hardware-node1.md`
+  (roughly 3-5GB RAM and a core per OSD, on top of VM workloads).
+- Enterprise SSDs with power-loss protection (PLP) matter far more here
+  than for plain ZFS — Ceph write latency is sensitive to it.
+- Once Ceph is live, local ZFS remains only for each node's boot pool —
+  it's not a fallback tier for VM storage going forward.

.claude/worktrees/debian-restructure/proxmox/docs/03-networking.md

@@ -0,0 +1,66 @@
+# Networking
+
+## pve1 as built (Stage 1, current)
+
+Single NIC (2.5GbE), single bridge `vmbr0` on the flat LAN
+(`192.168.2.0/24`), no VLANs. There is no corosync or Ceph traffic to
+separate yet — this node isn't clustered. Segmentation for Stage 1 is
+done at the firewall, not the network: `scripts/deploy-firewall.sh`
+restricts SSH (22) and the web UI (8006) to the management CIDR via the
+PVE datacenter firewall (default-deny inbound otherwise). That's
+sufficient until Stage 2 needs actual separate physical/VLAN paths for
+corosync and Ceph traffic — see below.
+
+## pve-test as built (sandbox, current)
+
+Different node, different design, not a Stage 1/2 example to generalize
+from: `pve-test` runs `vmbr0` bridged over a wifi NIC in 4addr client mode
+(active-backup bonded with a wired NIC as an automatic fallback). Full
+detail, including why this is normally impossible and how it was
+validated before trusting it with the management IP, in
+`06-pve-test-wifi-network.md`. This is intentionally a one-off for a
+standalone sandbox box — never extend it to a node that's clustered or
+Ceph-connected (see the Stage 2 note in `00-overview.md`).
+
+## Target design (Stage 2, future cluster)
+
+## Required separation
+
+Keep these on logically separate networks/VLANs, ideally separate NICs:
+
+- **Management** — web UI (8006), SSH
+- **Corosync** — cluster quorum. Low, *consistent* latency (well under
+  5ms) matters more than bandwidth. Never share with VM/storage traffic.
+- **Ceph public** — VM-to-OSD traffic (once Ceph is live)
+- **Ceph cluster/backend** — OSD-to-OSD replication, heaviest load
+
+## Practical layout
+
+2x 10/25GbE bonded or split:
+
+- Link pair A → Ceph (public + backend, or split further if 4 NICs
+  available)
+- Link pair B → management + corosync + VM traffic, with corosync on its
+  own VLAN even when sharing a physical NIC with the rest
+
+## Cluster join requirements
+
+- All nodes reachable to each other on SSH (22) and the corosync network
+- Same PVE version across nodes
+- NTP-synced clocks
+
+## Firewall
+
+Proxmox's built-in firewall operates at datacenter and node level.
+Default-deny, then whitelist:
+
+- SSH from the management network/VLAN only
+- Web UI (8006) from the management network/VLAN only
+- Corosync ports between cluster nodes
+- Ceph ports between cluster nodes (once Ceph is live)
+
+Enforce the network separation above at the firewall — corosync and Ceph
+traffic shouldn't be reachable from the VM network even if they end up
+sharing a physical link.
+
+See `config/pve-firewall/` for a starting rule set.

.claude/worktrees/debian-restructure/proxmox/docs/04-security-hardening.md

@@ -0,0 +1,112 @@
+# Security Hardening
+
+Stage 1 (see `00-overview.md`) — applies to any Proxmox host, independent
+of cluster plans. Proxmox has no `sudo` out of the box — everything
+defaults to root. That's the install default, not the recommended end
+state. Two layers to harden separately.
+
+## Checklist / script mapping
+
+Run `scripts/bootstrap.sh` for everything except the admin user (needs a
+username decision) and 2FA enrollment (must be done interactively via the
+web UI — there's no safe way to script TOTP secret generation over SSH).
+Then run `scripts/audit.sh` to verify. Order matters (matches
+`bootstrap.sh`):
+
+| # | Item | Script | Manual step required? |
+|---|------|--------|------------------------|
+| 1 | Remove enterprise repos, switch to no-subscription | `switch-to-no-subscription-repo.sh` | no |
+| 2 | Linux admin user with SSH key + sudo access | `setup-linux-admin-user.sh <user> <pubkey>` | yes — choose a username and provide your SSH public key. Run this **before** SSH hardening or pass `ADMIN_USER`/`ADMIN_SSH_KEY` to `bootstrap.sh` so it runs automatically at the right point |
+| 3 | Passwordless sudo for pvesh/qm/pct | `setup-admin-sudo.sh <username>` | yes — same username as step 2 |
+| 4 | SSH: key-only root login + fail2ban | `harden-ssh.sh` | no — step 2 ensures authorized_keys are in place first |
+| 5 | Unattended security upgrades, no auto-reboot | `setup-unattended-upgrades.sh` | no |
+| 6 | PVE firewall, default-deny, mgmt-only SSH/8006 | `deploy-firewall.sh` | needs `MGMT_CIDR` set |
+| 7 | Disable subscription nag (cosmetic) | `disable-subscription-nag.sh` | no |
+| 8 | Named PVE admin user, Administrator role | `create-admin-user.sh <username>` | yes — pick the username, change the generated password on first login |
+| 9 | 2FA/TOTP on that user and `root@pam` | — | yes — web UI only: Datacenter → Permissions → Two Factor, or user menu → TFA |
+| 10 | Verify everything above | `audit.sh` | no |
+
+## Linux/SSH layer
+
+- A named Linux system user (created by `setup-linux-admin-user.sh`) with an
+  SSH authorized key and `sudo` group membership is the primary SSH login
+  account. Root SSH is locked to key-only after `harden-ssh.sh` runs; the
+  Linux admin user is how you get shell access day-to-day without using root.
+- `setup-admin-sudo.sh` adds a narrower, password-free sudoers rule for the
+  Proxmox management tools (`pvesh`, `qm`, `pct`) specifically — needed for
+  non-interactive automation scripts that SSH in and run these tools without a
+  TTY.
+- `PermitRootLogin prohibit-password` in `sshd_config` — root can only
+  log in via SSH key, never password. Kills most brute-force attempts.
+- fail2ban jail for SSH on top of that.
+- Restrict SSH to the management VLAN/trusted IPs via the Proxmox
+  firewall (see `03-networking.md`) rather than exposing broadly.
+
+## PVE/web layer (the one that actually matters day-to-day)
+
+- Keep `root@pam` for emergencies only.
+- Create a named user (e.g. `wayne@pve`) with the Administrator role for
+  routine cluster management — `create-admin-user.sh` does this, or
+  Datacenter → Permissions → Users manually.
+- Enable 2FA (TOTP or hardware key) on both that account and `root@pam`:
+  Datacenter → Permissions → Realms/Users.
+- For API integrations (monitoring, automation, Terraform, etc.), issue
+  scoped API tokens with least-privilege roles (e.g. `PVEAuditor` or a
+  custom role) — never hand out root credentials.
+
+## Firewall
+
+Default-deny at datacenter/node level, whitelist only what's needed (see
+`03-networking.md` for the specifics). Template in
+`config/pve-firewall/cluster.fw.example`, applied by
+`scripts/deploy-firewall.sh`.
+
+**ICMP echo (ping) is explicitly allowed from the management network**,
+alongside SSH/8006 — not required for anything to function, but without
+it `policy_in: DROP` silently eats ping while SSH/web UI keep working.
+That split (ping dead, everything else fine) reads exactly like a real
+outage mid-troubleshooting; see `06-pve-test-wifi-network.md` for a case
+this caused genuine confusion after a network change. If you ever debug
+"can't ping but can SSH", check this firewall before assuming the
+network itself is broken.
+
+**This file is cluster-wide, not per-node**: `cluster.fw` lives in
+`/etc/pve/firewall/` — shared via pmxcfs across every node in a cluster.
+A node that joins a cluster inherits whatever's already there, and (per
+`docs/05-node-roles.md`) keeps its own local copy after leaving. Don't
+assume a node's firewall state matches what `deploy-firewall.sh` was
+last run with directly against it — check `pve-firewall status` /
+`/etc/pve/firewall/cluster.fw` on the actual node.
+
+## Repos and updates
+
+Fresh installs point at the enterprise repo, which fails on `apt update`
+without a subscription. `scripts/switch-to-no-subscription-repo.sh`
+removes the enterprise sources entirely (renamed `.disabled`, not just
+commented out) and switches to the no-subscription repo — handles both
+the legacy `.list` format and the deb822 `.sources` format current
+installers write. Keep the host patched — hypervisor CVEs are high-value
+targets; `scripts/setup-unattended-upgrades.sh` automates security
+patches (deliberately no auto-reboot on a hypervisor — check
+`/var/run/reboot-required` and reboot during a planned window).
+
+The web UI's "No valid subscription" popup and dashboard indicator are
+cosmetic upsell, not a security control, but with no subscription they'll
+nag on every login — `scripts/disable-subscription-nag.sh` patches
+`proxmox-widget-toolkit`'s JS to suppress them, and installs an apt
+`Post-Invoke` hook that reapplies the patch automatically after every
+`apt`/`dpkg` run, since a `proxmox-widget-toolkit` package upgrade
+overwrites the patched file.
+
+## Misc
+
+- Management interface on a network you trust, not the same broadcast
+  domain as guest VM traffic.
+- If the web UI is ever needed outside the LAN, put it behind a VPN —
+  don't port-forward 8006 directly.
+
+## Further reading / not yet automated here
+
+- CIS Benchmark for Proxmox VE
+- Community PVE hardening guides (kernel parameters, audit logging,
+  storage encryption)

.claude/worktrees/debian-restructure/proxmox/docs/05-node-roles.md

@@ -0,0 +1,83 @@
+# Node Roles: pve1 and pve-test
+
+Two Proxmox VE hosts exist on the LAN. They are **not interchangeable** —
+different purpose, different risk tolerance, different guardrails. See
+`CLAUDE.md` for the actual rules Claude follows; this doc is the
+background/context those rules assume.
+
+## pve1 (production)
+
+ASUS PN53 mini PC (Ryzen 7 7735HS, 32GB RAM) — see `01-hardware-node1.md`
+for the full hardware layout. Stage 1 base hardening applied and verified
+(`04-security-hardening.md`).
+
+Runs real production workloads:
+
+- `server-nixos` (VM) and LXC containers `pihole`, `claude`, `nix-cache`,
+  `docker`, `pdm`, `pxe-boot` — DNS, this Claude Code environment itself,
+  the LAN's Nix binary cache, etc.
+
+Off-limits to Claude by default — see `CLAUDE.md`.
+
+## pve-test (sandbox)
+
+A separate physical node kept for testing. Has a capable GPU (earmarked
+for future GPU workloads — transcoding, local inference, etc. — nothing
+built on it yet as of this writing). No production guests run here.
+
+Safe for Claude to create/destroy scratch VMs and containers on without
+asking — see `CLAUDE.md` for the exact boundary (node-level config is
+still not free-for-all even here).
+
+### History: briefly clustered with pve1, then deliberately separated
+
+On 2026-07-21/22, pve-test was joined to pve1 in a 2-node Proxmox cluster
+(corosync + shared `/etc/pve`) for a period, then the cluster was
+**intentionally destroyed** at the operator's explicit request — not a
+failure or accident. Rationale: the operator wanted pve-test moved onto
+wifi networking (see `06-pve-test-wifi-network.md`), and wifi is
+fundamentally unsuitable for corosync's latency/jitter requirements (token
+timeouts, flapping membership). Rather than fight that, the cluster was
+torn down first, then wifi was configured on the now-standalone node.
+
+Teardown procedure used (safe to repeat if pve-test or pve1 is ever
+reclustered and needs separating again):
+
+1. Un-manage any HA-assigned resources first if HA is in use
+   (`ha-manager remove <sid>` per resource) — **critical**: if a node has
+   HA-managed guests and its watchdog is still armed when it loses
+   quorum, HA's self-fence behavior can force-reboot it. Removing the
+   resource assignments first (not stopping the guests — just
+   un-managing them) prevents this.
+2. Stop `pve-ha-lrm`/`pve-ha-crm` on all nodes once no resources remain
+   assigned, confirm via `ha-manager status` that fencing is no longer
+   armed.
+3. From a node that will remain in the cluster: `pvecm delnode <name>`
+   for the node being removed. In a 2-node cluster with no QDevice, this
+   can itself cause the *remaining* node to transiently lose quorum
+   (killing a node's live corosync membership drops total votes below
+   expected votes before the nodelist file can be rewritten to match) —
+   this is expected, not a failure; proceed to step 4 regardless.
+4. On **each** node being separated (including one you just ran `delnode`
+   from, if it's also being decoupled): `systemctl stop pve-cluster
+   corosync`, `pmxcfs -l` (local mode, bypasses the quorum requirement
+   for local writes), remove `/etc/pve/corosync.conf` and
+   `/etc/corosync/*`, `killall pmxcfs`, `systemctl start pve-cluster`.
+   This is the official "separate a node without reinstalling" procedure
+   and doesn't touch any running VM/CT — they're independent OS
+   processes, unaffected by corosync/pmxcfs state either way.
+5. Optional cosmetic cleanup: the node(s) staying up may retain a stale
+   `/etc/pve/nodes/<removed-node>/` directory (cached VM/CT configs from
+   when they were clustered) — safe to `rm -rf` once separation is
+   confirmed; it's dead data, not a live reference to anything.
+
+Guests were never at risk during this procedure (confirmed: all 7 of
+pve1's guests ran continuously throughout) — the only real hazard is the
+HA/watchdog self-fence path in step 1, and losing corosync-provided quorum
+temporarily blocking `/etc/pve` writes (not guest execution) in step 3.
+
+### Current state: standalone, wifi-primary networking
+
+pve-test now runs entirely independently of pve1, on the network design
+documented in `06-pve-test-wifi-network.md`. **Do not re-cluster
+pve-test while it's on wifi** — see that doc for why.

.claude/worktrees/debian-restructure/proxmox/docs/06-pve-test-wifi-network.md

@@ -0,0 +1,191 @@
+# pve-test's Wifi-Primary Network
+
+Non-standard, deliberately chosen against normal Proxmox guidance (a
+hypervisor's management network is not supposed to be wireless). Applies
+**only** to `pve-test` as a standalone sandbox node — see
+`05-node-roles.md` for why this and cluster membership don't mix, and
+never apply this pattern to a node that's joined (or might join) a
+cluster.
+
+## Why this works at all
+
+Wifi (802.11) normally cannot be bridged the way Ethernet can: an access
+point only accepts frames whose source MAC matches the MAC that
+associated with it. A Linux bridge with a wifi port as a member sends
+frames tagged with whatever MAC actually generated them (the bridge's own
+MAC, a VM's MAC, etc.) — none of which match the wifi card's hardware
+MAC, so the AP silently drops them. This isn't a Linux/Proxmox limitation,
+it's how WiFi association works.
+
+**4addr (WDS) client mode** is the exception: if both the wifi driver and
+the AP support it, the client tags frames with a 4th address field
+carrying the original MAC, and the AP forwards them like a real Ethernet
+segment. Most consumer routers do **not** support this (it's common on
+OpenWrt/DD-WRT/enterprise APs, rare on stock ISP hardware) — it has to be
+confirmed empirically, per-AP, before trusting it with a management IP.
+
+pve-test's card (Intel AX200, `iwlwifi` driver, interface `wlp3s0`)
+supports 4addr client-side generically (this is a `mac80211` core
+feature, not driver-specific). The home AP it associates to
+(`nbn-fttp-net-5G`) was confirmed to honor it — see validation method
+below.
+
+## Validate before ever touching a live management IP
+
+Don't add the wifi NIC straight into your real bridge to "see if it
+works" — if 4addr isn't actually honored, you lose access to the box with
+no diagnostic trail (association succeeds; only the *data plane* silently
+fails). Test in an isolated scratch namespace first:
+
+```bash
+# assumes wlp3s0 already associated with 4addr on (see systemd unit below)
+ip link add testbr0 type bridge && ip link set testbr0 up
+ip link set wlp3s0 master testbr0 && ip link set wlp3s0 up
+
+ip netns add wifitest
+ip link add veth-host type veth peer name veth-ns
+ip link set veth-host master testbr0 && ip link set veth-host up
+ip link set veth-ns netns wifitest
+ip netns exec wifitest ip link set veth-ns address 02:11:22:33:44:55  # deliberately NOT the wifi card's MAC
+ip netns exec wifitest ip link set veth-ns up
+ip netns exec wifitest ip link set lo up
+
+ip netns exec wifitest dhclient -v -1 veth-ns   # real DHCP lease = AP forwards foreign MACs = 4addr genuinely works
+```
+
+If that gets a lease (or at minimum a ping reply from the gateway) from a
+MAC that was never the one that associated, the AP is forwarding
+arbitrary-MAC frames — real bridge mode will work. If not, don't proceed
+to bridge mode; fall back to a routed/NAT design instead (wifi gets its
+own IP via DHCP, VMs NAT out through it — never covered here since it
+wasn't needed, but keep this fallback in mind if reproducing on different
+hardware/AP).
+
+Tear the test namespace/bridge down afterwards (`ip netns del wifitest;
+ip link del veth-host; ip link set wlp3s0 nomaster; ip link del testbr0`)
+— it's scratch, not part of the real config.
+
+## Final architecture
+
+```
+wlp3s0 (wifi, 4addr client mode) ─┐
+                                    ├─ bond0 (active-backup, wlp3s0 primary) ─ vmbr0 (192.168.2.251/24)
+nic0 (wired, LAN backup)          ─┘
+```
+
+- **`wpa-4addr-<iface>.service`** (systemd unit, `Before=network-pre.target`) —
+  sets 4addr mode and starts `wpa_supplicant` *before* ifupdown2 processes
+  `/etc/network/interfaces`. Deliberately not using `wpasupplicant`'s own
+  `wpa-conf` ifupdown2 hook integration — its ordering relative to a
+  custom `pre-up iw ... set 4addr on` line is implementation-specific and
+  not worth gambling on for something this hard to debug if it's wrong.
+  The systemd unit gives full, deterministic control over sequencing
+  instead.
+- **`bond0`** — active-backup bonding, `wlp3s0` as `bond-primary`,
+  `nic0` as backup, `bond-miimon 100`. Bonding a wifi interface works here
+  because `mac80211`/`iwlwifi` properly reports carrier state
+  (`netif_carrier_on/off`) on association/disassociation, which is what
+  bonding's `miimon` (with the default `use_carrier=1`) actually watches
+  — it doesn't require a "real" MII-capable NIC.
+- **`vmbr0`** — unchanged IP (`192.168.2.251/24`), bridged over `bond0`
+  instead of directly over a physical NIC.
+
+Reproduce with `scripts/setup-wifi-bond-network.sh` — see its header
+comment for usage and required env vars (SSID/passphrase, management
+address/gateway). It stages `/etc/network/interfaces` and the systemd
+unit but does **not** auto-apply (`ifreload -a`) — see the next section
+for why that step needs care, not automation.
+
+## Applying a live bridge-port change: what went wrong once, and what worked
+
+Moving `vmbr0`'s underlying port live — while SSH'd in over the very
+address that lives on that bridge — is inherently risky: if the new port
+doesn't actually pass traffic, you lose the connection you're using to
+fix it.
+
+**What failed**: doing the port swap via raw `ip link set nic0 nomaster`
++ `ip link set nic0 down` (with `wlp3s0` already added as a second bridge
+member) caused an extended outage that needed a physical power-cycle to
+recover — `nic0` never came back up on its own even well past a 60s
+auto-revert watchdog's deadline. Importantly, the *host itself* never
+crashed — `journalctl -b -1` showed completely normal operation (a
+`pvestatd` polling loop, no gaps, no panic/OOM/watchdog trigger) right up
+until a keyboard was plugged in and it was manually reset. So the failure
+was specifically in the live network transition, not the OS — root cause
+unconfirmed, but plausibly bridge FDB/MAC-identity handling when a port
+carrying an already-live IP is swapped out via raw `ip link` rather than
+a coordinated reconfiguration.
+
+**What worked**: writing the final state into `/etc/network/interfaces`
+and applying with `ifreload -a` (Proxmox's own supported hot-reload path)
+instead. Same end state, same risk window, but ifupdown2 evidently
+sequences the transition in a way raw `ip link` surgery didn't. This
+succeeded on the first retry using this method.
+
+**Either way, expect ~30-60 seconds of apparent breakage even on a
+successful change** — the upstream switch/AP needs to relearn which port
+`192.168.2.251`'s MAC now lives behind. Don't judge success/failure
+before that window passes; a `ping`/SSH failure at the 5-10s mark is not
+yet a sign anything is wrong.
+
+**Recommended safety net for any future live change here**: arm a
+backgrounded auto-revert-on-timeout before applying, e.g.:
+
+```bash
+cp /etc/network/interfaces /etc/network/interfaces.bak
+cat > /root/revert.sh <<'EOF'
+#!/bin/bash
+cp /etc/network/interfaces.bak /etc/network/interfaces
+ifreload -a
+EOF
+chmod +x /root/revert.sh
+setsid nohup bash -c "sleep 45 && /root/revert.sh" >/root/revert.log 2>&1 < /dev/null &
+disown
+# now apply the real change, e.g.: ifreload -a
+# once confirmed working: pkill -f "sleep 45 && /root/revert.sh"
+```
+
+This only self-heals if the box stays responsive enough for the
+backgrounded job to run to completion — it is not a substitute for
+physical/console access being available, just a way to avoid needing it
+for the common case.
+
+## Troubleshooting: "can't ping it, web UI won't load" isn't necessarily an outage
+
+Two independent things can make `pve-test` look dead when it isn't:
+
+1. **Stale ARP after any change to `vmbr0`'s active port.** `vmbr0`'s MAC
+   address follows whichever interface is currently active in `bond0`
+   (`wlp3s0`'s MAC while wifi is primary, `nic0`'s if it fails over) — it
+   is *not* fixed. Other devices on the LAN (including your own
+   workstation) that cached the old MAC before a change won't notice
+   until their ARP entry naturally expires or gets flushed
+   (`ip neigh flush <ip>` / `arp -d <ip>`). Symptom: intermittent or
+   totally dead connectivity from one specific device while others (or
+   the host itself) are fine.
+2. **ICMP ping is blocked by the management firewall, unrelated to
+   network health.** See `04-security-hardening.md`'s firewall section —
+   `cluster.fw`'s `policy_in: DROP` only explicitly allows TCP 22/8006
+   (now also ICMP echo-request, after this was hit) from the mgmt
+   network. Before that rule was added, `ping` failed 100% *even with a
+   perfectly correct, fresh ARP entry and full SSH/web UI access* —
+   nothing to do with wifi, bonding, or the network being down. This is
+   exactly what happened once: alarming "can't ping, web UI not loading"
+   turned out to be a stale-ARP moment (which resolved itself) plus a
+   pre-existing firewall policy (ping was never going to work,
+   regardless of wifi).
+
+**When `pve-test` seems unreachable, check in this order before assuming
+a real outage**: SSH (`ssh root@pve-test.sweet.home`) → web UI via `curl
+-sk -o /dev/null -w '%{http_code}' https://pve-test.sweet.home:8006/` →
+only then worry about `ping` specifically, and check
+`/etc/pve/firewall/cluster.fw` before blaming the network.
+
+## Known limitation history
+
+`nic0` (the bond's backup slave) initially had no cable physically
+connected when this was first set up — the bond was correctly configured
+but inert until a cable was plugged in. As of this writing a cable is
+connected and the backup slave is live (confirm with `cat
+/proc/net/bonding/bond0`) — if reproducing this setup, verify the same
+before assuming LAN failover will actually work.