beatzaplenty/nixos
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor: full repo sweep — variables, docs, and comment cleanup
Check NixOS configurations / eval-hosts (pull_request) Successful in 10m31s
Details

Browse Source 
- variables.nix: switch to rec {}, extract giteaDomain/giteaRepoPath,
  extraAdminSshKeys, haLanNfsFqdn, tailscaleResolverIp, ports.dhcp,
  ports.dns; ipaServer now derives from homeDomain ref; section headers
- modules: use new vars throughout (pxe-boot, ts-dns-forwarder,
  cluster-config, configuration.nix, mount-pxe-images) — eval unchanged
- docs: delete ephemeral planning docs (AUDIT_REPORT, ha-network-audit,
  network-cutover); add docs/ha.md; drop migration reference table from
  ip-addressing.md; remove stale server example from beszel.md
- CLAUDE.md/README.md/AGENTS.md: fix build types (tailscale-router,
  ha-server, drop server); document scripts/ha/, scripts/ipa/, and
  all previously undocumented top-level and lib scripts

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
beatzaplenty
2026-07-30 06:28:52 +10:00
co-authored by Claude Sonnet 4.6
parent 71381ad990
commit cded77919d
15 changed files with 502 additions and 1162 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/beszel.md
-9
View File
@@ -82,15 +82,6 @@ services.beszel.agent.environment = {
};
```
The `server` host uses this to expose its ZFS data pool:
```nix
services.beszel.agent.environment = {
EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
LOG_LEVEL = "debug";
};
```
---
## Optional: monitoring Docker containers
docs/ha-network-audit.md
-261
View File
@@ -1,261 +0,0 @@
# Storage/Cluster Network Segmentation Audit — pve1.sweet.home
**Date:** 2026-07-29
**Scope:** Read-only discovery of pve1.sweet.home host networking, HA cluster VMs (200/201), and Docker CT (105). No changes made.
> **Implementation status — 2026-07-29:** All recommendations from this audit have been
> implemented in the same session. See `docs/ip-addressing.md` for the current state.
> Key decisions that diverged from the original recommendations:
> - VLAN IDs renumbered: cluster → VLAN 10 (192.168.10.x), storage-client → VLAN 20 (192.168.20.x)
> - Two Pacemaker VIPs: `vip-lan` (192.168.2.229, NFS for LAN) and `vip-storage` (192.168.20.229, NFS + iSCSI for VLAN 20)
> - NFS served on **both** VIPs (each firewalled to its own subnet); iSCSI available on VLAN 20 but NFS is preferred for docker to support future Docker Swarm multi-host access
> - `corosync.conf` ring1 added using LAN IPs (RF-1 resolved)
> - `vmbr2` created and NICs added to HA VMs and docker CT (RF-6/RF-7 resolved)
> - iSCSI portal remains on `[::0]`; firewall enforces VLAN 20 restriction (RF-4 mitigated)
> - STONITH still disabled (RF-3 deferred — accepted risk during development phase)
> - iSCSI ACLs not configured (RF-5 deferred — iSCSI not in active use)
---
## 1. Current State Summary
### pve1.sweet.home Host — Physical NICs
| Interface | Speed/Duplex | Notes |
|-----------|-------------|-------|
| `nic0` | 2500 Mb/s / Full (2.5GbE) | Only active physical NIC; sole bridge port for vmbr0 |
| `nic1` | (not connected / no data) | Present in config, not UP |
| `wlp4s0` | DOWN | WiFi, unused |
No bonding configured. Every guest's traffic ultimately funnels through the single 2.5GbE `nic0`.
### Proxmox Bridges
| Bridge | Physical NIC | Host IP | Subnet | VLAN-aware | Purpose (current) |
|--------|-------------|---------|--------|------------|-------------------|
| `vmbr0` | `nic0` (2.5GbE) | 192.168.2.245/24 | 192.168.2.0/24 | No | General LAN, management, **iSCSI/NFS VIP** |
| `vmbr1` | **none** (internal-only) | — | 192.168.4.224/29 | No | Corosync heartbeat + DRBD replication |
`vmbr1` has `bridge-ports none` in `/etc/network/interfaces.d/vmbr1.conf` — it is a purely software bridge with zero physical uplink. All traffic on it stays inside the hypervisor's memory.
### Guest NIC Assignments
| Guest | VMID | Role | NIC | Bridge | IP | Traffic type |
|-------|------|------|-----|--------|----|--------------|
| nix-cache | CT 102 | Build cache | eth0 | vmbr0 | DHCP/LAN | LAN |
| pxe-boot | CT 103 | PXE/TFTP | eth0 | vmbr0 | DHCP/LAN | LAN |
| tor-relay | CT 104 | Tor | eth0 | vmbr0 | DHCP/LAN | LAN |
| **docker** | **CT 105** | **Docker host** | **eth0** | **vmbr0** | **192.168.2.225/24** | **LAN only** |
| pdm | CT 106 | Proxmox mgmt | eth0 | vmbr0 | 192.168.2.220/24 | LAN |
| server | VM 101 | General server | net0 | vmbr0 | DHCP/LAN | LAN |
| tailscale-router | VM 107 | Tailscale exit | net0 | vmbr0 | DHCP/LAN | LAN |
| domain-controller | VM 108 | FreeIPA | net0 | vmbr0 | 192.168.2.253/24 | LAN |
| **ha-server-1** | **VM 200** | **HA primary** | **net0** | **vmbr0** | **192.168.2.228/24 + VIP 192.168.2.229/24** | **LAN + VIP** |
| **ha-server-1** | **VM 200** | **HA primary** | **net1** | **vmbr1** | **192.168.4.228/29** | **Corosync + DRBD** |
| **ha-server-2** | **VM 201** | **HA secondary** | **net0** | **vmbr0** | **192.168.2.227/24** | **LAN** |
| **ha-server-2** | **VM 201** | **HA secondary** | **net1** | **vmbr1** | **192.168.4.227/29** | **Corosync + DRBD** |
### Corosync/Pacemaker State
- **Transport:** knet/UDP
- **Rings:** 1 only — ring0 on `192.168.4.228` / `192.168.4.227` (vmbr1/ens19)
- **Cluster status:** Both nodes online, DC = ha-server-1, quorum achieved
- **STONITH:** `stonith-enabled: false`
- **no-quorum-policy:** `ignore`
- **Resources (all active on ha-server-1):**
- `ms-drbd0` — promotable DRBD clone (Primary: ha-server-1, Secondary: ha-server-2)
- `xfs-data` — XFS on `/dev/drbd0` → `/srv/ha-data`
- `iscsi-target` — targetctl service
- `nfs-server` — nfs-server service
- `vip` — IPaddr2 at **192.168.2.229/24** (no `nic=` parameter specified; floats to ens18/vmbr0 automatically based on subnet match)
- **Resource ordering:** ha-group starts only after DRBD is promoted; collocated with Promoted DRBD clone.
### DRBD State
- **Resource:** `ha-data` (DRBD 8.4.11 kernel module, config in `/etc/drbd.conf`)
- **Protocol:** C (synchronous)
- **Replication endpoints:**
- ha-server-1: `192.168.4.228:7789` (ens19 / vmbr1)
- ha-server-2: `192.168.4.227:7789` (ens19 / vmbr1)
- **State at audit time:** Initial sync in progress — ~20% complete, ~40 MB/s, ~34 min remaining (100 GB disk)
- **Fencing config:** `fencing resource-only` + fence-peer/unfence-peer handlers
### iSCSI Target
- **IQN:** `iqn.2026-01.home.sweet:ha-storage`
- **Portal:** `[::0]:3260` — confirmed listening on all interfaces (`ss -tnlp` shows `*:3260 *:*`)
- **LUN 0:** fileio backstore — `/srv/ha-data/iscsi-lun.img` (10 GiB, write-thru)
- **ACLs:** **None** (`no-gen-acls`, `no-auth`)
- **VIP (intended portal):** 192.168.2.229 — on vmbr0/LAN, no storage-NIC-specific binding
### NFS Exports
Served from the same `ha-group` as iSCSI (starts/stops together):
| Export path | Client subnet |
|------------|---------------|
| `/srv/ha-data/docker/{config,databases,volumes,nextcloud-data}` | 192.168.2.0/24 |
| `/srv/ha-data/proxmox/{iso,lxc}` | 192.168.2.0/24 |
| `/srv/ha-data/pxe-boot/images` | 192.168.2.0/24 |
| `/srv/ha-data/raspi/volumes` | 192.168.2.0/24 |
All NFS exports restrict to 192.168.2.0/24 and are served via the VIP at 192.168.2.229. `rw,sync,no_subtree_check,no_root_squash`.
### Docker Host Current State
- **Single NIC:** eth0 on vmbr0, 192.168.2.225/24, gateway 192.168.2.254
- **Route to storage network (192.168.4.x):** none — no NIC and no route
- **iSCSI sessions:** none
- **iSCSI nodes discovered:** none
- **Docker networks:** several active compose-project networks (core_traefik, core_nextcloud, core_passbolt, core_gramps, core_docker-socket-proxy, plus CI isolation networks)
---
## 2. Risk Flags
### RF-1: Corosync has only one ring (no heartbeat path redundancy)
`corosync.conf` defines only `ring0_addr` for each node, using 192.168.4.x on vmbr1. No `ring1_addr` / second knet link is configured. On a single Proxmox host, vmbr1 is a software bridge (no physical NIC), so physical link failure is not the concern — but a kernel network stack hiccup, a `pveproxy` restart dropping bridge state, or vmbr1 getting disrupted during heavy DRBD sync all leave corosync with zero fallback path. Missed heartbeats on a two-node cluster with `no-quorum-policy: ignore` do not cause a clean shutdown; they cause a false failover or split-brain.
Adding the LAN addresses (192.168.2.228 / 192.168.2.227 via ens18/vmbr0) as a second knet link would provide a backup path with no infrastructure changes needed.
### RF-2: Corosync heartbeat and DRBD replication share vmbr1 — no isolation between them
Both corosync (knet/UDP, ~1 kB heartbeat packets every ~100 ms) and DRBD replication (protocol C, synchronous, syncing at ~40 MB/s on a 100 GB initial fill at audit time) traverse the same `vmbr1` virtual bridge and terminate on the same ens19 NIC pair inside each HA VM. Under heavy DRBD write load, the guest-kernel scheduler's NIC transmit queue processes both flows together. While corosync's heartbeat is tiny, the absence of QoS/priority marking on vmbr1 means a DRBD burst can delay a heartbeat enough to trigger a ring fault warning. This is a latent risk that grows under high-write workloads.
### RF-3: STONITH disabled — split-brain protection relies solely on DRBD's resource-only fencing
`stonith-enabled: false` in the CIB. With `no-quorum-policy: ignore`, both nodes will continue running if corosync loses communication. DRBD's `fencing resource-only` does call `fence-peer` before allowing a Primary promotion, which provides some protection, but there is no hard external power fence to guarantee the other node actually stops. In a real split-brain (both nodes believe they are Primary), data corruption on the shared XFS filesystem is possible. **This is the highest-severity risk in the current setup.**
Getting STONITH to work on Proxmox-hosted VMs requires either a `fence_pve` agent (Proxmox API fencing) or `fence_virtd` (QEMU guest agent fencing). Neither is configured.
### RF-4: iSCSI portal bound to `[::0]:3260` — listens on every interface, not just the VIP
The targetcli portal is `[::0]:3260` (confirmed: `*:3260 *:*` in ss). This means the target is reachable on:
- 192.168.2.229 (VIP — correct, failover-safe)
- 192.168.2.228 (ha-server-1 LAN IP — does **not** move during failover; an initiator session connecting here would break on failover)
- 192.168.4.228 (storage NIC — not reachable by the Docker host today, but unintentionally exposed)
Binding the portal explicitly to the VIP IP instead of wildcard eliminates the non-VIP reachability risks.
### RF-5: iSCSI has zero ACLs and no authentication
`targetcli ls` shows `acls: 0`, `no-gen-acls`, `no-auth`. Any host that can reach port 3260 on any of the above IPs can log into the LUN with no credentials. The Docker host is not yet configured as an initiator — but neither is it blocked.
### RF-6: Docker host has no path to the storage network — iSCSI would traverse vmbr0/nic0
CT 105 (docker, 192.168.2.225) has one NIC, on vmbr0. To reach the VIP at 192.168.2.229, iSCSI traffic would travel:
```
docker (eth0/vmbr0) → nic0 (2.5GbE) → vmbr0 → tap200i0 (VM 200 net0/ens18)
```
All of the following share this same path over vmbr0 → nic0:
- Docker container traffic (outbound and inter-container)
- CI/CD runner traffic (Gitea Actions jobs visible in `docker network ls`)
- NFS mounts from pxe-boot, proxmox host itself, and other LAN clients
- iSCSI block traffic (protocol-sensitive to latency and retransmit)
A Nextcloud upload or a CI `nix build` job can saturate nic0 and starve the iSCSI session, causing command timeouts and filesystem errors on the Docker host.
### RF-7: VIP is on the LAN interface with no storage-specific binding
The Pacemaker `vip` resource specifies `ip=192.168.2.229, cidr_netmask=24` with no `nic=` parameter. Pacemaker's IPaddr2 agent selects the interface by longest-prefix match, landing it on ens18 (vmbr0/LAN). There is no way to keep this VIP from competing with general LAN traffic on nic0 without moving the VIP to a separate subnet on a different virtual bridge.
---
## 3. Recommended Target Layout
### Design constraints
- Single Proxmox host: all traffic ultimately shares nic0's bandwidth. The goal is QoS partitioning via separate bridges and subnets, not true physical isolation.
- Future physical split: bridge/VLAN IDs chosen here should map cleanly to physical uplink VLAN tags when the HA nodes move to separate hardware.
### Proposed bridge layout
| Bridge | Physical port | VLAN tag (future) | Subnet | Purpose |
|--------|-------------|-------------------|--------|---------|
| `vmbr0` | nic0 | untagged / VLAN 1 | 192.168.2.0/24 | **LAN/management only** — no storage traffic |
| `vmbr1` | (none / VLAN 10 on future trunk) | VLAN 10 | 192.168.4.224/29 | **Corosync heartbeat + DRBD replication** (current, keep) |
| `vmbr2` *(new)* | (none / VLAN 20 on future trunk) | VLAN 20 | 192.168.5.0/24 | **Storage: iSCSI + NFS client access** |
This is the minimum-disruption path: vmbr1 stays as-is (no DRBD reconfiguration needed), and the new vmbr2 gives the Docker host a direct path to the storage VIP without crossing vmbr0.
If stricter isolation is later desired, DRBD can be migrated from vmbr1 to vmbr2 in a separate maintenance window (see §5), leaving vmbr1 as corosync-only.
### Per-guest NIC assignments in target layout
| Guest | VMID | NIC | Bridge | Proposed IP | Purpose |
|-------|------|-----|--------|-------------|---------|
| ha-server-1 | VM 200 | net0 | vmbr0 | 192.168.2.228/24 | LAN/management (keep) |
| ha-server-1 | VM 200 | net1 | vmbr1 | 192.168.4.228/29 | Corosync + DRBD (keep) |
| ha-server-1 | VM 200 | **net2 (new)** | **vmbr2** | **192.168.5.1/24** | iSCSI + NFS storage client |
| ha-server-2 | VM 201 | net0 | vmbr0 | 192.168.2.227/24 | LAN/management (keep) |
| ha-server-2 | VM 201 | net1 | vmbr1 | 192.168.4.227/29 | Corosync + DRBD (keep) |
| ha-server-2 | VM 201 | **net2 (new)** | **vmbr2** | **192.168.5.2/24** | iSCSI + NFS storage client |
| docker | CT 105 | net0 | vmbr0 | 192.168.2.225/24 | LAN/management (keep) |
| docker | CT 105 | **net1 (new)** | **vmbr2** | **192.168.5.10/24** | iSCSI + NFS |
**VIP target:** `192.168.5.100/24` on vmbr2. The Pacemaker `vip` resource changes from `ip=192.168.2.229` to `ip=192.168.5.100, nic=<ens20>` (whichever name the new NIC gets inside the HA VMs). The existing `192.168.2.229` LAN VIP can optionally be retained as a separate static alias on ens18 for management-plane access, but should not be the iSCSI portal target.
**iSCSI portal:** Bind to `192.168.5.100:3260` instead of `[::0]:3260`. In targetcli: remove the wildcard portal, add `portals/ create 192.168.5.100`.
**NFS exports:** NFS is a file-level protocol and is fine being accessed over a routed path. After the VIP moves, non-docker LAN clients (proxmox host, pxe-boot, raspi) can reach NFS either via a static route to 192.168.5.0/24 or by keeping a secondary static alias at 192.168.2.229 on ens18 dedicated to NFS. Either approach works — NFS handles reconnect gracefully in ways iSCSI block I/O cannot.
**Corosync second ring (independent, low-disruption improvement):**
Add a second knet link using the LAN addresses as a backup heartbeat path. Edit `corosync.conf` on both nodes:
```
node { ring0_addr: 192.168.4.228; ring1_addr: 192.168.2.228; name: ha-server-1; nodeid: 1; }
node { ring0_addr: 192.168.4.227; ring1_addr: 192.168.2.227; name: ha-server-2; nodeid: 2; }
```
Requires a corosync service restart (brief cluster pause, ~5 seconds), no interface or bridge changes.
**Future physical-host split:**
When ha-server-1 and ha-server-2 move to separate physical machines, vmbr1 and vmbr2 become VLAN-tagged sub-interfaces on a physical trunk (e.g. VLAN 10 → cluster, VLAN 20 → storage). The bridge/subnet/IP layout above is designed so the tag numbers can be layered onto the existing addresses without renumbering.
---
## 4. Gap List
| Gap | Action needed |
|-----|--------------|
| `vmbr2` does not exist on pve1 | Create internal bridge: `/etc/network/interfaces.d/vmbr2.conf` with `bridge-ports none`, `inet manual` |
| VM 200 and VM 201 have no net2 | `qm set 200 --net2 virtio,bridge=vmbr2` / `qm set 201 --net2 virtio,bridge=vmbr2` (hot-plug, no reboot needed) |
| CT 105 has no net1 | `pct set 105 --net1 name=eth1,bridge=vmbr2,ip=192.168.5.10/24` |
| HA VMs have no OS config for the new NIC | NixOS `networking.interfaces.<ens20>` with `ipv4.addresses = [{address="192.168.5.1"; prefixLength=24;}]` per host (name may differ — check `ip link` after hotplug) |
| VIP needs to move to 192.168.5.100 on vmbr2 | `pcs resource update vip ip=192.168.5.100 cidr_netmask=24 nic=<ens20>` |
| iSCSI portal bound to `[::0]` | `targetcli /iscsi/iqn.2026-01.home.sweet:ha-storage/tpg1/portals delete ::0 3260` then `create 192.168.5.100`; save and restart via `pcs resource restart iscsi-target` |
| iSCSI ACLs empty | Get Docker initiator IQN via `iscsiadm -m iface` on CT 105, then add via targetcli `acls/ create <iqn>` |
| Docker host has no iSCSI initiator config | `iscsiadm -m discoverydb -t sendtargets -p 192.168.5.100 -D` then `iscsiadm -m node -l` once ACLs are set |
| Corosync single ring | Add `ring1_addr` entries in `corosync.conf` using LAN IPs; restart corosync cluster-wide (one node at a time) |
| STONITH not configured | Evaluate `fence_pve` (Proxmox API agent); document accepted risk if deferred |
| DRBD still on vmbr1 (optional, separate window) | Stop ms-drbd0 via pcs, edit `/etc/drbd.conf` on both nodes (change `192.168.4.x` → `192.168.5.x`), restart DRBD, re-enable via pcs |
---
## 5. Migration Notes
### Non-disruptive (no service impact)
- **Create vmbr2 on pve1:** Bridge definition edit only; no effect on existing bridges or guests.
- **Hot-add net2 to VMs 200/201:** Proxmox allows adding a NIC without reboot (`qm set 200 --net2 ...`). The NIC appears inside the VM immediately via QEMU hotplug but will be unconfigured (down) inside NixOS until the NixOS config is deployed — no impact on running services.
- **Add net1 to CT 105:** LXC NIC hotplug works similarly; CT does not need to restart.
- **Add corosync ring1:** Requires `systemctl restart corosync` on both nodes (one at a time). Pacemaker briefly sees corosync go offline and recover; with two nodes and `wait_for_all: 0`, this typically completes in under 5 seconds and resources stay running.
### Disruptive — requires maintenance window
- **Move VIP from 192.168.2.229 to 192.168.5.100:** `pcs resource update vip ip=192.168.5.100` causes Pacemaker to immediately stop the old VIP and start the new one. Any NFS mounts referencing 192.168.2.229 will stall until remounted at the new address (or a static alias is added at 192.168.2.229 on ens18). No iSCSI sessions exist yet, so no iSCSI disruption.
- **Change iSCSI portal from `[::0]` to VIP-specific:** Requires `pcs resource restart iscsi-target` after the targetcli portal change — brief target unavailability. Any initiator sessions (once configured) will need to re-login.
- **Migrate DRBD replication from 192.168.4.x to 192.168.5.x** (optional — only needed to give corosync sole ownership of vmbr1):
1. `pcs resource disable ms-drbd0` — demotes DRBD Primary, stops ha-group (unmounts XFS, stops iSCSI + NFS + VIP)
2. `drbdadm down ha-data` on both nodes
3. Edit `/etc/drbd.conf` on both nodes (change `address` lines)
4. `drbdadm up ha-data` on both nodes
5. `pcs resource enable ms-drbd0` — Pacemaker re-promotes, mounts, starts services
DRBD does **not** require a full resync when only the address changes — the disk data and metadata are unchanged; only the TCP connection endpoint changes. However, the initial sync was in progress at audit time (~20% at ~40 MB/s). Recommend waiting for that sync to complete before scheduling this migration.
docs/ha.md
+160
View File
@@ -0,0 +1,160 @@
# HA File-Server Cluster
Two `proxmox-ha-server-{1,2}` VMs form an active/passive file-server cluster:
DRBD replicates a block device between nodes; Corosync + Pacemaker manage
failover; XFS, LIO iSCSI, and NFS are brought up as a collocated resource
group on whichever node holds the DRBD Primary role.
NixOS modules: `modules/ha/`. Lifecycle scripts: `scripts/ha/`.
Cluster-wide constants: `variables.nix` (`haServer*` vars).
---
## Network layout
Three subnets — all internal to pve1 (`vmbr0`/`vmbr1`/`vmbr2`):
| Subnet | VLAN | CIDR | Bridge | Purpose |
|---|---|---|---|---|
| LAN | 2 | `192.168.2.0/24` | `vmbr0` | Management, LAN NFS |
| Cluster | 10 | `192.168.10.224/29` | `vmbr1` | Corosync ring0 + DRBD replication |
| Storage-client | 20 | `192.168.20.0/24` | `vmbr2` | NFS + iSCSI for docker/swarm |
Each HA VM has three NICs: `ens18` (LAN/vmbr0), `ens19` (cluster/vmbr1),
`ens20` (storage-client/vmbr2). See `docs/ip-addressing.md` for all IPs.
Corosync ring0 uses the cluster NIC; ring1 (backup heartbeat) uses the LAN
NIC. DRBD replicates over the cluster NIC. No storage traffic crosses the LAN.
---
## Pacemaker resources
All resources run collocated on whichever node is Primary, in this order:
```
ms-drbd0 (promotable DRBD clone)
→ xfs-data (XFS mount on /dev/drbd0 → /srv/ha-data)
→ iscsi-target (targetctl)
→ nfs-server (nfs-server.service)
→ vip-lan (192.168.2.229/24 on vmbr0 — NFS for LAN clients)
→ vip-storage (192.168.20.229/24 on vmbr2 — NFS + iSCSI for VLAN 20)
```
`vip-lan` serves pxe-boot and other LAN-only NFS clients.
`vip-storage` serves docker and any future swarm nodes; iSCSI is available on
VLAN 20 but NFS is preferred for multi-host volume sharing.
---
## DRBD fencing
`fencing resource-only` with `crm-fence-peer.sh`/`crm-unfence-peer.sh`
wrappers (`modules/ha/cluster-config.nix`). The DRBD kernel module invokes
these via the User Mode Helper with a minimal PATH; the wrappers prepend
`/run/current-system/sw/bin` before exec-ing the real handlers so Pacemaker
tools (`cibadmin`, `crm_mon`, etc.) are found.
STONITH is initially disabled (`stonith-enabled: false`,
`no-quorum-policy: ignore`). Enable it once the `fence_pve_ssh` fence agent
(`scripts/ha/fence-pve-ssh.py`) is deployed and authorised:
```bash
scripts/ha/cluster-enable-stonith.sh # run as root on ha-server-1
```
---
## Deploying the cluster from scratch
Use `scripts/ha/deploy.sh` — it orchestrates all phases:
```bash
# Against pve-test (safe — Claude's default target):
scripts/ha/deploy.sh --node "$PVE_TEST_HOST" [--dry-run]
# Against pve1 (production — requires explicit operator go-ahead):
scripts/ha/deploy.sh --node "$PVE1_HOST"
```
Phases (each skippable with `--skip-<phase>`):
1. `ensure-bridge` — creates `vmbr1`/`vmbr2` on the Proxmox node if absent
2. `sync-keys` — generates SSH host keys for both nodes; registers sops recipients
3. `create-vms` — builds disk images, creates VMs via `create-proxmox-resource.sh`
4. `add-hardware` — attaches storage NIC and DRBD data disk to each VM
5. `init-cluster` — runs `scripts/ha/cluster-init.sh` on ha-server-1
`--destroy` runs the teardown sequence.
---
## Day-to-day operations
```bash
# Read-only health check (safe from workstation):
scripts/ha/health.sh
# Graceful failover (prompts for confirmation):
scripts/ha/failover.sh [--to node1|node2]
# Online data-disk growth (no downtime):
scripts/ha/resize-data-disk.sh --size +20G
# Acceptance tests (run after any significant change):
scripts/ha/acceptance-tests.sh
```
---
## Adding FreeIPA host accounts
IPA host registration is automated:
```bash
scripts/ipa/create-nixos-ipa-host-account.sh <hostname>
```
This runs `ipa host-add`, fetches a keytab from the domain controller, and
writes a sops-encrypted `secrets/<hostname>.keytab` in one step. The module
`modules/ipa/client.nix` (imported by every host via
`modules/common/configuration.nix`) consumes the keytab via sops-nix.
---
## Storage layout
```
/srv/ha-data/
docker/
config/ NFS → docker:/mnt/docker/config
databases/ NFS → docker:/mnt/docker/databases
volumes/ NFS → docker:/mnt/docker/volumes
nextcloud-data/ NFS → docker:/mnt/docker/nextcloud-data
proxmox/
iso/ NFS → pve1 ISO storage
lxc/ NFS → pve1 CT template storage
pxe-boot/
images/ NFS → pxe-boot:/srv/pxe/http/images (PXE assets)
raspi/
volumes/ NFS → raspi NFS mounts
iscsi-lun.img iSCSI fileio backstore (VLAN 20 only, not in active use)
```
All shares are defined in `variables.nix` (`vars.nfsShares.*`). The NFS
export list lives in `modules/ha/nfs-exports.nix`.
---
## Key variables
| Variable | Description |
|---|---|
| `vars.haServer1Ip` / `vars.haServer2Ip` | LAN management IPs |
| `vars.haServer1StorageIp` / `vars.haServer2StorageIp` | Cluster NIC IPs (DRBD/Corosync ring0) |
| `vars.haServerLanVip` | Pacemaker `vip-lan` — NFS for LAN (192.168.2.229) |
| `vars.haServerVip` | Pacemaker `vip-storage` — NFS + iSCSI for VLAN 20 (192.168.20.229) |
| `vars.haLanNfsFqdn` | FQDN of `vip-lan`: `ha-vip-lan.sweet.home` |
| `vars.haStorageRoot` | XFS mount point: `/srv/ha-data` |
| `vars.haServerDrbdDisk` | Block device for DRBD backing store |
| `vars.haStorageCidr` | Cluster subnet CIDR (`192.168.10.224/29`) |
| `vars.haClientCidr` | Storage-client subnet CIDR (`192.168.20.0/24`) |
docs/ip-addressing.md
-39
View File
@@ -184,42 +184,3 @@ cannot reach either service on this VIP. The `vip-storage` endpoint is not reach
from the workstation directly (internal bridge only); health checks proxy through the
active HA node.
---
## Migration reference
Current → target IP for every host being renumbered.
| Host | Current IP | New IP | Config location |
|---|---|---|---|
| router | `192.168.2.254` | `192.168.2.254` | unchanged |
| domain-controller | `192.168.2.138` | `192.168.2.253` | `/etc/sysconfig/network-scripts/ifcfg-eth0` on guest |
| pve1 | `192.168.2.250` | `192.168.2.245` | `/etc/network/interfaces` on Proxmox host |
| pbs | `192.168.2.108` | `192.168.2.244` | static config on PBS host |
| nixos workstation | `192.168.2.119` | `192.168.2.243` | `networking.interfaces` / NetworkManager on guest |
| ha-node1 | — | `192.168.2.228` (LAN), `192.168.10.228` (cluster/VLAN 10), `192.168.20.228` (storage/VLAN 20) | active |
| ha-node2 | — | `192.168.2.227` (LAN), `192.168.10.227` (cluster/VLAN 10), `192.168.20.227` (storage/VLAN 20) | active |
| ha-vip-lan | — | `192.168.2.229` (vmbr0 / Pacemaker `vip-lan`) — NFS endpoint for LAN clients | active |
| ha-vip-storage | — | `192.168.20.229` (vmbr2 / Pacemaker `vip-storage`) — iSCSI endpoint for VLAN 20 clients | active |
| server | `192.168.2.252` | `192.168.2.226` | static config on guest |
| docker | `192.168.2.249` | `192.168.2.225` | static config on guest |
| nix-cache | `192.168.2.120` | `192.168.2.224` | static config on guest |
| pxe-boot | `192.168.2.247` | `192.168.2.223` | static config on guest; update `vars.pxeServerIp` in `variables.nix` ✓ |
| tailscale-router | `192.168.2.121` | `192.168.2.222` | static config on guest |
| tor-relay | `192.168.2.107` | `192.168.2.221` | static config on guest |
| pdm | `192.168.2.248` | `192.168.2.220` | static config on guest |
### Cutover notes
- **Do domain-controller first** — it becomes the DNS server; everything else depends on it
having its new IP and FreeIPA DNS configured before Pi-hole is retired.
- **pve1 last among physical hosts** — changing the Proxmox management IP drops the web UI
briefly; all guests keep running.
- **Update Pi-hole custom.list / FreeIPA DNS A records** to new IPs before flipping any host,
so name resolution stays valid throughout the migration.
- **variables.nix already updated** for `pxeServerIp` (.247→.223), `pbsIp` (.108→.244), and
new `domainControllerIp` (.253). Rebuild affected hosts after renumbering.
- **Router DHCP**: once domain-controller is at .253 and FreeIPA DNS is serving `sweet.home`,
switch router DHCP on with pool .10–.59 and DNS option pointing to .253; retire Pi-hole CT.
- **Pi-hole's iPXE dnsmasq config** (`99-ipxe-chainload.conf`) moves to the pxe-boot CT as a
dnsmasq proxy-mode config before Pi-hole is decommissioned.
docs/network-cutover.md
-448
View File
@@ -1,448 +0,0 @@
# Network Cutover Plan
Moves the LAN from the current flat/Pi-hole-managed state to the new IP scheme
defined in `docs/ip-addressing.md`. Works in five independent stages — each
stage is safe to pause after and resume later. Rollback steps are given at
every point where something can break.
**Before starting anything:** confirm you have
- SSH access to `192.168.2.138` (domain-controller, current IP)
- SSH access to `192.168.2.250` (pve1)
- Browser access to Pi-hole admin at `http://192.168.2.253`
- Browser access to router admin at `http://192.168.2.254`
- The FreeIPA `admin` password to hand
---
## Stage 1 — Prepare FreeIPA DNS (zero downtime)
Everything here is additive. Pi-hole keeps running. Nothing breaks if you stop
mid-stage.
### 1a. Add NextDNS forwarders
```bash
ssh wayne@192.168.2.138
kinit admin # enter FreeIPA admin password when prompted
ipa dnsconfig-mod \
--forwarder=45.90.28.142 \
--forwarder=45.90.30.142 \
--forward-policy=only
```
**Verify external resolution works through FreeIPA before continuing:**
```bash
dig @127.0.0.1 google.com +short # must return an IP, not SERVFAIL
```
### 1b. Add A records for every host at their CURRENT IPs
These represent the live state now. You'll update each record to the new IP
when you renumber that host in Stage 5.
```bash
ipa dnsrecord-add sweet.home pve1 --a-rec 192.168.2.250
ipa dnsrecord-add sweet.home pbs --a-rec 192.168.2.108
ipa dnsrecord-add sweet.home nixos --a-rec 192.168.2.119
ipa dnsrecord-add sweet.home server --a-rec 192.168.2.252
ipa dnsrecord-add sweet.home docker --a-rec 192.168.2.249
ipa dnsrecord-add sweet.home nix-cache --a-rec 192.168.2.120
ipa dnsrecord-add sweet.home pxe-boot --a-rec 192.168.2.247
ipa dnsrecord-add sweet.home tailscale-router --a-rec 192.168.2.121
ipa dnsrecord-add sweet.home tor-relay --a-rec 192.168.2.107
ipa dnsrecord-add sweet.home pdm --a-rec 192.168.2.248
ipa dnsrecord-add sweet.home router --a-rec 192.168.2.254
```
### 1c. Clean up stale reverse-zone PTR records
FreeIPA already has PTR records from an earlier import but some are wrong.
Fix them now so reverse DNS is accurate from day one.
```bash
# Remove stale "win11" entry at .250 (should be pve1)
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec win11.
ipa dnsrecord-add 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
# Fix unqualified PTR records (missing .sweet.home. suffix)
ipa dnsrecord-mod 2.168.192.in-addr.arpa 108 --ptr-rec pbs.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 248 --ptr-rec pdm.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 249 --ptr-rec docker.sweet.home.
ipa dnsrecord-mod 2.168.192.in-addr.arpa 252 --ptr-rec server.sweet.home.
# Add any missing PTR records
ipa dnsrecord-add 2.168.192.in-addr.arpa 119 --ptr-rec nixos.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 120 --ptr-rec nix-cache.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 121 --ptr-rec tailscale-router.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 247 --ptr-rec pxe-boot.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 254 --ptr-rec router.sweet.home.
```
### 1d. Point domain-controller's own DNS at itself
```bash
sudo nmcli connection modify "System eth0" ipv4.dns "127.0.0.1"
sudo nmcli connection up "System eth0"
```
**Verify:**
```bash
dig pve1.sweet.home +short # must return 192.168.2.250
dig google.com +short # must return an IP (NextDNS forwarding)
```
**Rollback 1d:** `sudo nmcli connection modify "System eth0" ipv4.dns "192.168.2.253" && sudo nmcli connection up "System eth0"`
---
## Stage 2 — Move pxe-boot DHCP options off Pi-hole (zero downtime)
Pi-hole's dnsmasq currently serves the iPXE boot options via
`99-ipxe-chainload.conf`. Before Pi-hole is retired, that config must move to
the pxe-boot CT running dnsmasq in proxy mode so PXE boot keeps working.
### 2a. Add dnsmasq proxy config to the pxe-boot NixOS module
In `modules/build-types/pxe-boot.nix`, add:
```nix
services.dnsmasq = {
enable = true;
settings = {
# Proxy mode: respond only to PXE DHCP requests, leave normal leases to router
dhcp-range = [ "192.168.2.0,proxy" ];
# iPXE client detection
dhcp-match = [
"set:ipxe,175"
"set:efi64,option:client-arch,7"
"set:efi64,option:client-arch,9"
];
dhcp-userclass = "set:ipxe,iPXE";
# Boot file selection
dhcp-boot = [
"tag:ipxe,tag:efi64,http://${vars.pxeServerIp}/boot.ipxe"
"tag:ipxe,http://${vars.pxeServerIp}/boot.ipxe"
"tag:efi64,ipxe.efi,,${vars.pxeServerIp}"
"undionly.kpxe,,${vars.pxeServerIp}"
];
};
};
```
### 2b. Rebuild and deploy the pxe-boot CT
```bash
# On pve1 — build the new tarball
nix build .#lxc-pxe-boot.config.system.build.tarball
# Verify dnsmasq starts correctly in the CT after deploy
ssh nixos@192.168.2.247 systemctl status dnsmasq
```
### 2c. Remove the iPXE config from Pi-hole
In the Pi-hole CT, remove `/etc/dnsmasq.d/99-ipxe-chainload.conf` and
restart the FTL service:
```bash
ssh wayne@pve1.sweet.home \
"sudo pct exec 100 -- bash -c 'rm /etc/dnsmasq.d/99-ipxe-chainload.conf && systemctl restart pihole-FTL'"
```
**Verify:** PXE boot a test machine — it should still get an iPXE response and
reach the boot menu.
**Rollback 2c:** restore the file from the Pi-hole config backup at
`/etc/pihole/config_backups/` and restart pihole-FTL.
---
## Stage 3 — DHCP migration: Pi-hole → router (brief maintenance window)
**Do this in the evening.** Existing DHCP leases stay valid during the
switchover so connected devices don't drop — only new lease requests fail
during the gap, which is under 60 seconds if you follow the steps in order.
The key: configure the router's DHCP DNS option to point at `.253` (Pi-hole's
current IP). This way, all new leases issued by the router still get the same
DNS server address — clients never need to change their DNS config. When Pi-hole
is retired and the DC takes `.253` in Stage 4, `.253` just starts answering
differently. No client reconfiguration.
### 3a. Pre-configure router DHCP (do not enable yet)
Log into `http://192.168.2.254`, find the DHCP settings and fill in — but
leave DHCP **disabled** until step 3b:
| Setting | Value |
|---|---|
| Start IP | 192.168.2.10 |
| End IP | 192.168.2.59 |
| Subnet mask | 255.255.255.0 |
| Gateway | 192.168.2.254 |
| Primary DNS | 192.168.2.253 |
| Secondary DNS | *(leave blank)* |
| Lease time | 24h |
Save without enabling.
### 3b. Switchover (do steps in quick succession)
1. **Disable Pi-hole DHCP:** Pi-hole admin UI → Settings → DHCP → uncheck
"DHCP server enabled" → Save
2. **Enable router DHCP** immediately after step 1
### 3c. Verify router DHCP is working
On a phone or laptop, disconnect from WiFi and reconnect (or run
`sudo dhclient -r && sudo dhclient` on a Linux host):
```bash
ip addr show # IP should be in 192.168.2.10–59 range
dig google.com # should resolve (Pi-hole DNS still running at .253)
dig pve1.sweet.home # should resolve via FreeIPA at .138 (relayed via Pi-hole)
```
Wait 10–15 minutes for the most active devices to renew their leases. There's
no need to wait for all leases to expire before proceeding.
**Rollback 3b:** Re-enable Pi-hole DHCP. Disable router DHCP. Done — existing
leases remain valid so most devices are unaffected.
---
## Stage 4 — Move domain-controller from .138 to .253
Pi-hole lives at `.253`. The DC must take `.253` the moment Pi-hole stops so
clients that still have `.253` as their DNS server don't notice the change.
Script these commands in advance and run them in rapid succession.
**Pre-stage: have this SSH command ready before running step 4a:**
```bash
ssh wayne@192.168.2.138 "
sudo nmcli connection modify 'System eth0' \
ipv4.addresses '192.168.2.253/24' \
ipv4.gateway '192.168.2.254' \
ipv4.dns '127.0.0.1' \
ipv4.method manual && \
sudo nmcli connection up 'System eth0'
"
```
**Also update the Proxmox VM config to match (run from pve1):**
```bash
sudo qm set 108 \
--ipconfig0 ip=192.168.2.253/24,gw=192.168.2.254 \
--nameserver 192.168.2.253
```
### 4a. Stop Pi-hole
```bash
ssh wayne@pve1.sweet.home "sudo pct stop 100"
```
### 4b. Immediately: change DC's IP to .253
Run the pre-staged SSH command from above. You have ~30 seconds before any
client notices Pi-hole is gone. If SSH to `.138` refuses (the IP is already
changing), open a Proxmox console to VM 108 and run the `nmcli` commands
there.
### 4c. Update Proxmox VM config
Run the pre-staged `qm set 108` command from above.
### 4d. Verify
```bash
ssh wayne@192.168.2.253 # must connect (new DC IP)
dig @192.168.2.253 pve1.sweet.home +short # must return 192.168.2.250
dig @192.168.2.253 google.com +short # must return an IP
```
From a client device that renewed its DHCP lease in Stage 3:
```bash
cat /etc/resolv.conf # should show 192.168.2.253
dig pve1.sweet.home # should resolve
```
**Rollback 4:** `ssh wayne@pve1.sweet.home "sudo pct start 100"`. Change DC IP
back to .138 via Proxmox console. This restores full Pi-hole DNS/DHCP service.
Leave Pi-hole CT stopped-but-intact for 48 hours before deleting it.
---
## Stage 5 — Host renumbering (one at a time, any order)
For each host:
1. Update FreeIPA DNS A record and PTR record to the new IP
2. Change the static IP on the host itself
3. Verify SSH to new IP
4. Update `variables.nix` if that host has an IP variable (pxe-boot, pbs — already done in this PR)
**FreeIPA record update template** (run as admin on domain-controller):
```bash
ipa dnsrecord-mod sweet.home <hostname> --a-rec <new-ip>
ipa dnsrecord-del 2.168.192.in-addr.arpa <old-last-octet> --ptr-rec <hostname>.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa <new-last-octet> --ptr-rec <hostname>.sweet.home.
```
### Renumbering order
| # | Host | Old IP | New IP | How to change IP |
|---|---|---|---|---|
| 1 | nixos workstation | .119 | .243 | NetworkManager on guest; or `nmcli connection modify` |
| 2 | nix-cache | .120 | .224 | `pct set 102 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.224/24,gw=192.168.2.254` then `pct reboot 102` |
| 3 | tailscale-router | .121 | .222 | Static config on guest; check Tailscale ACLs if IP is referenced there |
| 4 | tor-relay | .107 | .221 | `pct set 104 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.221/24,gw=192.168.2.254` then `pct reboot 104` |
| 5 | pdm | .248 | .220 | `pct set 106 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.220/24,gw=192.168.2.254` then `pct reboot 106` |
| 6 | pxe-boot | .247 | .223 | `pct set 103 --net0 name=eth0,bridge=vmbr0,ip=192.168.2.223/24,gw=192.168.2.254` then rebuild NixOS (already updated in variables.nix) |
| 7 | server | .252 | .226 | Static config on guest; NFS clients (docker) lose mounts briefly — they remount automatically |
| 8 | docker | .249 | .225 | Static config on guest; do this after server is at .226 |
| 9 | pbs | .108 | .244 | Static config on PBS host itself; update in `pbsIp` already done in variables.nix |
| 10 | pve1 | .250 | .245 | Edit `/etc/network/interfaces` on the Proxmox host — see below |
### pve1 renumber (step 10 — do last)
All guests keep running; only the Proxmox web UI is briefly unreachable.
```bash
ssh wayne@pve1.sweet.home
# Edit /etc/network/interfaces: change address from .250 to .245
sudo nano /etc/network/interfaces
# Change: address 192.168.2.250/24
# To: address 192.168.2.245/24
sudo systemctl restart networking
# SSH will drop here — reconnect to new IP
```
```bash
ssh wayne@192.168.2.245 # verify
```
Update FreeIPA DNS:
```bash
ipa dnsrecord-mod sweet.home pve1 --a-rec 192.168.2.245
ipa dnsrecord-del 2.168.192.in-addr.arpa 250 --ptr-rec pve1.sweet.home.
ipa dnsrecord-add 2.168.192.in-addr.arpa 245 --ptr-rec pve1.sweet.home.
```
**Rollback any step 5 host:** change the IP back on the guest and update the
FreeIPA record back to the old IP. The old IP is unoccupied so you can
temporarily use either.
---
## Stage 6 — HA storage cutover (docker NFS remount)
> **Prerequisites:**
> - HA cluster fully deployed and `vip-storage` (`nfs.storage.home` → 192.168.20.229) serving NFS ✓
> - DNS configured: `storage.home` zone populated, `nfs.storage.home` resolves to 192.168.20.229 ✓
> - docker CT has eth1 on vmbr2 (`docker.storage.home` → 192.168.20.225) ✓
> - Final rsync from server.sweet.home to `/srv/ha-data` complete before step 6b
docker.sweet.home currently NFS-mounts its persistent volumes from `server.sweet.home`
(`192.168.2.226:/tank/docker/...`). This stage moves those mounts to the HA cluster's
storage VIP so server can be decommissioned.
### 6a. Final rsync from server to HA cluster
Run from server.sweet.home (or over SSH from the workstation) to sync any data written
since the initial rsync:
```bash
# Confirm active HA node and mount point
ssh wayne@192.168.2.228 'sudo findmnt /srv/ha-data' # check which node is active
# rsync each dataset (adjust source paths to match /tank layout on server)
sudo rsync -av --delete /tank/docker/config/ wayne@<active-node-ip>:/srv/ha-data/docker/config/
sudo rsync -av --delete /tank/docker/databases/ wayne@<active-node-ip>:/srv/ha-data/docker/databases/
sudo rsync -av --delete /tank/docker/volumes/ wayne@<active-node-ip>:/srv/ha-data/docker/volumes/
sudo rsync -av --delete /tank/docker/nextcloud-data/ wayne@<active-node-ip>:/srv/ha-data/docker/nextcloud-data/
```
### 6b. Update docker NixOS config to mount from vip-storage
In `hosts/docker/host.nix` (or wherever the NFS mount fileSystems are declared), change
the NFS server from `server.sweet.home` / `192.168.2.226` to `nfs.storage.home`:
```nix
# Before:
fileSystems."/mnt/docker/config" = {
device = "server:/tank/docker/config"; # or 192.168.2.226:...
...
};
# After:
fileSystems."/mnt/docker/config" = {
device = "nfs.storage.home:/srv/ha-data/docker/config";
...
};
```
Using the DNS name (`nfs.storage.home`) rather than the VIP IP means the mount
config survives a future VIP renumber without touching the NixOS config.
Repeat for all four docker shares (`config`, `databases`, `volumes`, `nextcloud-data`).
Then rebuild docker:
```bash
# On the workstation — or via Switch-nix on docker itself
sudo nixos-rebuild switch --no-write-lock-file --refresh \
--flake "git+https://gitea.lan.ddnsgeek.com/beatzaplenty/nixos.git#lxc-docker"
```
### 6c. Verify mounts and container health
```bash
ssh wayne@192.168.2.225 'findmnt | grep 192.168.20' # mounts should show vip-storage
ssh wayne@192.168.2.225 'docker ps' # all containers running
```
Spot-check Nextcloud, Traefik, and any database containers for connectivity.
### 6d. Decommission server.sweet.home
Once docker is confirmed healthy on the HA NFS mounts:
```bash
# Stop server VM on pve1
ssh wayne@192.168.2.245 'sudo qm stop 101'
# (Optional) Archive the ZFS pool snapshot before destroying
# Then after a settling period:
ssh wayne@192.168.2.245 'sudo qm destroy 101 --destroy-unreferenced-disks 1'
```
---
## Stage 7 — Final cleanup
Once all hosts are at their new IPs and verified:
```bash
# Delete the Pi-hole CT (already stopped since Stage 4)
ssh wayne@pve1.sweet.home "sudo pct destroy 100"
# Remove stale FreeIPA records for retired addresses
ipa dnsrecord-del sweet.home pihole --del-all
ipa dnsrecord-del 2.168.192.in-addr.arpa 253 --ptr-rec pihole.sweet.home.
# Rebuild any NixOS hosts that reference pbsIp or pxeServerIp to pick up
# the updated variables.nix values (pxe-boot mandatory; others as convenient)
```
---
## Rollback summary
| What broke | How to roll back |
|---|---|
| FreeIPA DNS not resolving | Check `systemctl status named` on DC; restart if failed |
| FreeIPA DNS unreachable | `pct start 100` on pve1 (restores Pi-hole) |
| Router DHCP not handing out leases | Re-enable Pi-hole DHCP; disable router DHCP |
| DC unreachable after IP change | Proxmox console on VM 108 → `nmcli connection up "System eth0"` with old IP |
| Host unreachable after renumber | Proxmox console → revert IP; or `pct set <id> --net0 ...` old IP and reboot CT |
| pve1 web UI gone after renumber | SSH to .245 and check `/etc/network/interfaces`; if wrong, fix and restart networking |