beatzaplenty/nixos
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add Docker Swarm HA cluster: ha-docker-1 and ha-docker-2

Browse Source 
Two new NixOS Proxmox VMs (VMIDs 202/203) forming a dual-manager Docker
Swarm on dedicated vmbr3 (192.168.30.0/24, VLAN 30) for gossip and VXLAN,
with NFS via the storage-client network (vmbr2) from the existing HA cluster.

- nixos/variables.nix: add ha-docker IP/interface/port vars and swarm CIDR
- nixos/modules/build-types/ha-docker.nix: new build type — Docker 29,
  NFS mounts, beszel-agent, health monitoring, swarm firewall rules with
  checkReversePath = "loose" for VXLAN routing mesh
- nixos/hosts/ha-docker-{1,2}/host.nix: per-host identity — three NICs
  (LAN, storage, swarm), IPA dyndns pinned to LAN interface
- nixos/flake.nix: add proxmox-ha-docker-{1,2} targets; build-validated
  with nix build --dry-run (169 derivations, no errors)
- nixos/docs/ip-addressing.md: document VLAN 30 / swarm.home zone,
  ha-docker IP allocations across all three subnets
- nixos/scripts/docker-swarm/deploy.sh: 10-phase lifecycle script
  (bridge, keys, IPA, VMs, swarm init, DNS, verify); modelled on
  scripts/ha/deploy.sh with --destroy mode
- nixos/docs/internal/docker-swarm-cutover.md: service-by-service
  migration guide covering Traefik log rotation, Nextcloud cron sidecar,
  docker-health-to-gotify swarm awareness updates, Passbolt/Gitea steps,
  DNS cutover, and CT 105 decommission checklist

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
This commit is contained in:
beatzaplenty
2026-07-30 19:01:17 +10:00
co-authored by Claude Sonnet 4.6
parent 43314b6aa3
commit c96752e5d0
8 changed files with 1162 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ip-addressing.md
+47 -5
View File
@@ -7,6 +7,10 @@
| LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
| Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
| Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink |
| Swarm cluster | 30 | `192.168.30.0/24` | Docker Swarm gossip (TCP/UDP 7946) + VXLAN overlay (UDP 4789) | No — internal `vmbr3` only, no uplink |
When expanded to a second Proxmox node, VLAN 10 (cluster), VLAN 20 (storage-client), and VLAN 30 (swarm) all share
the same inter-node trunk NIC via 802.1q VLAN tagging — different VLAN IDs, same physical cable.
The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
@@ -27,9 +31,9 @@ is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20
## DNS Zones
FreeIPA (domain-controller.sweet.home) is authoritative for all zones. Three
zones correspond to the three subnets — one per VLAN. All zones are internal
only; no external delegation.
FreeIPA (domain-controller.sweet.home) is authoritative for all zones.
Four zones correspond to the four subnets. All zones are internal only; no external delegation.
### sweet.home — VLAN 2 (192.168.2.x)
@@ -132,12 +136,14 @@ All VMs and LXC containers run on pve1.
| `192.168.2.228` | ha-node1 | HA file server node 1 — management NIC | Active |
| `192.168.2.227` | ha-node2 | HA file server node 2 — management NIC | Active |
| `192.168.2.226` | server | Former NFS/ZFS file server — decommissioned | Removed from flake |
| `192.168.2.225` | docker | Docker / Traefik stack | Active |
| `192.168.2.225` | docker | Docker / Traefik stack (CT 105 — existing single-host) | Active |
| `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
| `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active |
| `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active |
| `192.168.2.221` | tor-relay | Tor relay | Active |
| `192.168.2.220` | pdm | Proxmox Deploy Manager | Active |
| `192.168.2.231` | ha-docker-2 | Docker Swarm node 2 — management NIC | Active |
| `192.168.2.230` | ha-docker-1 | Docker Swarm node 1 — management NIC | Active |
### Client DHCP pool (.10–.59)
@@ -175,7 +181,9 @@ Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached.
| `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
| `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
| `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — decommissioned |
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client |
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client (CT 105) |
| `192.168.20.231` | ha-docker-2 | Storage-client NIC (ens19 / vmbr2) — NFS client |
| `192.168.20.230` | ha-docker-1 | Storage-client NIC (ens19 / vmbr2) — NFS client |
| — | no gateway | Isolated — not routed to LAN or internet |
NFS clients mount from `192.168.20.229` (surviving failover transparently via the VIP).
@@ -184,3 +192,37 @@ cannot reach either service on this VIP. The `vip-storage` endpoint is not reach
from the workstation directly (internal bridge only); health checks proxy through the
active HA node.
---
## Swarm cluster network — VLAN 30 — 192.168.30.0/24
Internal to pve1 only. Proxmox bridge `vmbr3`, no physical NIC attached.
Carries Docker Swarm inter-node traffic only: Raft consensus (TCP 2377),
Serf gossip (TCP/UDP 7946), and VXLAN overlay data path (UDP 4789).
Docker Swarm is initialised with `--advertise-addr` and `--data-path-addr`
both pointing to this subnet so all cluster traffic stays on `vmbr3` and
never crosses the LAN.
| IP | Hostname | Interface / role |
|---|---|---|
| `192.168.30.231` | ha-docker-2 | Swarm cluster NIC (ens20 / vmbr3) |
| `192.168.30.230` | ha-docker-1 | Swarm cluster NIC (ens20 / vmbr3) |
| — | no gateway | Isolated — not routed to LAN or internet |
### DNS zone: `swarm.home` — VLAN 30 (192.168.30.x)
| Hostname | A record | Notes |
|---|---|---|
| `ha-docker-1.swarm.home` | `192.168.30.230` | Swarm NIC — debugging only |
| `ha-docker-2.swarm.home` | `192.168.30.231` | Swarm NIC — debugging only |
Operators reach the Docker API on the LAN IPs (`192.168.2.230`/`.231`), not these addresses.
The `swarm.home` records exist for diagnostic convenience (e.g. confirming `vmbr3` routing).
### Multi-node Proxmox expansion
When a second Proxmox node (pve2) is added, VLAN 10 (cluster), VLAN 20 (storage-client),
and VLAN 30 (swarm) all extend to pve2 via 802.1q VLAN tagging on the inter-node trunk
link. All three internal networks share the same physical NIC between hypervisors —
VLAN tags provide the logical separation.