Add Docker Swarm HA cluster: ha-docker-1 and ha-docker-2

Two new NixOS Proxmox VMs (VMIDs 202/203) forming a dual-manager Docker Swarm on dedicated vmbr3 (192.168.30.0/24, VLAN 30) for gossip and VXLAN, with NFS via the storage-client network (vmbr2) from the existing HA cluster. - nixos/variables.nix: add ha-docker IP/interface/port vars and swarm CIDR - nixos/modules/build-types/ha-docker.nix: new build type — Docker 29, NFS mounts, beszel-agent, health monitoring, swarm firewall rules with checkReversePath = "loose" for VXLAN routing mesh - nixos/hosts/ha-docker-{1,2}/host.nix: per-host identity — three NICs (LAN, storage, swarm), IPA dyndns pinned to LAN interface - nixos/flake.nix: add proxmox-ha-docker-{1,2} targets; build-validated with nix build --dry-run (169 derivations, no errors) - nixos/docs/ip-addressing.md: document VLAN 30 / swarm.home zone, ha-docker IP allocations across all three subnets - nixos/scripts/docker-swarm/deploy.sh: 10-phase lifecycle script (bridge, keys, IPA, VMs, swarm init, DNS, verify); modelled on scripts/ha/deploy.sh with --destroy mode - nixos/docs/internal/docker-swarm-cutover.md: service-by-service migration guide covering Traefik log rotation, Nextcloud cron sidecar, docker-health-to-gotify swarm awareness updates, Passbolt/Gitea steps, DNS cutover, and CT 105 decommission checklist Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
2026-07-30 19:01:17 +10:00
parent 43314b6aa3
commit c96752e5d0
8 changed files with 1162 additions and 5 deletions
@@ -0,0 +1,330 @@
+# Docker Swarm Cutover Plan
+
+Migration guide for moving containerised services from the existing single-host
+Docker LXC container (CT 105, `docker.sweet.home`, 192.168.2.225) to the new
+Docker Swarm cluster (`ha-docker-1` / `ha-docker-2`, 192.168.2.230–231).
+
+CT 105 stays running throughout.  Services migrate one stack at a time.
+Roll back any stack by restarting it on CT 105 if anything goes wrong.
+
+---
+
+## Prerequisites
+
+- Swarm cluster deployed and healthy (`scripts/docker-swarm/deploy.sh`).
+- Both nodes show `Ready / Active / Manager` in `docker node ls`.
+- NFS mounts healthy on both swarm nodes (`/mnt/docker/config`, `/mnt/docker/databases`, `/mnt/docker/volumes`).
+- Access to FreeIPA DNS admin to update A records during cutover.
+
+---
+
+## 1. Traefik — switch to Docker log rotation
+
+**Current state (CT 105):** Traefik writes access logs to the NFS volume at
+`/mnt/docker/volumes/traefik-data/logs/`.  `modules/traefik/rotate-logs.nix`
+rotates those files via `logrotate`.
+
+**Swarm approach:** Remove file-based access logging from Traefik's static
+config and rely on Docker's json-file log driver with built-in rotation.
+Traefik container logs (including access events) then live under
+`/var/lib/docker/containers/<id>/` on the node running Traefik.
+
+### Steps
+
+**1a.** In the Traefik stack definition, add logging config to the service:
+
+```yaml
+services:
+  traefik:
+    logging:
+      driver: "json-file"
+      options:
+        max-size: "100m"
+        max-file: "20"
+```
+
+**1b.** In `traefik.yml` (Traefik's static config), remove the `accessLog`
+file path if present.  To keep structured access logs, use Traefik's
+`accessLog.format: json` with no `filePath` — logs then go to stdout and are
+captured by the json-file driver above.
+
+**1c.** Deploy Traefik to the swarm:
+
+```bash
+# On either swarm manager:
+docker stack deploy -c /mnt/docker/config/traefik/docker-compose.yml traefik
+```
+
+Traefik should be deployed as a **global mode** service so it runs on all
+swarm nodes and handles ingress on whichever node a request arrives at:
+
+```yaml
+services:
+  traefik:
+    deploy:
+      mode: global
+      placement:
+        constraints:
+          - node.role == manager
+```
+
+**1d.** After confirming Traefik works on the swarm, remove
+`traefik/rotate-logs.nix` from the `docker` build type in
+`modules/build-types/docker.nix` and rebuild CT 105.
+
+**DNS:** Update `docker.sweet.home` and any service FQDNs that point at
+192.168.2.225 to a swarm VIP or round-robin A records once Traefik is running
+on the swarm.  See section 8 (DNS cutover).
+
+---
+
+## 2. Nextcloud — migrate cron job to sidecar container
+
+**Current state (CT 105):** `modules/docker/nextcloud-cron-job.nix` runs a
+systemd timer every 5 minutes that calls:
+```bash
+docker exec nextcloud-webapp php ./cron.php
+```
+
+**Swarm problem:** `docker exec` only works against the local daemon.  If
+Nextcloud is scheduled on the other swarm node, the exec fails silently and
+cron never runs.
+
+**Swarm approach:** Add a `nextcloud-cron` sidecar container to the Nextcloud
+stack definition, pinned to the same node as the main Nextcloud container via
+placement constraints.
+
+### Steps
+
+**2a.** Choose which swarm node will host Nextcloud (e.g. `ha-docker-1`).
+Label that node:
+
+```bash
+# On either swarm manager:
+docker node update --label-add nextcloud=true ha-docker-1
+```
+
+**2b.** In the Nextcloud stack compose file, add the sidecar and pin both
+services to the labelled node:
+
+```yaml
+services:
+  nextcloud-webapp:
+    image: nextcloud:production   # pin same version as CT 105
+    deploy:
+      replicas: 1
+      placement:
+        constraints:
+          - node.labels.nextcloud == true
+    # ... existing volumes, env, networks ...
+
+  nextcloud-cron:
+    image: nextcloud:production   # same image, different entrypoint
+    entrypoint: /cron.sh
+    deploy:
+      replicas: 1
+      placement:
+        constraints:
+          - node.labels.nextcloud == true   # must co-locate with webapp
+    volumes:
+      # Same data volume as nextcloud-webapp so cron sees the same files.
+      - nextcloud-data:/var/www/html
+    # No ports exposed — cron only runs PHP inside the container.
+```
+
+`/cron.sh` is Nextcloud's built-in cron entrypoint.  It runs
+`php -f /var/www/html/cron.php` in a loop, sleeping for 5 minutes between
+runs — identical to the current systemd timer.
+
+**2c.** Migrate Nextcloud's data volume to the swarm:
+
+```
+/mnt/docker/volumes/nextcloud-data/  →  already on NFS, no migration needed
+/mnt/docker/databases/nextcloud/     →  already on NFS, no migration needed
+```
+
+The NFS paths are identical on the swarm nodes (`mount-data.nix` mounts the
+same shares from the same VIP).  Stop Nextcloud on CT 105, deploy on the
+swarm, confirm it starts cleanly.
+
+**2d.** Remove `nextcloud-cron-job.nix` from `modules/build-types/docker.nix`
+and rebuild CT 105 after confirming Nextcloud works on the swarm.
+
+---
+
+## 3. docker-health-to-gotify — update for swarm awareness
+
+**Current state (CT 105):** The script at
+`/home/nixos/docker/monitoring/gotify/docker-health-to-gotify.sh` runs every
+minute, calls `docker ps --filter health=unhealthy`, and notifies Gotify.
+
+**Swarm behaviour:** The same script runs on both swarm nodes independently,
+each monitoring its own local Docker daemon.  This gives per-node coverage
+across the swarm.
+
+**Changes needed in the script** (edit the copy on the NFS volume — it takes
+effect on both nodes simultaneously on the next timer fire):
+
+### 3a. Strip the Swarm task suffix from service names
+
+In swarm mode, `docker ps --format '{{.Names}}'` returns names like
+`nextcloud-webapp.1.abc123xyz`.  The notification should show `nextcloud-webapp`,
+not the full task name.
+
+```bash
+# Before:
+CONTAINER_NAME=$(docker ps --format '{{.Names}}' ...)
+
+# After:
+CONTAINER_NAME=$(docker ps --format '{{.Names}}' ... | cut -d. -f1)
+```
+
+### 3b. Include the reporting node in the Gotify message
+
+Add `$(hostname)` to the notification payload so you know which swarm node
+detected the problem:
+
+```bash
+MESSAGE="[$(hostname)] ${CONTAINER_NAME} is unhealthy"
+```
+
+### 3c. Extend to catch swarm service replica failures
+
+`docker ps` only shows what's running locally.  If a service has zero healthy
+replicas (task crash-looping) it may not show up on either node's `docker ps`
+at the same moment.  Add a swarm-level check:
+
+```bash
+# Run only on managers (both ha-docker nodes are managers):
+if docker info --format '{{.Swarm.ControlAvailable}}' 2>/dev/null | grep -q true; then
+  # Find services where running replicas < desired replicas
+  docker service ls --format '{{.Name}}\t{{.Replicas}}' | \
+    awk -F'\t' '$2 !~ /^[0-9]+\/[0-9]+$/ || split($2,a,"/") && a[1] < a[2] { print $1, $2 }' | \
+    while read -r svc_name replicas; do
+      # Send Gotify notification for degraded service
+      curl -s -X POST "${GOTIFY_URL}/message" \
+        -H "X-Gotify-Key: ${GOTIFY_TOKEN}" \
+        -d "title=Swarm service degraded" \
+        -d "message=[$(hostname)] ${svc_name}: ${replicas} replicas"
+    done
+fi
+```
+
+This catches the case where a service's desired replicas are not running
+(e.g. OOM kill, image pull failure) — a failure mode that doesn't produce a
+Docker health event on any node.
+
+---
+
+## 4. Passbolt migration
+
+Passbolt has strict data integrity requirements.  Migrate with care:
+
+1. **Backup first** — `docker exec passbolt-webapp php /usr/share/php/passbolt/bin/cake passbolt export_keys` and a database dump.
+2. Database is on NFS (`/mnt/docker/databases/passbolt/`) — no data copy needed.
+3. Pin Passbolt to a specific node: `docker node update --label-add passbolt=true ha-docker-1`
+4. Add placement constraint `node.labels.passbolt == true` to the Passbolt stack.
+5. Stop on CT 105, deploy on swarm, verify login works.
+6. Test email delivery and 2FA.
+
+---
+
+## 5. Gitea migration
+
+Gitea's data directory is on NFS (`/mnt/docker/volumes/gitea-data/`).
+
+1. Stop Gitea on CT 105: `docker stop gitea`
+2. Deploy to swarm with placement constraint (pin to `ha-docker-1` initially).
+3. Verify web UI and SSH clone/push work.
+4. Update DNS: `gitea.lan.ddnsgeek.com` → swarm Traefik endpoint.
+5. Update the flake remote URL in `variables.nix` (`giteaDomain`) if the address changes.
+
+---
+
+## 6. Other services
+
+Deploy remaining services (Grafana, InfluxDB, NodeRed, Prometheus, etc.)
+as swarm stacks.  Most have no special migration concern — they use NFS
+volumes already on the shared storage.
+
+Services with stateful databases (PostgreSQL, MariaDB) should follow the
+pattern: stop on CT 105, confirm NFS database directory is intact, deploy on
+swarm, verify.
+
+---
+
+## 7. Monitoring — Beszel
+
+The Beszel hub runs on CT 105 (`docker.sweet.home:8090`).  Both swarm nodes
+run `beszel-agent` (from `modules/beszel/enable-agent.nix`), pointing at the
+existing hub URL.
+
+No migration needed for Beszel itself during the container migration.  Once
+all services are on the swarm, you may wish to move the Beszel hub too (as a
+swarm service with a placement constraint) but this is optional.
+
+---
+
+## 8. DNS cutover
+
+When a service is confirmed working on the swarm, update the FreeIPA DNS
+A record from the CT 105 IP (192.168.2.225) to a swarm node IP or, when a
+shared Traefik frontend is in place, to a round-robin record across both nodes.
+
+**Recommended approach — Traefik as the single entry point:**
+
+```
+service.lan.ddnsgeek.com  →  Traefik on swarm (global mode)
+docker.sweet.home         →  keep as 192.168.2.225 (CT 105) until fully decommissioned
+```
+
+For LAN-only services using `*.sweet.home` names, update FreeIPA directly:
+
+```bash
+# On domain-controller (or via SSH):
+ipa dnsrecord-mod sweet.home nextcloud --a-rec=192.168.2.230
+# Add 192.168.2.231 as a second A record for round-robin (optional):
+ipa dnsrecord-add sweet.home nextcloud --a-rec=192.168.2.231
+```
+
+Services behind Traefik don't need their own DNS updates — only Traefik's
+own entry point IPs need to change.
+
+---
+
+## 9. NixOS cleanup — CT 105
+
+Once all services are migrated:
+
+**Remove from `modules/build-types/docker.nix`:**
+- `../docker/nextcloud-cron-job.nix` — replaced by sidecar container
+- `../traefik/rotate-logs.nix` — replaced by Docker log driver
+
+**Keep in `modules/build-types/docker.nix` until CT 105 is decommissioned:**
+- `../docker/docker-health-to-gotify.nix` — still monitors CT 105's own daemon
+- Everything else
+
+**When decommissioning CT 105:**
+1. Confirm all NFS volumes are in use only by swarm services (not CT 105).
+2. Stop CT 105: `pct stop 105` on pve1.
+3. Archive/remove the `lxc-docker` and `proxmox-docker` targets from `flake.nix`.
+4. Remove `hosts/docker/`, `modules/build-types/docker.nix`, and `modules/docker/`.
+5. Update `variables.nix` to remove `dockerIp`, `dockerStorageIp`, `dockerHost`
+   (or reassign `dockerHost` to point at a swarm node for Beszel hub resolution).
+
+---
+
+## Rollback
+
+Any stack can be rolled back to CT 105 independently:
+
+```bash
+# On CT 105:
+docker start <service-name>
+# Update DNS A record back to 192.168.2.225
+ipa dnsrecord-mod sweet.home <service> --a-rec=192.168.2.225
+```
+
+CT 105 remains running throughout the cutover.  Only decommission it after
+every service is confirmed stable on the swarm and you have run one full
+backup cycle from the new hosts.
@@ -7,6 +7,10 @@
 | LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
 | Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
 | Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink |
+| Swarm cluster | 30 | `192.168.30.0/24` | Docker Swarm gossip (TCP/UDP 7946) + VXLAN overlay (UDP 4789) | No — internal `vmbr3` only, no uplink |
+
+When expanded to a second Proxmox node, VLAN 10 (cluster), VLAN 20 (storage-client), and VLAN 30 (swarm) all share
+the same inter-node trunk NIC via 802.1q VLAN tagging — different VLAN IDs, same physical cable.

 The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
 bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
@@ -27,9 +31,9 @@ is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20

 ## DNS Zones

-FreeIPA (domain-controller.sweet.home) is authoritative for all zones. Three
-zones correspond to the three subnets — one per VLAN. All zones are internal
-only; no external delegation.
+FreeIPA (domain-controller.sweet.home) is authoritative for all zones.
+
+Four zones correspond to the four subnets. All zones are internal only; no external delegation.

 ### sweet.home — VLAN 2 (192.168.2.x)

@@ -132,12 +136,14 @@ All VMs and LXC containers run on pve1.
 | `192.168.2.228` | ha-node1 | HA file server node 1 — management NIC | Active |
 | `192.168.2.227` | ha-node2 | HA file server node 2 — management NIC | Active |
 | `192.168.2.226` | server | Former NFS/ZFS file server — decommissioned | Removed from flake |
-| `192.168.2.225` | docker | Docker / Traefik stack | Active |
+| `192.168.2.225` | docker | Docker / Traefik stack (CT 105 — existing single-host) | Active |
 | `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
 | `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active |
 | `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active |
 | `192.168.2.221` | tor-relay | Tor relay | Active |
 | `192.168.2.220` | pdm | Proxmox Deploy Manager | Active |
+| `192.168.2.231` | ha-docker-2 | Docker Swarm node 2 — management NIC | Active |
+| `192.168.2.230` | ha-docker-1 | Docker Swarm node 1 — management NIC | Active |

 ### Client DHCP pool (.10–.59)

@@ -175,7 +181,9 @@ Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached.
 | `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
 | `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
 | `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — decommissioned |
-| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client |
+| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client (CT 105) |
+| `192.168.20.231` | ha-docker-2 | Storage-client NIC (ens19 / vmbr2) — NFS client |
+| `192.168.20.230` | ha-docker-1 | Storage-client NIC (ens19 / vmbr2) — NFS client |
 | — | no gateway | Isolated — not routed to LAN or internet |

 NFS clients mount from `192.168.20.229` (surviving failover transparently via the VIP).
@@ -184,3 +192,37 @@ cannot reach either service on this VIP. The `vip-storage` endpoint is not reach
 from the workstation directly (internal bridge only); health checks proxy through the
 active HA node.

+---
+
+## Swarm cluster network — VLAN 30 — 192.168.30.0/24
+
+Internal to pve1 only. Proxmox bridge `vmbr3`, no physical NIC attached.
+Carries Docker Swarm inter-node traffic only: Raft consensus (TCP 2377),
+Serf gossip (TCP/UDP 7946), and VXLAN overlay data path (UDP 4789).
+Docker Swarm is initialised with `--advertise-addr` and `--data-path-addr`
+both pointing to this subnet so all cluster traffic stays on `vmbr3` and
+never crosses the LAN.
+
+| IP | Hostname | Interface / role |
+|---|---|---|
+| `192.168.30.231` | ha-docker-2 | Swarm cluster NIC (ens20 / vmbr3) |
+| `192.168.30.230` | ha-docker-1 | Swarm cluster NIC (ens20 / vmbr3) |
+| — | no gateway | Isolated — not routed to LAN or internet |
+
+### DNS zone: `swarm.home` — VLAN 30 (192.168.30.x)
+
+| Hostname | A record | Notes |
+|---|---|---|
+| `ha-docker-1.swarm.home` | `192.168.30.230` | Swarm NIC — debugging only |
+| `ha-docker-2.swarm.home` | `192.168.30.231` | Swarm NIC — debugging only |
+
+Operators reach the Docker API on the LAN IPs (`192.168.2.230`/`.231`), not these addresses.
+The `swarm.home` records exist for diagnostic convenience (e.g. confirming `vmbr3` routing).
+
+### Multi-node Proxmox expansion
+
+When a second Proxmox node (pve2) is added, VLAN 10 (cluster), VLAN 20 (storage-client),
+and VLAN 30 (swarm) all extend to pve2 via 802.1q VLAN tagging on the inter-node trunk
+link. All three internal networks share the same physical NIC between hypervisors —
+VLAN tags provide the logical separation.
+