diff --git a/docs/internal/docker-swarm-cutover.md b/docs/internal/docker-swarm-cutover.md new file mode 100644 index 0000000..abf4236 --- /dev/null +++ b/docs/internal/docker-swarm-cutover.md @@ -0,0 +1,330 @@ +# Docker Swarm Cutover Plan + +Migration guide for moving containerised services from the existing single-host +Docker LXC container (CT 105, `docker.sweet.home`, 192.168.2.225) to the new +Docker Swarm cluster (`ha-docker-1` / `ha-docker-2`, 192.168.2.230–231). + +CT 105 stays running throughout. Services migrate one stack at a time. +Roll back any stack by restarting it on CT 105 if anything goes wrong. + +--- + +## Prerequisites + +- Swarm cluster deployed and healthy (`scripts/docker-swarm/deploy.sh`). +- Both nodes show `Ready / Active / Manager` in `docker node ls`. +- NFS mounts healthy on both swarm nodes (`/mnt/docker/config`, `/mnt/docker/databases`, `/mnt/docker/volumes`). +- Access to FreeIPA DNS admin to update A records during cutover. + +--- + +## 1. Traefik — switch to Docker log rotation + +**Current state (CT 105):** Traefik writes access logs to the NFS volume at +`/mnt/docker/volumes/traefik-data/logs/`. `modules/traefik/rotate-logs.nix` +rotates those files via `logrotate`. + +**Swarm approach:** Remove file-based access logging from Traefik's static +config and rely on Docker's json-file log driver with built-in rotation. +Traefik container logs (including access events) then live under +`/var/lib/docker/containers//` on the node running Traefik. + +### Steps + +**1a.** In the Traefik stack definition, add logging config to the service: + +```yaml +services: + traefik: + logging: + driver: "json-file" + options: + max-size: "100m" + max-file: "20" +``` + +**1b.** In `traefik.yml` (Traefik's static config), remove the `accessLog` +file path if present. To keep structured access logs, use Traefik's +`accessLog.format: json` with no `filePath` — logs then go to stdout and are +captured by the json-file driver above. + +**1c.** Deploy Traefik to the swarm: + +```bash +# On either swarm manager: +docker stack deploy -c /mnt/docker/config/traefik/docker-compose.yml traefik +``` + +Traefik should be deployed as a **global mode** service so it runs on all +swarm nodes and handles ingress on whichever node a request arrives at: + +```yaml +services: + traefik: + deploy: + mode: global + placement: + constraints: + - node.role == manager +``` + +**1d.** After confirming Traefik works on the swarm, remove +`traefik/rotate-logs.nix` from the `docker` build type in +`modules/build-types/docker.nix` and rebuild CT 105. + +**DNS:** Update `docker.sweet.home` and any service FQDNs that point at +192.168.2.225 to a swarm VIP or round-robin A records once Traefik is running +on the swarm. See section 8 (DNS cutover). + +--- + +## 2. Nextcloud — migrate cron job to sidecar container + +**Current state (CT 105):** `modules/docker/nextcloud-cron-job.nix` runs a +systemd timer every 5 minutes that calls: +```bash +docker exec nextcloud-webapp php ./cron.php +``` + +**Swarm problem:** `docker exec` only works against the local daemon. If +Nextcloud is scheduled on the other swarm node, the exec fails silently and +cron never runs. + +**Swarm approach:** Add a `nextcloud-cron` sidecar container to the Nextcloud +stack definition, pinned to the same node as the main Nextcloud container via +placement constraints. + +### Steps + +**2a.** Choose which swarm node will host Nextcloud (e.g. `ha-docker-1`). +Label that node: + +```bash +# On either swarm manager: +docker node update --label-add nextcloud=true ha-docker-1 +``` + +**2b.** In the Nextcloud stack compose file, add the sidecar and pin both +services to the labelled node: + +```yaml +services: + nextcloud-webapp: + image: nextcloud:production # pin same version as CT 105 + deploy: + replicas: 1 + placement: + constraints: + - node.labels.nextcloud == true + # ... existing volumes, env, networks ... + + nextcloud-cron: + image: nextcloud:production # same image, different entrypoint + entrypoint: /cron.sh + deploy: + replicas: 1 + placement: + constraints: + - node.labels.nextcloud == true # must co-locate with webapp + volumes: + # Same data volume as nextcloud-webapp so cron sees the same files. + - nextcloud-data:/var/www/html + # No ports exposed — cron only runs PHP inside the container. +``` + +`/cron.sh` is Nextcloud's built-in cron entrypoint. It runs +`php -f /var/www/html/cron.php` in a loop, sleeping for 5 minutes between +runs — identical to the current systemd timer. + +**2c.** Migrate Nextcloud's data volume to the swarm: + +``` +/mnt/docker/volumes/nextcloud-data/ → already on NFS, no migration needed +/mnt/docker/databases/nextcloud/ → already on NFS, no migration needed +``` + +The NFS paths are identical on the swarm nodes (`mount-data.nix` mounts the +same shares from the same VIP). Stop Nextcloud on CT 105, deploy on the +swarm, confirm it starts cleanly. + +**2d.** Remove `nextcloud-cron-job.nix` from `modules/build-types/docker.nix` +and rebuild CT 105 after confirming Nextcloud works on the swarm. + +--- + +## 3. docker-health-to-gotify — update for swarm awareness + +**Current state (CT 105):** The script at +`/home/nixos/docker/monitoring/gotify/docker-health-to-gotify.sh` runs every +minute, calls `docker ps --filter health=unhealthy`, and notifies Gotify. + +**Swarm behaviour:** The same script runs on both swarm nodes independently, +each monitoring its own local Docker daemon. This gives per-node coverage +across the swarm. + +**Changes needed in the script** (edit the copy on the NFS volume — it takes +effect on both nodes simultaneously on the next timer fire): + +### 3a. Strip the Swarm task suffix from service names + +In swarm mode, `docker ps --format '{{.Names}}'` returns names like +`nextcloud-webapp.1.abc123xyz`. The notification should show `nextcloud-webapp`, +not the full task name. + +```bash +# Before: +CONTAINER_NAME=$(docker ps --format '{{.Names}}' ...) + +# After: +CONTAINER_NAME=$(docker ps --format '{{.Names}}' ... | cut -d. -f1) +``` + +### 3b. Include the reporting node in the Gotify message + +Add `$(hostname)` to the notification payload so you know which swarm node +detected the problem: + +```bash +MESSAGE="[$(hostname)] ${CONTAINER_NAME} is unhealthy" +``` + +### 3c. Extend to catch swarm service replica failures + +`docker ps` only shows what's running locally. If a service has zero healthy +replicas (task crash-looping) it may not show up on either node's `docker ps` +at the same moment. Add a swarm-level check: + +```bash +# Run only on managers (both ha-docker nodes are managers): +if docker info --format '{{.Swarm.ControlAvailable}}' 2>/dev/null | grep -q true; then + # Find services where running replicas < desired replicas + docker service ls --format '{{.Name}}\t{{.Replicas}}' | \ + awk -F'\t' '$2 !~ /^[0-9]+\/[0-9]+$/ || split($2,a,"/") && a[1] < a[2] { print $1, $2 }' | \ + while read -r svc_name replicas; do + # Send Gotify notification for degraded service + curl -s -X POST "${GOTIFY_URL}/message" \ + -H "X-Gotify-Key: ${GOTIFY_TOKEN}" \ + -d "title=Swarm service degraded" \ + -d "message=[$(hostname)] ${svc_name}: ${replicas} replicas" + done +fi +``` + +This catches the case where a service's desired replicas are not running +(e.g. OOM kill, image pull failure) — a failure mode that doesn't produce a +Docker health event on any node. + +--- + +## 4. Passbolt migration + +Passbolt has strict data integrity requirements. Migrate with care: + +1. **Backup first** — `docker exec passbolt-webapp php /usr/share/php/passbolt/bin/cake passbolt export_keys` and a database dump. +2. Database is on NFS (`/mnt/docker/databases/passbolt/`) — no data copy needed. +3. Pin Passbolt to a specific node: `docker node update --label-add passbolt=true ha-docker-1` +4. Add placement constraint `node.labels.passbolt == true` to the Passbolt stack. +5. Stop on CT 105, deploy on swarm, verify login works. +6. Test email delivery and 2FA. + +--- + +## 5. Gitea migration + +Gitea's data directory is on NFS (`/mnt/docker/volumes/gitea-data/`). + +1. Stop Gitea on CT 105: `docker stop gitea` +2. Deploy to swarm with placement constraint (pin to `ha-docker-1` initially). +3. Verify web UI and SSH clone/push work. +4. Update DNS: `gitea.lan.ddnsgeek.com` → swarm Traefik endpoint. +5. Update the flake remote URL in `variables.nix` (`giteaDomain`) if the address changes. + +--- + +## 6. Other services + +Deploy remaining services (Grafana, InfluxDB, NodeRed, Prometheus, etc.) +as swarm stacks. Most have no special migration concern — they use NFS +volumes already on the shared storage. + +Services with stateful databases (PostgreSQL, MariaDB) should follow the +pattern: stop on CT 105, confirm NFS database directory is intact, deploy on +swarm, verify. + +--- + +## 7. Monitoring — Beszel + +The Beszel hub runs on CT 105 (`docker.sweet.home:8090`). Both swarm nodes +run `beszel-agent` (from `modules/beszel/enable-agent.nix`), pointing at the +existing hub URL. + +No migration needed for Beszel itself during the container migration. Once +all services are on the swarm, you may wish to move the Beszel hub too (as a +swarm service with a placement constraint) but this is optional. + +--- + +## 8. DNS cutover + +When a service is confirmed working on the swarm, update the FreeIPA DNS +A record from the CT 105 IP (192.168.2.225) to a swarm node IP or, when a +shared Traefik frontend is in place, to a round-robin record across both nodes. + +**Recommended approach — Traefik as the single entry point:** + +``` +service.lan.ddnsgeek.com → Traefik on swarm (global mode) +docker.sweet.home → keep as 192.168.2.225 (CT 105) until fully decommissioned +``` + +For LAN-only services using `*.sweet.home` names, update FreeIPA directly: + +```bash +# On domain-controller (or via SSH): +ipa dnsrecord-mod sweet.home nextcloud --a-rec=192.168.2.230 +# Add 192.168.2.231 as a second A record for round-robin (optional): +ipa dnsrecord-add sweet.home nextcloud --a-rec=192.168.2.231 +``` + +Services behind Traefik don't need their own DNS updates — only Traefik's +own entry point IPs need to change. + +--- + +## 9. NixOS cleanup — CT 105 + +Once all services are migrated: + +**Remove from `modules/build-types/docker.nix`:** +- `../docker/nextcloud-cron-job.nix` — replaced by sidecar container +- `../traefik/rotate-logs.nix` — replaced by Docker log driver + +**Keep in `modules/build-types/docker.nix` until CT 105 is decommissioned:** +- `../docker/docker-health-to-gotify.nix` — still monitors CT 105's own daemon +- Everything else + +**When decommissioning CT 105:** +1. Confirm all NFS volumes are in use only by swarm services (not CT 105). +2. Stop CT 105: `pct stop 105` on pve1. +3. Archive/remove the `lxc-docker` and `proxmox-docker` targets from `flake.nix`. +4. Remove `hosts/docker/`, `modules/build-types/docker.nix`, and `modules/docker/`. +5. Update `variables.nix` to remove `dockerIp`, `dockerStorageIp`, `dockerHost` + (or reassign `dockerHost` to point at a swarm node for Beszel hub resolution). + +--- + +## Rollback + +Any stack can be rolled back to CT 105 independently: + +```bash +# On CT 105: +docker start +# Update DNS A record back to 192.168.2.225 +ipa dnsrecord-mod sweet.home --a-rec=192.168.2.225 +``` + +CT 105 remains running throughout the cutover. Only decommission it after +every service is confirmed stable on the swarm and you have run one full +backup cycle from the new hosts. diff --git a/docs/ip-addressing.md b/docs/ip-addressing.md index 2a6554a..e6b84c8 100644 --- a/docs/ip-addressing.md +++ b/docs/ip-addressing.md @@ -7,6 +7,10 @@ | LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) | | Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink | | Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink | +| Swarm cluster | 30 | `192.168.30.0/24` | Docker Swarm gossip (TCP/UDP 7946) + VXLAN overlay (UDP 4789) | No — internal `vmbr3` only, no uplink | + +When expanded to a second Proxmox node, VLAN 10 (cluster), VLAN 20 (storage-client), and VLAN 30 (swarm) all share +the same inter-node trunk NIC via 802.1q VLAN tagging — different VLAN IDs, same physical cable. The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux bridges with no physical port attached; traffic between guests on each bridge stays in-kernel. @@ -27,9 +31,9 @@ is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20 ## DNS Zones -FreeIPA (domain-controller.sweet.home) is authoritative for all zones. Three -zones correspond to the three subnets — one per VLAN. All zones are internal -only; no external delegation. +FreeIPA (domain-controller.sweet.home) is authoritative for all zones. + +Four zones correspond to the four subnets. All zones are internal only; no external delegation. ### sweet.home — VLAN 2 (192.168.2.x) @@ -132,12 +136,14 @@ All VMs and LXC containers run on pve1. | `192.168.2.228` | ha-node1 | HA file server node 1 — management NIC | Active | | `192.168.2.227` | ha-node2 | HA file server node 2 — management NIC | Active | | `192.168.2.226` | server | Former NFS/ZFS file server — decommissioned | Removed from flake | -| `192.168.2.225` | docker | Docker / Traefik stack | Active | +| `192.168.2.225` | docker | Docker / Traefik stack (CT 105 — existing single-host) | Active | | `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active | | `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active | | `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active | | `192.168.2.221` | tor-relay | Tor relay | Active | | `192.168.2.220` | pdm | Proxmox Deploy Manager | Active | +| `192.168.2.231` | ha-docker-2 | Docker Swarm node 2 — management NIC | Active | +| `192.168.2.230` | ha-docker-1 | Docker Swarm node 1 — management NIC | Active | ### Client DHCP pool (.10–.59) @@ -175,7 +181,9 @@ Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached. | `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) | | `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) | | `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — decommissioned | -| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client | +| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client (CT 105) | +| `192.168.20.231` | ha-docker-2 | Storage-client NIC (ens19 / vmbr2) — NFS client | +| `192.168.20.230` | ha-docker-1 | Storage-client NIC (ens19 / vmbr2) — NFS client | | — | no gateway | Isolated — not routed to LAN or internet | NFS clients mount from `192.168.20.229` (surviving failover transparently via the VIP). @@ -184,3 +192,37 @@ cannot reach either service on this VIP. The `vip-storage` endpoint is not reach from the workstation directly (internal bridge only); health checks proxy through the active HA node. +--- + +## Swarm cluster network — VLAN 30 — 192.168.30.0/24 + +Internal to pve1 only. Proxmox bridge `vmbr3`, no physical NIC attached. +Carries Docker Swarm inter-node traffic only: Raft consensus (TCP 2377), +Serf gossip (TCP/UDP 7946), and VXLAN overlay data path (UDP 4789). +Docker Swarm is initialised with `--advertise-addr` and `--data-path-addr` +both pointing to this subnet so all cluster traffic stays on `vmbr3` and +never crosses the LAN. + +| IP | Hostname | Interface / role | +|---|---|---| +| `192.168.30.231` | ha-docker-2 | Swarm cluster NIC (ens20 / vmbr3) | +| `192.168.30.230` | ha-docker-1 | Swarm cluster NIC (ens20 / vmbr3) | +| — | no gateway | Isolated — not routed to LAN or internet | + +### DNS zone: `swarm.home` — VLAN 30 (192.168.30.x) + +| Hostname | A record | Notes | +|---|---|---| +| `ha-docker-1.swarm.home` | `192.168.30.230` | Swarm NIC — debugging only | +| `ha-docker-2.swarm.home` | `192.168.30.231` | Swarm NIC — debugging only | + +Operators reach the Docker API on the LAN IPs (`192.168.2.230`/`.231`), not these addresses. +The `swarm.home` records exist for diagnostic convenience (e.g. confirming `vmbr3` routing). + +### Multi-node Proxmox expansion + +When a second Proxmox node (pve2) is added, VLAN 10 (cluster), VLAN 20 (storage-client), +and VLAN 30 (swarm) all extend to pve2 via 802.1q VLAN tagging on the inter-node trunk +link. All three internal networks share the same physical NIC between hypervisors — +VLAN tags provide the logical separation. + diff --git a/flake.nix b/flake.nix index 27f6833..c0f363c 100644 --- a/flake.nix +++ b/flake.nix @@ -130,6 +130,9 @@ proxmox-ha-server-1 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-1/host.nix; nameSuffix = "-1"; }; proxmox-ha-server-2 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-2/host.nix; nameSuffix = "-2"; }; + + proxmox-ha-docker-1 = mkTarget { platform = "proxmox"; buildType = "ha-docker"; hostPath = ./hosts/ha-docker-1/host.nix; nameSuffix = "-1"; }; + proxmox-ha-docker-2 = mkTarget { platform = "proxmox"; buildType = "ha-docker"; hostPath = ./hosts/ha-docker-2/host.nix; nameSuffix = "-2"; }; }; # Auto-install environments (migrated from the former nix-auto-installer diff --git a/hosts/ha-docker-1/host.nix b/hosts/ha-docker-1/host.nix new file mode 100644 index 0000000..4ae2a48 --- /dev/null +++ b/hosts/ha-docker-1/host.nix @@ -0,0 +1,34 @@ +{ vars, ... }: +{ + networking = { + hostName = vars.haDocker1Host; + hostId = "a1d0c4e1"; + useDHCP = false; + interfaces = { + # ens18 — LAN management NIC (vmbr0, 192.168.2.0/24) + ${vars.vmLanInterface}.ipv4.addresses = [{ + address = vars.haDocker1Ip; + prefixLength = vars.lanPrefixLength; + }]; + # ens19 — storage-client NIC (vmbr2, 192.168.20.0/24) — NFS from HA cluster + ${vars.haDockerStorageInterface}.ipv4.addresses = [{ + address = vars.haDocker1StorageIp; + prefixLength = vars.haClientPrefixLength; + }]; + # ens20 — swarm cluster NIC (vmbr3, 192.168.30.0/24) — Docker gossip + VXLAN + ${vars.haDockerSwarmInterface}.ipv4.addresses = [{ + address = vars.haDocker1SwarmIp; + prefixLength = vars.haDockerSwarmPrefixLength; + }]; + }; + defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; }; + nameservers = [ vars.domainControllerIp ]; + }; + + # Only register the LAN IP with IPA DNS. Without this, sssd dyndns + # would also register Docker bridge IPs (172.x.x.x) and the storage/swarm + # NIC IPs as A records for ha-docker-1.sweet.home. + security.ipa.dyndns.interface = vars.vmLanInterface; + + system.stateVersion = "26.05"; +} diff --git a/hosts/ha-docker-2/host.nix b/hosts/ha-docker-2/host.nix new file mode 100644 index 0000000..c3e8c18 --- /dev/null +++ b/hosts/ha-docker-2/host.nix @@ -0,0 +1,32 @@ +{ vars, ... }: +{ + networking = { + hostName = vars.haDocker2Host; + hostId = "a2d0c4e2"; + useDHCP = false; + interfaces = { + # ens18 — LAN management NIC (vmbr0, 192.168.2.0/24) + ${vars.vmLanInterface}.ipv4.addresses = [{ + address = vars.haDocker2Ip; + prefixLength = vars.lanPrefixLength; + }]; + # ens19 — storage-client NIC (vmbr2, 192.168.20.0/24) — NFS from HA cluster + ${vars.haDockerStorageInterface}.ipv4.addresses = [{ + address = vars.haDocker2StorageIp; + prefixLength = vars.haClientPrefixLength; + }]; + # ens20 — swarm cluster NIC (vmbr3, 192.168.30.0/24) — Docker gossip + VXLAN + ${vars.haDockerSwarmInterface}.ipv4.addresses = [{ + address = vars.haDocker2SwarmIp; + prefixLength = vars.haDockerSwarmPrefixLength; + }]; + }; + defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; }; + nameservers = [ vars.domainControllerIp ]; + }; + + # Only register the LAN IP with IPA DNS — same reasoning as ha-docker-1. + security.ipa.dyndns.interface = vars.vmLanInterface; + + system.stateVersion = "26.05"; +} diff --git a/modules/build-types/ha-docker.nix b/modules/build-types/ha-docker.nix new file mode 100644 index 0000000..968f06f --- /dev/null +++ b/modules/build-types/ha-docker.nix @@ -0,0 +1,84 @@ +# Docker Swarm node build type. +# +# Produces NixOS hosts that form a Docker Swarm manager cluster. Two nodes +# (ha-docker-1, ha-docker-2) are both managers so either can accept Docker +# API and `docker stack` commands. +# +# Key differences from the existing `docker` build type (used by CT 105): +# - nextcloud-cron-job.nix is EXCLUDED — `docker exec` breaks in swarm +# because the target container may be on the other node. The cron job +# is replaced by a nextcloud-cron sidecar in the Nextcloud stack. +# See docs/internal/docker-swarm-cutover.md. +# - traefik/rotate-logs.nix is EXCLUDED — log rotation moves to Docker's +# json-file log driver (max-size/max-file on the Traefik service +# definition). See docs/internal/docker-swarm-cutover.md. +# - raspi/mount-data.nix is EXCLUDED — specific to CT 105's backup role. +# - Swarm firewall ports (2377/tcp, 7946/tcp+udp, 4789/udp) are opened +# on the swarm NIC (ens20/vmbr3) only. +# - checkReversePath = "loose" is required for the Swarm ingress routing +# mesh: VXLAN return traffic is asymmetric (arrives ens20, exits ens18). +# - beszel-agent is enabled for host-level monitoring. +{ pkgs, vars, ... }: + +{ + # Pin Docker Engine to version 29, matching CT 105, so image layers cached + # on NFS volumes remain compatible across old and new hosts. + nixpkgs.overlays = [ + (final: prev: { + docker = prev.docker_29; + docker_cli = prev.docker_29; + }) + ]; + + imports = [ + ../docker/enable-service.nix + ../docker/mount-data.nix + ../docker/docker-health-to-gotify.nix + ../beszel/enable-agent.nix + ../services/enable-rpcbind.nix + ]; + + environment.systemPackages = with pkgs; [ + nfs-utils + ]; + + boot.supportedFilesystems = [ "nfs" ]; + + systemd.tmpfiles.rules = [ + # Symlink ~/docker → NFS config mount so the docker-health-to-gotify + # script (and operator convenience) resolves ~/docker/... correctly. + "L+ /home/${vars.primaryUser}/docker - - - - ${vars.nfsShares.dockerConfig.mountpoint}" + "d /mnt/docker 0755 ${vars.primaryUser} users -" + ]; + + users.users.${vars.primaryUser}.extraGroups = [ "docker" ]; + + networking.firewall = { + # LAN-facing service ports — same as the existing docker build type. + allowedTCPPorts = [ + vars.ports.dockerHttp + vars.ports.dockerHttps + vars.ports.dockerExtra + vars.ports.beszelHub + ]; + + # Swarm inter-node ports restricted to the swarm NIC (ens20/vmbr3). + # vmbr3 is an isolated internal bridge — no LAN reachability. + interfaces.${vars.haDockerSwarmInterface} = { + allowedTCPPorts = [ + vars.ports.dockerSwarmMgmt # 2377 — Raft + cluster management + vars.ports.dockerSwarmDisc # 7946 — Serf gossip (TCP half) + ]; + allowedUDPPorts = [ + vars.ports.dockerSwarmDisc # 7946 — Serf gossip (UDP half) + vars.ports.dockerSwarmVxlan # 4789 — VXLAN overlay data path + ]; + }; + + # Docker Swarm ingress routing mesh creates asymmetric routes: a request + # arrives on ens18 (LAN) for a container that lives on ens20's VXLAN + # overlay; the return path differs from the incoming interface. Strict + # rp_filter drops these packets. "loose" allows them. + checkReversePath = "loose"; + }; +} diff --git a/scripts/docker-swarm/deploy.sh b/scripts/docker-swarm/deploy.sh new file mode 100755 index 0000000..be2b434 --- /dev/null +++ b/scripts/docker-swarm/deploy.sh @@ -0,0 +1,585 @@ +#!/usr/bin/env bash +# deploy.sh — Full lifecycle management for the Docker Swarm HA cluster. +# +# Provisions two NixOS Proxmox VMs (ha-docker-1, ha-docker-2) as dual-manager +# Docker Swarm nodes sharing NFS storage from the existing HA file-server +# cluster. Both nodes are managers so either can accept Docker API and +# `docker stack` commands. +# +# Usage: +# scripts/docker-swarm/deploy.sh [options] +# scripts/docker-swarm/deploy.sh --destroy [options] +# +# Phases (all run by default; skip any with --skip-): +# 1. ensure-bridge Create vmbr3 (swarm cluster bridge) on the Proxmox node. +# 2. sync-keys Generate SSH host keys for both nodes (clan vars). +# 3. ipa-hosts Create IPA host objects + sops-encrypted keytabs. +# 4. create-vms Build NixOS disk images and create VMs via create-proxmox-resource.sh. +# 5. add-hardware Attach vmbr2 (storage) and vmbr3 (swarm) NICs; start VMs. +# 6. boot-wait Wait for SSH on both LAN IPs. +# 7. refresh-sops-keys Detect disko key drift; re-encrypt secrets; commit. +# 8. init-swarm docker swarm init on node1; manager join on node2; label nodes. +# 9. dns Register storage.home and swarm.home A records in FreeIPA. +# 10. verify docker node ls; NFS mount check; swarm health. +# +# Options: +# --node Proxmox host (default: pve1.sweet.home) +# --vmid1 VMID for ha-docker-1 (default: 202) +# --vmid2 VMID for ha-docker-2 (default: 203) +# --storage Proxmox storage pool (default: local-zfs) +# --swarm-bridge
Bridge for Docker Swarm cluster network (default: vmbr3) +# --storage-bridge
Bridge for NFS storage network (default: vmbr2) +# --memory RAM per node (default: 4096) +# --cores vCPUs per node (default: 4) +# --skip-ensure-bridge Skip vmbr3 creation/check +# --skip-sync-keys Skip sync-host-keys.sh (clan vars already exist) +# --skip-ipa-hosts Skip IPA host account creation (keytabs already exist) +# --skip-create-vms Skip VM creation (VMs already exist) +# --skip-add-hardware Skip NIC attachment (already attached) +# --skip-boot-wait Skip boot/SSH wait (VMs already running) +# --skip-refresh-sops-keys Skip sops host-key drift fix +# --skip-init-swarm Skip swarm initialisation (already initialised) +# --skip-dns Skip FreeIPA DNS record creation +# --skip-verify Skip post-deploy health checks +# --force-rebuild Pass --force-rebuild to create-proxmox-resource.sh +# --destroy Stop and delete both VMs (skip all other phases) +# --dry-run Print what would run without executing +# -h|--help Show this message +# +# Prerequisites: +# - SSH access to the Proxmox node as $PROXMOX_SSH_USER (wayne). +# - sops age key in the standard location (used by sync-host-keys.sh). +# - SSH access to domain-controller.sweet.home as $PROXMOX_SSH_USER for DNS phase. +# - For --skip-sync-keys: clan vars already in vars/per-machine/proxmox-ha-docker-{1,2}/. +# - For --skip-ipa-hosts: secrets/ha-docker-{1,2}.keytab already exist and are committed. + +set -euo pipefail + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)" + +# shellcheck source=../env.sh +source "${REPO_ROOT}/scripts/env.sh" + +# ── Defaults ────────────────────────────────────────────────────────────────── + +NODE="${PVE1_HOST}" # deploy.sh targets pve1 by default (authorised for this cluster) +VMID1=202 +VMID2=203 +STORAGE="${PROXMOX_STORAGE:-local-zfs}" +SWARM_BRIDGE="vmbr3" +STORAGE_BRIDGE="vmbr2" +MEMORY_MB=4096 +CORES=4 + +SKIP_ENSURE_BRIDGE=false +SKIP_SYNC_KEYS=false +SKIP_IPA_HOSTS=false +SKIP_CREATE_VMS=false +SKIP_ADD_HARDWARE=false +SKIP_BOOT_WAIT=false +SKIP_REFRESH_SOPS_KEYS=false +SKIP_INIT_SWARM=false +SKIP_DNS=false +SKIP_VERIFY=false +FORCE_REBUILD=false +DESTROY=false +DRY_RUN=false + +# ── Variables from repo (mirrors variables.nix) ─────────────────────────────── + +NODE1_HOST="ha-docker-1" +NODE2_HOST="ha-docker-2" +NODE1_LAN_IP="192.168.2.230" +NODE2_LAN_IP="192.168.2.231" +NODE1_SWARM_IP="192.168.30.230" +NODE2_SWARM_IP="192.168.30.231" +NODE1_STORAGE_IP="192.168.20.230" +NODE2_STORAGE_IP="192.168.20.231" +SWARM_CIDR="192.168.30.0/24" +STORAGE_CIDR="192.168.20.0/24" +STORAGE_ZONE="storage.home" +SWARM_ZONE="swarm.home" +SSH_USER="${PROXMOX_SSH_USER:-wayne}" +DC_HOST="${IPA_SERVER:-domain-controller.sweet.home}" + +# ── Argument parsing ────────────────────────────────────────────────────────── + +usage() { + sed -n '/^# Usage:/,/^[^#]/{ /^#/{ s/^# \?//; p } }' "$0" + exit "${1:-0}" +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --node) NODE="$2"; shift 2 ;; + --vmid1) VMID1="$2"; shift 2 ;; + --vmid2) VMID2="$2"; shift 2 ;; + --storage) STORAGE="$2"; shift 2 ;; + --swarm-bridge) SWARM_BRIDGE="$2"; shift 2 ;; + --storage-bridge) STORAGE_BRIDGE="$2"; shift 2 ;; + --memory) MEMORY_MB="$2"; shift 2 ;; + --cores) CORES="$2"; shift 2 ;; + --skip-ensure-bridge) SKIP_ENSURE_BRIDGE=true; shift ;; + --skip-sync-keys) SKIP_SYNC_KEYS=true; shift ;; + --skip-ipa-hosts) SKIP_IPA_HOSTS=true; shift ;; + --skip-create-vms) SKIP_CREATE_VMS=true; shift ;; + --skip-add-hardware) SKIP_ADD_HARDWARE=true; shift ;; + --skip-boot-wait) SKIP_BOOT_WAIT=true; shift ;; + --skip-refresh-sops-keys) SKIP_REFRESH_SOPS_KEYS=true; shift ;; + --skip-init-swarm) SKIP_INIT_SWARM=true; shift ;; + --skip-dns) SKIP_DNS=true; shift ;; + --skip-verify) SKIP_VERIFY=true; shift ;; + --force-rebuild) FORCE_REBUILD=true; shift ;; + --destroy) DESTROY=true; shift ;; + --dry-run) DRY_RUN=true; shift ;; + -h|--help) usage 0 ;; + *) echo "Unknown option: $1" >&2; usage 1 ;; + esac +done + +# ── Helpers ─────────────────────────────────────────────────────────────────── + +log() { echo "==> $*"; } +logn() { echo " $*"; } +err() { echo "ERROR: $*" >&2; exit 1; } + +run() { + if $DRY_RUN; then + echo "[dry-run] $*" + else + "$@" + fi +} + +pve() { + if $DRY_RUN; then + echo "[dry-run] ssh ${SSH_USER}@${NODE} sudo $*" + else + ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*" + fi +} + +pve_check() { + # Read-only probe — always executes even in dry-run. + ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*" +} + +SWARM_USER="nixos" + +n1() { + ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \ + "${SWARM_USER}@${NODE1_LAN_IP}" sudo "$@" 2>/dev/null +} + +n2() { + ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \ + "${SWARM_USER}@${NODE2_LAN_IP}" sudo "$@" 2>/dev/null +} + +dc() { + # Run ipa commands on domain-controller as $SSH_USER. + if $DRY_RUN; then + echo "[dry-run] ssh ${SSH_USER}@${DC_HOST} $*" + return 0 + fi + ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${DC_HOST}" "$@" +} + +wait_for_ssh() { + local ip="$1" label="$2" + if $DRY_RUN; then + logn "[dry-run] Skipping SSH wait for ${label} (${ip})" + return 0 + fi + local deadline=$(( $(date +%s) + 300 )) + log "Waiting for SSH on ${label} (${ip}) — up to 5 min..." + while [[ $(date +%s) -lt $deadline ]]; do + if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=3 \ + -o BatchMode=yes "${SWARM_USER}@${ip}" true 2>/dev/null; then + logn "${label} is up." + return 0 + fi + sleep 5 + done + err "Timed out waiting for SSH on ${label} (${ip})" +} + +# ── Destroy mode ────────────────────────────────────────────────────────────── + +if $DESTROY; then + log "Destroying Docker Swarm VMs (${VMID1}=${NODE1_HOST}, ${VMID2}=${NODE2_HOST}) on ${NODE}" + for vmid in "$VMID1" "$VMID2"; do + STATUS=$(pve "qm status ${vmid} 2>/dev/null" 2>/dev/null || true) + if echo "$STATUS" | grep -q "running"; then + log "Stopping VMID ${vmid}..." + pve "qm stop ${vmid} --skiplock 1" + sleep 5 + fi + if $DRY_RUN || pve "qm config ${vmid} >/dev/null 2>&1"; then + log "Deleting VMID ${vmid}..." + run pve "qm destroy ${vmid} --purge 1" + else + logn "VMID ${vmid} not found — already gone." + fi + done + log "Done — swarm VMs destroyed." + exit 0 +fi + +# ── Phase 1: Ensure swarm bridge ────────────────────────────────────────────── + +if ! $SKIP_ENSURE_BRIDGE; then + log "Phase 1: Ensuring swarm bridge ${SWARM_BRIDGE} on ${NODE}" + if pve_check "test -d /sys/class/net/${SWARM_BRIDGE}" &>/dev/null; then + logn "${SWARM_BRIDGE} already exists — skipping." + else + logn "Creating isolated internal bridge ${SWARM_BRIDGE} (no upstream port, ${SWARM_CIDR})" + BRIDGE_CONF="auto ${SWARM_BRIDGE} +iface ${SWARM_BRIDGE} inet manual + bridge-ports none + bridge-stp off + bridge-fd 0" + if $DRY_RUN; then + echo "[dry-run] Would write /etc/network/interfaces.d/${SWARM_BRIDGE}.conf and ifup it" + else + ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \ + "echo '${BRIDGE_CONF}' | sudo tee /etc/network/interfaces.d/${SWARM_BRIDGE}.conf > /dev/null && sudo ifup ${SWARM_BRIDGE}" + logn "${SWARM_BRIDGE} created and brought up." + fi + fi +fi + +# ── Phase 2: Sync host keys ─────────────────────────────────────────────────── + +if ! $SKIP_SYNC_KEYS; then + log "Phase 2: Syncing SSH host keys for both swarm targets" + for target in proxmox-ha-docker-1 proxmox-ha-docker-2; do + CLAN_DIR="${REPO_ROOT}/vars/per-machine/${target}/openssh" + if [[ -d "$CLAN_DIR" ]]; then + logn "Clan vars for ${target} already exist — skipping." + else + logn "Generating host keys for ${target}..." + run bash "${REPO_ROOT}/scripts/secrets/sync-host-keys.sh" "$target" + fi + done +fi + +# ── Phase 3: IPA host accounts ──────────────────────────────────────────────── + +if ! $SKIP_IPA_HOSTS; then + log "Phase 3: Creating IPA host accounts and keytabs" + IPA_SCRIPT="${REPO_ROOT}/scripts/ipa/create-nixos-ipa-host-account.sh" + for host in "${NODE1_HOST}" "${NODE2_HOST}"; do + KEYTAB="${REPO_ROOT}/secrets/${host}.keytab" + if [[ -f "$KEYTAB" ]]; then + logn "Keytab for ${host} already exists — skipping." + else + logn "Creating IPA host account and keytab for ${host}..." + run bash "$IPA_SCRIPT" "$host" + fi + done + + if ! $DRY_RUN; then + # Keytabs must be committed and pushed before VMs rebuild from Gitea. + CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)" + logn "Committing keytabs and pushing to Gitea (branch: ${CURRENT_BRANCH})..." + (cd "${REPO_ROOT}" && \ + git add secrets/ha-docker-1.keytab secrets/ha-docker-2.keytab .sops.yaml && \ + git commit -m "secrets(ha-docker): add IPA keytabs for ha-docker-1 and ha-docker-2" || true && \ + git push origin "${CURRENT_BRANCH}") + logn "Pushed." + fi +fi + +# ── Phase 3.5: Prepare Proxmox node for building ───────────────────────────── + +if ! $SKIP_CREATE_VMS && ! $DRY_RUN; then + CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)" + + local_ssh() { ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "$*"; } + if local_ssh "test -d /nix" &>/dev/null && ! local_ssh "test -w /nix" &>/dev/null; then + logn "/nix exists but not writable by ${SSH_USER} — fixing ownership with sudo..." + local_ssh "sudo chown -R ${SSH_USER} /nix" + logn "Done." + fi + unset -f local_ssh + + REMOTE_REPO="/home/${SSH_USER}/nixos" + if pve_check "test -d ${REMOTE_REPO}/.git" &>/dev/null; then + REMOTE_BRANCH=$(ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \ + "cd ${REMOTE_REPO} && git rev-parse --abbrev-ref HEAD 2>/dev/null") + if [[ "$REMOTE_BRANCH" != "$CURRENT_BRANCH" ]]; then + logn "Remote clone is on '${REMOTE_BRANCH}', switching to '${CURRENT_BRANCH}'..." + ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \ + "cd ${REMOTE_REPO} && git fetch origin && git checkout '${CURRENT_BRANCH}' && git pull --ff-only" + logn "Done." + fi + fi +fi + +# ── Phase 4: Create VMs ─────────────────────────────────────────────────────── + +if ! $SKIP_CREATE_VMS; then + log "Phase 4: Building and creating swarm VMs on ${NODE}" + CREATE="${REPO_ROOT}/scripts/proxmox/create-proxmox-resource.sh" + REBUILD_FLAG="" + $FORCE_REBUILD && REBUILD_FLAG="--force-rebuild" + + for spec in "${VMID1}:${NODE1_HOST}:proxmox-ha-docker-1" "${VMID2}:${NODE2_HOST}:proxmox-ha-docker-2"; do + IFS=: read -r vmid host_name flake_target <<< "$spec" + log "Creating ${flake_target} (VMID ${vmid}) on ${NODE}..." + # --force-rebuild is always passed: create-proxmox-resource.sh only calls + # sync_remote_host_keys (which bakes the clan-var SSH key into the disk + # image) when it actually builds. Reusing a cached image skips that step + # and leaves the VM unable to decrypt sops secrets on first boot. + run bash "$CREATE" \ + --type vm \ + --host "$host_name" \ + --vmid "$vmid" \ + --node "$NODE" \ + --storage "$STORAGE" \ + --memory "$MEMORY_MB" \ + --cores "$CORES" \ + --force-rebuild + done +fi + +# ── Phase 5: Add NICs and start VMs ────────────────────────────────────────── + +if ! $SKIP_ADD_HARDWARE; then + log "Phase 5: Attaching storage (${STORAGE_BRIDGE}) and swarm (${SWARM_BRIDGE}) NICs" + for vmid in "$VMID1" "$VMID2"; do + logn "VMID ${vmid}: stopping to add NICs..." + pve "qm stop ${vmid} --skiplock 1 2>/dev/null; sleep 3" || true + + logn "Adding net1 (${STORAGE_BRIDGE} — NFS storage)..." + pve "qm set ${vmid} --net1 virtio,bridge=${STORAGE_BRIDGE},firewall=0" + + logn "Adding net2 (${SWARM_BRIDGE} — Docker Swarm)..." + pve "qm set ${vmid} --net2 virtio,bridge=${SWARM_BRIDGE},firewall=0" + + logn "Starting VMID ${vmid}..." + pve "qm start ${vmid}" + done +fi + +# ── Phase 6: Wait for SSH ───────────────────────────────────────────────────── + +if ! $SKIP_BOOT_WAIT; then + log "Phase 6: Waiting for both nodes to come up on LAN IPs" + wait_for_ssh "$NODE1_LAN_IP" "$NODE1_HOST" + wait_for_ssh "$NODE2_LAN_IP" "$NODE2_HOST" + logn "Both nodes are SSHable." + sleep 10 # let systemd finish activation +fi + +# ── Phase 7: Refresh sops host-key registrations ───────────────────────────── +# +# Disko builds raw disk images: each new VM boots with a freshly-generated SSH +# host key, not the one pre-seeded in clan vars. Scan the running VMs; if +# their ed25519 keys differ from the clan var, update the clan var, rewrite +# the .sops.yaml anchor, and re-encrypt all affected sops files. + +if ! $SKIP_REFRESH_SOPS_KEYS; then + if $DRY_RUN; then + logn "[dry-run] Would scan VM host keys and refresh .sops.yaml / secrets if needed" + else + log "Phase 7: Refreshing sops host-key registrations (disko key drift fix)" + SOPS_UPDATED=false + + for spec in \ + "${NODE1_LAN_IP}:proxmox-ha-docker-1:${NODE1_HOST}" \ + "${NODE2_LAN_IP}:proxmox-ha-docker-2:${NODE2_HOST}"; do + IFS=: read -r node_ip flake_target host_name <<< "$spec" + CLAN_PUB="${REPO_ROOT}/vars/per-machine/${flake_target}/openssh/ssh_host_ed25519_key.pub/value" + + logn "Scanning ed25519 host key from ${host_name} (${node_ip})..." + RAW=$(ssh-keyscan -t ed25519 "${node_ip}" 2>/dev/null | grep -v "^#") || true + if [[ -z "$RAW" ]]; then + logn "WARNING: no ed25519 key returned for ${node_ip} — skipping" + continue + fi + SCANNED_TYPE=$(awk '{print $2}' <<< "$RAW") + SCANNED_KEY=$(awk '{print $3}' <<< "$RAW") + SCANNED_PUBKEY="${SCANNED_TYPE} ${SCANNED_KEY} ${host_name}" + + CURRENT=$(tr -d '\n' < "$CLAN_PUB" 2>/dev/null || true) + if [[ "$SCANNED_PUBKEY" == "$CURRENT" ]]; then + logn "${host_name}: clan var matches running key — no update needed" + continue + fi + + logn "${host_name}: key drift detected — updating clan var" + logn " old: ${CURRENT}" + logn " new: ${SCANNED_PUBKEY}" + echo "$SCANNED_PUBKEY" > "$CLAN_PUB" + SOPS_UPDATED=true + + ANCHOR="${flake_target}" + NEW_AGE=$(echo "$SCANNED_PUBKEY" | \ + nix run --quiet --no-warn-dirty nixpkgs#ssh-to-age 2>/dev/null) + [[ -z "$NEW_AGE" ]] && err "ssh-to-age produced no output for ${host_name}" + logn " new age key: ${NEW_AGE}" + sed -i "/&${ANCHOR} /s| age[a-z0-9]*$| ${NEW_AGE}|" "${REPO_ROOT}/.sops.yaml" + done + + if $SOPS_UPDATED; then + logn "Running sops updatekeys on affected secrets..." + SOPS="nix run --quiet --no-warn-dirty nixpkgs#sops --" + (cd "${REPO_ROOT}" && \ + $SOPS updatekeys -y secrets/common.yaml && \ + $SOPS updatekeys -y secrets/ha-docker-1.keytab && \ + $SOPS updatekeys -y secrets/ha-docker-2.keytab) + + logn "Committing refreshed host keys and re-encrypted secrets..." + (cd "${REPO_ROOT}" && \ + git add \ + vars/per-machine/proxmox-ha-docker-1/openssh/ssh_host_ed25519_key.pub/value \ + vars/per-machine/proxmox-ha-docker-2/openssh/ssh_host_ed25519_key.pub/value \ + .sops.yaml \ + secrets/common.yaml \ + secrets/ha-docker-1.keytab \ + secrets/ha-docker-2.keytab && \ + git commit -m "secrets(ha-docker): refresh sops host-key registrations for new VM instances" || true) + logn "Sops keys refreshed and committed." + fi + fi +fi + +# ── Phase 8: Initialise Docker Swarm ───────────────────────────────────────── + +if ! $SKIP_INIT_SWARM; then + log "Phase 8: Initialising Docker Swarm" + + if $DRY_RUN; then + logn "[dry-run] Would run: docker swarm init --advertise-addr ${NODE1_SWARM_IP} --data-path-addr ${NODE1_SWARM_IP} on ${NODE1_HOST}" + logn "[dry-run] Would join ${NODE2_HOST} as manager" + logn "[dry-run] Would label both nodes" + else + # Check if node1 is already a swarm manager. + if n1 "docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null | grep -q "active"; then + logn "${NODE1_HOST} is already in a swarm — skipping init." + else + logn "Initialising swarm on ${NODE1_HOST} (advertise: ${NODE1_SWARM_IP})..." + n1 "docker swarm init \ + --advertise-addr ${NODE1_SWARM_IP} \ + --data-path-addr ${NODE1_SWARM_IP}" + logn "Swarm initialised on ${NODE1_HOST}." + fi + + # Check if node2 is already joined. + if n2 "docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null | grep -q "active"; then + logn "${NODE2_HOST} is already in the swarm — skipping join." + else + logn "Fetching manager join token from ${NODE1_HOST}..." + JOIN_TOKEN=$(n1 "docker swarm join-token manager -q") + [[ -z "$JOIN_TOKEN" ]] && err "Failed to get swarm manager join token from ${NODE1_HOST}" + + logn "Joining ${NODE2_HOST} as manager (advertise: ${NODE2_SWARM_IP})..." + n2 "docker swarm join \ + --token ${JOIN_TOKEN} \ + --advertise-addr ${NODE2_SWARM_IP} \ + --data-path-addr ${NODE2_SWARM_IP} \ + ${NODE1_SWARM_IP}:2377" + logn "${NODE2_HOST} joined as manager." + fi + + # Label nodes for service placement constraints. + logn "Labelling swarm nodes..." + n1 "docker node update --label-add node=${NODE1_HOST} ${NODE1_HOST}" || true + n1 "docker node update --label-add node=${NODE2_HOST} ${NODE2_HOST}" || true + logn "Labels applied." + fi +fi + +# ── Phase 9: DNS registration ───────────────────────────────────────────────── + +if ! $SKIP_DNS; then + log "Phase 9: Registering DNS records in FreeIPA" + + if $DRY_RUN; then + logn "[dry-run] Would create/verify ${SWARM_ZONE} zone and add A records" + else + # Check for and create the swarm.home zone if absent. + if ! dc "ipa dnszone-show ${SWARM_ZONE}" >/dev/null 2>&1; then + logn "Creating ${SWARM_ZONE} DNS zone..." + dc "ipa dnszone-add ${SWARM_ZONE} \ + --name-server=${DC_HOST}. \ + --admin-email=hostmaster@${SWARM_ZONE}" + # Reverse zone for 192.168.30.x + dc "ipa dnszone-add 30.168.192.in-addr.arpa \ + --name-server=${DC_HOST}. \ + --admin-email=hostmaster@${SWARM_ZONE}" 2>/dev/null || \ + logn " (reverse zone 30.168.192.in-addr.arpa already exists or skipped)" + else + logn "${SWARM_ZONE} zone already exists." + fi + + # storage.home A records (zone already exists from HA cluster setup). + for spec in "${NODE1_HOST}:${NODE1_STORAGE_IP}" "${NODE2_HOST}:${NODE2_STORAGE_IP}"; do + IFS=: read -r hostname ip <<< "$spec" + logn "Adding ${hostname}.${STORAGE_ZONE} → ${ip}" + dc "ipa dnsrecord-add ${STORAGE_ZONE} ${hostname} --a-rec=${ip} --a-create-reverse" 2>/dev/null || \ + logn " (record already exists or reverse zone missing — continuing)" + done + + # swarm.home A records. + for spec in "${NODE1_HOST}:${NODE1_SWARM_IP}" "${NODE2_HOST}:${NODE2_SWARM_IP}"; do + IFS=: read -r hostname ip <<< "$spec" + logn "Adding ${hostname}.${SWARM_ZONE} → ${ip}" + dc "ipa dnsrecord-add ${SWARM_ZONE} ${hostname} --a-rec=${ip} --a-create-reverse" 2>/dev/null || \ + logn " (record already exists — continuing)" + done + fi +fi + +# ── Phase 10: Verify ────────────────────────────────────────────────────────── + +if ! $SKIP_VERIFY; then + log "Phase 10: Verifying swarm health" + + if $DRY_RUN; then + logn "[dry-run] Would verify swarm node list and NFS mounts" + else + logn "Swarm node list:" + n1 "docker node ls" || err "docker node ls failed on ${NODE1_HOST}" + + logn "Checking swarm state on both nodes..." + for spec in "${NODE1_LAN_IP}:${NODE1_HOST}" "${NODE2_LAN_IP}:${NODE2_HOST}"; do + IFS=: read -r ip hostname <<< "$spec" + STATE=$(ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \ + "${SWARM_USER}@${ip}" "sudo docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null) + if [[ "$STATE" != "active" ]]; then + err "${hostname} swarm state is '${STATE}', expected 'active'" + fi + logn " ${hostname}: swarm=${STATE} ✓" + done + + logn "Checking NFS mounts on both nodes..." + for spec in "${NODE1_LAN_IP}:${NODE1_HOST}" "${NODE2_LAN_IP}:${NODE2_HOST}"; do + IFS=: read -r ip hostname <<< "$spec" + NFS_OK=$(ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \ + "${SWARM_USER}@${ip}" "df -h /mnt/docker/config 2>/dev/null | grep -c nfs || echo 0" 2>/dev/null) + if [[ "$NFS_OK" -ge 1 ]]; then + logn " ${hostname}: /mnt/docker/config NFS mount ✓" + else + logn " WARNING: ${hostname}: /mnt/docker/config does not appear to be NFS-mounted" + logn " (automount may still be pending — try: ssh nixos@${ip} 'ls /mnt/docker/config')" + fi + done + + logn "Checking overlay network..." + NETWORKS=$(n1 "docker network ls --filter driver=overlay --format '{{.Name}}'") + if echo "$NETWORKS" | grep -q "ingress"; then + logn " ingress overlay network present ✓" + else + logn " WARNING: ingress overlay network not found — swarm may not be fully initialised" + fi + fi +fi + +log "Deploy complete. Both nodes are ready for 'docker stack deploy'." +log "Connect to either manager:" +log " ssh nixos@${NODE1_LAN_IP} (${NODE1_HOST})" +log " ssh nixos@${NODE2_LAN_IP} (${NODE2_HOST})" diff --git a/variables.nix b/variables.nix index d2fbbd0..f169df8 100644 --- a/variables.nix +++ b/variables.nix @@ -188,6 +188,46 @@ rec { # the data disk; drive-scsi0 is the OS disk. haServerDrbdDisk = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1"; + # ── Docker Swarm cluster ────────────────────────────────────────────────── + # + # Three network segments, all internal to pve1: + # LAN vmbr0 192.168.2.0/24 — management; SSH + external service traffic + # Storage-client vmbr2 192.168.20.0/24 — NFS from HA cluster VIP (shared with HA nodes) + # Swarm cluster vmbr3 192.168.30.0/24 — Docker Swarm gossip + VXLAN overlay + # + # Host octet consistent across subnets: node1 = .230, node2 = .231. + # IPs from the .230–.239 expansion buffer documented in docs/ip-addressing.md. + # + # Docker Swarm uses --advertise-addr and --data-path-addr on the swarm NIC + # (ens20/vmbr3) so all inter-node cluster traffic stays on the isolated + # internal bridge and never crosses the LAN. + # + # When expanding to a second Proxmox node, vmbr3 (VLAN 30) and vmbr1 + # (VLAN 10) share the same inter-node trunk NIC via VLAN tagging — same + # physical wire, different VLAN IDs. + + haDocker1Host = "ha-docker-1"; + haDocker2Host = "ha-docker-2"; + + haDocker1Ip = "192.168.2.230"; # LAN management NIC (ens18, vmbr0) + haDocker2Ip = "192.168.2.231"; + + haDocker1StorageIp = "192.168.20.230"; # storage-client NIC (ens19, vmbr2) + haDocker2StorageIp = "192.168.20.231"; + + haDocker1SwarmIp = "192.168.30.230"; # swarm cluster NIC (ens20, vmbr3) + haDocker2SwarmIp = "192.168.30.231"; + + haDockerSwarmCidr = "192.168.30.0/24"; + haDockerSwarmPrefixLength = 24; + + # NIC names for ha-docker VMs. ens19/ens20 occupy the same guest bus + # positions as vmStorageInterface/vmStorageClientInterface on ha-server VMs + # but are attached to different bridges — storage (vmbr2) and swarm (vmbr3) + # respectively. Kept as named variables to avoid bare literals in modules. + haDockerStorageInterface = "ens19"; # vmbr2 — NFS client + haDockerSwarmInterface = "ens20"; # vmbr3 — Docker Swarm gossip + VXLAN + # ── Storage / NFS ───────────────────────────────────────────────────────── # NFS share definitions — used by ha-server.nix (exports), docker/mount-data.nix, @@ -263,6 +303,13 @@ rec { dockerHttps = 443; dockerExtra = 8080; + # Docker Swarm inter-node ports (modules/build-types/ha-docker.nix). + # Firewalled to haDockerSwarmCidr only — vmbr3 is an isolated bridge + # with no physical uplink, so these ports are unreachable from LAN. + dockerSwarmMgmt = 2377; # TCP — Raft consensus + cluster management + dockerSwarmDisc = 7946; # TCP+UDP — Serf gossip (container network discovery) + dockerSwarmVxlan = 4789; # UDP — VXLAN overlay data path + # Beszel monitoring hub on docker.sweet.home, reached by every agent # (modules/beszel/enable-agent.nix, hosts/nixos/home.nix) beszelHub = 8090;