nixos/docs/internal/docker-swarm-cutover.md

Docker Swarm Cutover Plan

Migration guide for moving containerised services from the existing single-host Docker LXC container (CT 105, docker.sweet.home, 192.168.2.225) to the new Docker Swarm cluster (ha-docker-1 / ha-docker-2, 192.168.2.230–231).

CT 105 stays running throughout. Services migrate one stack at a time. Roll back any stack by restarting it on CT 105 if anything goes wrong.

---

Prerequisites

• Swarm cluster deployed and healthy (scripts/docker-swarm/deploy.sh).
• Both nodes show Ready / Active / Manager in docker node ls.
• NFS mounts healthy on both swarm nodes (/mnt/docker/config, /mnt/docker/databases, /mnt/docker/volumes).
• Access to FreeIPA DNS admin to update A records during cutover.

---

1. Traefik — switch to Docker log rotation

Current state (CT 105): Traefik writes access logs to the NFS volume at /mnt/docker/volumes/traefik-data/logs/. modules/traefik/rotate-logs.nix rotates those files via logrotate.

Swarm approach: Remove file-based access logging from Traefik's static config and rely on Docker's json-file log driver with built-in rotation. Traefik container logs (including access events) then live under /var/lib/docker/containers/<id>/ on the node running Traefik.

Steps

1a. In the Traefik stack definition, add logging config to the service:

services:
  traefik:
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "20"

1b. In traefik.yml (Traefik's static config), remove the accessLog file path if present. To keep structured access logs, use Traefik's accessLog.format: json with no filePath — logs then go to stdout and are captured by the json-file driver above.

1c. Deploy Traefik to the swarm:

# On either swarm manager:
docker stack deploy -c /mnt/docker/config/traefik/docker-compose.yml traefik

Traefik should be deployed as a global mode service so it runs on all swarm nodes and handles ingress on whichever node a request arrives at:

services:
  traefik:
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == manager

1d. After confirming Traefik works on the swarm, remove traefik/rotate-logs.nix from the docker build type in modules/build-types/docker.nix and rebuild CT 105.

DNS: Update docker.sweet.home and any service FQDNs that point at 192.168.2.225 to a swarm VIP or round-robin A records once Traefik is running on the swarm. See section 8 (DNS cutover).

---

2. Nextcloud — migrate cron job to sidecar container

Current state (CT 105): modules/docker/nextcloud-cron-job.nix runs a systemd timer every 5 minutes that calls:

docker exec nextcloud-webapp php ./cron.php

Swarm problem: docker exec only works against the local daemon. If Nextcloud is scheduled on the other swarm node, the exec fails silently and cron never runs.

Swarm approach: Add a nextcloud-cron sidecar container to the Nextcloud stack definition, pinned to the same node as the main Nextcloud container via placement constraints.

Steps

2a. Choose which swarm node will host Nextcloud (e.g. ha-docker-1). Label that node:

# On either swarm manager:
docker node update --label-add nextcloud=true ha-docker-1

2b. In the Nextcloud stack compose file, add the sidecar and pin both services to the labelled node:

services:
  nextcloud-webapp:
    image: nextcloud:production   # pin same version as CT 105
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.nextcloud == true
    # ... existing volumes, env, networks ...

  nextcloud-cron:
    image: nextcloud:production   # same image, different entrypoint
    entrypoint: /cron.sh
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.nextcloud == true   # must co-locate with webapp
    volumes:
      # Same data volume as nextcloud-webapp so cron sees the same files.
      - nextcloud-data:/var/www/html
    # No ports exposed — cron only runs PHP inside the container.

/cron.sh is Nextcloud's built-in cron entrypoint. It runs php -f /var/www/html/cron.php in a loop, sleeping for 5 minutes between runs — identical to the current systemd timer.

2c. Migrate Nextcloud's data volume to the swarm:

/mnt/docker/volumes/nextcloud-data/  →  already on NFS, no migration needed
/mnt/docker/databases/nextcloud/     →  already on NFS, no migration needed

The NFS paths are identical on the swarm nodes (mount-data.nix mounts the same shares from the same VIP). Stop Nextcloud on CT 105, deploy on the swarm, confirm it starts cleanly.

2d. Remove nextcloud-cron-job.nix from modules/build-types/docker.nix and rebuild CT 105 after confirming Nextcloud works on the swarm.

---

3. docker-health-to-gotify — update for swarm awareness

Current state (CT 105): The script at /home/nixos/docker/monitoring/gotify/docker-health-to-gotify.sh runs every minute, calls docker ps --filter health=unhealthy, and notifies Gotify.

Swarm behaviour: The same script runs on both swarm nodes independently, each monitoring its own local Docker daemon. This gives per-node coverage across the swarm.

Changes needed in the script (edit the copy on the NFS volume — it takes effect on both nodes simultaneously on the next timer fire):

3a. Strip the Swarm task suffix from service names

In swarm mode, docker ps --format '{{.Names}}' returns names like nextcloud-webapp.1.abc123xyz. The notification should show nextcloud-webapp, not the full task name.

# Before:
CONTAINER_NAME=$(docker ps --format '{{.Names}}' ...)

# After:
CONTAINER_NAME=$(docker ps --format '{{.Names}}' ... | cut -d. -f1)

3b. Include the reporting node in the Gotify message

Add $(hostname) to the notification payload so you know which swarm node detected the problem:

MESSAGE="[$(hostname)] ${CONTAINER_NAME} is unhealthy"

3c. Extend to catch swarm service replica failures

docker ps only shows what's running locally. If a service has zero healthy replicas (task crash-looping) it may not show up on either node's docker ps at the same moment. Add a swarm-level check:

# Run only on managers (both ha-docker nodes are managers):
if docker info --format '{{.Swarm.ControlAvailable}}' 2>/dev/null | grep -q true; then
  # Find services where running replicas < desired replicas
  docker service ls --format '{{.Name}}\t{{.Replicas}}' | \
    awk -F'\t' '$2 !~ /^[0-9]+\/[0-9]+$/ || split($2,a,"/") && a[1] < a[2] { print $1, $2 }' | \
    while read -r svc_name replicas; do
      # Send Gotify notification for degraded service
      curl -s -X POST "${GOTIFY_URL}/message" \
        -H "X-Gotify-Key: ${GOTIFY_TOKEN}" \
        -d "title=Swarm service degraded" \
        -d "message=[$(hostname)] ${svc_name}: ${replicas} replicas"
    done
fi

This catches the case where a service's desired replicas are not running (e.g. OOM kill, image pull failure) — a failure mode that doesn't produce a Docker health event on any node.

---

4. Passbolt migration

Passbolt has strict data integrity requirements. Migrate with care:

1. Backup first — docker exec passbolt-webapp php /usr/share/php/passbolt/bin/cake passbolt export_keys and a database dump.
2. Database is on NFS (/mnt/docker/databases/passbolt/) — no data copy needed.
3. Pin Passbolt to a specific node: docker node update --label-add passbolt=true ha-docker-1
4. Add placement constraint node.labels.passbolt == true to the Passbolt stack.
5. Stop on CT 105, deploy on swarm, verify login works.
6. Test email delivery and 2FA.

---

5. Gitea migration

Gitea's data directory is on NFS (/mnt/docker/volumes/gitea-data/).

1. Stop Gitea on CT 105: docker stop gitea
2. Deploy to swarm with placement constraint (pin to ha-docker-1 initially).
3. Verify web UI and SSH clone/push work.
4. Update DNS: gitea.lan.ddnsgeek.com → swarm Traefik endpoint.
5. Update the flake remote URL in variables.nix (giteaDomain) if the address changes.

---

6. Other services

Deploy remaining services (Grafana, InfluxDB, NodeRed, Prometheus, etc.) as swarm stacks. Most have no special migration concern — they use NFS volumes already on the shared storage.

Services with stateful databases (PostgreSQL, MariaDB) should follow the pattern: stop on CT 105, confirm NFS database directory is intact, deploy on swarm, verify.

---

7. Monitoring — Beszel

The Beszel hub runs on CT 105 (docker.sweet.home:8090). Both swarm nodes run beszel-agent (from modules/beszel/enable-agent.nix), pointing at the existing hub URL.

No migration needed for Beszel itself during the container migration. Once all services are on the swarm, you may wish to move the Beszel hub too (as a swarm service with a placement constraint) but this is optional.

---

8. DNS cutover

When a service is confirmed working on the swarm, update the FreeIPA DNS A record from the CT 105 IP (192.168.2.225) to a swarm node IP or, when a shared Traefik frontend is in place, to a round-robin record across both nodes.

Recommended approach — Traefik as the single entry point:

service.lan.ddnsgeek.com  →  Traefik on swarm (global mode)
docker.sweet.home         →  keep as 192.168.2.225 (CT 105) until fully decommissioned

For LAN-only services using *.sweet.home names, update FreeIPA directly:

# On domain-controller (or via SSH):
ipa dnsrecord-mod sweet.home nextcloud --a-rec=192.168.2.230
# Add 192.168.2.231 as a second A record for round-robin (optional):
ipa dnsrecord-add sweet.home nextcloud --a-rec=192.168.2.231

Services behind Traefik don't need their own DNS updates — only Traefik's own entry point IPs need to change.

---

9. NixOS cleanup — CT 105

Once all services are migrated:

Remove from modules/build-types/docker.nix:

• ../docker/nextcloud-cron-job.nix — replaced by sidecar container
• ../traefik/rotate-logs.nix — replaced by Docker log driver

Keep in modules/build-types/docker.nix until CT 105 is decommissioned:

• ../docker/docker-health-to-gotify.nix — still monitors CT 105's own daemon
• Everything else

When decommissioning CT 105:

1. Confirm all NFS volumes are in use only by swarm services (not CT 105).
2. Stop CT 105: pct stop 105 on pve1.
3. Archive/remove the lxc-docker and proxmox-docker targets from flake.nix.
4. Remove hosts/docker/, modules/build-types/docker.nix, and modules/docker/.
5. Update variables.nix to remove dockerIp, dockerStorageIp, dockerHost (or reassign dockerHost to point at a swarm node for Beszel hub resolution).

---

Rollback

Any stack can be rolled back to CT 105 independently:

# On CT 105:
docker start <service-name>
# Update DNS A record back to 192.168.2.225
ipa dnsrecord-mod sweet.home <service> --a-rec=192.168.2.225

CT 105 remains running throughout the cutover. Only decommission it after every service is confirmed stable on the swarm and you have run one full backup cycle from the new hosts.