nixos/docs/internal/docker-swarm-cutover.md

# Docker Swarm Cutover Plan

Migration guide for moving containerised services from the existing single-host
Docker LXC container (CT 105, `docker.sweet.home`, 192.168.2.225) to the new
Docker Swarm cluster (`ha-docker-1` / `ha-docker-2`, 192.168.2.230–231).

CT 105 stays running throughout.  Services migrate one stack at a time.
Roll back any stack by restarting it on CT 105 if anything goes wrong.

---

## Prerequisites

- Swarm cluster deployed and healthy (`scripts/docker-swarm/deploy.sh`).
- Both nodes show `Ready / Active / Manager` in `docker node ls`.
- NFS mounts healthy on both swarm nodes (`/mnt/docker/config`, `/mnt/docker/databases`, `/mnt/docker/volumes`).
- Access to FreeIPA DNS admin to update A records during cutover.

---

## 1. Traefik — switch to Docker log rotation

**Current state (CT 105):** Traefik writes access logs to the NFS volume at
`/mnt/docker/volumes/traefik-data/logs/`.  `modules/traefik/rotate-logs.nix`
rotates those files via `logrotate`.

**Swarm approach:** Remove file-based access logging from Traefik's static
config and rely on Docker's json-file log driver with built-in rotation.
Traefik container logs (including access events) then live under
`/var/lib/docker/containers/<id>/` on the node running Traefik.

### Steps

**1a.** In the Traefik stack definition, add logging config to the service:

```yaml
services:
  traefik:
    logging:
      driver: "json-file"
      options:
        max-size: "100m"
        max-file: "20"
```

**1b.** In `traefik.yml` (Traefik's static config), remove the `accessLog`
file path if present.  To keep structured access logs, use Traefik's
`accessLog.format: json` with no `filePath` — logs then go to stdout and are
captured by the json-file driver above.

**1c.** Deploy Traefik to the swarm:

```bash
# On either swarm manager:
docker stack deploy -c /mnt/docker/config/traefik/docker-compose.yml traefik
```

Traefik should be deployed as a **global mode** service so it runs on all
swarm nodes and handles ingress on whichever node a request arrives at:

```yaml
services:
  traefik:
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == manager
```

**1d.** After confirming Traefik works on the swarm, remove
`traefik/rotate-logs.nix` from the `docker` build type in
`modules/build-types/docker.nix` and rebuild CT 105.

**DNS:** Update `docker.sweet.home` and any service FQDNs that point at
192.168.2.225 to a swarm VIP or round-robin A records once Traefik is running
on the swarm.  See section 8 (DNS cutover).

---

## 2. Nextcloud — migrate cron job to sidecar container

**Current state (CT 105):** `modules/docker/nextcloud-cron-job.nix` runs a
systemd timer every 5 minutes that calls:
```bash
docker exec nextcloud-webapp php ./cron.php
```

**Swarm problem:** `docker exec` only works against the local daemon.  If
Nextcloud is scheduled on the other swarm node, the exec fails silently and
cron never runs.

**Swarm approach:** Add a `nextcloud-cron` sidecar container to the Nextcloud
stack definition, pinned to the same node as the main Nextcloud container via
placement constraints.

### Steps

**2a.** Choose which swarm node will host Nextcloud (e.g. `ha-docker-1`).
Label that node:

```bash
# On either swarm manager:
docker node update --label-add nextcloud=true ha-docker-1
```

**2b.** In the Nextcloud stack compose file, add the sidecar and pin both
services to the labelled node:

```yaml
services:
  nextcloud-webapp:
    image: nextcloud:production   # pin same version as CT 105
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.nextcloud == true
    # ... existing volumes, env, networks ...

  nextcloud-cron:
    image: nextcloud:production   # same image, different entrypoint
    entrypoint: /cron.sh
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.labels.nextcloud == true   # must co-locate with webapp
    volumes:
      # Same data volume as nextcloud-webapp so cron sees the same files.
      - nextcloud-data:/var/www/html
    # No ports exposed — cron only runs PHP inside the container.
```

`/cron.sh` is Nextcloud's built-in cron entrypoint.  It runs
`php -f /var/www/html/cron.php` in a loop, sleeping for 5 minutes between
runs — identical to the current systemd timer.

**2c.** Migrate Nextcloud's data volume to the swarm:

```
/mnt/docker/volumes/nextcloud-data/  →  already on NFS, no migration needed
/mnt/docker/databases/nextcloud/     →  already on NFS, no migration needed
```

The NFS paths are identical on the swarm nodes (`mount-data.nix` mounts the
same shares from the same VIP).  Stop Nextcloud on CT 105, deploy on the
swarm, confirm it starts cleanly.

**2d.** Remove `nextcloud-cron-job.nix` from `modules/build-types/docker.nix`
and rebuild CT 105 after confirming Nextcloud works on the swarm.

---

## 3. docker-health-to-gotify — update for swarm awareness

**Current state (CT 105):** The script at
`/home/nixos/docker/monitoring/gotify/docker-health-to-gotify.sh` runs every
minute, calls `docker ps --filter health=unhealthy`, and notifies Gotify.

**Swarm behaviour:** The same script runs on both swarm nodes independently,
each monitoring its own local Docker daemon.  This gives per-node coverage
across the swarm.

**Changes needed in the script** (edit the copy on the NFS volume — it takes
effect on both nodes simultaneously on the next timer fire):

### 3a. Strip the Swarm task suffix from service names

In swarm mode, `docker ps --format '{{.Names}}'` returns names like
`nextcloud-webapp.1.abc123xyz`.  The notification should show `nextcloud-webapp`,
not the full task name.

```bash
# Before:
CONTAINER_NAME=$(docker ps --format '{{.Names}}' ...)

# After:
CONTAINER_NAME=$(docker ps --format '{{.Names}}' ... | cut -d. -f1)
```

### 3b. Include the reporting node in the Gotify message

Add `$(hostname)` to the notification payload so you know which swarm node
detected the problem:

```bash
MESSAGE="[$(hostname)] ${CONTAINER_NAME} is unhealthy"
```

### 3c. Extend to catch swarm service replica failures

`docker ps` only shows what's running locally.  If a service has zero healthy
replicas (task crash-looping) it may not show up on either node's `docker ps`
at the same moment.  Add a swarm-level check:

```bash
# Run only on managers (both ha-docker nodes are managers):
if docker info --format '{{.Swarm.ControlAvailable}}' 2>/dev/null | grep -q true; then
  # Find services where running replicas < desired replicas
  docker service ls --format '{{.Name}}\t{{.Replicas}}' | \
    awk -F'\t' '$2 !~ /^[0-9]+\/[0-9]+$/ || split($2,a,"/") && a[1] < a[2] { print $1, $2 }' | \
    while read -r svc_name replicas; do
      # Send Gotify notification for degraded service
      curl -s -X POST "${GOTIFY_URL}/message" \
        -H "X-Gotify-Key: ${GOTIFY_TOKEN}" \
        -d "title=Swarm service degraded" \
        -d "message=[$(hostname)] ${svc_name}: ${replicas} replicas"
    done
fi
```

This catches the case where a service's desired replicas are not running
(e.g. OOM kill, image pull failure) — a failure mode that doesn't produce a
Docker health event on any node.

---

## 4. Passbolt migration

Passbolt has strict data integrity requirements.  Migrate with care:

1. **Backup first** — `docker exec passbolt-webapp php /usr/share/php/passbolt/bin/cake passbolt export_keys` and a database dump.
2. Database is on NFS (`/mnt/docker/databases/passbolt/`) — no data copy needed.
3. Pin Passbolt to a specific node: `docker node update --label-add passbolt=true ha-docker-1`
4. Add placement constraint `node.labels.passbolt == true` to the Passbolt stack.
5. Stop on CT 105, deploy on swarm, verify login works.
6. Test email delivery and 2FA.

---

## 5. Gitea migration

Gitea's data directory is on NFS (`/mnt/docker/volumes/gitea-data/`).

1. Stop Gitea on CT 105: `docker stop gitea`
2. Deploy to swarm with placement constraint (pin to `ha-docker-1` initially).
3. Verify web UI and SSH clone/push work.
4. Update DNS: `gitea.lan.ddnsgeek.com` → swarm Traefik endpoint.
5. Update the flake remote URL in `variables.nix` (`giteaDomain`) if the address changes.

---

## 6. Other services

Deploy remaining services (Grafana, InfluxDB, NodeRed, Prometheus, etc.)
as swarm stacks.  Most have no special migration concern — they use NFS
volumes already on the shared storage.

Services with stateful databases (PostgreSQL, MariaDB) should follow the
pattern: stop on CT 105, confirm NFS database directory is intact, deploy on
swarm, verify.

---

## 7. Monitoring — Beszel

The Beszel hub runs on CT 105 (`docker.sweet.home:8090`).  Both swarm nodes
run `beszel-agent` (from `modules/beszel/enable-agent.nix`), pointing at the
existing hub URL.

No migration needed for Beszel itself during the container migration.  Once
all services are on the swarm, you may wish to move the Beszel hub too (as a
swarm service with a placement constraint) but this is optional.

---

## 8. DNS cutover

When a service is confirmed working on the swarm, update the FreeIPA DNS
A record from the CT 105 IP (192.168.2.225) to a swarm node IP or, when a
shared Traefik frontend is in place, to a round-robin record across both nodes.

**Recommended approach — Traefik as the single entry point:**

```
service.lan.ddnsgeek.com  →  Traefik on swarm (global mode)
docker.sweet.home         →  keep as 192.168.2.225 (CT 105) until fully decommissioned
```

For LAN-only services using `*.sweet.home` names, update FreeIPA directly:

```bash
# On domain-controller (or via SSH):
ipa dnsrecord-mod sweet.home nextcloud --a-rec=192.168.2.230
# Add 192.168.2.231 as a second A record for round-robin (optional):
ipa dnsrecord-add sweet.home nextcloud --a-rec=192.168.2.231
```

Services behind Traefik don't need their own DNS updates — only Traefik's
own entry point IPs need to change.

---

## 9. NixOS cleanup — CT 105

Once all services are migrated:

**Remove from `modules/build-types/docker.nix`:**
- `../docker/nextcloud-cron-job.nix` — replaced by sidecar container
- `../traefik/rotate-logs.nix` — replaced by Docker log driver

**Keep in `modules/build-types/docker.nix` until CT 105 is decommissioned:**
- `../docker/docker-health-to-gotify.nix` — still monitors CT 105's own daemon
- Everything else

**When decommissioning CT 105:**
1. Confirm all NFS volumes are in use only by swarm services (not CT 105).
2. Stop CT 105: `pct stop 105` on pve1.
3. Archive/remove the `lxc-docker` and `proxmox-docker` targets from `flake.nix`.
4. Remove `hosts/docker/`, `modules/build-types/docker.nix`, and `modules/docker/`.
5. Update `variables.nix` to remove `dockerIp`, `dockerStorageIp`, `dockerHost`
   (or reassign `dockerHost` to point at a swarm node for Beszel hub resolution).

---

## Rollback

Any stack can be rolled back to CT 105 independently:

```bash
# On CT 105:
docker start <service-name>
# Update DNS A record back to 192.168.2.225
ipa dnsrecord-mod sweet.home <service> --a-rec=192.168.2.225
```

CT 105 remains running throughout the cutover.  Only decommission it after
every service is confirmed stable on the swarm and you have run one full
backup cycle from the new hosts.