Archived
Add Docker Swarm HA cluster: ha-docker-1 and ha-docker-2
Two new NixOS Proxmox VMs (VMIDs 202/203) forming a dual-manager Docker
Swarm on dedicated vmbr3 (192.168.30.0/24, VLAN 30) for gossip and VXLAN,
with NFS via the storage-client network (vmbr2) from the existing HA cluster.
- nixos/variables.nix: add ha-docker IP/interface/port vars and swarm CIDR
- nixos/modules/build-types/ha-docker.nix: new build type — Docker 29,
NFS mounts, beszel-agent, health monitoring, swarm firewall rules with
checkReversePath = "loose" for VXLAN routing mesh
- nixos/hosts/ha-docker-{1,2}/host.nix: per-host identity — three NICs
(LAN, storage, swarm), IPA dyndns pinned to LAN interface
- nixos/flake.nix: add proxmox-ha-docker-{1,2} targets; build-validated
with nix build --dry-run (169 derivations, no errors)
- nixos/docs/ip-addressing.md: document VLAN 30 / swarm.home zone,
ha-docker IP allocations across all three subnets
- nixos/scripts/docker-swarm/deploy.sh: 10-phase lifecycle script
(bridge, keys, IPA, VMs, swarm init, DNS, verify); modelled on
scripts/ha/deploy.sh with --destroy mode
- nixos/docs/internal/docker-swarm-cutover.md: service-by-service
migration guide covering Traefik log rotation, Nextcloud cron sidecar,
docker-health-to-gotify swarm awareness updates, Passbolt/Gitea steps,
DNS cutover, and CT 105 decommission checklist
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
This commit is contained in:
Claude Sonnet 4.6
@@ -0,0 +1,330 @@
|
|||||||
|
# Docker Swarm Cutover Plan
|
||||||
|
|
||||||
|
Migration guide for moving containerised services from the existing single-host
|
||||||
|
Docker LXC container (CT 105, `docker.sweet.home`, 192.168.2.225) to the new
|
||||||
|
Docker Swarm cluster (`ha-docker-1` / `ha-docker-2`, 192.168.2.230–231).
|
||||||
|
|
||||||
|
CT 105 stays running throughout. Services migrate one stack at a time.
|
||||||
|
Roll back any stack by restarting it on CT 105 if anything goes wrong.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- Swarm cluster deployed and healthy (`scripts/docker-swarm/deploy.sh`).
|
||||||
|
- Both nodes show `Ready / Active / Manager` in `docker node ls`.
|
||||||
|
- NFS mounts healthy on both swarm nodes (`/mnt/docker/config`, `/mnt/docker/databases`, `/mnt/docker/volumes`).
|
||||||
|
- Access to FreeIPA DNS admin to update A records during cutover.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Traefik — switch to Docker log rotation
|
||||||
|
|
||||||
|
**Current state (CT 105):** Traefik writes access logs to the NFS volume at
|
||||||
|
`/mnt/docker/volumes/traefik-data/logs/`. `modules/traefik/rotate-logs.nix`
|
||||||
|
rotates those files via `logrotate`.
|
||||||
|
|
||||||
|
**Swarm approach:** Remove file-based access logging from Traefik's static
|
||||||
|
config and rely on Docker's json-file log driver with built-in rotation.
|
||||||
|
Traefik container logs (including access events) then live under
|
||||||
|
`/var/lib/docker/containers/<id>/` on the node running Traefik.
|
||||||
|
|
||||||
|
### Steps
|
||||||
|
|
||||||
|
**1a.** In the Traefik stack definition, add logging config to the service:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
traefik:
|
||||||
|
logging:
|
||||||
|
driver: "json-file"
|
||||||
|
options:
|
||||||
|
max-size: "100m"
|
||||||
|
max-file: "20"
|
||||||
|
```
|
||||||
|
|
||||||
|
**1b.** In `traefik.yml` (Traefik's static config), remove the `accessLog`
|
||||||
|
file path if present. To keep structured access logs, use Traefik's
|
||||||
|
`accessLog.format: json` with no `filePath` — logs then go to stdout and are
|
||||||
|
captured by the json-file driver above.
|
||||||
|
|
||||||
|
**1c.** Deploy Traefik to the swarm:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On either swarm manager:
|
||||||
|
docker stack deploy -c /mnt/docker/config/traefik/docker-compose.yml traefik
|
||||||
|
```
|
||||||
|
|
||||||
|
Traefik should be deployed as a **global mode** service so it runs on all
|
||||||
|
swarm nodes and handles ingress on whichever node a request arrives at:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
traefik:
|
||||||
|
deploy:
|
||||||
|
mode: global
|
||||||
|
placement:
|
||||||
|
constraints:
|
||||||
|
- node.role == manager
|
||||||
|
```
|
||||||
|
|
||||||
|
**1d.** After confirming Traefik works on the swarm, remove
|
||||||
|
`traefik/rotate-logs.nix` from the `docker` build type in
|
||||||
|
`modules/build-types/docker.nix` and rebuild CT 105.
|
||||||
|
|
||||||
|
**DNS:** Update `docker.sweet.home` and any service FQDNs that point at
|
||||||
|
192.168.2.225 to a swarm VIP or round-robin A records once Traefik is running
|
||||||
|
on the swarm. See section 8 (DNS cutover).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Nextcloud — migrate cron job to sidecar container
|
||||||
|
|
||||||
|
**Current state (CT 105):** `modules/docker/nextcloud-cron-job.nix` runs a
|
||||||
|
systemd timer every 5 minutes that calls:
|
||||||
|
```bash
|
||||||
|
docker exec nextcloud-webapp php ./cron.php
|
||||||
|
```
|
||||||
|
|
||||||
|
**Swarm problem:** `docker exec` only works against the local daemon. If
|
||||||
|
Nextcloud is scheduled on the other swarm node, the exec fails silently and
|
||||||
|
cron never runs.
|
||||||
|
|
||||||
|
**Swarm approach:** Add a `nextcloud-cron` sidecar container to the Nextcloud
|
||||||
|
stack definition, pinned to the same node as the main Nextcloud container via
|
||||||
|
placement constraints.
|
||||||
|
|
||||||
|
### Steps
|
||||||
|
|
||||||
|
**2a.** Choose which swarm node will host Nextcloud (e.g. `ha-docker-1`).
|
||||||
|
Label that node:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On either swarm manager:
|
||||||
|
docker node update --label-add nextcloud=true ha-docker-1
|
||||||
|
```
|
||||||
|
|
||||||
|
**2b.** In the Nextcloud stack compose file, add the sidecar and pin both
|
||||||
|
services to the labelled node:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
nextcloud-webapp:
|
||||||
|
image: nextcloud:production # pin same version as CT 105
|
||||||
|
deploy:
|
||||||
|
replicas: 1
|
||||||
|
placement:
|
||||||
|
constraints:
|
||||||
|
- node.labels.nextcloud == true
|
||||||
|
# ... existing volumes, env, networks ...
|
||||||
|
|
||||||
|
nextcloud-cron:
|
||||||
|
image: nextcloud:production # same image, different entrypoint
|
||||||
|
entrypoint: /cron.sh
|
||||||
|
deploy:
|
||||||
|
replicas: 1
|
||||||
|
placement:
|
||||||
|
constraints:
|
||||||
|
- node.labels.nextcloud == true # must co-locate with webapp
|
||||||
|
volumes:
|
||||||
|
# Same data volume as nextcloud-webapp so cron sees the same files.
|
||||||
|
- nextcloud-data:/var/www/html
|
||||||
|
# No ports exposed — cron only runs PHP inside the container.
|
||||||
|
```
|
||||||
|
|
||||||
|
`/cron.sh` is Nextcloud's built-in cron entrypoint. It runs
|
||||||
|
`php -f /var/www/html/cron.php` in a loop, sleeping for 5 minutes between
|
||||||
|
runs — identical to the current systemd timer.
|
||||||
|
|
||||||
|
**2c.** Migrate Nextcloud's data volume to the swarm:
|
||||||
|
|
||||||
|
```
|
||||||
|
/mnt/docker/volumes/nextcloud-data/ → already on NFS, no migration needed
|
||||||
|
/mnt/docker/databases/nextcloud/ → already on NFS, no migration needed
|
||||||
|
```
|
||||||
|
|
||||||
|
The NFS paths are identical on the swarm nodes (`mount-data.nix` mounts the
|
||||||
|
same shares from the same VIP). Stop Nextcloud on CT 105, deploy on the
|
||||||
|
swarm, confirm it starts cleanly.
|
||||||
|
|
||||||
|
**2d.** Remove `nextcloud-cron-job.nix` from `modules/build-types/docker.nix`
|
||||||
|
and rebuild CT 105 after confirming Nextcloud works on the swarm.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 3. docker-health-to-gotify — update for swarm awareness
|
||||||
|
|
||||||
|
**Current state (CT 105):** The script at
|
||||||
|
`/home/nixos/docker/monitoring/gotify/docker-health-to-gotify.sh` runs every
|
||||||
|
minute, calls `docker ps --filter health=unhealthy`, and notifies Gotify.
|
||||||
|
|
||||||
|
**Swarm behaviour:** The same script runs on both swarm nodes independently,
|
||||||
|
each monitoring its own local Docker daemon. This gives per-node coverage
|
||||||
|
across the swarm.
|
||||||
|
|
||||||
|
**Changes needed in the script** (edit the copy on the NFS volume — it takes
|
||||||
|
effect on both nodes simultaneously on the next timer fire):
|
||||||
|
|
||||||
|
### 3a. Strip the Swarm task suffix from service names
|
||||||
|
|
||||||
|
In swarm mode, `docker ps --format '{{.Names}}'` returns names like
|
||||||
|
`nextcloud-webapp.1.abc123xyz`. The notification should show `nextcloud-webapp`,
|
||||||
|
not the full task name.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Before:
|
||||||
|
CONTAINER_NAME=$(docker ps --format '{{.Names}}' ...)
|
||||||
|
|
||||||
|
# After:
|
||||||
|
CONTAINER_NAME=$(docker ps --format '{{.Names}}' ... | cut -d. -f1)
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3b. Include the reporting node in the Gotify message
|
||||||
|
|
||||||
|
Add `$(hostname)` to the notification payload so you know which swarm node
|
||||||
|
detected the problem:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
MESSAGE="[$(hostname)] ${CONTAINER_NAME} is unhealthy"
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3c. Extend to catch swarm service replica failures
|
||||||
|
|
||||||
|
`docker ps` only shows what's running locally. If a service has zero healthy
|
||||||
|
replicas (task crash-looping) it may not show up on either node's `docker ps`
|
||||||
|
at the same moment. Add a swarm-level check:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run only on managers (both ha-docker nodes are managers):
|
||||||
|
if docker info --format '{{.Swarm.ControlAvailable}}' 2>/dev/null | grep -q true; then
|
||||||
|
# Find services where running replicas < desired replicas
|
||||||
|
docker service ls --format '{{.Name}}\t{{.Replicas}}' | \
|
||||||
|
awk -F'\t' '$2 !~ /^[0-9]+\/[0-9]+$/ || split($2,a,"/") && a[1] < a[2] { print $1, $2 }' | \
|
||||||
|
while read -r svc_name replicas; do
|
||||||
|
# Send Gotify notification for degraded service
|
||||||
|
curl -s -X POST "${GOTIFY_URL}/message" \
|
||||||
|
-H "X-Gotify-Key: ${GOTIFY_TOKEN}" \
|
||||||
|
-d "title=Swarm service degraded" \
|
||||||
|
-d "message=[$(hostname)] ${svc_name}: ${replicas} replicas"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
```
|
||||||
|
|
||||||
|
This catches the case where a service's desired replicas are not running
|
||||||
|
(e.g. OOM kill, image pull failure) — a failure mode that doesn't produce a
|
||||||
|
Docker health event on any node.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 4. Passbolt migration
|
||||||
|
|
||||||
|
Passbolt has strict data integrity requirements. Migrate with care:
|
||||||
|
|
||||||
|
1. **Backup first** — `docker exec passbolt-webapp php /usr/share/php/passbolt/bin/cake passbolt export_keys` and a database dump.
|
||||||
|
2. Database is on NFS (`/mnt/docker/databases/passbolt/`) — no data copy needed.
|
||||||
|
3. Pin Passbolt to a specific node: `docker node update --label-add passbolt=true ha-docker-1`
|
||||||
|
4. Add placement constraint `node.labels.passbolt == true` to the Passbolt stack.
|
||||||
|
5. Stop on CT 105, deploy on swarm, verify login works.
|
||||||
|
6. Test email delivery and 2FA.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 5. Gitea migration
|
||||||
|
|
||||||
|
Gitea's data directory is on NFS (`/mnt/docker/volumes/gitea-data/`).
|
||||||
|
|
||||||
|
1. Stop Gitea on CT 105: `docker stop gitea`
|
||||||
|
2. Deploy to swarm with placement constraint (pin to `ha-docker-1` initially).
|
||||||
|
3. Verify web UI and SSH clone/push work.
|
||||||
|
4. Update DNS: `gitea.lan.ddnsgeek.com` → swarm Traefik endpoint.
|
||||||
|
5. Update the flake remote URL in `variables.nix` (`giteaDomain`) if the address changes.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 6. Other services
|
||||||
|
|
||||||
|
Deploy remaining services (Grafana, InfluxDB, NodeRed, Prometheus, etc.)
|
||||||
|
as swarm stacks. Most have no special migration concern — they use NFS
|
||||||
|
volumes already on the shared storage.
|
||||||
|
|
||||||
|
Services with stateful databases (PostgreSQL, MariaDB) should follow the
|
||||||
|
pattern: stop on CT 105, confirm NFS database directory is intact, deploy on
|
||||||
|
swarm, verify.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 7. Monitoring — Beszel
|
||||||
|
|
||||||
|
The Beszel hub runs on CT 105 (`docker.sweet.home:8090`). Both swarm nodes
|
||||||
|
run `beszel-agent` (from `modules/beszel/enable-agent.nix`), pointing at the
|
||||||
|
existing hub URL.
|
||||||
|
|
||||||
|
No migration needed for Beszel itself during the container migration. Once
|
||||||
|
all services are on the swarm, you may wish to move the Beszel hub too (as a
|
||||||
|
swarm service with a placement constraint) but this is optional.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 8. DNS cutover
|
||||||
|
|
||||||
|
When a service is confirmed working on the swarm, update the FreeIPA DNS
|
||||||
|
A record from the CT 105 IP (192.168.2.225) to a swarm node IP or, when a
|
||||||
|
shared Traefik frontend is in place, to a round-robin record across both nodes.
|
||||||
|
|
||||||
|
**Recommended approach — Traefik as the single entry point:**
|
||||||
|
|
||||||
|
```
|
||||||
|
service.lan.ddnsgeek.com → Traefik on swarm (global mode)
|
||||||
|
docker.sweet.home → keep as 192.168.2.225 (CT 105) until fully decommissioned
|
||||||
|
```
|
||||||
|
|
||||||
|
For LAN-only services using `*.sweet.home` names, update FreeIPA directly:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On domain-controller (or via SSH):
|
||||||
|
ipa dnsrecord-mod sweet.home nextcloud --a-rec=192.168.2.230
|
||||||
|
# Add 192.168.2.231 as a second A record for round-robin (optional):
|
||||||
|
ipa dnsrecord-add sweet.home nextcloud --a-rec=192.168.2.231
|
||||||
|
```
|
||||||
|
|
||||||
|
Services behind Traefik don't need their own DNS updates — only Traefik's
|
||||||
|
own entry point IPs need to change.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 9. NixOS cleanup — CT 105
|
||||||
|
|
||||||
|
Once all services are migrated:
|
||||||
|
|
||||||
|
**Remove from `modules/build-types/docker.nix`:**
|
||||||
|
- `../docker/nextcloud-cron-job.nix` — replaced by sidecar container
|
||||||
|
- `../traefik/rotate-logs.nix` — replaced by Docker log driver
|
||||||
|
|
||||||
|
**Keep in `modules/build-types/docker.nix` until CT 105 is decommissioned:**
|
||||||
|
- `../docker/docker-health-to-gotify.nix` — still monitors CT 105's own daemon
|
||||||
|
- Everything else
|
||||||
|
|
||||||
|
**When decommissioning CT 105:**
|
||||||
|
1. Confirm all NFS volumes are in use only by swarm services (not CT 105).
|
||||||
|
2. Stop CT 105: `pct stop 105` on pve1.
|
||||||
|
3. Archive/remove the `lxc-docker` and `proxmox-docker` targets from `flake.nix`.
|
||||||
|
4. Remove `hosts/docker/`, `modules/build-types/docker.nix`, and `modules/docker/`.
|
||||||
|
5. Update `variables.nix` to remove `dockerIp`, `dockerStorageIp`, `dockerHost`
|
||||||
|
(or reassign `dockerHost` to point at a swarm node for Beszel hub resolution).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Rollback
|
||||||
|
|
||||||
|
Any stack can be rolled back to CT 105 independently:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# On CT 105:
|
||||||
|
docker start <service-name>
|
||||||
|
# Update DNS A record back to 192.168.2.225
|
||||||
|
ipa dnsrecord-mod sweet.home <service> --a-rec=192.168.2.225
|
||||||
|
```
|
||||||
|
|
||||||
|
CT 105 remains running throughout the cutover. Only decommission it after
|
||||||
|
every service is confirmed stable on the swarm and you have run one full
|
||||||
|
backup cycle from the new hosts.
|
||||||
+47
-5
@@ -7,6 +7,10 @@
|
|||||||
| LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
|
| LAN | 2 (native/untagged) | `192.168.2.0/24` | General LAN — clients and infrastructure | Yes (gateway .254) |
|
||||||
| Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
|
| Cluster | 10 | `192.168.10.224/29` | HA file server DRBD replication + Corosync heartbeat | No — internal `vmbr1` only, no uplink |
|
||||||
| Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink |
|
| Storage client | 20 | `192.168.20.0/24` | HA file server NFS (and iSCSI if needed) — docker and swarm nodes mount from VIP here | No — internal `vmbr2` only, no uplink |
|
||||||
|
| Swarm cluster | 30 | `192.168.30.0/24` | Docker Swarm gossip (TCP/UDP 7946) + VXLAN overlay (UDP 4789) | No — internal `vmbr3` only, no uplink |
|
||||||
|
|
||||||
|
When expanded to a second Proxmox node, VLAN 10 (cluster), VLAN 20 (storage-client), and VLAN 30 (swarm) all share
|
||||||
|
the same inter-node trunk NIC via 802.1q VLAN tagging — different VLAN IDs, same physical cable.
|
||||||
|
|
||||||
The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
|
The cluster and storage-client subnets never leave pve1. `vmbr1` and `vmbr2` are Proxmox Linux
|
||||||
bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
|
bridges with no physical port attached; traffic between guests on each bridge stays in-kernel.
|
||||||
@@ -27,9 +31,9 @@ is always `.228`: `192.168.2.228` (LAN), `192.168.10.228` (cluster), `192.168.20
|
|||||||
|
|
||||||
## DNS Zones
|
## DNS Zones
|
||||||
|
|
||||||
FreeIPA (domain-controller.sweet.home) is authoritative for all zones. Three
|
FreeIPA (domain-controller.sweet.home) is authoritative for all zones.
|
||||||
zones correspond to the three subnets — one per VLAN. All zones are internal
|
|
||||||
only; no external delegation.
|
Four zones correspond to the four subnets. All zones are internal only; no external delegation.
|
||||||
|
|
||||||
### sweet.home — VLAN 2 (192.168.2.x)
|
### sweet.home — VLAN 2 (192.168.2.x)
|
||||||
|
|
||||||
@@ -132,12 +136,14 @@ All VMs and LXC containers run on pve1.
|
|||||||
| `192.168.2.228` | ha-node1 | HA file server node 1 — management NIC | Active |
|
| `192.168.2.228` | ha-node1 | HA file server node 1 — management NIC | Active |
|
||||||
| `192.168.2.227` | ha-node2 | HA file server node 2 — management NIC | Active |
|
| `192.168.2.227` | ha-node2 | HA file server node 2 — management NIC | Active |
|
||||||
| `192.168.2.226` | server | Former NFS/ZFS file server — decommissioned | Removed from flake |
|
| `192.168.2.226` | server | Former NFS/ZFS file server — decommissioned | Removed from flake |
|
||||||
| `192.168.2.225` | docker | Docker / Traefik stack | Active |
|
| `192.168.2.225` | docker | Docker / Traefik stack (CT 105 — existing single-host) | Active |
|
||||||
| `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
|
| `192.168.2.224` | nix-cache | Nix binary cache + remote builder | Active |
|
||||||
| `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active |
|
| `192.168.2.223` | pxe-boot | PXE / TFTP / HTTP netboot server | Active |
|
||||||
| `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active |
|
| `192.168.2.222` | tailscale-router | Tailscale exit node / router | Active |
|
||||||
| `192.168.2.221` | tor-relay | Tor relay | Active |
|
| `192.168.2.221` | tor-relay | Tor relay | Active |
|
||||||
| `192.168.2.220` | pdm | Proxmox Deploy Manager | Active |
|
| `192.168.2.220` | pdm | Proxmox Deploy Manager | Active |
|
||||||
|
| `192.168.2.231` | ha-docker-2 | Docker Swarm node 2 — management NIC | Active |
|
||||||
|
| `192.168.2.230` | ha-docker-1 | Docker Swarm node 1 — management NIC | Active |
|
||||||
|
|
||||||
### Client DHCP pool (.10–.59)
|
### Client DHCP pool (.10–.59)
|
||||||
|
|
||||||
@@ -175,7 +181,9 @@ Internal to pve1 only. Proxmox bridge `vmbr2`, no physical NIC attached.
|
|||||||
| `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
|
| `192.168.20.228` | ha-node1 | Storage-client NIC (ens20 / vmbr2) |
|
||||||
| `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
|
| `192.168.20.227` | ha-node2 | Storage-client NIC (ens20 / vmbr2) |
|
||||||
| `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — decommissioned |
|
| `192.168.20.226` | server | Storage-client NIC (ens19 / vmbr2) — decommissioned |
|
||||||
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client |
|
| `192.168.20.225` | docker | Storage-client NIC (eth1 / vmbr2) — NFS client (CT 105) |
|
||||||
|
| `192.168.20.231` | ha-docker-2 | Storage-client NIC (ens19 / vmbr2) — NFS client |
|
||||||
|
| `192.168.20.230` | ha-docker-1 | Storage-client NIC (ens19 / vmbr2) — NFS client |
|
||||||
| — | no gateway | Isolated — not routed to LAN or internet |
|
| — | no gateway | Isolated — not routed to LAN or internet |
|
||||||
|
|
||||||
NFS clients mount from `192.168.20.229` (surviving failover transparently via the VIP).
|
NFS clients mount from `192.168.20.229` (surviving failover transparently via the VIP).
|
||||||
@@ -184,3 +192,37 @@ cannot reach either service on this VIP. The `vip-storage` endpoint is not reach
|
|||||||
from the workstation directly (internal bridge only); health checks proxy through the
|
from the workstation directly (internal bridge only); health checks proxy through the
|
||||||
active HA node.
|
active HA node.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Swarm cluster network — VLAN 30 — 192.168.30.0/24
|
||||||
|
|
||||||
|
Internal to pve1 only. Proxmox bridge `vmbr3`, no physical NIC attached.
|
||||||
|
Carries Docker Swarm inter-node traffic only: Raft consensus (TCP 2377),
|
||||||
|
Serf gossip (TCP/UDP 7946), and VXLAN overlay data path (UDP 4789).
|
||||||
|
Docker Swarm is initialised with `--advertise-addr` and `--data-path-addr`
|
||||||
|
both pointing to this subnet so all cluster traffic stays on `vmbr3` and
|
||||||
|
never crosses the LAN.
|
||||||
|
|
||||||
|
| IP | Hostname | Interface / role |
|
||||||
|
|---|---|---|
|
||||||
|
| `192.168.30.231` | ha-docker-2 | Swarm cluster NIC (ens20 / vmbr3) |
|
||||||
|
| `192.168.30.230` | ha-docker-1 | Swarm cluster NIC (ens20 / vmbr3) |
|
||||||
|
| — | no gateway | Isolated — not routed to LAN or internet |
|
||||||
|
|
||||||
|
### DNS zone: `swarm.home` — VLAN 30 (192.168.30.x)
|
||||||
|
|
||||||
|
| Hostname | A record | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `ha-docker-1.swarm.home` | `192.168.30.230` | Swarm NIC — debugging only |
|
||||||
|
| `ha-docker-2.swarm.home` | `192.168.30.231` | Swarm NIC — debugging only |
|
||||||
|
|
||||||
|
Operators reach the Docker API on the LAN IPs (`192.168.2.230`/`.231`), not these addresses.
|
||||||
|
The `swarm.home` records exist for diagnostic convenience (e.g. confirming `vmbr3` routing).
|
||||||
|
|
||||||
|
### Multi-node Proxmox expansion
|
||||||
|
|
||||||
|
When a second Proxmox node (pve2) is added, VLAN 10 (cluster), VLAN 20 (storage-client),
|
||||||
|
and VLAN 30 (swarm) all extend to pve2 via 802.1q VLAN tagging on the inter-node trunk
|
||||||
|
link. All three internal networks share the same physical NIC between hypervisors —
|
||||||
|
VLAN tags provide the logical separation.
|
||||||
|
|
||||||
|
|||||||
@@ -130,6 +130,9 @@
|
|||||||
|
|
||||||
proxmox-ha-server-1 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-1/host.nix; nameSuffix = "-1"; };
|
proxmox-ha-server-1 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-1/host.nix; nameSuffix = "-1"; };
|
||||||
proxmox-ha-server-2 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-2/host.nix; nameSuffix = "-2"; };
|
proxmox-ha-server-2 = mkTarget { platform = "proxmox"; buildType = "ha-server"; hostPath = ./hosts/ha-server-2/host.nix; nameSuffix = "-2"; };
|
||||||
|
|
||||||
|
proxmox-ha-docker-1 = mkTarget { platform = "proxmox"; buildType = "ha-docker"; hostPath = ./hosts/ha-docker-1/host.nix; nameSuffix = "-1"; };
|
||||||
|
proxmox-ha-docker-2 = mkTarget { platform = "proxmox"; buildType = "ha-docker"; hostPath = ./hosts/ha-docker-2/host.nix; nameSuffix = "-2"; };
|
||||||
};
|
};
|
||||||
|
|
||||||
# Auto-install environments (migrated from the former nix-auto-installer
|
# Auto-install environments (migrated from the former nix-auto-installer
|
||||||
|
|||||||
@@ -0,0 +1,34 @@
|
|||||||
|
{ vars, ... }:
|
||||||
|
{
|
||||||
|
networking = {
|
||||||
|
hostName = vars.haDocker1Host;
|
||||||
|
hostId = "a1d0c4e1";
|
||||||
|
useDHCP = false;
|
||||||
|
interfaces = {
|
||||||
|
# ens18 — LAN management NIC (vmbr0, 192.168.2.0/24)
|
||||||
|
${vars.vmLanInterface}.ipv4.addresses = [{
|
||||||
|
address = vars.haDocker1Ip;
|
||||||
|
prefixLength = vars.lanPrefixLength;
|
||||||
|
}];
|
||||||
|
# ens19 — storage-client NIC (vmbr2, 192.168.20.0/24) — NFS from HA cluster
|
||||||
|
${vars.haDockerStorageInterface}.ipv4.addresses = [{
|
||||||
|
address = vars.haDocker1StorageIp;
|
||||||
|
prefixLength = vars.haClientPrefixLength;
|
||||||
|
}];
|
||||||
|
# ens20 — swarm cluster NIC (vmbr3, 192.168.30.0/24) — Docker gossip + VXLAN
|
||||||
|
${vars.haDockerSwarmInterface}.ipv4.addresses = [{
|
||||||
|
address = vars.haDocker1SwarmIp;
|
||||||
|
prefixLength = vars.haDockerSwarmPrefixLength;
|
||||||
|
}];
|
||||||
|
};
|
||||||
|
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
|
||||||
|
nameservers = [ vars.domainControllerIp ];
|
||||||
|
};
|
||||||
|
|
||||||
|
# Only register the LAN IP with IPA DNS. Without this, sssd dyndns
|
||||||
|
# would also register Docker bridge IPs (172.x.x.x) and the storage/swarm
|
||||||
|
# NIC IPs as A records for ha-docker-1.sweet.home.
|
||||||
|
security.ipa.dyndns.interface = vars.vmLanInterface;
|
||||||
|
|
||||||
|
system.stateVersion = "26.05";
|
||||||
|
}
|
||||||
@@ -0,0 +1,32 @@
|
|||||||
|
{ vars, ... }:
|
||||||
|
{
|
||||||
|
networking = {
|
||||||
|
hostName = vars.haDocker2Host;
|
||||||
|
hostId = "a2d0c4e2";
|
||||||
|
useDHCP = false;
|
||||||
|
interfaces = {
|
||||||
|
# ens18 — LAN management NIC (vmbr0, 192.168.2.0/24)
|
||||||
|
${vars.vmLanInterface}.ipv4.addresses = [{
|
||||||
|
address = vars.haDocker2Ip;
|
||||||
|
prefixLength = vars.lanPrefixLength;
|
||||||
|
}];
|
||||||
|
# ens19 — storage-client NIC (vmbr2, 192.168.20.0/24) — NFS from HA cluster
|
||||||
|
${vars.haDockerStorageInterface}.ipv4.addresses = [{
|
||||||
|
address = vars.haDocker2StorageIp;
|
||||||
|
prefixLength = vars.haClientPrefixLength;
|
||||||
|
}];
|
||||||
|
# ens20 — swarm cluster NIC (vmbr3, 192.168.30.0/24) — Docker gossip + VXLAN
|
||||||
|
${vars.haDockerSwarmInterface}.ipv4.addresses = [{
|
||||||
|
address = vars.haDocker2SwarmIp;
|
||||||
|
prefixLength = vars.haDockerSwarmPrefixLength;
|
||||||
|
}];
|
||||||
|
};
|
||||||
|
defaultGateway = { address = vars.lanGateway; interface = vars.vmLanInterface; };
|
||||||
|
nameservers = [ vars.domainControllerIp ];
|
||||||
|
};
|
||||||
|
|
||||||
|
# Only register the LAN IP with IPA DNS — same reasoning as ha-docker-1.
|
||||||
|
security.ipa.dyndns.interface = vars.vmLanInterface;
|
||||||
|
|
||||||
|
system.stateVersion = "26.05";
|
||||||
|
}
|
||||||
@@ -0,0 +1,84 @@
|
|||||||
|
# Docker Swarm node build type.
|
||||||
|
#
|
||||||
|
# Produces NixOS hosts that form a Docker Swarm manager cluster. Two nodes
|
||||||
|
# (ha-docker-1, ha-docker-2) are both managers so either can accept Docker
|
||||||
|
# API and `docker stack` commands.
|
||||||
|
#
|
||||||
|
# Key differences from the existing `docker` build type (used by CT 105):
|
||||||
|
# - nextcloud-cron-job.nix is EXCLUDED — `docker exec` breaks in swarm
|
||||||
|
# because the target container may be on the other node. The cron job
|
||||||
|
# is replaced by a nextcloud-cron sidecar in the Nextcloud stack.
|
||||||
|
# See docs/internal/docker-swarm-cutover.md.
|
||||||
|
# - traefik/rotate-logs.nix is EXCLUDED — log rotation moves to Docker's
|
||||||
|
# json-file log driver (max-size/max-file on the Traefik service
|
||||||
|
# definition). See docs/internal/docker-swarm-cutover.md.
|
||||||
|
# - raspi/mount-data.nix is EXCLUDED — specific to CT 105's backup role.
|
||||||
|
# - Swarm firewall ports (2377/tcp, 7946/tcp+udp, 4789/udp) are opened
|
||||||
|
# on the swarm NIC (ens20/vmbr3) only.
|
||||||
|
# - checkReversePath = "loose" is required for the Swarm ingress routing
|
||||||
|
# mesh: VXLAN return traffic is asymmetric (arrives ens20, exits ens18).
|
||||||
|
# - beszel-agent is enabled for host-level monitoring.
|
||||||
|
{ pkgs, vars, ... }:
|
||||||
|
|
||||||
|
{
|
||||||
|
# Pin Docker Engine to version 29, matching CT 105, so image layers cached
|
||||||
|
# on NFS volumes remain compatible across old and new hosts.
|
||||||
|
nixpkgs.overlays = [
|
||||||
|
(final: prev: {
|
||||||
|
docker = prev.docker_29;
|
||||||
|
docker_cli = prev.docker_29;
|
||||||
|
})
|
||||||
|
];
|
||||||
|
|
||||||
|
imports = [
|
||||||
|
../docker/enable-service.nix
|
||||||
|
../docker/mount-data.nix
|
||||||
|
../docker/docker-health-to-gotify.nix
|
||||||
|
../beszel/enable-agent.nix
|
||||||
|
../services/enable-rpcbind.nix
|
||||||
|
];
|
||||||
|
|
||||||
|
environment.systemPackages = with pkgs; [
|
||||||
|
nfs-utils
|
||||||
|
];
|
||||||
|
|
||||||
|
boot.supportedFilesystems = [ "nfs" ];
|
||||||
|
|
||||||
|
systemd.tmpfiles.rules = [
|
||||||
|
# Symlink ~/docker → NFS config mount so the docker-health-to-gotify
|
||||||
|
# script (and operator convenience) resolves ~/docker/... correctly.
|
||||||
|
"L+ /home/${vars.primaryUser}/docker - - - - ${vars.nfsShares.dockerConfig.mountpoint}"
|
||||||
|
"d /mnt/docker 0755 ${vars.primaryUser} users -"
|
||||||
|
];
|
||||||
|
|
||||||
|
users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
|
||||||
|
|
||||||
|
networking.firewall = {
|
||||||
|
# LAN-facing service ports — same as the existing docker build type.
|
||||||
|
allowedTCPPorts = [
|
||||||
|
vars.ports.dockerHttp
|
||||||
|
vars.ports.dockerHttps
|
||||||
|
vars.ports.dockerExtra
|
||||||
|
vars.ports.beszelHub
|
||||||
|
];
|
||||||
|
|
||||||
|
# Swarm inter-node ports restricted to the swarm NIC (ens20/vmbr3).
|
||||||
|
# vmbr3 is an isolated internal bridge — no LAN reachability.
|
||||||
|
interfaces.${vars.haDockerSwarmInterface} = {
|
||||||
|
allowedTCPPorts = [
|
||||||
|
vars.ports.dockerSwarmMgmt # 2377 — Raft + cluster management
|
||||||
|
vars.ports.dockerSwarmDisc # 7946 — Serf gossip (TCP half)
|
||||||
|
];
|
||||||
|
allowedUDPPorts = [
|
||||||
|
vars.ports.dockerSwarmDisc # 7946 — Serf gossip (UDP half)
|
||||||
|
vars.ports.dockerSwarmVxlan # 4789 — VXLAN overlay data path
|
||||||
|
];
|
||||||
|
};
|
||||||
|
|
||||||
|
# Docker Swarm ingress routing mesh creates asymmetric routes: a request
|
||||||
|
# arrives on ens18 (LAN) for a container that lives on ens20's VXLAN
|
||||||
|
# overlay; the return path differs from the incoming interface. Strict
|
||||||
|
# rp_filter drops these packets. "loose" allows them.
|
||||||
|
checkReversePath = "loose";
|
||||||
|
};
|
||||||
|
}
|
||||||
Executable
+585
@@ -0,0 +1,585 @@
|
|||||||
|
#!/usr/bin/env bash
|
||||||
|
# deploy.sh — Full lifecycle management for the Docker Swarm HA cluster.
|
||||||
|
#
|
||||||
|
# Provisions two NixOS Proxmox VMs (ha-docker-1, ha-docker-2) as dual-manager
|
||||||
|
# Docker Swarm nodes sharing NFS storage from the existing HA file-server
|
||||||
|
# cluster. Both nodes are managers so either can accept Docker API and
|
||||||
|
# `docker stack` commands.
|
||||||
|
#
|
||||||
|
# Usage:
|
||||||
|
# scripts/docker-swarm/deploy.sh [options]
|
||||||
|
# scripts/docker-swarm/deploy.sh --destroy [options]
|
||||||
|
#
|
||||||
|
# Phases (all run by default; skip any with --skip-<phase>):
|
||||||
|
# 1. ensure-bridge Create vmbr3 (swarm cluster bridge) on the Proxmox node.
|
||||||
|
# 2. sync-keys Generate SSH host keys for both nodes (clan vars).
|
||||||
|
# 3. ipa-hosts Create IPA host objects + sops-encrypted keytabs.
|
||||||
|
# 4. create-vms Build NixOS disk images and create VMs via create-proxmox-resource.sh.
|
||||||
|
# 5. add-hardware Attach vmbr2 (storage) and vmbr3 (swarm) NICs; start VMs.
|
||||||
|
# 6. boot-wait Wait for SSH on both LAN IPs.
|
||||||
|
# 7. refresh-sops-keys Detect disko key drift; re-encrypt secrets; commit.
|
||||||
|
# 8. init-swarm docker swarm init on node1; manager join on node2; label nodes.
|
||||||
|
# 9. dns Register storage.home and swarm.home A records in FreeIPA.
|
||||||
|
# 10. verify docker node ls; NFS mount check; swarm health.
|
||||||
|
#
|
||||||
|
# Options:
|
||||||
|
# --node <host> Proxmox host (default: pve1.sweet.home)
|
||||||
|
# --vmid1 <n> VMID for ha-docker-1 (default: 202)
|
||||||
|
# --vmid2 <n> VMID for ha-docker-2 (default: 203)
|
||||||
|
# --storage <pool> Proxmox storage pool (default: local-zfs)
|
||||||
|
# --swarm-bridge <br> Bridge for Docker Swarm cluster network (default: vmbr3)
|
||||||
|
# --storage-bridge <br> Bridge for NFS storage network (default: vmbr2)
|
||||||
|
# --memory <MB> RAM per node (default: 4096)
|
||||||
|
# --cores <n> vCPUs per node (default: 4)
|
||||||
|
# --skip-ensure-bridge Skip vmbr3 creation/check
|
||||||
|
# --skip-sync-keys Skip sync-host-keys.sh (clan vars already exist)
|
||||||
|
# --skip-ipa-hosts Skip IPA host account creation (keytabs already exist)
|
||||||
|
# --skip-create-vms Skip VM creation (VMs already exist)
|
||||||
|
# --skip-add-hardware Skip NIC attachment (already attached)
|
||||||
|
# --skip-boot-wait Skip boot/SSH wait (VMs already running)
|
||||||
|
# --skip-refresh-sops-keys Skip sops host-key drift fix
|
||||||
|
# --skip-init-swarm Skip swarm initialisation (already initialised)
|
||||||
|
# --skip-dns Skip FreeIPA DNS record creation
|
||||||
|
# --skip-verify Skip post-deploy health checks
|
||||||
|
# --force-rebuild Pass --force-rebuild to create-proxmox-resource.sh
|
||||||
|
# --destroy Stop and delete both VMs (skip all other phases)
|
||||||
|
# --dry-run Print what would run without executing
|
||||||
|
# -h|--help Show this message
|
||||||
|
#
|
||||||
|
# Prerequisites:
|
||||||
|
# - SSH access to the Proxmox node as $PROXMOX_SSH_USER (wayne).
|
||||||
|
# - sops age key in the standard location (used by sync-host-keys.sh).
|
||||||
|
# - SSH access to domain-controller.sweet.home as $PROXMOX_SSH_USER for DNS phase.
|
||||||
|
# - For --skip-sync-keys: clan vars already in vars/per-machine/proxmox-ha-docker-{1,2}/.
|
||||||
|
# - For --skip-ipa-hosts: secrets/ha-docker-{1,2}.keytab already exist and are committed.
|
||||||
|
|
||||||
|
set -euo pipefail
|
||||||
|
|
||||||
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||||
|
REPO_ROOT="$(cd "${SCRIPT_DIR}/../.." && pwd)"
|
||||||
|
|
||||||
|
# shellcheck source=../env.sh
|
||||||
|
source "${REPO_ROOT}/scripts/env.sh"
|
||||||
|
|
||||||
|
# ── Defaults ──────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
NODE="${PVE1_HOST}" # deploy.sh targets pve1 by default (authorised for this cluster)
|
||||||
|
VMID1=202
|
||||||
|
VMID2=203
|
||||||
|
STORAGE="${PROXMOX_STORAGE:-local-zfs}"
|
||||||
|
SWARM_BRIDGE="vmbr3"
|
||||||
|
STORAGE_BRIDGE="vmbr2"
|
||||||
|
MEMORY_MB=4096
|
||||||
|
CORES=4
|
||||||
|
|
||||||
|
SKIP_ENSURE_BRIDGE=false
|
||||||
|
SKIP_SYNC_KEYS=false
|
||||||
|
SKIP_IPA_HOSTS=false
|
||||||
|
SKIP_CREATE_VMS=false
|
||||||
|
SKIP_ADD_HARDWARE=false
|
||||||
|
SKIP_BOOT_WAIT=false
|
||||||
|
SKIP_REFRESH_SOPS_KEYS=false
|
||||||
|
SKIP_INIT_SWARM=false
|
||||||
|
SKIP_DNS=false
|
||||||
|
SKIP_VERIFY=false
|
||||||
|
FORCE_REBUILD=false
|
||||||
|
DESTROY=false
|
||||||
|
DRY_RUN=false
|
||||||
|
|
||||||
|
# ── Variables from repo (mirrors variables.nix) ───────────────────────────────
|
||||||
|
|
||||||
|
NODE1_HOST="ha-docker-1"
|
||||||
|
NODE2_HOST="ha-docker-2"
|
||||||
|
NODE1_LAN_IP="192.168.2.230"
|
||||||
|
NODE2_LAN_IP="192.168.2.231"
|
||||||
|
NODE1_SWARM_IP="192.168.30.230"
|
||||||
|
NODE2_SWARM_IP="192.168.30.231"
|
||||||
|
NODE1_STORAGE_IP="192.168.20.230"
|
||||||
|
NODE2_STORAGE_IP="192.168.20.231"
|
||||||
|
SWARM_CIDR="192.168.30.0/24"
|
||||||
|
STORAGE_CIDR="192.168.20.0/24"
|
||||||
|
STORAGE_ZONE="storage.home"
|
||||||
|
SWARM_ZONE="swarm.home"
|
||||||
|
SSH_USER="${PROXMOX_SSH_USER:-wayne}"
|
||||||
|
DC_HOST="${IPA_SERVER:-domain-controller.sweet.home}"
|
||||||
|
|
||||||
|
# ── Argument parsing ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
usage() {
|
||||||
|
sed -n '/^# Usage:/,/^[^#]/{ /^#/{ s/^# \?//; p } }' "$0"
|
||||||
|
exit "${1:-0}"
|
||||||
|
}
|
||||||
|
|
||||||
|
while [[ $# -gt 0 ]]; do
|
||||||
|
case "$1" in
|
||||||
|
--node) NODE="$2"; shift 2 ;;
|
||||||
|
--vmid1) VMID1="$2"; shift 2 ;;
|
||||||
|
--vmid2) VMID2="$2"; shift 2 ;;
|
||||||
|
--storage) STORAGE="$2"; shift 2 ;;
|
||||||
|
--swarm-bridge) SWARM_BRIDGE="$2"; shift 2 ;;
|
||||||
|
--storage-bridge) STORAGE_BRIDGE="$2"; shift 2 ;;
|
||||||
|
--memory) MEMORY_MB="$2"; shift 2 ;;
|
||||||
|
--cores) CORES="$2"; shift 2 ;;
|
||||||
|
--skip-ensure-bridge) SKIP_ENSURE_BRIDGE=true; shift ;;
|
||||||
|
--skip-sync-keys) SKIP_SYNC_KEYS=true; shift ;;
|
||||||
|
--skip-ipa-hosts) SKIP_IPA_HOSTS=true; shift ;;
|
||||||
|
--skip-create-vms) SKIP_CREATE_VMS=true; shift ;;
|
||||||
|
--skip-add-hardware) SKIP_ADD_HARDWARE=true; shift ;;
|
||||||
|
--skip-boot-wait) SKIP_BOOT_WAIT=true; shift ;;
|
||||||
|
--skip-refresh-sops-keys) SKIP_REFRESH_SOPS_KEYS=true; shift ;;
|
||||||
|
--skip-init-swarm) SKIP_INIT_SWARM=true; shift ;;
|
||||||
|
--skip-dns) SKIP_DNS=true; shift ;;
|
||||||
|
--skip-verify) SKIP_VERIFY=true; shift ;;
|
||||||
|
--force-rebuild) FORCE_REBUILD=true; shift ;;
|
||||||
|
--destroy) DESTROY=true; shift ;;
|
||||||
|
--dry-run) DRY_RUN=true; shift ;;
|
||||||
|
-h|--help) usage 0 ;;
|
||||||
|
*) echo "Unknown option: $1" >&2; usage 1 ;;
|
||||||
|
esac
|
||||||
|
done
|
||||||
|
|
||||||
|
# ── Helpers ───────────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
log() { echo "==> $*"; }
|
||||||
|
logn() { echo " $*"; }
|
||||||
|
err() { echo "ERROR: $*" >&2; exit 1; }
|
||||||
|
|
||||||
|
run() {
|
||||||
|
if $DRY_RUN; then
|
||||||
|
echo "[dry-run] $*"
|
||||||
|
else
|
||||||
|
"$@"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
pve() {
|
||||||
|
if $DRY_RUN; then
|
||||||
|
echo "[dry-run] ssh ${SSH_USER}@${NODE} sudo $*"
|
||||||
|
else
|
||||||
|
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
pve_check() {
|
||||||
|
# Read-only probe — always executes even in dry-run.
|
||||||
|
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "sudo $*"
|
||||||
|
}
|
||||||
|
|
||||||
|
SWARM_USER="nixos"
|
||||||
|
|
||||||
|
n1() {
|
||||||
|
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
|
||||||
|
"${SWARM_USER}@${NODE1_LAN_IP}" sudo "$@" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
n2() {
|
||||||
|
ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=5 \
|
||||||
|
"${SWARM_USER}@${NODE2_LAN_IP}" sudo "$@" 2>/dev/null
|
||||||
|
}
|
||||||
|
|
||||||
|
dc() {
|
||||||
|
# Run ipa commands on domain-controller as $SSH_USER.
|
||||||
|
if $DRY_RUN; then
|
||||||
|
echo "[dry-run] ssh ${SSH_USER}@${DC_HOST} $*"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${DC_HOST}" "$@"
|
||||||
|
}
|
||||||
|
|
||||||
|
wait_for_ssh() {
|
||||||
|
local ip="$1" label="$2"
|
||||||
|
if $DRY_RUN; then
|
||||||
|
logn "[dry-run] Skipping SSH wait for ${label} (${ip})"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
local deadline=$(( $(date +%s) + 300 ))
|
||||||
|
log "Waiting for SSH on ${label} (${ip}) — up to 5 min..."
|
||||||
|
while [[ $(date +%s) -lt $deadline ]]; do
|
||||||
|
if ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no -o ConnectTimeout=3 \
|
||||||
|
-o BatchMode=yes "${SWARM_USER}@${ip}" true 2>/dev/null; then
|
||||||
|
logn "${label} is up."
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
err "Timed out waiting for SSH on ${label} (${ip})"
|
||||||
|
}
|
||||||
|
|
||||||
|
# ── Destroy mode ──────────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
if $DESTROY; then
|
||||||
|
log "Destroying Docker Swarm VMs (${VMID1}=${NODE1_HOST}, ${VMID2}=${NODE2_HOST}) on ${NODE}"
|
||||||
|
for vmid in "$VMID1" "$VMID2"; do
|
||||||
|
STATUS=$(pve "qm status ${vmid} 2>/dev/null" 2>/dev/null || true)
|
||||||
|
if echo "$STATUS" | grep -q "running"; then
|
||||||
|
log "Stopping VMID ${vmid}..."
|
||||||
|
pve "qm stop ${vmid} --skiplock 1"
|
||||||
|
sleep 5
|
||||||
|
fi
|
||||||
|
if $DRY_RUN || pve "qm config ${vmid} >/dev/null 2>&1"; then
|
||||||
|
log "Deleting VMID ${vmid}..."
|
||||||
|
run pve "qm destroy ${vmid} --purge 1"
|
||||||
|
else
|
||||||
|
logn "VMID ${vmid} not found — already gone."
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
log "Done — swarm VMs destroyed."
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 1: Ensure swarm bridge ──────────────────────────────────────────────
|
||||||
|
|
||||||
|
if ! $SKIP_ENSURE_BRIDGE; then
|
||||||
|
log "Phase 1: Ensuring swarm bridge ${SWARM_BRIDGE} on ${NODE}"
|
||||||
|
if pve_check "test -d /sys/class/net/${SWARM_BRIDGE}" &>/dev/null; then
|
||||||
|
logn "${SWARM_BRIDGE} already exists — skipping."
|
||||||
|
else
|
||||||
|
logn "Creating isolated internal bridge ${SWARM_BRIDGE} (no upstream port, ${SWARM_CIDR})"
|
||||||
|
BRIDGE_CONF="auto ${SWARM_BRIDGE}
|
||||||
|
iface ${SWARM_BRIDGE} inet manual
|
||||||
|
bridge-ports none
|
||||||
|
bridge-stp off
|
||||||
|
bridge-fd 0"
|
||||||
|
if $DRY_RUN; then
|
||||||
|
echo "[dry-run] Would write /etc/network/interfaces.d/${SWARM_BRIDGE}.conf and ifup it"
|
||||||
|
else
|
||||||
|
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
|
||||||
|
"echo '${BRIDGE_CONF}' | sudo tee /etc/network/interfaces.d/${SWARM_BRIDGE}.conf > /dev/null && sudo ifup ${SWARM_BRIDGE}"
|
||||||
|
logn "${SWARM_BRIDGE} created and brought up."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 2: Sync host keys ───────────────────────────────────────────────────
|
||||||
|
|
||||||
|
if ! $SKIP_SYNC_KEYS; then
|
||||||
|
log "Phase 2: Syncing SSH host keys for both swarm targets"
|
||||||
|
for target in proxmox-ha-docker-1 proxmox-ha-docker-2; do
|
||||||
|
CLAN_DIR="${REPO_ROOT}/vars/per-machine/${target}/openssh"
|
||||||
|
if [[ -d "$CLAN_DIR" ]]; then
|
||||||
|
logn "Clan vars for ${target} already exist — skipping."
|
||||||
|
else
|
||||||
|
logn "Generating host keys for ${target}..."
|
||||||
|
run bash "${REPO_ROOT}/scripts/secrets/sync-host-keys.sh" "$target"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 3: IPA host accounts ────────────────────────────────────────────────
|
||||||
|
|
||||||
|
if ! $SKIP_IPA_HOSTS; then
|
||||||
|
log "Phase 3: Creating IPA host accounts and keytabs"
|
||||||
|
IPA_SCRIPT="${REPO_ROOT}/scripts/ipa/create-nixos-ipa-host-account.sh"
|
||||||
|
for host in "${NODE1_HOST}" "${NODE2_HOST}"; do
|
||||||
|
KEYTAB="${REPO_ROOT}/secrets/${host}.keytab"
|
||||||
|
if [[ -f "$KEYTAB" ]]; then
|
||||||
|
logn "Keytab for ${host} already exists — skipping."
|
||||||
|
else
|
||||||
|
logn "Creating IPA host account and keytab for ${host}..."
|
||||||
|
run bash "$IPA_SCRIPT" "$host"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
if ! $DRY_RUN; then
|
||||||
|
# Keytabs must be committed and pushed before VMs rebuild from Gitea.
|
||||||
|
CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"
|
||||||
|
logn "Committing keytabs and pushing to Gitea (branch: ${CURRENT_BRANCH})..."
|
||||||
|
(cd "${REPO_ROOT}" && \
|
||||||
|
git add secrets/ha-docker-1.keytab secrets/ha-docker-2.keytab .sops.yaml && \
|
||||||
|
git commit -m "secrets(ha-docker): add IPA keytabs for ha-docker-1 and ha-docker-2" || true && \
|
||||||
|
git push origin "${CURRENT_BRANCH}")
|
||||||
|
logn "Pushed."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 3.5: Prepare Proxmox node for building ─────────────────────────────
|
||||||
|
|
||||||
|
if ! $SKIP_CREATE_VMS && ! $DRY_RUN; then
|
||||||
|
CURRENT_BRANCH="$(git -C "$REPO_ROOT" rev-parse --abbrev-ref HEAD)"
|
||||||
|
|
||||||
|
local_ssh() { ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" "$*"; }
|
||||||
|
if local_ssh "test -d /nix" &>/dev/null && ! local_ssh "test -w /nix" &>/dev/null; then
|
||||||
|
logn "/nix exists but not writable by ${SSH_USER} — fixing ownership with sudo..."
|
||||||
|
local_ssh "sudo chown -R ${SSH_USER} /nix"
|
||||||
|
logn "Done."
|
||||||
|
fi
|
||||||
|
unset -f local_ssh
|
||||||
|
|
||||||
|
REMOTE_REPO="/home/${SSH_USER}/nixos"
|
||||||
|
if pve_check "test -d ${REMOTE_REPO}/.git" &>/dev/null; then
|
||||||
|
REMOTE_BRANCH=$(ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
|
||||||
|
"cd ${REMOTE_REPO} && git rev-parse --abbrev-ref HEAD 2>/dev/null")
|
||||||
|
if [[ "$REMOTE_BRANCH" != "$CURRENT_BRANCH" ]]; then
|
||||||
|
logn "Remote clone is on '${REMOTE_BRANCH}', switching to '${CURRENT_BRANCH}'..."
|
||||||
|
ssh -i ~/.ssh/id_ed25519 "${SSH_USER}@${NODE}" \
|
||||||
|
"cd ${REMOTE_REPO} && git fetch origin && git checkout '${CURRENT_BRANCH}' && git pull --ff-only"
|
||||||
|
logn "Done."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 4: Create VMs ───────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
if ! $SKIP_CREATE_VMS; then
|
||||||
|
log "Phase 4: Building and creating swarm VMs on ${NODE}"
|
||||||
|
CREATE="${REPO_ROOT}/scripts/proxmox/create-proxmox-resource.sh"
|
||||||
|
REBUILD_FLAG=""
|
||||||
|
$FORCE_REBUILD && REBUILD_FLAG="--force-rebuild"
|
||||||
|
|
||||||
|
for spec in "${VMID1}:${NODE1_HOST}:proxmox-ha-docker-1" "${VMID2}:${NODE2_HOST}:proxmox-ha-docker-2"; do
|
||||||
|
IFS=: read -r vmid host_name flake_target <<< "$spec"
|
||||||
|
log "Creating ${flake_target} (VMID ${vmid}) on ${NODE}..."
|
||||||
|
# --force-rebuild is always passed: create-proxmox-resource.sh only calls
|
||||||
|
# sync_remote_host_keys (which bakes the clan-var SSH key into the disk
|
||||||
|
# image) when it actually builds. Reusing a cached image skips that step
|
||||||
|
# and leaves the VM unable to decrypt sops secrets on first boot.
|
||||||
|
run bash "$CREATE" \
|
||||||
|
--type vm \
|
||||||
|
--host "$host_name" \
|
||||||
|
--vmid "$vmid" \
|
||||||
|
--node "$NODE" \
|
||||||
|
--storage "$STORAGE" \
|
||||||
|
--memory "$MEMORY_MB" \
|
||||||
|
--cores "$CORES" \
|
||||||
|
--force-rebuild
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 5: Add NICs and start VMs ──────────────────────────────────────────
|
||||||
|
|
||||||
|
if ! $SKIP_ADD_HARDWARE; then
|
||||||
|
log "Phase 5: Attaching storage (${STORAGE_BRIDGE}) and swarm (${SWARM_BRIDGE}) NICs"
|
||||||
|
for vmid in "$VMID1" "$VMID2"; do
|
||||||
|
logn "VMID ${vmid}: stopping to add NICs..."
|
||||||
|
pve "qm stop ${vmid} --skiplock 1 2>/dev/null; sleep 3" || true
|
||||||
|
|
||||||
|
logn "Adding net1 (${STORAGE_BRIDGE} — NFS storage)..."
|
||||||
|
pve "qm set ${vmid} --net1 virtio,bridge=${STORAGE_BRIDGE},firewall=0"
|
||||||
|
|
||||||
|
logn "Adding net2 (${SWARM_BRIDGE} — Docker Swarm)..."
|
||||||
|
pve "qm set ${vmid} --net2 virtio,bridge=${SWARM_BRIDGE},firewall=0"
|
||||||
|
|
||||||
|
logn "Starting VMID ${vmid}..."
|
||||||
|
pve "qm start ${vmid}"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 6: Wait for SSH ─────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
if ! $SKIP_BOOT_WAIT; then
|
||||||
|
log "Phase 6: Waiting for both nodes to come up on LAN IPs"
|
||||||
|
wait_for_ssh "$NODE1_LAN_IP" "$NODE1_HOST"
|
||||||
|
wait_for_ssh "$NODE2_LAN_IP" "$NODE2_HOST"
|
||||||
|
logn "Both nodes are SSHable."
|
||||||
|
sleep 10 # let systemd finish activation
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 7: Refresh sops host-key registrations ─────────────────────────────
|
||||||
|
#
|
||||||
|
# Disko builds raw disk images: each new VM boots with a freshly-generated SSH
|
||||||
|
# host key, not the one pre-seeded in clan vars. Scan the running VMs; if
|
||||||
|
# their ed25519 keys differ from the clan var, update the clan var, rewrite
|
||||||
|
# the .sops.yaml anchor, and re-encrypt all affected sops files.
|
||||||
|
|
||||||
|
if ! $SKIP_REFRESH_SOPS_KEYS; then
|
||||||
|
if $DRY_RUN; then
|
||||||
|
logn "[dry-run] Would scan VM host keys and refresh .sops.yaml / secrets if needed"
|
||||||
|
else
|
||||||
|
log "Phase 7: Refreshing sops host-key registrations (disko key drift fix)"
|
||||||
|
SOPS_UPDATED=false
|
||||||
|
|
||||||
|
for spec in \
|
||||||
|
"${NODE1_LAN_IP}:proxmox-ha-docker-1:${NODE1_HOST}" \
|
||||||
|
"${NODE2_LAN_IP}:proxmox-ha-docker-2:${NODE2_HOST}"; do
|
||||||
|
IFS=: read -r node_ip flake_target host_name <<< "$spec"
|
||||||
|
CLAN_PUB="${REPO_ROOT}/vars/per-machine/${flake_target}/openssh/ssh_host_ed25519_key.pub/value"
|
||||||
|
|
||||||
|
logn "Scanning ed25519 host key from ${host_name} (${node_ip})..."
|
||||||
|
RAW=$(ssh-keyscan -t ed25519 "${node_ip}" 2>/dev/null | grep -v "^#") || true
|
||||||
|
if [[ -z "$RAW" ]]; then
|
||||||
|
logn "WARNING: no ed25519 key returned for ${node_ip} — skipping"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
SCANNED_TYPE=$(awk '{print $2}' <<< "$RAW")
|
||||||
|
SCANNED_KEY=$(awk '{print $3}' <<< "$RAW")
|
||||||
|
SCANNED_PUBKEY="${SCANNED_TYPE} ${SCANNED_KEY} ${host_name}"
|
||||||
|
|
||||||
|
CURRENT=$(tr -d '\n' < "$CLAN_PUB" 2>/dev/null || true)
|
||||||
|
if [[ "$SCANNED_PUBKEY" == "$CURRENT" ]]; then
|
||||||
|
logn "${host_name}: clan var matches running key — no update needed"
|
||||||
|
continue
|
||||||
|
fi
|
||||||
|
|
||||||
|
logn "${host_name}: key drift detected — updating clan var"
|
||||||
|
logn " old: ${CURRENT}"
|
||||||
|
logn " new: ${SCANNED_PUBKEY}"
|
||||||
|
echo "$SCANNED_PUBKEY" > "$CLAN_PUB"
|
||||||
|
SOPS_UPDATED=true
|
||||||
|
|
||||||
|
ANCHOR="${flake_target}"
|
||||||
|
NEW_AGE=$(echo "$SCANNED_PUBKEY" | \
|
||||||
|
nix run --quiet --no-warn-dirty nixpkgs#ssh-to-age 2>/dev/null)
|
||||||
|
[[ -z "$NEW_AGE" ]] && err "ssh-to-age produced no output for ${host_name}"
|
||||||
|
logn " new age key: ${NEW_AGE}"
|
||||||
|
sed -i "/&${ANCHOR} /s| age[a-z0-9]*$| ${NEW_AGE}|" "${REPO_ROOT}/.sops.yaml"
|
||||||
|
done
|
||||||
|
|
||||||
|
if $SOPS_UPDATED; then
|
||||||
|
logn "Running sops updatekeys on affected secrets..."
|
||||||
|
SOPS="nix run --quiet --no-warn-dirty nixpkgs#sops --"
|
||||||
|
(cd "${REPO_ROOT}" && \
|
||||||
|
$SOPS updatekeys -y secrets/common.yaml && \
|
||||||
|
$SOPS updatekeys -y secrets/ha-docker-1.keytab && \
|
||||||
|
$SOPS updatekeys -y secrets/ha-docker-2.keytab)
|
||||||
|
|
||||||
|
logn "Committing refreshed host keys and re-encrypted secrets..."
|
||||||
|
(cd "${REPO_ROOT}" && \
|
||||||
|
git add \
|
||||||
|
vars/per-machine/proxmox-ha-docker-1/openssh/ssh_host_ed25519_key.pub/value \
|
||||||
|
vars/per-machine/proxmox-ha-docker-2/openssh/ssh_host_ed25519_key.pub/value \
|
||||||
|
.sops.yaml \
|
||||||
|
secrets/common.yaml \
|
||||||
|
secrets/ha-docker-1.keytab \
|
||||||
|
secrets/ha-docker-2.keytab && \
|
||||||
|
git commit -m "secrets(ha-docker): refresh sops host-key registrations for new VM instances" || true)
|
||||||
|
logn "Sops keys refreshed and committed."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 8: Initialise Docker Swarm ─────────────────────────────────────────
|
||||||
|
|
||||||
|
if ! $SKIP_INIT_SWARM; then
|
||||||
|
log "Phase 8: Initialising Docker Swarm"
|
||||||
|
|
||||||
|
if $DRY_RUN; then
|
||||||
|
logn "[dry-run] Would run: docker swarm init --advertise-addr ${NODE1_SWARM_IP} --data-path-addr ${NODE1_SWARM_IP} on ${NODE1_HOST}"
|
||||||
|
logn "[dry-run] Would join ${NODE2_HOST} as manager"
|
||||||
|
logn "[dry-run] Would label both nodes"
|
||||||
|
else
|
||||||
|
# Check if node1 is already a swarm manager.
|
||||||
|
if n1 "docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null | grep -q "active"; then
|
||||||
|
logn "${NODE1_HOST} is already in a swarm — skipping init."
|
||||||
|
else
|
||||||
|
logn "Initialising swarm on ${NODE1_HOST} (advertise: ${NODE1_SWARM_IP})..."
|
||||||
|
n1 "docker swarm init \
|
||||||
|
--advertise-addr ${NODE1_SWARM_IP} \
|
||||||
|
--data-path-addr ${NODE1_SWARM_IP}"
|
||||||
|
logn "Swarm initialised on ${NODE1_HOST}."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Check if node2 is already joined.
|
||||||
|
if n2 "docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null | grep -q "active"; then
|
||||||
|
logn "${NODE2_HOST} is already in the swarm — skipping join."
|
||||||
|
else
|
||||||
|
logn "Fetching manager join token from ${NODE1_HOST}..."
|
||||||
|
JOIN_TOKEN=$(n1 "docker swarm join-token manager -q")
|
||||||
|
[[ -z "$JOIN_TOKEN" ]] && err "Failed to get swarm manager join token from ${NODE1_HOST}"
|
||||||
|
|
||||||
|
logn "Joining ${NODE2_HOST} as manager (advertise: ${NODE2_SWARM_IP})..."
|
||||||
|
n2 "docker swarm join \
|
||||||
|
--token ${JOIN_TOKEN} \
|
||||||
|
--advertise-addr ${NODE2_SWARM_IP} \
|
||||||
|
--data-path-addr ${NODE2_SWARM_IP} \
|
||||||
|
${NODE1_SWARM_IP}:2377"
|
||||||
|
logn "${NODE2_HOST} joined as manager."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Label nodes for service placement constraints.
|
||||||
|
logn "Labelling swarm nodes..."
|
||||||
|
n1 "docker node update --label-add node=${NODE1_HOST} ${NODE1_HOST}" || true
|
||||||
|
n1 "docker node update --label-add node=${NODE2_HOST} ${NODE2_HOST}" || true
|
||||||
|
logn "Labels applied."
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 9: DNS registration ─────────────────────────────────────────────────
|
||||||
|
|
||||||
|
if ! $SKIP_DNS; then
|
||||||
|
log "Phase 9: Registering DNS records in FreeIPA"
|
||||||
|
|
||||||
|
if $DRY_RUN; then
|
||||||
|
logn "[dry-run] Would create/verify ${SWARM_ZONE} zone and add A records"
|
||||||
|
else
|
||||||
|
# Check for and create the swarm.home zone if absent.
|
||||||
|
if ! dc "ipa dnszone-show ${SWARM_ZONE}" >/dev/null 2>&1; then
|
||||||
|
logn "Creating ${SWARM_ZONE} DNS zone..."
|
||||||
|
dc "ipa dnszone-add ${SWARM_ZONE} \
|
||||||
|
--name-server=${DC_HOST}. \
|
||||||
|
--admin-email=hostmaster@${SWARM_ZONE}"
|
||||||
|
# Reverse zone for 192.168.30.x
|
||||||
|
dc "ipa dnszone-add 30.168.192.in-addr.arpa \
|
||||||
|
--name-server=${DC_HOST}. \
|
||||||
|
--admin-email=hostmaster@${SWARM_ZONE}" 2>/dev/null || \
|
||||||
|
logn " (reverse zone 30.168.192.in-addr.arpa already exists or skipped)"
|
||||||
|
else
|
||||||
|
logn "${SWARM_ZONE} zone already exists."
|
||||||
|
fi
|
||||||
|
|
||||||
|
# storage.home A records (zone already exists from HA cluster setup).
|
||||||
|
for spec in "${NODE1_HOST}:${NODE1_STORAGE_IP}" "${NODE2_HOST}:${NODE2_STORAGE_IP}"; do
|
||||||
|
IFS=: read -r hostname ip <<< "$spec"
|
||||||
|
logn "Adding ${hostname}.${STORAGE_ZONE} → ${ip}"
|
||||||
|
dc "ipa dnsrecord-add ${STORAGE_ZONE} ${hostname} --a-rec=${ip} --a-create-reverse" 2>/dev/null || \
|
||||||
|
logn " (record already exists or reverse zone missing — continuing)"
|
||||||
|
done
|
||||||
|
|
||||||
|
# swarm.home A records.
|
||||||
|
for spec in "${NODE1_HOST}:${NODE1_SWARM_IP}" "${NODE2_HOST}:${NODE2_SWARM_IP}"; do
|
||||||
|
IFS=: read -r hostname ip <<< "$spec"
|
||||||
|
logn "Adding ${hostname}.${SWARM_ZONE} → ${ip}"
|
||||||
|
dc "ipa dnsrecord-add ${SWARM_ZONE} ${hostname} --a-rec=${ip} --a-create-reverse" 2>/dev/null || \
|
||||||
|
logn " (record already exists — continuing)"
|
||||||
|
done
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
# ── Phase 10: Verify ──────────────────────────────────────────────────────────
|
||||||
|
|
||||||
|
if ! $SKIP_VERIFY; then
|
||||||
|
log "Phase 10: Verifying swarm health"
|
||||||
|
|
||||||
|
if $DRY_RUN; then
|
||||||
|
logn "[dry-run] Would verify swarm node list and NFS mounts"
|
||||||
|
else
|
||||||
|
logn "Swarm node list:"
|
||||||
|
n1 "docker node ls" || err "docker node ls failed on ${NODE1_HOST}"
|
||||||
|
|
||||||
|
logn "Checking swarm state on both nodes..."
|
||||||
|
for spec in "${NODE1_LAN_IP}:${NODE1_HOST}" "${NODE2_LAN_IP}:${NODE2_HOST}"; do
|
||||||
|
IFS=: read -r ip hostname <<< "$spec"
|
||||||
|
STATE=$(ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
|
||||||
|
"${SWARM_USER}@${ip}" "sudo docker info --format '{{.Swarm.LocalNodeState}}'" 2>/dev/null)
|
||||||
|
if [[ "$STATE" != "active" ]]; then
|
||||||
|
err "${hostname} swarm state is '${STATE}', expected 'active'"
|
||||||
|
fi
|
||||||
|
logn " ${hostname}: swarm=${STATE} ✓"
|
||||||
|
done
|
||||||
|
|
||||||
|
logn "Checking NFS mounts on both nodes..."
|
||||||
|
for spec in "${NODE1_LAN_IP}:${NODE1_HOST}" "${NODE2_LAN_IP}:${NODE2_HOST}"; do
|
||||||
|
IFS=: read -r ip hostname <<< "$spec"
|
||||||
|
NFS_OK=$(ssh -i ~/.ssh/id_ed25519 -o StrictHostKeyChecking=no \
|
||||||
|
"${SWARM_USER}@${ip}" "df -h /mnt/docker/config 2>/dev/null | grep -c nfs || echo 0" 2>/dev/null)
|
||||||
|
if [[ "$NFS_OK" -ge 1 ]]; then
|
||||||
|
logn " ${hostname}: /mnt/docker/config NFS mount ✓"
|
||||||
|
else
|
||||||
|
logn " WARNING: ${hostname}: /mnt/docker/config does not appear to be NFS-mounted"
|
||||||
|
logn " (automount may still be pending — try: ssh nixos@${ip} 'ls /mnt/docker/config')"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
logn "Checking overlay network..."
|
||||||
|
NETWORKS=$(n1 "docker network ls --filter driver=overlay --format '{{.Name}}'")
|
||||||
|
if echo "$NETWORKS" | grep -q "ingress"; then
|
||||||
|
logn " ingress overlay network present ✓"
|
||||||
|
else
|
||||||
|
logn " WARNING: ingress overlay network not found — swarm may not be fully initialised"
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
|
||||||
|
log "Deploy complete. Both nodes are ready for 'docker stack deploy'."
|
||||||
|
log "Connect to either manager:"
|
||||||
|
log " ssh nixos@${NODE1_LAN_IP} (${NODE1_HOST})"
|
||||||
|
log " ssh nixos@${NODE2_LAN_IP} (${NODE2_HOST})"
|
||||||
@@ -188,6 +188,46 @@ rec {
|
|||||||
# the data disk; drive-scsi0 is the OS disk.
|
# the data disk; drive-scsi0 is the OS disk.
|
||||||
haServerDrbdDisk = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1";
|
haServerDrbdDisk = "/dev/disk/by-id/scsi-0QEMU_QEMU_HARDDISK_drive-scsi1";
|
||||||
|
|
||||||
|
# ── Docker Swarm cluster ──────────────────────────────────────────────────
|
||||||
|
#
|
||||||
|
# Three network segments, all internal to pve1:
|
||||||
|
# LAN vmbr0 192.168.2.0/24 — management; SSH + external service traffic
|
||||||
|
# Storage-client vmbr2 192.168.20.0/24 — NFS from HA cluster VIP (shared with HA nodes)
|
||||||
|
# Swarm cluster vmbr3 192.168.30.0/24 — Docker Swarm gossip + VXLAN overlay
|
||||||
|
#
|
||||||
|
# Host octet consistent across subnets: node1 = .230, node2 = .231.
|
||||||
|
# IPs from the .230–.239 expansion buffer documented in docs/ip-addressing.md.
|
||||||
|
#
|
||||||
|
# Docker Swarm uses --advertise-addr and --data-path-addr on the swarm NIC
|
||||||
|
# (ens20/vmbr3) so all inter-node cluster traffic stays on the isolated
|
||||||
|
# internal bridge and never crosses the LAN.
|
||||||
|
#
|
||||||
|
# When expanding to a second Proxmox node, vmbr3 (VLAN 30) and vmbr1
|
||||||
|
# (VLAN 10) share the same inter-node trunk NIC via VLAN tagging — same
|
||||||
|
# physical wire, different VLAN IDs.
|
||||||
|
|
||||||
|
haDocker1Host = "ha-docker-1";
|
||||||
|
haDocker2Host = "ha-docker-2";
|
||||||
|
|
||||||
|
haDocker1Ip = "192.168.2.230"; # LAN management NIC (ens18, vmbr0)
|
||||||
|
haDocker2Ip = "192.168.2.231";
|
||||||
|
|
||||||
|
haDocker1StorageIp = "192.168.20.230"; # storage-client NIC (ens19, vmbr2)
|
||||||
|
haDocker2StorageIp = "192.168.20.231";
|
||||||
|
|
||||||
|
haDocker1SwarmIp = "192.168.30.230"; # swarm cluster NIC (ens20, vmbr3)
|
||||||
|
haDocker2SwarmIp = "192.168.30.231";
|
||||||
|
|
||||||
|
haDockerSwarmCidr = "192.168.30.0/24";
|
||||||
|
haDockerSwarmPrefixLength = 24;
|
||||||
|
|
||||||
|
# NIC names for ha-docker VMs. ens19/ens20 occupy the same guest bus
|
||||||
|
# positions as vmStorageInterface/vmStorageClientInterface on ha-server VMs
|
||||||
|
# but are attached to different bridges — storage (vmbr2) and swarm (vmbr3)
|
||||||
|
# respectively. Kept as named variables to avoid bare literals in modules.
|
||||||
|
haDockerStorageInterface = "ens19"; # vmbr2 — NFS client
|
||||||
|
haDockerSwarmInterface = "ens20"; # vmbr3 — Docker Swarm gossip + VXLAN
|
||||||
|
|
||||||
# ── Storage / NFS ─────────────────────────────────────────────────────────
|
# ── Storage / NFS ─────────────────────────────────────────────────────────
|
||||||
|
|
||||||
# NFS share definitions — used by ha-server.nix (exports), docker/mount-data.nix,
|
# NFS share definitions — used by ha-server.nix (exports), docker/mount-data.nix,
|
||||||
@@ -263,6 +303,13 @@ rec {
|
|||||||
dockerHttps = 443;
|
dockerHttps = 443;
|
||||||
dockerExtra = 8080;
|
dockerExtra = 8080;
|
||||||
|
|
||||||
|
# Docker Swarm inter-node ports (modules/build-types/ha-docker.nix).
|
||||||
|
# Firewalled to haDockerSwarmCidr only — vmbr3 is an isolated bridge
|
||||||
|
# with no physical uplink, so these ports are unreachable from LAN.
|
||||||
|
dockerSwarmMgmt = 2377; # TCP — Raft consensus + cluster management
|
||||||
|
dockerSwarmDisc = 7946; # TCP+UDP — Serf gossip (container network discovery)
|
||||||
|
dockerSwarmVxlan = 4789; # UDP — VXLAN overlay data path
|
||||||
|
|
||||||
# Beszel monitoring hub on docker.sweet.home, reached by every agent
|
# Beszel monitoring hub on docker.sweet.home, reached by every agent
|
||||||
# (modules/beszel/enable-agent.nix, hosts/nixos/home.nix)
|
# (modules/beszel/enable-agent.nix, hosts/nixos/home.nix)
|
||||||
beszelHub = 8090;
|
beszelHub = 8090;
|
||||||
|
|||||||
Reference in New Issue
Block a user