Archived
Add Docker Swarm HA cluster: ha-docker-1 and ha-docker-2
Two new NixOS Proxmox VMs (VMIDs 202/203) forming a dual-manager Docker
Swarm on dedicated vmbr3 (192.168.30.0/24, VLAN 30) for gossip and VXLAN,
with NFS via the storage-client network (vmbr2) from the existing HA cluster.
- nixos/variables.nix: add ha-docker IP/interface/port vars and swarm CIDR
- nixos/modules/build-types/ha-docker.nix: new build type — Docker 29,
NFS mounts, beszel-agent, health monitoring, swarm firewall rules with
checkReversePath = "loose" for VXLAN routing mesh
- nixos/hosts/ha-docker-{1,2}/host.nix: per-host identity — three NICs
(LAN, storage, swarm), IPA dyndns pinned to LAN interface
- nixos/flake.nix: add proxmox-ha-docker-{1,2} targets; build-validated
with nix build --dry-run (169 derivations, no errors)
- nixos/docs/ip-addressing.md: document VLAN 30 / swarm.home zone,
ha-docker IP allocations across all three subnets
- nixos/scripts/docker-swarm/deploy.sh: 10-phase lifecycle script
(bridge, keys, IPA, VMs, swarm init, DNS, verify); modelled on
scripts/ha/deploy.sh with --destroy mode
- nixos/docs/internal/docker-swarm-cutover.md: service-by-service
migration guide covering Traefik log rotation, Nextcloud cron sidecar,
docker-health-to-gotify swarm awareness updates, Passbolt/Gitea steps,
DNS cutover, and CT 105 decommission checklist
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
This commit is contained in:
@@ -0,0 +1,84 @@
|
||||
# Docker Swarm node build type.
|
||||
#
|
||||
# Produces NixOS hosts that form a Docker Swarm manager cluster. Two nodes
|
||||
# (ha-docker-1, ha-docker-2) are both managers so either can accept Docker
|
||||
# API and `docker stack` commands.
|
||||
#
|
||||
# Key differences from the existing `docker` build type (used by CT 105):
|
||||
# - nextcloud-cron-job.nix is EXCLUDED — `docker exec` breaks in swarm
|
||||
# because the target container may be on the other node. The cron job
|
||||
# is replaced by a nextcloud-cron sidecar in the Nextcloud stack.
|
||||
# See docs/internal/docker-swarm-cutover.md.
|
||||
# - traefik/rotate-logs.nix is EXCLUDED — log rotation moves to Docker's
|
||||
# json-file log driver (max-size/max-file on the Traefik service
|
||||
# definition). See docs/internal/docker-swarm-cutover.md.
|
||||
# - raspi/mount-data.nix is EXCLUDED — specific to CT 105's backup role.
|
||||
# - Swarm firewall ports (2377/tcp, 7946/tcp+udp, 4789/udp) are opened
|
||||
# on the swarm NIC (ens20/vmbr3) only.
|
||||
# - checkReversePath = "loose" is required for the Swarm ingress routing
|
||||
# mesh: VXLAN return traffic is asymmetric (arrives ens20, exits ens18).
|
||||
# - beszel-agent is enabled for host-level monitoring.
|
||||
{ pkgs, vars, ... }:
|
||||
|
||||
{
|
||||
# Pin Docker Engine to version 29, matching CT 105, so image layers cached
|
||||
# on NFS volumes remain compatible across old and new hosts.
|
||||
nixpkgs.overlays = [
|
||||
(final: prev: {
|
||||
docker = prev.docker_29;
|
||||
docker_cli = prev.docker_29;
|
||||
})
|
||||
];
|
||||
|
||||
imports = [
|
||||
../docker/enable-service.nix
|
||||
../docker/mount-data.nix
|
||||
../docker/docker-health-to-gotify.nix
|
||||
../beszel/enable-agent.nix
|
||||
../services/enable-rpcbind.nix
|
||||
];
|
||||
|
||||
environment.systemPackages = with pkgs; [
|
||||
nfs-utils
|
||||
];
|
||||
|
||||
boot.supportedFilesystems = [ "nfs" ];
|
||||
|
||||
systemd.tmpfiles.rules = [
|
||||
# Symlink ~/docker → NFS config mount so the docker-health-to-gotify
|
||||
# script (and operator convenience) resolves ~/docker/... correctly.
|
||||
"L+ /home/${vars.primaryUser}/docker - - - - ${vars.nfsShares.dockerConfig.mountpoint}"
|
||||
"d /mnt/docker 0755 ${vars.primaryUser} users -"
|
||||
];
|
||||
|
||||
users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
|
||||
|
||||
networking.firewall = {
|
||||
# LAN-facing service ports — same as the existing docker build type.
|
||||
allowedTCPPorts = [
|
||||
vars.ports.dockerHttp
|
||||
vars.ports.dockerHttps
|
||||
vars.ports.dockerExtra
|
||||
vars.ports.beszelHub
|
||||
];
|
||||
|
||||
# Swarm inter-node ports restricted to the swarm NIC (ens20/vmbr3).
|
||||
# vmbr3 is an isolated internal bridge — no LAN reachability.
|
||||
interfaces.${vars.haDockerSwarmInterface} = {
|
||||
allowedTCPPorts = [
|
||||
vars.ports.dockerSwarmMgmt # 2377 — Raft + cluster management
|
||||
vars.ports.dockerSwarmDisc # 7946 — Serf gossip (TCP half)
|
||||
];
|
||||
allowedUDPPorts = [
|
||||
vars.ports.dockerSwarmDisc # 7946 — Serf gossip (UDP half)
|
||||
vars.ports.dockerSwarmVxlan # 4789 — VXLAN overlay data path
|
||||
];
|
||||
};
|
||||
|
||||
# Docker Swarm ingress routing mesh creates asymmetric routes: a request
|
||||
# arrives on ens18 (LAN) for a container that lives on ens20's VXLAN
|
||||
# overlay; the return path differs from the incoming interface. Strict
|
||||
# rp_filter drops these packets. "loose" allows them.
|
||||
checkReversePath = "loose";
|
||||
};
|
||||
}
|
||||
Reference in New Issue
Block a user