Adds a DNS Zones section to ip-addressing.md covering the three FreeIPA
zones (sweet.home / cluster.home / storage.home) with all A records and PTR
records as configured. Updates Stage 6 in network-cutover.md to reference
nfs.storage.home by DNS name rather than the raw VIP IP, and marks DNS as a
completed prerequisite for the HA storage cutover.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
ip-addressing.md:
- Storage-client network purpose updated: NFS primary, iSCSI available
- Protocol separation box updated: NFS on both VIPs (each CIDR-restricted);
iSCSI on VLAN 20 only but not in active use
- docker role updated to NFS client (not iSCSI initiator)
- vip-storage description updated to NFS + iSCSI endpoint
ha-network-audit.md:
- Implementation status header added noting all audit items resolved
- Records decisions that diverged from original recommendations
(VLAN renumbering, dual VIP, NFS-over-iSCSI for docker/swarm)
- Notes deferred items (STONITH, iSCSI ACLs)
network-cutover.md:
- New Stage 6: HA storage cutover — docker NFS remount from
server.sweet.home to vip-storage (192.168.20.229), final rsync
steps, and server.sweet.home decommission procedure
- Former Stage 6 (final cleanup) renumbered to Stage 7
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
vip-storage (192.168.20.229) now serves NFS as well as iSCSI, firewalled
to haClientCidr (192.168.20.0/24) only. This allows docker and future
swarm nodes to NFS-mount shared volumes from the storage network rather
than the LAN, keeping storage traffic off vmbr0.
LAN NFS (vip-lan 192.168.2.229) remains for pxe-boot and other LAN
clients. Protocol and subnet boundaries enforced by firewall on both nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Storage VIP (192.168.20.229) is on vmbr2 — an internal-only bridge with no
physical uplink, unreachable from the LAN. Test it via SSH to the active node
(which has VLAN 20 connectivity) instead of directly. LAN VIP (192.168.2.229)
continues to be tested directly from wherever the script runs.
Also adds VIP_LAN variable to config section.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Renumber internal HA bridges so VLAN ID matches IP third octet:
vmbr1 (cluster): 192.168.4.x → 192.168.10.x (VLAN 10, /29)
vmbr2 (storage): 192.168.5.x → 192.168.20.x (VLAN 20, /24)
Enforce protocol separation at the firewall on both HA nodes:
- iSCSI (3260/tcp): storage-client subnet (192.168.20.0/24) only
- NFS (111,2049,20048 tcp+udp): LAN subnet (192.168.2.0/24) only
- Cluster subnet (192.168.10.224/29) accepted wholesale (DRBD+Corosync)
Moves from allowedTCPPorts/allowedUDPPorts to source-restricted extraCommands
rules on the nixos-fw chain.
NFS exports reduced to LAN-only (drop haClientCidr lines from mkNfsExports).
Storage-client network clients use iSCSI only; no NFS on VLAN 20.
All HA script VIP/storage IP defaults updated to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
mkTarget computed flakeTarget as "${platform}-${buildType}" which gives
"proxmox-ha-server" for both nodes, but the actual nixosConfigurations
keys are "proxmox-ha-server-1" / "proxmox-ha-server-2". This caused
Switch-nix to fail immediately with an "unknown flake attribute" error.
Add an optional nameSuffix parameter to mkTarget and pass "-1" / "-2"
for the ha-server entries so /etc/flake-target matches the real
attribute name on each node.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Both VIPs fail over together in ha-group:
vip-lan: 192.168.2.229/24 on ens18 (vmbr0) — LAN iSCSI + NFS access
vip-storage: 192.168.5.229/24 on ens20 (vmbr2) — storage-client iSCSI + NFS
variables.nix: haServerLanVip = "192.168.2.229"
cluster-init.sh: both VIP primitives in ha-group XML; VIP_LAN var added
ip-addressing.md: both VIP entries documented
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Infrastructure changes already applied to pve1:
- vmbr2 internal bridge created (192.168.5.0/24, no physical uplink)
- VM 200 (ha-server-1): net2 added → vmbr2 (ens20)
- VM 201 (ha-server-2): net2 added → vmbr2 (ens20)
- CT 105 (docker): net1 added → vmbr2 (eth1)
- VM 101 (server): net1 added → vmbr2 (ens19) — needs reboot to activate
NixOS config (deploy to ha nodes to complete; docker/server at cutover):
- ha-server-{1,2}/host.nix: ens20 with 192.168.5.{228,227}/24
- docker/host.nix: eth1 with 192.168.5.225/24
- server/host.nix: ens19 with 192.168.5.226/24
- cluster-config.nix: corosync ring1 on LAN IPs as backup heartbeat path
- cluster-config.nix: allow haClientCidr (192.168.5.0/24) in iptables
- ha-server.nix: NFS exports now allow both lanCidr and haClientCidr
- VIP moves from 192.168.2.229 (vmbr0/LAN) to 192.168.5.229 (vmbr2)
- iSCSI portal to be rebound from [::0] to 192.168.5.229 at cutover
variables.nix: haStorageCidr corrected to 192.168.4.224/29; new vars:
vmStorageClientInterface, lxcStorageInterface, haServer{1,2}ClientIp,
dockerStorageIp, serverStorageIp, haClientCidr/PrefixLength; haServerVip
updated to 192.168.5.229.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Read-only discovery of pve1 bridges, HA cluster VMs (200/201), and
Docker CT (105) — current state, risk flags, recommended vmbr2 storage
bridge layout, gap list, and migration notes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pve1 runs non-NixOS guests (FreeIPA domain-controller, Proxmox Data Manager, etc.)
alongside NixOS ones. Validate each discovered hostname against the set of
hostnames defined in nixosConfigurations before adding it to the gc list.
Brings back the flake eval but uses it correctly: extract all hostname values
(not target-name keys) to build a filter set, then only include pve1 guests
whose name matches a flake-managed NixOS host.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The by-id lookup is read-only so it's safe to run in dry-run mode.
Previously it was gated behind `if ! $DRY_RUN`, which always triggered
the sdb fallback warning in dry-run — making it look like the device
path wasn't reliable when the symlink actually exists on both servers.
Now the lookup always runs and the script errors out with a clear message
if the by-id symlink is genuinely missing, instead of silently falling
back to a guessed /dev/sd* name.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The flake hostname map approach was wrong: create-proxmox-resource.sh names
every guest after its NixOS hostname (pct create --hostname <nixos-hostname>,
qm create --name <nixos-hostname>), so pct/qm list already output the final
hostname directly. Looking those names up in a map keyed by flake target names
(lxc-docker, proxmox-server, ...) found nothing and the dynamic list was
always empty.
Drop the flake eval entirely; use the pct/qm list output as-is, filtering
only nix-cache and already-seen entries.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
qm lives at /usr/sbin/qm, which is not in the default PATH for
non-interactive SSH sessions as a non-root user. Add PVE_SUDO (set to
"sudo" when PVE_SSH_USER != root, matching create-proxmox-resource.sh's
own sudo_prefix pattern) and prepend it to all three qm invocations in
the script.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two bugs:
1. Dynamic discovery used BatchMode=yes for the pct/qm list SSH call, which
silently suppressed any sudo password prompt and returned empty output.
Fix: split SSH_OPTS into SSH_OPTS (BatchMode, for gc) and SSH_QUERY_OPTS
(no BatchMode, for discovery) so sudo can prompt when needed.
2. pve1 gc used a non-login SSH session which doesn't source /etc/profile,
so nix-collect-garbage was never on PATH. The Nix installer registers
itself via /etc/profile.d/nix-daemon.sh which only runs in login shells.
Fix: use "bash -l -c 'nix-collect-garbage -d'" instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
BatchMode SSH sessions don't source /etc/profile on non-NixOS hosts, so
nix-collect-garbage isn't on PATH for the wayne user. Source the nix-daemon
profile script explicitly, matching the pattern in scripts/lib/nix-bootstrap.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the static host list with dynamic discovery: workstation (nixos)
and pve1 are hard-wired first and second; remaining hosts are discovered
on every run by SSHing to pve1, listing running VMs/containers via
pct/qm list, and resolving their NixOS hostnames from a single flake eval.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Runs nix-collect-garbage -d on all deployed NixOS hosts and pve1 in
parallel, skipping nix-cache to avoid evicting shared cache paths.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Generated /home/wayne/.ssh/id_ed25519 on pve1 (single-user Nix install
owned by wayne, no daemon — builds run as wayne not root). Adds the
public key to remoteBuilderAuthorizedKeys so nix-cache accepts it.
After deploying nix-cache, re-run configure-nix-cache-client.sh as
wayne on pve1 to wire in the builder line and known_hosts entry.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nfs-utils changed ConditionPathExists from !/etc/krb5.keytab to
/etc/krb5.keytab, so on IPA-joined hosts the service now starts instead
of skipping. Exports use standard auth (no sec=krb5) so the nfs/ Kerberos
principal is never provisioned and the service fails.
enable = false masks the unit so nfs-server's Wants= can't pull it in.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
KEY (hub's SSH public key) is now stored in variables.nix as beszelHubKey
and set by enable-agent.nix — no host file ever needs to touch it.
TOKEN was already universal; this removes the last per-host beszel config.
Importing enable-agent.nix in a build type is now the only step needed to
add a new host to beszel monitoring.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Five files contained only beszel-token and are deleted entirely.
nix-cache.yaml retains cache-priv-key; beszel-token removed from it.
The universal token in secrets/common.yaml replaces all of these.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace per-host host-token.nix imports with a single beszel-token secret
in secrets/common.yaml, wired once in enable-agent.nix. Host files now only
need services.beszel.agent.environment.KEY — no imports block required.
Delete modules/beszel/host-token.nix (no longer referenced anywhere).
Action needed: run `sops secrets/common.yaml` and add `beszel-token: <value>`
from the beszel hub UI before deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents how to wire beszel-agent into any flake build: the two-module
split (enable-agent.nix in build types, host-token.nix in host files),
sops secret creation, hub pairing flow, and optional EXTRA_FILESYSTEMS /
Docker monitoring knobs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Three-phase online resize: Proxmox qm resize → guest block device
rescan → drbdadm resize + xfs_growfs. No downtime required.
Detects active node via crm_mon, pre-checks DRBD UpToDate/UpToDate,
auto-discovers VMIDs from qm list, and resolves the block device name
from the stable scsi-0QEMU... by-id symlink with a slot-index fallback.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
crm_mon 2.x formats the Promoted line as " * Promoted: [ node ]" — the
asterisk bullet means the previous grep -E '^\s*(Promoted|Masters):' never
matched, so active-node detection silently returned empty in health.sh,
failover.sh, and acceptance-tests.sh.
Fix: pipe through grep -v Unpromoted first, then grep -E '(Promoted|Masters):'
without anchoring to start-of-line.
Also: remove the SSH_OPTS=-i ~/.ssh/... variable pattern in health.sh and
failover.sh; tilde is not expanded inside double-quoted strings, so $SSH_OPTS
was passing a literal ~ to SSH. Inline the key path in each function
definition instead (same as acceptance-tests.sh already did).
Also: drop the 2>/dev/null embedded in the crm_mon argument string — the
outer 2>/dev/null on the n1/n2 call already suppresses SSH stderr; the
embedded one was harmless but noisy to reason about.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
failover.sh: graceful active→standby resource migration with pre-checks
(DRBD UpToDate/UpToDate, quorum), spinner wait for XFS mount on target,
rollback on timeout, --to/--force/--dry-run/--timeout flags.
health.sh: read-only status panel — node reachability, quorum, DRBD
role/dstate/cs per node, Pacemaker resources, failure history, XFS mount
usage, and service port checks (NFS 2049, iSCSI 3260) via VIP.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
