Merge pull request 'chore: full sweep — docs sync, SSH key module extraction, NFS dedup, dead code removal' (#89) from worktree-full-sweep into main

Reviewed-on: #89

AGENTS.md

@@ -6,12 +6,14 @@ This repository contains flake-based NixOS configurations for Wayne's LAN
 servers and workstation.
 
 The flake exposes NixOS configurations named `<platform>-<buildtype>`
-(platforms: `linode`, `proxmox`, `lxc`; build types: `minimal`, `nix-cache`,
+(platforms: `linode`, `proxmox`, `lxc`, `baremetal`; build types: `minimal`,
-`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`, `tor-relay`), generated from `modules/platforms/*`
+`nix-cache`, `server`, `docker`, `gui`, `pxe-boot`, `tailscale-router`,
-and `modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not
+`tor-relay`, `ha-server`), generated from `modules/platforms/*` and
-every combination is built — `pxe-boot` has no `linode` variant. See
+`modules/build-types/*` by the `mkTarget` function in `flake.nix`. Not every
-`README.md` for the full current target list; treat `flake.nix` as the
+combination is built — `pxe-boot` has no `linode` variant, `ha-server` only
-source of truth since this list can drift.
+exists on `proxmox`, and `tor-relay` only exists on `lxc`. See `README.md`
+for the full current target list; treat `flake.nix` as the source of truth
+since this list can drift.
 
 Do not deploy, switch, reboot, repartition, format disks, or run destructive
 install commands from this repository unless explicitly asked.

README.md

@@ -10,13 +10,13 @@ pieces composed in `flake.nix`:
 
 - **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`, `baremetal`
 - **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
-  `gui`, `pxe-boot`, `tailscale-exit-node`, `tor-relay`
+  `gui`, `pxe-boot`, `tailscale-router`, `tor-relay`, `ha-server`
 
 Not every combination exists — `pxe-boot` has no `linode` variant, since
 PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have,
-`tor-relay` currently only exists as `lxc-tor-relay`, and `baremetal`
+`tor-relay` and `ha-server` currently only exist on `lxc`/`proxmox`, and
-currently only exists as `baremetal-gui` (the real gui-host hardware). The
+`baremetal` currently only exists as `baremetal-gui` (the real gui-host
-full list:
+hardware). The full list:
 
 | Target | Purpose |
 | --- | --- |
@@ -29,8 +29,9 @@ full list:
 | `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target |
 | `baremetal-gui` | Same Cinnamon desktop workstation, on the real gui-host hardware — ZFS RAID0 root, systemd-boot |
 | `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host — previously the flat `pxe-boot` target |
-| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node |
+| `linode-tailscale-router` / `proxmox-tailscale-router` / `lxc-tailscale-router` | Tailscale subnet router + MagicDNS forwarder for the LAN |
 | `lxc-tor-relay` | Tor middle relay |
+| `proxmox-ha-server-1` / `proxmox-ha-server-2` | HA file-server cluster nodes — DRBD + XFS + iSCSI + NFS, managed by Corosync + Pacemaker |
 
 Which variant of a given buildtype is actually deployed isn't tracked
 anywhere in this repo — that's live infrastructure state, not something a
@@ -47,8 +48,7 @@ section for which is which.
 
 Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
 (hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay
-fixed regardless of which platform it's built for — see
+fixed regardless of which platform it's built for. Every deployed host
-`flake-target-refactor-spec.md` for the full rationale. Every deployed host
 stamps its own active target name into `/etc/flake-target` at build time, so
 `nixos-rebuild switch --flake .#$(cat /etc/flake-target)` always picks up the
 right one even after a platform migration changes the flake attribute name.
@@ -167,7 +167,6 @@ per-boot host key for sops-nix to derive from on ephemeral media) — see
 (`vars/per-machine/<target>/openssh/`, committed and sops-encrypted) for
 their SSH host keys.
 
-This repository's git *history* still contains secrets committed before this
+This repository's git *history* still contains secrets committed before the
-migration (see `remove-sensetive-info-refactor.md`) — those are being
+sops-nix migration — those are being scrubbed and rotated separately; don't
-scrubbed and rotated separately; don't treat the repo as safe to make public
+treat the repo as safe to make public until that's finished.
-until that's finished.

docs/auto-installer.md

@@ -22,7 +22,7 @@ see "LXC hosts" immediately below for why those are different.**
 ## LXC hosts
 
 `lxc-*` targets (`lxc-minimal`, `lxc-nix-cache`, `lxc-server`, `lxc-docker`,
-`lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-exit-node`, `lxc-tor-relay`) are **not** installed via `auto-install.sh` — the
+`lxc-gui`, `lxc-pxe-boot`, `lxc-tailscale-router`, `lxc-tor-relay`) are **not** installed via `auto-install.sh` — the
 interactive menu deliberately excludes them. Don't try to select one there;
 `nixos-install` would bind-mount `/` onto `/mnt` (LXC containers have no raw
 disk to partition) and then refuse to touch the filesystem it's currently
@@ -133,13 +133,15 @@ Flake outputs:
 ```nix
 nixosConfigurations.installer   # ISO/netboot installer image
 
-packages.x86_64-linux.iso   # installer ISO/netboot image
+packages.x86_64-linux.iso       # installer ISO/netboot image
-packages.x86_64-linux.pxe   # netboot-ipxe + netboot-initrd + netboot-kernel, bundled
+packages.x86_64-linux.pxe       # auto-installer netboot bundle (kernel + initrd + ipxe script)
+packages.x86_64-linux.pxe-minimal  # vanilla NixOS minimal netboot bundle (no installer wiring)
 ```
 
 ```sh
 nix build .#iso
 nix build .#pxe
+nix build .#pxe-minimal
 ```
 
 There's no `nixosConfigurations.proxmox-lxc` (installer-boots-as-an-LXC-

hosts/nix-cache/host.nix

@@ -20,7 +20,6 @@
   };
 
   services.beszel.agent.environment = {
-    #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
     KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
   };
 

hosts/server/host.nix

@@ -21,7 +21,6 @@
   };
 
   services.beszel.agent.environment = {
-    #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
     KEY = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFPR9kwtC4TAeTRu46A7+opZsYpxqkRJ+x/ZyB2GWCeG";
     EXTRA_FILESYSTEMS = "${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath}";
     LOG_LEVEL = "debug";

modules/build-types/ha-server.nix

@@ -15,6 +15,16 @@
 #   under services.beszel.agent.environment.KEY once the hub accepts the
 #   new agents, following the pattern in hosts/server/host.nix.
 { lib, vars, ... }:
+
+let
+  # Generates /etc/exports lines for all nfsShares data entries. Shared
+  # pattern with modules/build-types/server.nix — both export the same
+  # set of shares, differing only in the storage root they serve from.
+  mkNfsExports = storageRoot:
+    lib.concatMapStrings
+      (share: "  ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n")
+      (lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares));
+in
 {
   imports = [
     ../ha/pacemaker-stack.nix
@@ -25,16 +35,7 @@
 
   services.nfs.server = {
     enable = true;
-    exports = ''
+    exports = mkNfsExports vars.haStorageRoot;
-      ${vars.haStorageRoot}/${vars.nfsShares.dockerConfig.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.haStorageRoot}/${vars.nfsShares.dockerVolumes.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.haStorageRoot}/${vars.nfsShares.dockerDatabases.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.haStorageRoot}/${vars.nfsShares.nextcloudData.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.haStorageRoot}/${vars.nfsShares.raspiVolumes.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.haStorageRoot}/${vars.nfsShares.proxmoxIsos.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.haStorageRoot}/${vars.nfsShares.proxmoxLxcImages.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.haStorageRoot}/${vars.nfsShares.pxebootImages.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-    '';
   };
 
   # Pacemaker controls nfs-server — prevent systemd from starting it at boot

modules/build-types/server.nix

@@ -13,6 +13,15 @@ let
     lib.concatMap (share: ancestors share.subpath)
       (lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares))
   );
+
+  # Generates /etc/exports lines for all nfsShares data entries (every
+  # attrset value — excludes the bare `options` string). Both server and
+  # ha-server export the same share set from different storage roots, so
+  # this helper is the single source of truth for the export line format.
+  mkNfsExports = storageRoot:
+    lib.concatMapStrings
+      (share: "  ${storageRoot}/${share.subpath} ${vars.lanCidr}${vars.nfsShares.options}\n")
+      (lib.filter builtins.isAttrs (lib.attrValues vars.nfsShares));
 in
 {
   imports = [
@@ -95,16 +104,7 @@ in
 
   services.nfs.server = {
     enable = true;
-    exports = ''
+    exports = mkNfsExports vars.storageRoot;
-      ${vars.storageRoot}/${vars.nfsShares.dockerConfig.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.storageRoot}/${vars.nfsShares.dockerVolumes.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.storageRoot}/${vars.nfsShares.dockerDatabases.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.storageRoot}/${vars.nfsShares.nextcloudData.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.storageRoot}/${vars.nfsShares.raspiVolumes.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.storageRoot}/${vars.nfsShares.proxmoxIsos.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.storageRoot}/${vars.nfsShares.proxmoxLxcImages.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-      ${vars.storageRoot}/${vars.nfsShares.pxebootImages.subpath} ${vars.lanCidr}${vars.nfsShares.options}
-    '';
   };
 
   # mountd (20048) is needed for showmount/NFSv3 mount protocol — without it

modules/common/configuration.nix

@@ -25,13 +25,10 @@ let
   '';
 in
 {
-  imports =
+  imports = [
-    [
+    ./set-locale.nix
-      # Include the results of the hardware scan.
+    ../ipa/client.nix
-      #      ./hardware-configuration.nix
+  ];
-      ./set-locale.nix
-      ../ipa/client.nix
-    ];
 
   # System-wide shell config so all users (including IPA accounts) get the
   # same management aliases as the local nixos user's Home Manager provides.
@@ -42,11 +39,7 @@ in
     };
     interactiveShellInit = buildImageFn;
   };
-  # Use the GRUB 2 boot loader.
+  networking.networkmanager.enable = true;
-  # boot.loader.grub.enable = true;
-  #boot.loader.grub.device = "/dev/sda"; # or "nodev" for efi only
-
-  networking.networkmanager.enable = true; # Easiest to use and most distros use this by default.
 
   # Recommended over the true default (bypasses ZFS's own import safeguards)
   # per the option's own docs; matches hosts/docker/host.nix and

modules/common/preserve-ssh-host-key.nix

@@ -0,0 +1,35 @@
+# Shared activation-script logic to preserve the SSH host key across
+# nixos-rebuild on platforms that embed the key via environment.etc (lxc and
+# proxmox).  When NIXOS_HOST_KEYS_DIR is not set the key is absent from
+# environment.etc, and NixOS's etc activation removes any /etc file not in
+# the new generation — which would destroy the live key and break sops-nix
+# decryption permanently.  These scripts save the key to /run before etc
+# removes it, then restore it afterward.
+#
+# Explicit deps enforce the correct ordering: without them the topological
+# sort places preserveSshHostKey after etc (confirmed live on lxc-tor-relay:
+# position 7 vs etc's position 5), so the key is gone before it can be saved.
+_: {
+  system.activationScripts = {
+    preserveSshHostKey = ''
+      if [ -f /etc/ssh/ssh_host_ed25519_key ]; then
+        cp /etc/ssh/ssh_host_ed25519_key /run/sshd-host-key-preserve.tmp
+        cp /etc/ssh/ssh_host_ed25519_key.pub /run/sshd-host-key-preserve.pub.tmp
+      fi
+    '';
+
+    restoreSshHostKey = {
+      deps = [ "etc" ];
+      text = ''
+        if [ ! -f /etc/ssh/ssh_host_ed25519_key ] && [ -f /run/sshd-host-key-preserve.tmp ]; then
+          install -m 0600 /run/sshd-host-key-preserve.tmp /etc/ssh/ssh_host_ed25519_key
+          install -m 0644 /run/sshd-host-key-preserve.pub.tmp /etc/ssh/ssh_host_ed25519_key.pub
+        fi
+        rm -f /run/sshd-host-key-preserve.tmp /run/sshd-host-key-preserve.pub.tmp
+      '';
+    };
+
+    etc = { deps = [ "preserveSshHostKey" ]; };
+    setupSecrets = { deps = [ "restoreSshHostKey" ]; };
+  };
+}

modules/docker/enable-service.nix

@@ -1,19 +1,9 @@
 { pkgs, vars, ... }:
 
 {
-  #  virtualisation.docker.enable = true;
   virtualisation.docker = {
     enable = true;
     package = pkgs.docker;
-    #  listenOptions = [
-    #  "unix:///var/run/docker.sock"
-    #  "tcp://0.0.0.0:2375"
-    #];
-
-    #  daemon.settings = {
-    #    metrics-addr = "0.0.0.0:9323";
-    #    experimental = true;
-    #  };
   };
   users.users.${vars.primaryUser}.extraGroups = [ "docker" ];
   environment.systemPackages = with pkgs; [

modules/platforms/lxc.nix

@@ -52,6 +52,7 @@ in
   # LXC container does).
   imports = [
     (modulesPath + "/virtualisation/proxmox-lxc.nix")
+    ../common/preserve-ssh-host-key.nix
   ];
 
   proxmoxLXC = {
@@ -105,49 +106,6 @@ in
     };
   };
 
-  # NixOS's etc activation removes any /etc file that was in the previous
-  # generation's environment.etc but is absent from the current one — even
-  # real (non-symlink) copies.  On every routine nixos-rebuild switch/test that
-  # lacks NIXOS_HOST_KEYS_DIR the key is absent from environment.etc, so it
-  # gets removed as "obsolete".  sops-nix derives its age decryption key from
-  # /etc/ssh/ssh_host_ed25519_key; deletion cascades into every sops secret
-  # failing with "Error getting data key: 0 successful groups required, got 0".
-  #
-  # Fix: activation scripts that bracket the etc step, with explicit deps
-  # to enforce the correct ordering. Without deps the topological sort places
-  # preserveSshHostKey AFTER etc (confirmed live on a deployed lxc-tor-relay:
-  # position 7 vs etc's position 5) -- the key is already gone by the time it
-  # tries to save it. The etc/setupSecrets entries ADD to existing deps
-  # (types.listOf concatenates across module definitions).
-  system.activationScripts = {
-    # Saves the live key to /run before etc can delete it.
-    preserveSshHostKey = ''
-      if [ -f /etc/ssh/ssh_host_ed25519_key ]; then
-        cp /etc/ssh/ssh_host_ed25519_key /run/sshd-host-key-preserve.tmp
-        cp /etc/ssh/ssh_host_ed25519_key.pub /run/sshd-host-key-preserve.pub.tmp
-      fi
-    '';
-
-    # Reinstalls the key after etc runs if it was removed as "obsolete".
-    # The resulting file is not registered in environment.etc for either
-    # generation, so subsequent rebuilds leave it alone permanently.
-    restoreSshHostKey = {
-      deps = [ "etc" ];
-      text = ''
-        if [ ! -f /etc/ssh/ssh_host_ed25519_key ] && [ -f /run/sshd-host-key-preserve.tmp ]; then
-          install -m 0600 /run/sshd-host-key-preserve.tmp /etc/ssh/ssh_host_ed25519_key
-          install -m 0644 /run/sshd-host-key-preserve.pub.tmp /etc/ssh/ssh_host_ed25519_key.pub
-        fi
-        rm -f /run/sshd-host-key-preserve.tmp /run/sshd-host-key-preserve.pub.tmp
-      '';
-    };
-
-    # Force etc to wait until the key is saved, and sops to wait until the
-    # key is restored. Without these the topological sort breaks the chain.
-    etc = { deps = [ "preserveSshHostKey" ]; };
-    setupSecrets = { deps = [ "restoreSshHostKey" ]; };
-  };
-
   # virtualisation/proxmox-lxc.nix (imported above) registers the Nix
   # store DB via a systemd service (register-nix-paths) -- it never runs
   # an activation script at all. Confirmed live this means neither

modules/platforms/proxmox.nix

@@ -34,6 +34,7 @@ in
     ../hardware-configuration/vm/proxmox.nix
     ../boot/efi.nix
     ../disko/proxmox.nix
+    ../common/preserve-ssh-host-key.nix
   ];
 
   environment.etc = lib.mkIf hasKeyForThisTarget {
@@ -46,36 +47,4 @@ in
       mode = "0644";
     };
   };
-
-  # NixOS's etc activation removes any /etc file that was in the previous
-  # generation's environment.etc but is absent from the current one. Since
-  # the SSH key is only in environment.etc during the --impure build (when
-  # NIXOS_HOST_KEYS_DIR is set), normal rebuilds would remove it as
-  # "obsolete". These scripts mirror lxc.nix's approach: save the live key
-  # before etc runs, restore it after. Without the explicit deps, the
-  # topological sort places preserveSshHostKey after etc (confirmed live on
-  # lxc-tor-relay: position 7 vs etc's position 5), so the key is gone
-  # before it can be saved.
-  system.activationScripts = {
-    preserveSshHostKey = ''
-      if [ -f /etc/ssh/ssh_host_ed25519_key ]; then
-        cp /etc/ssh/ssh_host_ed25519_key /run/sshd-host-key-preserve.tmp
-        cp /etc/ssh/ssh_host_ed25519_key.pub /run/sshd-host-key-preserve.pub.tmp
-      fi
-    '';
-
-    restoreSshHostKey = {
-      deps = [ "etc" ];
-      text = ''
-        if [ ! -f /etc/ssh/ssh_host_ed25519_key ] && [ -f /run/sshd-host-key-preserve.tmp ]; then
-          install -m 0600 /run/sshd-host-key-preserve.tmp /etc/ssh/ssh_host_ed25519_key
-          install -m 0644 /run/sshd-host-key-preserve.pub.tmp /etc/ssh/ssh_host_ed25519_key.pub
-        fi
-        rm -f /run/sshd-host-key-preserve.tmp /run/sshd-host-key-preserve.pub.tmp
-      '';
-    };
-
-    etc = { deps = [ "preserveSshHostKey" ]; };
-    setupSecrets = { deps = [ "restoreSshHostKey" ]; };
-  };
 }

variables.nix

@@ -16,7 +16,7 @@
   serverIp = "192.168.2.226"; # server (NFS/ZFS) Proxmox VM LAN IP
   dockerIp = "192.168.2.225"; # docker Proxmox VM LAN IP
   pbsIp = "192.168.2.244"; # Proxmox Backup Server LAN IP (not NixOS-managed)
-  domainControllerIp = "192.168.2.253"; # FreeIPA domain controller / primary DNS (not NixOS-managed)
+  domainControllerIp = "192.168.2.253"; # FreeIPA domain controller — authoritative DNS for sweet.home (not NixOS-managed)
   ipaServer = "domain-controller.sweet.home"; # FreeIPA server hostname (used by security.ipa and Kerberos; must be a resolvable FQDN, not an IP)
 
   # Cross-host references (LAN hostnames/users other hosts reach over the network)