beatzaplentyandClaude Sonnet 5
20f9475a7d
The reachability check used `cat < /dev/tcp/${NIX_CACHE_HOST}/22`, which
blocks forever reading for EOF that never comes -- sshd sends its banner
and then holds the connection open waiting for the client to speak next.
Every single check hit the 3s timeout and reported "unreachable"
unconditionally, regardless of whether the remote builder was actually up.
Confirmed live: a plain TCP connect (`exec 3<>/dev/tcp/...`, no read)
returns in ~60ms against a healthy nix-cache instead of always timing out.
Fixing that exposed a second, previously-dormant bug: `printf -v
NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"` on a genuinely empty NIX_OPTS array
still runs one format pass and yields the literal `'' ` rather than an
empty string. A subprocess (e.g. sync-host-keys.sh) reusing this
process's decision via `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` then rebuilt
a 1-element array holding an empty string instead of a 0-element array,
which broke `nix-shell "${NIX_OPTS[@]}" -p <pkg>` with a bogus positional
argument the moment NIX_OPTS was legitimately empty (nix-cache reachable)
-- something the first bug had made impossible to ever hit before.
Also adds a couple of retries (1s apart) to both checks as a secondary
safety net against genuine multi-second blips, on top of fixing the
checks themselves.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
NixOS LAN Configurations
Flake-based NixOS configuration repository for Wayne's LAN servers and workstation.
Hosts
Targets are named <platform>-<buildtype>, generated from two orthogonal
pieces composed in flake.nix:
- Platforms (what it runs on):
linode,proxmox,lxc - Build types (what it's for):
minimal,nix-cache,server,docker,gui,pxe-boot,tailscale-exit-node
Not every combination exists — pxe-boot has no linode variant, since
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have. The full
list:
| Target | Purpose |
|---|---|
linode-minimal |
Minimal NixOS host profile on a Linode VPS (real, deployed) |
proxmox-minimal |
Minimal NixOS host profile on Proxmox (real, deployed — previously the flat nix-minimal target) |
lxc-minimal |
Minimal NixOS host profile in a Proxmox LXC container |
linode-nix-cache / proxmox-nix-cache / lxc-nix-cache |
Local Nix binary cache and remote builder (proxmox-nix-cache is the real, deployed one — previously the flat nix-cache target) |
linode-server / proxmox-server / lxc-server |
Storage, NFS, backup, and monitoring exporter host (proxmox-server is the real, deployed one — previously the flat server target) |
linode-docker / proxmox-docker / lxc-docker |
Docker host for the main container stack (proxmox-docker is the real, deployed one — previously the flat docker target) |
linode-gui / proxmox-gui / lxc-gui |
Cinnamon desktop workstation (proxmox-gui is the real, deployed one — previously the flat nixos target) |
proxmox-pxe-boot / lxc-pxe-boot |
HTTP/iPXE boot asset host (proxmox-pxe-boot is the real, deployed one — previously the flat pxe-boot target) |
linode-tailscale-exit-node / proxmox-tailscale-exit-node / lxc-tailscale-exit-node |
Tailscale exit node (no deployed target yet; lxc-tailscale-exit-node is the one planned for actual use) |
The "(real, deployed)" targets above are also tracked machine-readably in
variables.nix's deployedTargets — keep both in sync when a deployment
changes. scripts/create-proxmox-resource.sh reads that list to refuse
creating a same-identity duplicate of an already-deployed host by accident.
Each buildtype's hosts/<name>/host.nix carries the per-machine identity
(hostname, hostId, per-machine secrets, system.stateVersion) that must stay
fixed regardless of which platform it's built for — see
flake-target-refactor-spec.md for the full rationale. Every deployed host
stamps its own active target name into /etc/flake-target at build time, so
nixos-rebuild switch --flake .#$(cat /etc/flake-target) always picks up the
right one even after a platform migration changes the flake attribute name.
List hosts with:
nix eval --json .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]'
Layout
| Path | Purpose |
|---|---|
flake.nix |
Flake inputs, the mkTarget platform × build-type generator, and nixosConfigurations outputs |
variables.nix |
Single source of truth for shared values (LAN domain/CIDR, hostnames, timezone, primary username, storage root, NFS share subpaths/mountpoints, service ports, ...) — passed to every module and Home Manager config as the vars argument via specialArgs/extraSpecialArgs |
hosts/<name>/host.nix |
Per-machine identity: hostname, hostId, per-machine secrets, system.stateVersion |
hosts/nixos/home.nix |
Workstation-specific Home Manager config (used by the gui build type) |
modules/platforms/ |
Platform-specific config: virtualisation guest tools, boot method, hardware config (linode.nix, proxmox.nix, lxc.nix) |
modules/build-types/ |
Build-type-specific config: what makes a system minimal/server/docker/gui/pxe-boot/nix-cache |
modules/common/ |
Shared NixOS config, Home Manager, aliases imported by every host |
modules/nix-cache/ |
Binary cache and remote builder client/server modules |
modules/installer/ |
Auto-installer environment (ISO, also served as PXE netboot) — see docs/auto-installer.md |
host-keys/ |
Gitignored, locally-generated SSH host keys for the auto-installer — see docs/auto-installer.md |
docs/ |
Operational notes for cache, builders, lock updates, boot services, the auto-installer, and Proxmox image builds |
scripts/ |
Codex setup, validation, host-key, release-bump, and Proxmox resource helpers |
Validation
Safe validation commands for Codex and local review:
bash scripts/codex-setup.sh
bash scripts/codex-maintenance.sh dry-run
bash scripts/codex-maintenance.sh
For individual host evaluation:
nix eval .#nixosConfigurations.<host>.config.system.build.toplevel.drvPath --raw
Use nix build --dry-run --no-link when build planning is needed. Do not run
deployment, install, disk formatting, mount, or reboot commands from automated
review sessions.
Operations
- Host rebuilds should consume the committed
flake.lock. - Routine dependency updates should happen through the flake lock automation
described in
docs/flake-lock-automation.md. nix-cacheserves substitutes over HTTP and can act as a remote builder for client hosts.pxe-bootserves iPXE boot files over HTTP from/srv/pxe.
Deploying a new host
Three different paths depending on target, none of them involving a manual
nixos-rebuild switch from this repo:
- Most hosts: boot the auto-installer, pick the target from its menu — see
docs/auto-installer.md. Every menu target has a Disko config the installer formats unconditionally (docs/auto-installer.md's "Storage" section covers how this stays non-destructive forlinode-*, whose disks Linode itself provisions ahead of time). lxc-*targets: not installed at all — build a ready-to-run container tarball andpct createit as a CT template directly.docs/auto-installer.mdcovers why (and the installer's menu excludes them for the same reason).proxmox-*targets: can alternatively be built as a standalone.rawdisk image and attached to a new VM with no install step — seedocs/proxmox-images.md.
scripts/create-proxmox-resource.sh --type lxc|vm --host <name> automates
either of the last two end to end (build, host-key registration, upload,
pct create/qm create), with --dry-run and a guard against duplicating
an already-deployed host's identity. See its --help.
Security Notes
Do not commit tokens, private keys, live credentials, or new password hashes
as plaintext. Secrets are managed with sops-nix:
encrypted files live under secrets/, recipients (per-host age keys derived
from each host's existing SSH host key, plus an admin key) are declared in
.sops.yaml. To add or edit a secret:
nix-shell -p sops --run "sops secrets/<file>.yaml"
then reference it from a module via config.sops.secrets."<name>".path
(or sops.templates for values that need to be embedded in a rendered
config file, e.g. nix.conf's access-tokens). Never write a secret value
directly into a tracked .nix file. A pre-commit hook (.githooks/,
enabled via git config core.hooksPath .githooks, done automatically by
scripts/codex-setup.sh) runs gitleaks protect --staged to catch mistakes
before they're committed.
The auto-installer environment is the one deliberate exception to
sops-nix-everywhere: it has a hardcoded login password instead (no stable
per-boot host key for sops-nix to derive from on ephemeral media) — see
"Host keys" in docs/auto-installer.md for why, and how the private keys it
does pre-seed for target hosts stay out of git via the gitignored
host-keys/ directory.
This repository's git history still contains secrets committed before this
migration (see remove-sensetive-info-refactor.md) — those are being
scrubbed and rotated separately; don't treat the repo as safe to make public
until that's finished.