Two new NixOS Proxmox VMs (VMIDs 202/203) forming a dual-manager Docker
Swarm on dedicated vmbr3 (192.168.30.0/24, VLAN 30) for gossip and VXLAN,
with NFS via the storage-client network (vmbr2) from the existing HA cluster.
- nixos/variables.nix: add ha-docker IP/interface/port vars and swarm CIDR
- nixos/modules/build-types/ha-docker.nix: new build type — Docker 29,
NFS mounts, beszel-agent, health monitoring, swarm firewall rules with
checkReversePath = "loose" for VXLAN routing mesh
- nixos/hosts/ha-docker-{1,2}/host.nix: per-host identity — three NICs
(LAN, storage, swarm), IPA dyndns pinned to LAN interface
- nixos/flake.nix: add proxmox-ha-docker-{1,2} targets; build-validated
with nix build --dry-run (169 derivations, no errors)
- nixos/docs/ip-addressing.md: document VLAN 30 / swarm.home zone,
ha-docker IP allocations across all three subnets
- nixos/scripts/docker-swarm/deploy.sh: 10-phase lifecycle script
(bridge, keys, IPA, VMs, swarm init, DNS, verify); modelled on
scripts/ha/deploy.sh with --destroy mode
- nixos/docs/internal/docker-swarm-cutover.md: service-by-service
migration guide covering Traefik log rotation, Nextcloud cron sidecar,
docker-health-to-gotify swarm awareness updates, Passbolt/Gitea steps,
DNS cutover, and CT 105 decommission checklist
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01DASH15okNvWeY1rVJmyJoJ
- variables.nix: switch to rec {}, extract giteaDomain/giteaRepoPath,
extraAdminSshKeys, haLanNfsFqdn, tailscaleResolverIp, ports.dhcp,
ports.dns; ipaServer now derives from homeDomain ref; section headers
- modules: use new vars throughout (pxe-boot, ts-dns-forwarder,
cluster-config, configuration.nix, mount-pxe-images) — eval unchanged
- docs: delete ephemeral planning docs (AUDIT_REPORT, ha-network-audit,
network-cutover); add docs/ha.md; drop migration reference table from
ip-addressing.md; remove stale server example from beszel.md
- CLAUDE.md/README.md/AGENTS.md: fix build types (tailscale-router,
ha-server, drop server); document scripts/ha/, scripts/ipa/, and
all previously undocumented top-level and lib scripts
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Removes all flake artifacts for the retired NFS/ZFS file server:
- flake.nix: drop linode-server, proxmox-server, lxc-server targets
- hosts/server/: delete host identity
- modules/build-types/server.nix: delete build type (ZFS pool + NFS exports)
- vars/per-machine/{linode,lxc,proxmox}-server/: delete clan vars
- secrets/server.keytab: delete IPA keytab
- .sops.yaml: remove server key anchors, strip from common.yaml recipients,
delete server.yaml and server.keytab creation rules
- variables.nix: remove serverIp, serverStorageIp (nfsServerHost/storageRoot
retained — pxe-boot/mount-pxe-images.nix still references them)
- docs: mark server decommissioned in ip-addressing.md
Docker NFS mounts already pointed at nfs.storage.home (previous commit).
VM destruction and DNS cleanup (server.sweet.home, server.storage.home
A/PTR records) to follow once the VM is confirmed stopped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Adds a DNS Zones section to ip-addressing.md covering the three FreeIPA
zones (sweet.home / cluster.home / storage.home) with all A records and PTR
records as configured. Updates Stage 6 in network-cutover.md to reference
nfs.storage.home by DNS name rather than the raw VIP IP, and marks DNS as a
completed prerequisite for the HA storage cutover.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
ip-addressing.md:
- Storage-client network purpose updated: NFS primary, iSCSI available
- Protocol separation box updated: NFS on both VIPs (each CIDR-restricted);
iSCSI on VLAN 20 only but not in active use
- docker role updated to NFS client (not iSCSI initiator)
- vip-storage description updated to NFS + iSCSI endpoint
ha-network-audit.md:
- Implementation status header added noting all audit items resolved
- Records decisions that diverged from original recommendations
(VLAN renumbering, dual VIP, NFS-over-iSCSI for docker/swarm)
- Notes deferred items (STONITH, iSCSI ACLs)
network-cutover.md:
- New Stage 6: HA storage cutover — docker NFS remount from
server.sweet.home to vip-storage (192.168.20.229), final rsync
steps, and server.sweet.home decommission procedure
- Former Stage 6 (final cleanup) renumbered to Stage 7
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Renumber internal HA bridges so VLAN ID matches IP third octet:
vmbr1 (cluster): 192.168.4.x → 192.168.10.x (VLAN 10, /29)
vmbr2 (storage): 192.168.5.x → 192.168.20.x (VLAN 20, /24)
Enforce protocol separation at the firewall on both HA nodes:
- iSCSI (3260/tcp): storage-client subnet (192.168.20.0/24) only
- NFS (111,2049,20048 tcp+udp): LAN subnet (192.168.2.0/24) only
- Cluster subnet (192.168.10.224/29) accepted wholesale (DRBD+Corosync)
Moves from allowedTCPPorts/allowedUDPPorts to source-restricted extraCommands
rules on the nixos-fw chain.
NFS exports reduced to LAN-only (drop haClientCidr lines from mkNfsExports).
Storage-client network clients use iSCSI only; no NFS on VLAN 20.
All HA script VIP/storage IP defaults updated to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01J8djTWdXVzXZc99iujU6T2
Both VIPs fail over together in ha-group:
vip-lan: 192.168.2.229/24 on ens18 (vmbr0) — LAN iSCSI + NFS access
vip-storage: 192.168.5.229/24 on ens20 (vmbr2) — storage-client iSCSI + NFS
variables.nix: haServerLanVip = "192.168.2.229"
cluster-init.sh: both VIP primitives in ha-group XML; VIP_LAN var added
ip-addressing.md: both VIP entries documented
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Infrastructure changes already applied to pve1:
- vmbr2 internal bridge created (192.168.5.0/24, no physical uplink)
- VM 200 (ha-server-1): net2 added → vmbr2 (ens20)
- VM 201 (ha-server-2): net2 added → vmbr2 (ens20)
- CT 105 (docker): net1 added → vmbr2 (eth1)
- VM 101 (server): net1 added → vmbr2 (ens19) — needs reboot to activate
NixOS config (deploy to ha nodes to complete; docker/server at cutover):
- ha-server-{1,2}/host.nix: ens20 with 192.168.5.{228,227}/24
- docker/host.nix: eth1 with 192.168.5.225/24
- server/host.nix: ens19 with 192.168.5.226/24
- cluster-config.nix: corosync ring1 on LAN IPs as backup heartbeat path
- cluster-config.nix: allow haClientCidr (192.168.5.0/24) in iptables
- ha-server.nix: NFS exports now allow both lanCidr and haClientCidr
- VIP moves from 192.168.2.229 (vmbr0/LAN) to 192.168.5.229 (vmbr2)
- iSCSI portal to be rebound from [::0] to 192.168.5.229 at cutover
variables.nix: haStorageCidr corrected to 192.168.4.224/29; new vars:
vmStorageClientInterface, lxcStorageInterface, haServer{1,2}ClientIp,
dockerStorageIp, serverStorageIp, haClientCidr/PrefixLength; haServerVip
updated to 192.168.5.229.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Read-only discovery of pve1 bridges, HA cluster VMs (200/201), and
Docker CT (105) — current state, risk flags, recommended vmbr2 storage
bridge layout, gap list, and migration notes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
KEY (hub's SSH public key) is now stored in variables.nix as beszelHubKey
and set by enable-agent.nix — no host file ever needs to touch it.
TOKEN was already universal; this removes the last per-host beszel config.
Importing enable-agent.nix in a build type is now the only step needed to
add a new host to beszel monitoring.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace per-host host-token.nix imports with a single beszel-token secret
in secrets/common.yaml, wired once in enable-agent.nix. Host files now only
need services.beszel.agent.environment.KEY — no imports block required.
Delete modules/beszel/host-token.nix (no longer referenced anywhere).
Action needed: run `sops secrets/common.yaml` and add `beszel-token: <value>`
from the beszel hub UI before deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documents how to wire beszel-agent into any flake build: the two-module
split (enable-agent.nix in build types, host-token.nix in host files),
sops secret creation, hub pairing flow, and optional EXTRA_FILESYSTEMS /
Docker monitoring knobs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Documentation fixes:
- README/AGENTS: rename tailscale-exit-node → tailscale-router, add ha-server
build type and proxmox-ha-server-{1,2} host table rows, add baremetal to
platform list, remove references to non-existent flake-target-refactor-spec.md
and remove-sensetive-info-refactor.md
- docs/auto-installer.md: fix lxc-tailscale-exit-node → lxc-tailscale-router,
add pxe-minimal to the flake outputs list
- variables.nix: fix domainControllerIp comment — IPA is the authoritative DNS
at .253 (Pi-hole is gone), not a forwarding intermediary
Code deduplication:
- Extract duplicate SSH host-key preservation activation scripts from
modules/platforms/lxc.nix and modules/platforms/proxmox.nix into a shared
modules/common/preserve-ssh-host-key.nix; both platforms now import it
- Replace 8-line hand-enumerated NFS export lists in server.nix and ha-server.nix
with a mkNfsExports helper that generates exports from vars.nfsShares — adding
a share to variables.nix now propagates to both exporters automatically
Dead code removal:
- modules/common/configuration.nix: remove leftover NixOS skeleton comments
(hardware-configuration import, grub lines) that were never used
- modules/docker/enable-service.nix: remove commented-out listenOptions and
daemon.settings blocks
- hosts/server/host.nix, hosts/nix-cache/host.nix: remove #DOCKER_HOST comments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds an unattended install option to the PXE boot menu that installs
Rocky Linux 9 and configures FreeIPA on the domain-controller.sweet.home
host without any operator interaction after selecting the menu entry.
How it works:
- fetch-rocky-pxeboot.service downloads the Rocky 9 Anaconda pxeboot
kernel and initrd from the Rocky mirror on first pxe-boot deploy
(idempotent, same pattern as fetch-debian-netboot)
- rocky-freeipa.ipxe boots Anaconda with inst.ks pointing at the
hosted Kickstart and net.ifnames=0 biosdevname=0 for stable eth0
- rocky-freeipa.ks (generated, includes vars.adminSshKey) performs:
- Minimal Rocky 9 install with ipa-server + ipa-server-dns
- Static IP 192.168.2.138 via NM connection file written in %post
- /etc/hosts fixed for FreeIPA FQDN requirement
- Random DM + admin passwords generated and saved to
/root/ipa-credentials.txt (chmod 600, never hardcoded)
- freeipa-first-boot.service oneshot enabled to run
ipa-server-install on the first real boot (~20 min)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015Jbvxx4xbHVcx1NkK3vtmK
- Correct the NFS path from /tank/proxmox/pxe-images to /tank/pxe-boot/images
(matches variables.nix's proxmoxPxeImages.subpath)
- Clarify that LXC uses NFSv3+nolock while VM uses NFSv4.2+automount
- Add explicit note that lxc-pxe-boot needs features: nesting=1,mount=nfs and
why: nesting=1 is required by systemd 260+ for userns/credential isolation
(AppArmor denies userns_create without it), mount=nfs for NFSv3 access.
pct set replaces the whole features string — include both or the container
will fail to boot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds modules/pxe-boot/mount-pxe-images.nix, which mounts
server.sweet.home:/tank/proxmox/pxe-images at /mnt/pxe-images via NFSv4.2
(x-systemd.automount on Proxmox VMs, nofail on LXC containers — same pattern
as docker/mount-data.nix). The pxe-boot build-type now imports this module and
replaces the previous local /srv/pxe/http/images directory rule with an L+
symlink pointing to /mnt/pxe-images, so large images (ISOs, disk images) live
on the NFS share rather than the host's own root disk.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds a `fetch-debian-netboot.service` oneshot that downloads the Debian
bookworm netboot kernel and initrd from deb.debian.org on first boot,
stages them under /srv/pxe/http/debian/, and serves them via a generated
debian.ipxe chain script. The service is idempotent — it skips the
download if both files are already present.
Also merges the previously split systemd.tmpfiles.rules and
systemd.services blocks into a single systemd = { ... } attrset to
satisfy statix's repeated-keys lint.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If neither /etc/host-keys nor /root/host-keys has the target's SSH
host key, auto-install.sh previously went straight to "continue
without one anyway?". Added a third option in between, gated on
[[ -t 0 ]] (only offered when there's an actual operator at stdin, never
in an unattended/non-interactive run): prompt for an arbitrary
directory (USB stick, other mount, etc.), and if the key pair is
there, copy it into /root/host-keys and install it to /mnt same as the
existing pre-seeded-key path. Falls through to the original
warning+confirm if the prompt is skipped, the path doesn't have the
key, or the run isn't interactive at all.
docs/auto-installer.md updated to mention the new fallback. Quick
bash -n + shellcheck pass only, per request.
Two real bugs, both hit live:
1. Shebang: #!/run/current-system/sw/bin/bash only resolves on an
already-activated NixOS system -- running the checked-out script
directly (e.g. from a stock ISO, cloned repo) failed with "cannot
execute: required file not found" on a non-NixOS box. Switched to
#!/usr/bin/env bash, which resolves identically on NixOS
(environment.usrbinenv's own default) and any normal Linux distro.
Also fixed the file's missing executable bit.
2. FLAKE_BASE_URL: previously depended on pkgs.replaceVars substituting
a Nix-templated @lanDomain@ placeholder at build time -- meaning it
only ever worked when baked into the built installer image, not when
run straight from a checkout (the literal, unexpanded "@lanDomain@"
string reached git as a bogus hostname). Replaced with LAN_DOMAIN in
scripts/env.sh (manually kept in sync with variables.nix's lanDomain,
same pattern as NIX_CACHE_HOST/nixCacheHost already), sourced by the
script itself like every other script in scripts/. Dropped
pkgs.replaceVars from modules/installer/common.nix entirely --
scripts/env.sh is now baked into the image alongside auto-install.sh
at a matching relative path (/etc/nixos-installer/env.sh next to
/etc/nixos-installer/installer/auto-install.sh) so the script's own
relative `source` line resolves the same way in both contexts.
loginShellInit's invocation path and docs/auto-installer.md updated to
match. Verified: shellcheck clean on both scripts, the baked files are
byte-identical to their checked-in sources (no templating left to
verify), and codex-maintenance.sh (secret grep, fmt, statix, full eval
of every host/package including the installer/pxe artifacts) passes
clean.
The iPXE menu's "nixos" entry actually chain-loads this flake's own
custom auto-installer image, not a stock NixOS image — rename it to
"auto-installer" (label "NixOS Auto-Installer") so the menu says what it
boots, and set networking.hostName on netbootSystem to match, so the
generated system name (nixos-system-auto-installer-*) and staged
directory (/srv/pxe/http/auto-installer) agree with the menu entry too.
Add a second, genuinely vanilla NixOS minimal netboot image
(netbootMinimalSystem in flake.nix — nixpkgs' netboot-minimal.nix on its
own, none of modules/installer/common.nix's auto-installer wiring),
built from source the same way as the auto-installer image and exposed
as packages.x86_64-linux.pxe-minimal. Staged and menu-wired the same
way, as "nixos-minimal" (item, hostname, and directory all matching).
modules/pxe-boot/stage-installer-artifacts.nix is generalized to stage
both images via a shared rule-builder instead of one hardcoded set of
paths.
Verified: nix eval confirms both images' config.system.name matches
their menu entry/directory names, the pxe-boot host itself builds
clean with the new menu.ipxe, and the new pxe-minimal image was booted
directly under QEMU (kernel+initrd, no KVM) to a working login shell
with hostname nixos-minimal, no hang.
CI was running a full eval of every host + package on every push/PR,
which was slow enough to routinely time out the Gitea runner. Default
mode now diffs against a base ref and scopes nixpkgs-fmt/statix/eval to
the files that changed and the hosts/packages they can affect; a change
to flake.nix/flake.lock/variables.nix/modules/common/* (or any other
modules/*.nix outside platforms//build-types, whose blast radius isn't
inferable from the path) falls back to evaluating everything. The old
full sweep moves behind --full-check, which CI never passes; --dry-run
adds build-planning on top of whichever scope is active.
Also trims codex-setup.sh's redundant full host eval loop -- that's
what codex-maintenance.sh is for; setup should just install tooling.
variables.nix's nixCacheHostKey no longer matched nix-cache's actual SSH
host key (confirmed via ssh-keyscan against the live container), so every
declaratively-configured client's programs.ssh.knownHosts trusted the
wrong key -- distributed builds would fail host-key verification. Also,
modules/nix-cache/remote-builder-client.nix hardcoded sshKey to
/root/.ssh/nixremote, but the `server` host only has its own default
/root/.ssh/id_ed25519 installed (confirmed live via qm guest-agent) --
that file was never even present, so the build machine config pointed at
nothing. Standardize on each client's own default identity, matching the
per-host-key pattern vars.remoteBuilderAuthorizedKeys already uses instead
of a shared/differently-named keypair, and add
scripts/secrets/sync-nix-cache-host-key.sh (wired into
codex-maintenance.sh's --check) so the host-key drift doesn't silently
recur next time nix-cache is rebuilt or recreated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V7yVH71vGrDVzovh9UaMu8
scripts/ had grown to 10 top-level scripts covering three distinct
concerns (sops/age + SSH host-key management, Proxmox deployment, and
repo-wide bootstrap/CI) with no grouping. Move the key-management scripts
(backup-admin-key.sh, rotate-admin-key.sh, prepare-host-key.sh,
sync-host-keys.sh) into scripts/secrets/, and the Proxmox scripts
(create-proxmox-resource.sh, configure-nix-cache-client.sh) into
scripts/proxmox/; leave env.sh, codex-setup.sh, codex-maintenance.sh, and
bump-nixpkgs-release.sh at the top level (frequently hand-typed or pure
shared config) and scripts/lib/ as-is.
Updates every cross-reference: each moved script's repo_root computation
(now one directory deeper), shellcheck source= directives, inter-script
paths (create-proxmox-resource.sh's call into sync-host-keys.sh and its
remote bootstrap of configure-nix-cache-client.sh on the Proxmox node),
and every doc/module mention (CLAUDE.md's Scripts section reorganized to
match, README.md, docs/auto-installer.md, docs/proxmox-images.md,
modules/installer/common.nix, modules/platforms/lxc.nix). CI workflows
need no change -- they only invoke codex-maintenance.sh, which didn't
move. Verified via bash -n, shellcheck (no new warnings beyond the
pre-existing SC1091/SC2029/SC2095 baseline), and live dry-runs of
sync-host-keys.sh --all and create-proxmox-resource.sh --list from their
new paths.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New tor-relay build type (currently lxc-only) running a plain Tor
middle relay via modules/tor/enable-relay.nix, plus nyx for
interactive monitoring over the relay's control socket.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
create-proxmox-resource.sh no longer builds locally and scp's a
multi-gigabyte image over -- it now clones/pulls this repo onto the
Proxmox node itself (bootstrapping build tooling via the existing
codex-setup.sh on first use) and runs the nix build / disko image
script there, staging the result straight into the node's own import
directory. host-keys/ (gitignored) is copied over separately since a
git pull doesn't carry it. --image still uploads an explicit local
file for the case where you don't want a build at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Same class of problem as the deployedTargets/README fixes: hand-maintained
prose that drifted from reality and nobody was obligated to update.
- CLAUDE.md: "18 hosts" was a stale hardcoded count (actually 20); reworded
to not need updating as hosts are added. Also added the missing
tailscale-exit-node build type to a list that had it everywhere else in
the file except one bullet.
- AGENTS.md: same missing tailscale-exit-node build type.
- docs/auto-installer.md: the hand-enumerated lxc-* list was missing
lxc-tailscale-exit-node.
- flake-target-refactor-spec.md: added a "Status: implemented" note so this
completed historical spec (referenced elsewhere purely for rationale)
can't be mistaken for an open plan with unresolved Open Questions.
- remove-sensetive-info-refactor.md: the "Definition of done" checklist was
entirely unchecked despite most of the work being done. Checked off what's
actually done (sops-nix migration, history scrub just performed, the
pre-commit gitleaks hook), and left rotation of the GitHub PAT found in
history explicitly flagged as the one still-open item -- an operator
action against GitHub, not something this repo can attest to itself.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Distributed builds failed with "Host key verification failed" on any
client that had never manually SSH'd to nix-cache before, since
nothing populated root's known_hosts for it. Wire nix-cache's host
public key into programs.ssh.knownHosts via a new vars.nixCacheHostKey
so every client picks it up automatically on rebuild.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The real root cause behind the original nix-cache 502, traced all the way
through: modules/installer/host-keys.nix (which NIXOS_HOST_KEYS_DIR=...
--impure actually wires up) is only ever imported by the installer's own
modules/installer/common.nix -- modules/platforms/lxc.nix, which every
real lxc-* host build actually uses, never imported anything like it.
docs/auto-installer.md previously claimed NIXOS_HOST_KEYS_DIR bakes a key
into lxc-* tarballs "the same way it does for the ISO/PXE installer
images" -- that was never actually true; I wrote it without verifying the
mechanism existed for lxc.nix specifically.
In practice this meant every lxc-* container booted with a freshly
self-generated SSH host key that could never match whatever .sops.yaml
actually trusts for that target, so *every* secret -- not just
cache-priv-key -- silently failed to decrypt. No error surfaces in the
boot log for this: the activation step that installs secrets only runs
on a genuinely fresh first activation and silently no-ops once
/run/current-system already exists, so by the time anyone looks the
window has closed. Found by manually invoking sops-install-secrets
directly: "Error getting data key: 0 successful groups required, got 0".
Fixed by giving modules/platforms/lxc.nix the same key-baking mechanism
the installer has, but keyed to its own exact flake target and placing
the key directly at /etc/ssh/ssh_host_ed25519_key (no copy step to stage
for, unlike the installer's /etc/host-keys/ staging area -- an lxc-*
tarball has no install step). The target name comes in via
specialArgs.flakeTarget (new, set by flake.nix's mkTarget) rather than
being read back from config.environment.etc."flake-target" -- reading
that back from within a module that also contributes to
environment.etc is circular (confirmed: "infinite recursion
encountered").
Verified live end-to-end against the real test container (lxc-nix-cache,
VMID 100 on pve.sweet.home): destroyed it, rebuilt the tarball fresh with
the fix, recreated it, and confirmed /run/secrets/ now has all three
secrets this host needs (beszel-token, cache-priv-key, nix-github-token),
nix-serve is active (running), and curl http://localhost/nix-cache-info
succeeds both directly and through nginx.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Found and fixed live against a real test container (VMID 100, lxc-nix-cache
on pve.sweet.home) after the previous pct-restore-to-pct-create fix still
produced a container that booted into garbled console output:
1. pct create's own CLI default for --unprivileged is privileged (unlike
the web UI, whose checkbox defaults the other way), but
modules/platforms/lxc.nix sets proxmoxLXC.privileged = false, so the
image assumes it's running unprivileged. Real mismatch -- now passes
--unprivileged 1 explicitly.
2. The actual root cause of the garbled console: modern (v247+) systemd
routinely uses nested user namespaces and credential mounts (even
plain getty units, via LoadCredential=-style mechanisms), which
AppArmor's default LXC confinement denies without --features
nesting=1,keyctl=1. Confirmed via the host's kernel audit log: every
getty unit was crash-looping on a denied /run/credentials/* mount
every ~3s, and core services like nsncd failed userns_create the same
way -- the system never finished activating. Fixed live (pct set +
restart on the running test container) before committing the script
change: systemctl is-system-running went from never completing to
"running" with zero failed units.
3. --memory doesn't touch swap -- confirmed live it silently stayed at
Proxmox's own 512M default with --memory 2048. Now defaults --swap to
whatever --memory resolves to.
docs/auto-installer.md's manual pct create walkthrough gets the same
fixes, with the "why" for each flag, since a human following it by hand
would hit the identical bugs.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Two independent problems found while actually running the script:
1. nix build/nix-shell retry each unreachable substituter/builder up to
5x with backoff, per store path -- with nix-cache down this compounds
into minutes of noise. scripts/env.sh gains nix_extra_opts(), which
probes http://nix-cache and nixremote@nix-cache:22 once via plain
curl/TCP (bypassing Nix's own retry logic entirely -- confirmed
nix store ping still retries 5x even with a short connect-timeout)
and exports the decision so create-proxmox-resource.sh and the
sync-host-keys.sh subprocess it shells out to both reuse it instead
of probing independently.
2. The actual failure: "archive contains no configuration file". pct
restore expects a vzdump backup archive with embedded config;
config.system.build.tarball is a plain CT template tarball -- wrong
Proxmox mechanism entirely. Fixed to pct create against it as a vztmpl
template instead, uploaded to /var/lib/vz/template/cache/ rather than
/var/lib/vz/dump/. This same wrong claim had propagated into
docs/auto-installer.md, README.md, and CLAUDE.md from when the script
was first written -- corrected everywhere.
Also: checks for an already-uploaded image on the node (fixed
<flake_target>.tar.xz/.raw naming) before building, skipping build+upload
entirely if found (--force-rebuild to always rebuild).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
sync-host-keys.sh: generates/registers SSH host keys and their
.sops.yaml/secrets/*.yaml recipients for flake targets, idempotently.
--all, <target>, --remove, --regenerate-all-keys, all with --dry-run
(verified zero-side-effect via a sandboxed git-status check across every
mode). Only ever touches anchors with a corresponding host-keys/ file --
&admin and any hand-registered real-host anchor are never listed,
removed, or regenerated. Supersedes running prepare-host-key.sh one host
at a time for any target that already has a flake entry.
create-proxmox-resource.sh: builds a lxc-*/proxmox-* target's
tarball/disk image and creates it on a real Proxmox node, or reconfigures
an existing resource's cores/memory/disk (--modify, always requires
typing the VMID back to confirm). Refuses to create a new resource for a
VMID that already exists, and refuses to duplicate a host identity that
already has a real deployment elsewhere (variables.nix's new
deployedTargets, checked by hostName so it also catches cross-platform
duplicates) unless --allow-duplicate-host is passed. --dry-run throughout.
scripts/env.sh centralizes the Proxmox connection config both scripts
(and future ones) share. Also fixes an unrelated gap found along the way:
proxmox-* Disko image builds write their .raw file straight into the
repo root, and .gitignore never covered it.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
nix-serve's secretKeyFile was a manual, undocumented-outside-a-comment
`nix-store --generate-binary-cache-key` step per host -- easy to miss on
a fresh nix-cache instance (as lxc-nix-cache testing just found: systemd
fails the unit with EXIT_CREDENTIALS when LoadCredential can't find the
source file, which nginx then reports as a 502 from clients). It also
can't be regenerated per-host safely: modules/nix-cache/client.nix
hardcodes every client's trust in one specific public key, so every
nix-cache instance has to share the exact same keypair.
Sourced from secrets/nix-cache.yaml's new cache-priv-key entry instead,
via the same sops-nix pattern every other secret in this repo already
uses. Verified the added value derives to the exact public key
modules/nix-cache/client.nix already trusts before committing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Linode provisions and sizes /dev/sda (root) and /dev/sdb (swap) itself
as whole, unpartitioned block devices before the OS ever boots.
modules/disko/linode.nix declares them with destroy = false (skips
Disko's wipe stage for these disks entirely) and a bare filesystem/swap
content type matching that existing layout, so re-running it against an
already-provisioned disk only mkfs/mkswaps if blkid shows it isn't
formatted yet -- never repartitions or destroys data.
With every host reachable through the installer menu now carrying a
Disko config, auto-install.sh no longer needs to probe the flake and
branch between `disko --mode destroy,format,mount` and a bind-mount
fallback -- it just always runs Disko.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Both existed only so the installer could boot as an LXC container and
nixos-install some other host from within it, but lxc-* targets are
already excluded from the install menu (nixos-install can't touch its
own running root filesystem), and now have their own direct tarball
path anyway. That left the installer's own LXC form with no real use
case, and packages.all with only two members worth bundling.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Both check-nixos.yml workflows (GitHub + Gitea) now call
scripts/codex-maintenance.sh instead of a hand-rolled eval-only loop,
closing a real gap: CI previously enforced none of the secret grep,
nixpkgs-fmt, or statix checks that codex-maintenance.sh already runs
locally — nothing was stopping that from regressing. One script now
backs both, instead of two copies that can drift from each other.
codex-maintenance.sh itself is extended to cover buildable surface
that wasn't validated anywhere before: packages.x86_64-linux.*, plus
config.system.build.tarball (lxc-* hosts) and
config.system.build.diskoImagesScript (proxmox-*, excluding the
installer's own proxmox-lxc target, which has no disko config).
Also:
- scripts/prepare-host-key.sh: dropped the redundant
[path-to-nixos-repo] parameter — it always defaults to the repo the
script itself lives in now, so a second argument never made sense
after the nix-auto-installer migration.
- Removed prepare.sh (dead pre-disko manual parted/mkfs/mkswap
partitioning, fully superseded) and
scripts/create-linode-installer-disk.sh (incomplete draft for an
abandoned dd-via-rescue-mode approach; Linode hosts already deploy
fine through the normal auto-installer flow).
- docs/pxe-boot.md: fixed a stale `nixosConfigurations.pxe-boot` eval
command (pre-refactor flat name, not a real flake attribute
anymore) and added a cross-reference to docs/auto-installer.md.
- CLAUDE.md/README.md: full documentation pass reconciling this
session's changes — modules/installer/, modules/pxe-boot/, the
LXC/Proxmox image-building deployment paths, corrected the
password-hash/SSH-key locations in the safety-rules section (both
had drifted to reference files/paths that no longer exist), and
added session-workflow guidance to prefer targeted host evals over
full-repo sweeps for incremental changes (explicitly scoped to
interactive sessions, not CI).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
modules/disko/proxmox.nix gains imageSize (20G default) and a
per-host imageName (networking.hostName, so every proxmox-* host
produces a distinctly named image instead of an identical main.raw).
This is the same disko.devices config already used to format a real
disk on install, so it's available for every proxmox-* target with no
per-host changes needed:
nix build .#nixosConfigurations.<host>.config.system.build.diskoImagesScript
sudo ./result --build-memory 2048
docs/proxmox-images.md covers building, host-key pre-seeding via
disko's --pre-format-files (same host-keys/ workflow as the installer
and LXC tarball paths), and the qm import/attach sequence for
deploying the result to Proxmox.
Also fixes a real bug in auto-install.sh found while testing: the
disko confirmation bypass used --yes, which disko's CLI doesn't
recognize at all (the actual flag is --yes-wipe-all-disks) — so the
"skip confirmation" flag was silently a no-op and the interactive
prompt kept appearing regardless.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
LXC hosts (device busy fix):
modules/platforms/lxc.nix now imports nixpkgs' own
virtualisation/proxmox-lxc.nix, giving every lxc-* host a real
config.system.build.tarball output — a directly `pct restore`-able
Proxmox container image. This is the actual bug fix behind the
"cannot remove real root directory: device busy or in use" error:
lxc-* targets were only reachable through nixos-install, which
bind-mounts / onto /mnt for containers (no raw disk to partition)
and then correctly refuses to modify the filesystem it's currently
running on. auto-install.sh's menu now excludes lxc-* targets
entirely (they deploy via nix build + pct restore instead, see
docs/auto-installer.md) — and, on the same reasoning, also excludes
`installer`/`proxmox-lxc`, which are the installer image's own flake
targets, not deployable hosts.
manageHostName = true keeps host.nix's declared hostnames (upstream's
default would let Proxmox's ambient container config win instead);
privileged = false matches how these containers are actually created.
Eval warnings, now zero across all 19 nixosConfigurations + 4 packages:
- Multiple password options (root/nixos in the installer): nixpkgs'
own installer profile sets initialHashedPassword = "" for
passwordless login, conflicting with our explicit hashedPassword.
Force-nulled the upstream option rather than adopting passwordless
login, since this image now also boots over LAN PXE with
PasswordAuthentication enabled.
- boot.zfs.forceImportRoot default value: set explicitly to false
(matching the two places that already did) in
modules/common/configuration.nix and modules/installer/common.nix,
covering every host and the installer alike.
- Deprecated pkgs.system in modules/build-types/gui.nix: switched to
pkgs.stdenv.hostPlatform.system.
All confirmed non-behavioral where it matters: unrelated hosts'
drvPaths are byte-identical to their pre-existing baselines throughout.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Finishes the nix-auto-installer migration: scripts/prepare-host-key.sh
and the local host-keys/ directory (gitignored, private key material,
never committed — moved as plain files, not through git history)
weren't carried over in the initial migration.
Also implements automatic key staging, replacing the manual
scp-after-boot step:
- modules/installer/host-keys.nix reads host-keys/ via
builtins.getEnv, which Nix silently returns as "" under normal
(non---impure) evaluation — the module is a no-op by default, safe
for CI, until explicitly opted into:
NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build .#iso --impure
When built this way every key present gets baked into the image at
/etc/host-keys/, and auto-install.sh installs whichever one matches
the flake target selected at install time — no manual per-host scp.
- This deliberately includes the PXE netboot variant, even though
pxe-boot serves it unauthenticated over LAN HTTP: accepted
explicitly as a reasonable trade-off for a network that sits behind
LAN-only infrastructure, not the open internet. auto-install.sh
still falls back to /root/host-keys (manual scp) if a key isn't
baked in, so images built without --impure keep working exactly as
before.
- docs/auto-installer.md replaces nix-auto-installer's README,
updated for in-repo paths and the new build flow.
Verified: normal `nix eval` (no --impure) evaluates identically across
all 19 nixosConfigurations + 4 packages with zero host-keys/* entries
(CI-unaffected); with --impure + the env var set, all three installer
variants (installer/ISO, proxmox-lxc, pxe) correctly embed every key
in host-keys/.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Adds debian@surface's existing SSH key (already used as its admin
key elsewhere in this repo) to vars.remoteBuilderAuthorizedKeys so
nix-cache will accept it as a distributed-build client once deployed.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
CLAUDE.md's "Composition pattern" section still described the
pre-refactor layout (hosts/<host>/configuration.nix as a thin imports
list, hardware-configuration wired in from flake.nix) from before the
platform x build-type matrix landed. Rewrite it to match the current
mkTarget/host.nix architecture and the module moves from the prior
commit. Also fixes docs/nix-cache.md, which referenced a
modules/nix/ path that never existed in this repo.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
Generates all nixosConfigurations from mkTarget(platform, buildType,
hostPath) instead of hand-written per-host blocks, so adding a new
platform or build type is a one-line addition. Per-machine identity
(hostname, hostId, secrets, stateVersion) moves into hosts/<name>/host.nix;
platform-specific config (hardware, boot, networking) into
modules/platforms/*.nix; build-type config (minimal/server/docker/gui/
nix-cache/pxe-boot) into modules/build-types/*.nix.
Old flat targets (nixos, docker, server, nix-cache, nix-minimal, pxe-boot)
are replaced by the 17-target <platform>-<buildtype> matrix; each new
target was verified to evaluate before its old counterpart was removed.
CI workflows and docs/aliases now discover hosts dynamically via
nixosConfigurations attrNames and /etc/flake-target instead of hardcoded
lists, so they can't drift from flake.nix again.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Remove the nonexistent kuma host and add the missing linode-minimal
host across README, AGENTS.md, docs, and CI eval workflows so they
match flake.nix's nixosConfigurations. Also add CLAUDE.md with
architecture/safety guidance for future Claude Code sessions.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
