The netboot build composed ./modules/installer/iso.nix (which pulls in
nixpkgs' installation-cd-minimal.nix) together with nixpkgs'
netboot-minimal.nix. Both installation-cd-base.nix and netboot.nix set
fileSystems."/" via the identical lib.mkImageMediaOverride (mkOverride
60) priority - genuinely conflicting root-filesystem strategies
(ISO-by-label vs. netboot-tmpfs) at the same priority, and the ISO one
was winning. Every netboot boot hung waiting for a device that can
never exist outside a real CD/USB:
A start job is running for /dev/disk/by-label/nixos-minimal-...
Reproduced live: deployed a scratch lxc-pxe-boot on pve-test, pulled its
built kernel/initrd, and booted them directly with QEMU to confirm the
hang and capture full console output. netboot-minimal.nix's own chain
(netboot-base.nix) already imports profiles/installation-device.nix
independently, so common.nix's initialHashedPassword override still
applies correctly with iso.nix removed from this composition. Rebuilt
and re-booted the same way after the fix - full boot to a working shell
with SSH up, no hang.
Backs up a VM/CT on pve1 (snapshot mode by default, so the source stays
online), relays the archive to pve-test, restores it there with fresh
MAC addresses (--unique), and deletes both the source and relayed
backup copies afterward -- no ad hoc backup files left behind on
either node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
pve1.sweet.home is production; scripts/env.sh now also defines
PVE_TEST_HOST for a separate sandbox node, individually targetable via
--node/PROXMOX_HOST. Tooling defaults are unchanged (still pve1) -- the
new restriction (Claude defaults to pve-test unless explicitly told to
use pve1) is documented as policy in CLAUDE.md, not enforced in the
scripts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The kernel's NFS client filesystem doesn't set FS_USERNS_MOUNT, so mounting
NFS from inside any non-init user namespace -- exactly what an unprivileged
LXC container's UID-mapped root runs in -- is rejected at the VFS layer
with EPERM, regardless of Proxmox's mount=nfs;nfs4 container feature (which
only patches the AppArmor layer). Confirmed live on the redeployed lxc-docker
container: TCP to the NFS server's port 2049 succeeds, the server's export
table matches the container's IP, and mount.nfs: Operation not permitted
still fires immediately with no corresponding denial anywhere in the
server's own logs -- a kernel-level rejection that no amount of DNS/
automount/export tweaking (this branch's earlier commits) could ever fix.
modules/platforms/lxc.nix now keys proxmoxLXC.privileged off hostName
("docker" -> true) rather than a blanket false, since build-types/docker.nix
is also composed for linode-docker/proxmox-docker, which don't import
proxmox-lxc.nix at all -- setting this option there would break their eval.
create-proxmox-resource.sh reads the value back via a new
flake_target_lxc_privileged helper instead of hardcoding --unprivileged 1,
so the two stay in sync automatically for every lxc-* target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
Codifies read-only access to existing Proxmox config/VMs/containers,
allows scratch test VMs/containers as long as they're torn down again,
and forbids any change to production on the node.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
create-proxmox-resource.sh's `qm create` never passed --agent, so despite
services.qemuGuest.enable = true being set on every host, Proxmox never
created the virtio-serial channel the guest agent needs -- qm guest exec
and the UI's IP-address display silently never worked for any VM this
script created. Found while live-testing every lxc-*/proxmox-* build type
against pve.sweet.home for an end-to-end flake audit.
Also registers a fresh sops age key for lxc-gui (no prior registration
existed), generated while testing that target live -- needed before
lxc-gui can be deployed with working secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZHwwAM7cacQKkqKfQ5e8a
The previous commit's networking.search fix was wrong. Confirmed live
on lxc-docker (vmid 105) after redeploying with it: `resolvectl query
server.sweet.home` started failing again, even though
`resolvectl query --interface=eth0 server.sweet.home` still resolved
correctly to the right IP via the LAN's real DNS server. The debug log
showed why -- adding a *global* search domain via networking.search
gave systemd-resolved a domain-matched but server-less "global" scope,
which it now prioritizes over eth0's correctly-configured scope for
every "*.sweet.home" query, silently sending them to public fallback
DNS (1.1.1.1 et al) instead, which of course returns NXDOMAIN for an
internal-only name. Bare single-label names (e.g. "server") were never
going to work either way -- systemd-resolved only ever tries LLMNR for
those, never DNS search-suffixing, regardless of configuration.
Reverts the networking.search addition and instead has
modules/docker/mount-data.nix build each NFS device string from
"${vars.nfsServerHost}.${vars.homeDomain}" (a plain FQDN, no dependency
on search-domain behavior at all) -- the same pattern
modules/raspi/mount-data.nix already uses for the Raspberry Pi's share
and for the identical reason.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
Cross-host references throughout this repo (vars.nfsServerHost,
vars.nixCacheHost, vars.dockerHost) are bare short names, not FQDNs.
Resolving them has always depended on whatever network stack happens
to be in play picking up the DHCP-advertised domain as a search
suffix -- NetworkManager does this by default, which is why it went
unnoticed everywhere else, but LXC containers force-disable
NetworkManager and get their systemd-networkd config written directly
by Proxmox instead, which never sets one.
Confirmed live on lxc-docker (vmid 105) after today's earlier fix for
the automount/mount=nfs bugs: systemd-resolved had no search domain
for eth0, so "server" failed to resolve ("Name or service not known")
while "server.sweet.home" resolved fine via the same DNS server --
every NFS mount in modules/docker/mount-data.nix was still failing.
networking.search sets systemd-resolved's own Domains= globally
rather than depending on the per-link DHCP path, so it isn't at the
mercy of whichever component owns a given host's interface file.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
CI was running a full eval of every host + package on every push/PR,
which was slow enough to routinely time out the Gitea runner. Default
mode now diffs against a base ref and scopes nixpkgs-fmt/statix/eval to
the files that changed and the hosts/packages they can affect; a change
to flake.nix/flake.lock/variables.nix/modules/common/* (or any other
modules/*.nix outside platforms//build-types, whose blast radius isn't
inferable from the path) falls back to evaluating everything. The old
full sweep moves behind --full-check, which CI never passes; --dry-run
adds build-planning on top of whichever scope is active.
Also trims codex-setup.sh's redundant full host eval loop -- that's
what codex-maintenance.sh is for; setup should just install tooling.
variables.nix's nixCacheHostKey no longer matched nix-cache's actual SSH
host key (confirmed via ssh-keyscan against the live container), so every
declaratively-configured client's programs.ssh.knownHosts trusted the
wrong key -- distributed builds would fail host-key verification. Also,
modules/nix-cache/remote-builder-client.nix hardcoded sshKey to
/root/.ssh/nixremote, but the `server` host only has its own default
/root/.ssh/id_ed25519 installed (confirmed live via qm guest-agent) --
that file was never even present, so the build machine config pointed at
nothing. Standardize on each client's own default identity, matching the
per-host-key pattern vars.remoteBuilderAuthorizedKeys already uses instead
of a shared/differently-named keypair, and add
scripts/secrets/sync-nix-cache-host-key.sh (wired into
codex-maintenance.sh's --check) so the host-key drift doesn't silently
recur next time nix-cache is rebuilt or recreated.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V7yVH71vGrDVzovh9UaMu8
Following the existing per-client-key pattern in
vars.remoteBuilderAuthorizedKeys (one entry per host's own default root
key, not a shared dedicated keypair -- docs/nix-cache.md's "./nixremote"
filename is just a generic placeholder in the instructions). This box
had no entry yet, which is why its earlier `ssh-copy-id nixremote@nix-cache`
never stuck: nix-cache's nixremote authorized_keys is NixOS-declarative
and gets overwritten on every rebuild.
Public key only; requires a nix-cache rebuild to take effect.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scripts/ had grown to 10 top-level scripts covering three distinct
concerns (sops/age + SSH host-key management, Proxmox deployment, and
repo-wide bootstrap/CI) with no grouping. Move the key-management scripts
(backup-admin-key.sh, rotate-admin-key.sh, prepare-host-key.sh,
sync-host-keys.sh) into scripts/secrets/, and the Proxmox scripts
(create-proxmox-resource.sh, configure-nix-cache-client.sh) into
scripts/proxmox/; leave env.sh, codex-setup.sh, codex-maintenance.sh, and
bump-nixpkgs-release.sh at the top level (frequently hand-typed or pure
shared config) and scripts/lib/ as-is.
Updates every cross-reference: each moved script's repo_root computation
(now one directory deeper), shellcheck source= directives, inter-script
paths (create-proxmox-resource.sh's call into sync-host-keys.sh and its
remote bootstrap of configure-nix-cache-client.sh on the Proxmox node),
and every doc/module mention (CLAUDE.md's Scripts section reorganized to
match, README.md, docs/auto-installer.md, docs/proxmox-images.md,
modules/installer/common.nix, modules/platforms/lxc.nix). CI workflows
need no change -- they only invoke codex-maintenance.sh, which didn't
move. Verified via bash -n, shellcheck (no new warnings beyond the
pre-existing SC1091/SC2029/SC2095 baseline), and live dry-runs of
sync-host-keys.sh --all and create-proxmox-resource.sh --list from their
new paths.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
scripts/backup-admin-key.sh, rotate-admin-key.sh, and sync-host-keys.sh
each independently resolved sops/age's default key-file path, derived an
age pubkey from an identity file, and (two of them) ran `sops updatekeys`
the same way -- now shared via scripts/lib/sops-age.sh. Also extracted the
"type X to confirm" prompt duplicated across create-proxmox-resource.sh
and sync-host-keys.sh into scripts/lib/confirm.sh. Pure extraction, no
behavior change -- each call site produces identical commands/output to
before.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
.sops.yaml's &lxc-nix-cache anchor was rotated (commit b3c8145) after
today's LXC rebuild, but secrets/nix-cache.yaml was never re-run through
sops updatekeys -- it stayed encrypted for the pre-rotation key. Per
docs/nix-cache.md and docs/auto-installer.md, sops-nix decrypt failures
are silent: cache-priv-key never materialized, nix-serve had nothing to
bind its secretKeyFile to and failed to start, and nginx proxy_pass had
no upstream -- the 502 Bad Gateway seen from clients.
Confirmed the new key matches the currently-deployed host by deriving
the age identity from host-keys/lxc-nix-cache_ssh_host_ed25519_key.pub
via ssh-to-age and comparing to .sops.yaml.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Run it once, right after a node's first-time Nix bootstrap (not on every
invocation, and not inside codex-setup.sh/codex-maintenance.sh themselves),
so a freshly-bootstrapped Proxmox node substitutes from and can offload
builds to nix-cache on every subsequent run. Non-fatal on failure -- the
build still proceeds, just without nix-cache.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Non-NixOS machines with just the Nix package manager installed have no
module system to pick up modules/nix-cache/client.nix, so this edits
/etc/nix/nix.conf directly (extra-substituters/extra-trusted-public-keys,
plus the SSH remote-builder config once the nixremote key is installed),
falling back to cache.nixos.org when nix-cache is unreachable.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Per-run tooling verification (added in the previous commit) was hardcoded
to checking just `nix`. Turn it into a small array instead, so a future
remote build step needing another tool extends that list rather than
growing a second parallel check.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Two bugs found running scripts/create-proxmox-resource.sh against a real
Proxmox node for the first time:
- The Nix installer's single-user root path still shells out to `sudo`
to create /nix even though it already detected it's running as root,
which fails outright on a minimal, sudo-less Debian/Proxmox node.
codex-setup.sh now pre-creates /nix itself so that branch of the
installer is skipped.
- ensure_remote_repo() only ran scripts/codex-setup.sh right after a
fresh git clone, so a bootstrap that cloned the repo but then failed
installing Nix (exactly the failure above) left every subsequent run
silently building with a `nix` that was never actually installed,
since the repo already existing skipped tooling setup entirely. It
now checks `command -v nix` (via the same ensure_nix_profile used
elsewhere, since a non-interactive ssh session won't otherwise have a
single-user install on PATH) on every run and re-bootstraps if it's
missing. Both remote build heredocs also now source
scripts/lib/nix-bootstrap.sh themselves for the same PATH reason.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
New tor-relay build type (currently lxc-only) running a plain Tor
middle relay via modules/tor/enable-relay.nix, plus nyx for
interactive monitoring over the relay's control socket.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
create-proxmox-resource.sh no longer builds locally and scp's a
multi-gigabyte image over -- it now clones/pulls this repo onto the
Proxmox node itself (bootstrapping build tooling via the existing
codex-setup.sh on first use) and runs the nix build / disko image
script there, staging the result straight into the node's own import
directory. host-keys/ (gitignored) is copied over separately since a
git pull doesn't carry it. --image still uploads an explicit local
file for the case where you don't want a build at all.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
--allow-duplicate-host previously just skipped the existing-resource
check entirely, so re-running e.g. --type lxc --host docker while an
lxc-docker container already existed created a second container
sharing the same hostname/identity instead of replacing it -- both
then fight over DNS/DHCP for that hostname, and it's easy to end up
testing the stale one without realizing.
Now splits matches into "exact" (same --type as the one being
created, e.g. another lxc-docker) and "cross-type" (a different
platform sharing this host identity, e.g. a proxmox-docker VM
alongside an lxc-docker container -- a deliberate, valid coexistence
this script has never managed and still won't). Only an exact match is
destroyed and replaced, after typing the hostname back to confirm; a
cross-type match is always left untouched. Without
--allow-duplicate-host, both cases still refuse to run exactly as
before.
Verified live against pve.sweet.home: correctly split VMID 103 (a
stopped proxmox-docker VM, cross-type -- left untouched) from VMID 105
(the running lxc-docker container, exact-type -- flagged for
destroy+replace), and confirmed the destroy prompt safely aborts on a
non-matching confirmation, leaving both resources untouched.
Two compounding bugs, confirmed live on the running lxc-docker
container (vmid 102 on pve.sweet.home):
1. x-systemd.automount never works inside any Linux container --
systemd logs "Starting of <unit>.automount unsupported" for every
share and never mounts them. modules/docker/mount-data.nix and
modules/raspi/mount-data.nix now key off config.boot.isContainer
(set true by nixpkgs' proxmox-lxc.nix) to mount eagerly with
`nofail` there instead, while VM-based docker targets keep automount
unchanged.
2. The container's Proxmox `features` never included `mount=nfs`, so
AppArmor blanket-denies the nfs/rpc_pipefs mount syscalls NFS
needs ("permission denied"). scripts/env.sh's
PROXMOX_DEFAULT_LXC_FEATURES now includes mount=nfs;nfs4 for future
lxc-* containers -- the semicolon required quoting the --features
value in create-proxmox-resource.sh's remote pct-create command,
since it's sent as a raw string for the remote shell to parse and an
unquoted `;` would be read as a command separator.
The already-running container needs a matching `pct set --features`
plus a restart to pick this up -- that's an operator step outside this
repo.
Companion to rotate-admin-key.sh: copies whatever age identity sops/age
itself would resolve (or an explicit --key-file) to a given destination
path with 0600 perms, validating it's a real identity and round-tripping
the derived public key before/after the write so a corrupted copy is
caught immediately rather than discovered later during a restore.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
beatzaplenty
