Replace the hardcoded hostname check (docker, pxe-boot) with a check
on config.fileSystems: any lxc-* host whose NixOS config declares an
NFS fileSystem entry is automatically made privileged. The script
already reads proxmoxLXC.privileged dynamically via
flake_target_lxc_privileged, so no logic change is needed there —
only the comment is updated to describe the new derivation.
Result: lxc-docker and lxc-pxe-boot (the two with NFS mounts) evaluate
as privileged=true; lxc-nix-cache, lxc-minimal, lxc-server,
lxc-tailscale-router, lxc-tor-relay evaluate as privileged=false.
Any future lxc-* host that declares an NFS mount gets the correct
privilege level for free without a separate manual edit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--pre-format-files placed the key on the QEMU builder VM's rootfs, not the
target disk. nixos-install chroots into the target and runs sshd-keygen, which
found no key in the chroot and generated a fresh (unregistered) one. sops then
could not decrypt on first boot because the key didn't match .sops.yaml, leaving
both root and nixos with '!' in /etc/shadow even after mutableUsers = false was
set (hashedPasswordFile pointed to paths sops never wrote).
Fix modules/platforms/proxmox.nix to embed the clan SSH host key in
environment.etc via NIXOS_HOST_KEYS_DIR at eval time -- the same pattern
lxc.nix uses. nixos-install's own activation places the key on the target disk,
sshd-keygen finds it already present and skips generation, and sops decrypts
correctly on first boot. Includes the same preserveSshHostKey/restoreSshHostKey
activation scripts as lxc.nix so subsequent nixos-rebuild switch calls (without
NIXOS_HOST_KEYS_DIR) don't remove the key as "obsolete" from environment.etc.
Update create-proxmox-resource.sh: switch VM builds from
./result-<target> --pre-format-files ... --build-memory 2048
to
NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build --impure ... diskoImagesScript
./result-<target> --build-memory 2048
matching the LXC build path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
Three bugs combined to leave every VM build with a shell but no boot disk:
1. The remote build script moved the raw image to /var/lib/vz/import/ before
qm importdisk could use it. If the mv failed (cross-filesystem copy, sudo
path, or any other reason) the remote script exited non-zero -- but the
local script's set -e handling of the SSH heredoc was inconsistent, so
qm create sometimes ran anyway, leaving a diskless VM shell.
Fix: skip the mv entirely. The diskoImagesScript writes <hostname>.raw into
its CWD (the remote repo dir, $out = $PWD at invocation). Import directly
from that path; clean it up after a successful import.
2. The qm importdisk output regex expected "Successfully imported disk as '...'"
but current Proxmox emits "unusedN: successfully imported disk '...'"
(lowercase, no "as"). The grep returned no match and exited 1.
3. The disk_id assignment used $(... | grep ...) without || true inside the
substitution. With set -euo pipefail, a non-zero grep exit aborts the
script before the fallback could run -- so the VM was always left with an
unattached unused0 disk.
Fix: update the primary regex to match the actual PVE format; add || true
inside the substitution so set -e never fires on a grep miss; add a qm
config fallback (scan for unusedN: lines) that works regardless of PVE
output format changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
qm importdisk in QEMU 11.x outputs lowercase "successfully imported disk
as '...'" rather than the capitalised form the original grep expected.
The case mismatch made disk_id always empty, which caused the script to
exit 1 after qm create had already run -- leaving the VM with only an
EFI disk, no scsi0, and boot order still set to net0.
Fix by adding -i (case-insensitive) to the grep. Both the old capitalised
format (where the disk id had an "unused0:" prefix inside the quotes) and
the new lowercase format are handled correctly: the sed strip of unused0:
is preserved for backward compatibility, and the regex result is identical
either way.
Also add an early warning when --disk-size is passed for --type vm: the
flag is LXC-only for create mode and was silently ignored, leaving users
expecting a different size than the proxmoxImageSize in variables.nix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
All actively deployed lxc-* hosts now have clan vars. Remove the legacy
scp -pr host-keys/ fallback in sync_remote_host_keys(): instead of
silently copying the gitignored directory, error clearly if no clan var
exists for the target and tell the operator how to generate one.
Also extend the uncommitted-changes check to cover vars/per-machine/ in
addition to .sops.yaml and secrets/, since clan vars must be committed
before the remote build git-pulls them.
Update the script header and sync_remote_host_keys comment to reflect
the new clan-only key flow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Replaces the gitignored host-keys/ directory with clan vars as the
authoritative storage for SSH host keys. Keys are now generated as
sops-binary-encrypted clan var files (admin-key only) and checked into
vars/per-machine/<target>/openssh/, eliminating the plaintext private
key that previously had to live outside the repo.
Changes:
- modules/clan/ssh-host-key.nix: clan vars generator for the ed25519
SSH host key pair (neededFor="activation" — not mapped to sops.secrets,
delivered via tarball baking for LXC or --pre-format-files for VMs)
- flake.nix: add clanCore module + required settings to every mkTarget;
deduplicate bundled disko/sops-nix via follows; all 27 hosts eval clean
- flake.lock: updated to reflect the new follows constraints
- scripts/lib/clan-vars.sh: new helper library with
clan_ssh_key_exists / clan_ssh_pubkey_path / clan_decrypt_ssh_key /
clan_generate_ssh_key for use by the provisioning and sync scripts
- scripts/secrets/sync-host-keys.sh: queue_host_sync() now checks clan
vars first; generates via clan_generate_ssh_key if no key exists;
derives age fingerprint from clan pub key for .sops.yaml registration
- scripts/proxmox/create-proxmox-resource.sh: key management simplified
(sync-host-keys.sh now generates the key if missing, so the inline
prepare-host-key.sh call is gone); sync_remote_host_keys() decrypts
the clan key into a temp dir and scps just the two files to the node
when a clan key exists, falling back to the old host-keys/ scp for
any remaining legacy entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Three ordering-related fixes to the Proxmox provisioning flow:
1. prepare-host-key.sh: make idempotent -- if the key already exists, print
a note and exit 0 instead of erroring. The caller (create-proxmox-resource.sh)
already guards standalone calls, but the script itself should be safe to
run directly on a host that was already keyed.
2. create-proxmox-resource.sh: after sync-host-keys.sh updates .sops.yaml /
secrets/, detect uncommitted changes and block with a prompt until the
operator confirms they've committed and pushed. The PVE node's git pull
only picks up committed+pushed state; without this gate, a new host's sops
recipient is missing from the secrets files the image build uses, so the
host can't decrypt secrets on first boot.
3. create-proxmox-resource.sh: add an explicit existence check for the host
key in both the LXC and VM remote build heredocs, before it's passed as
--pre-format-files / NIXOS_HOST_KEYS_DIR input. Gives a clear error
pointing at sync-host-keys.sh instead of a raw `cp: cannot stat` from
disko deep in the build.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The disko images script does `cd "$TMPDIR"` before parsing its arguments,
so relative paths passed to --pre-format-files resolve against the temp
dir instead of the repo root. Use $(pwd) to capture the absolute repo
path before the disko script changes directory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If host-keys/<target>_ssh_host_ed25519_key doesn't exist, run
prepare-host-key.sh to generate it before sync-host-keys.sh runs.
Prevents sync-host-keys.sh from hitting its SKIP/exit-1 path (anchor
in .sops.yaml but no local key) and the downstream disko build failure
when --pre-format-files can't find the key file.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The post-build cp/mv into /var/lib/vz/template/cache (LXC) and
/var/lib/vz/import (VM) are Proxmox-owned root directories -- they need
sudo_pfx just like pct/qm/pvesh do. nix build writes to the nix store
as the SSH user, but staging into /var/lib/vz/ requires root.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nix was installed as root on pve1 (the codex-setup.sh root path, meant
for container/Codex environments), making nix build require sudo there.
After cleaning up the root install and reinstalling as the SSH user
(wayne), nix is owned by that user and runs directly without sudo.
create-proxmox-resource.sh: drop sudo_pfx from nix build in both
remote scripts. The SSH user owns the store after reinstall; nix build
goes through the nix daemon-or-store directly. sudo stays on pct/qm/pvesh
(cluster IPC) and the disko image-writer script (writes to disk).
codex-setup.sh: add build-users-group = (empty) to the user nix.conf
written by the non-root install path. Guards against a stale
/etc/nix/nix.conf from a prior root install (which sets
build-users-group = nixbld) silently breaking single-user builds.
Manual cleanup required once on each Proxmox node that had root's nix:
sudo rm -rf /nix /etc/nix
sudo rm -f /etc/profile.d/nix.sh /etc/profile.d/nix-daemon.sh
for i in $(seq 1 10); do sudo userdel nixbld$i 2>/dev/null||true; done
sudo groupdel nixbld 2>/dev/null || true
After that, the next create-proxmox-resource.sh run auto-reinstalls
nix as the SSH user via codex-setup.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nix on pve1 was installed as root (single-user), so wayne can't access
/nix/var/nix/db/big-lock without root -- nix build genuinely needs sudo
there. The previous fix to drop sudo_pfx was wrong.
The real fix is node config: add nix to wayne's NOPASSWD rules in
sudoers on pve1 (see below). pct/qm/pvesh already have NOPASSWD and
work fine in non-interactive SSH heredocs; nix was just missing from
that list.
On pve1 as root:
echo 'wayne ALL=(root) NOPASSWD: ALL' | tee /etc/sudoers.d/wayne-nopasswd
chmod 440 /etc/sudoers.d/wayne-nopasswd
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nix build runs through the nix daemon and doesn't need root; the
tooling-check step already confirms the SSH user can run nix directly
(ensure_nix_profile + command -v nix). sudo without a TTY blocks
non-interactive SSH heredoc sessions with "a terminal is required".
Keep sudo on pct/qm/pvesh (cluster IPC) and the disko image-writer
script (writes to block devices) -- those actually require root.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single-user Nix installations are owned by root. When PROXMOX_SSH_USER
is not root, prefix the remote nix build command with sudo_prefix, same
as the Proxmox tool invocations. Passes sudo_prefix as an extra arg to
both the LXC tarball and VM disko image build heredocs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PROXMOX_SSH_USER was changed from root to wayne, but all remote pvesh/qm/pct
invocations assumed root. When run as a non-root user these commands fail with
ipcc_send_rec errors because they can't reach the pve-cluster IPC socket.
Adds a global sudo_prefix (empty when PROXMOX_SSH_USER=root, "sudo" otherwise)
and applies it to every remote Proxmox command in the script, including the
duplicate-host heredoc check, pvesh nextid, vmid existence checks, resource
destruction, and all create/start commands. Removes the now-redundant local
sudo_prefix definition that was previously only in the VM image build branch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pve1.sweet.home is production; scripts/env.sh now also defines
PVE_TEST_HOST for a separate sandbox node, individually targetable via
--node/PROXMOX_HOST. Tooling defaults are unchanged (still pve1) -- the
new restriction (Claude defaults to pve-test unless explicitly told to
use pve1) is documented as policy in CLAUDE.md, not enforced in the
scripts.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The kernel's NFS client filesystem doesn't set FS_USERNS_MOUNT, so mounting
NFS from inside any non-init user namespace -- exactly what an unprivileged
LXC container's UID-mapped root runs in -- is rejected at the VFS layer
with EPERM, regardless of Proxmox's mount=nfs;nfs4 container feature (which
only patches the AppArmor layer). Confirmed live on the redeployed lxc-docker
container: TCP to the NFS server's port 2049 succeeds, the server's export
table matches the container's IP, and mount.nfs: Operation not permitted
still fires immediately with no corresponding denial anywhere in the
server's own logs -- a kernel-level rejection that no amount of DNS/
automount/export tweaking (this branch's earlier commits) could ever fix.
modules/platforms/lxc.nix now keys proxmoxLXC.privileged off hostName
("docker" -> true) rather than a blanket false, since build-types/docker.nix
is also composed for linode-docker/proxmox-docker, which don't import
proxmox-lxc.nix at all -- setting this option there would break their eval.
create-proxmox-resource.sh reads the value back via a new
flake_target_lxc_privileged helper instead of hardcoding --unprivileged 1,
so the two stay in sync automatically for every lxc-* target.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T48qgH3VTvs8wvwj44FEbE
create-proxmox-resource.sh's `qm create` never passed --agent, so despite
services.qemuGuest.enable = true being set on every host, Proxmox never
created the virtio-serial channel the guest agent needs -- qm guest exec
and the UI's IP-address display silently never worked for any VM this
script created. Found while live-testing every lxc-*/proxmox-* build type
against pve.sweet.home for an end-to-end flake audit.
Also registers a fresh sops age key for lxc-gui (no prior registration
existed), generated while testing that target live -- needed before
lxc-gui can be deployed with working secrets.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZHwwAM7cacQKkqKfQ5e8a
scripts/ had grown to 10 top-level scripts covering three distinct
concerns (sops/age + SSH host-key management, Proxmox deployment, and
repo-wide bootstrap/CI) with no grouping. Move the key-management scripts
(backup-admin-key.sh, rotate-admin-key.sh, prepare-host-key.sh,
sync-host-keys.sh) into scripts/secrets/, and the Proxmox scripts
(create-proxmox-resource.sh, configure-nix-cache-client.sh) into
scripts/proxmox/; leave env.sh, codex-setup.sh, codex-maintenance.sh, and
bump-nixpkgs-release.sh at the top level (frequently hand-typed or pure
shared config) and scripts/lib/ as-is.
Updates every cross-reference: each moved script's repo_root computation
(now one directory deeper), shellcheck source= directives, inter-script
paths (create-proxmox-resource.sh's call into sync-host-keys.sh and its
remote bootstrap of configure-nix-cache-client.sh on the Proxmox node),
and every doc/module mention (CLAUDE.md's Scripts section reorganized to
match, README.md, docs/auto-installer.md, docs/proxmox-images.md,
modules/installer/common.nix, modules/platforms/lxc.nix). CI workflows
need no change -- they only invoke codex-maintenance.sh, which didn't
move. Verified via bash -n, shellcheck (no new warnings beyond the
pre-existing SC1091/SC2029/SC2095 baseline), and live dry-runs of
sync-host-keys.sh --all and create-proxmox-resource.sh --list from their
new paths.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
