cibadmin raw XML fails schema validation under pacemaker-4.0: globally-unique
is not a valid direct <clone> attribute, and master-max/master-node-max are
renamed. Switch to crm configure commands which are schema-version-aware:
- promotable clone with promoted-max/promoted-node-max
- order/colocation constraints using Promoted role (Pacemaker 4.0 naming)
- crm configure handles schema differences automatically
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
clearconfig does not reliably clear kernel LIO configfs state — the kernel
still holds backing-file references, blocking umount. Replace clearconfig
with explicit targetcli delete commands (/iscsi delete, /backstores/fileio
delete) which do release kernel state. Also make DRBD secondary demote
idempotent (skip if already Secondary).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Remove VIP-specific portal binding — the VIP doesn't exist until Pacemaker
assigns it; the default all-IPs portal (::0:3260) is correct for Pacemaker
to manage
- Clear existing LIO targets before re-running targetcli (idempotent on
partial failures)
- Tear down LIO kernel objects after saveconfig so umount succeeds (LIO holds
the backing file open otherwise)
- Guard mount with mountpoint check so re-runs don't fail when already mounted
- Use --replace for cibadmin constraints (idempotent vs --create)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- ha-server.nix: add xfsprogs to systemPackages so mkfs.xfs is on PATH for
root (needed by cluster-init.sh during initial setup)
- cluster-config.nix: create /var/lib/drbd via tmpfiles to silence
lk_bdev_save warnings from drbd-utils
- cluster-init.sh: dynamically find xfsprogs in /nix/store if not on PATH
(fallback for running VMs before xfsprogs is in the system profile)
- cluster-init.sh: fix DRBD metadata check on node2 — broken regex now uses
grep -E for ERE alternation to correctly skip create-md when DRBD is already
set up (previous regex would have triggered create-md on a live secondary)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Root SSH was failing because only the RSA admin key was authorized but the
local dev box only has an ed25519 key. Fix:
- cluster-config.nix: add ed25519 keys to root (same set as nixos user) so
future deployments work without the temp-key workaround
- deploy.sh/acceptance-tests.sh: SSH as nixos user with sudo instead of root@
- cluster-init.sh: HA_USER/HA_KEY env vars + n2_ssh()/n2_scp() helpers so
inter-node SSH works regardless of whether root-to-root is available
- deploy.sh Phase 6: generate temp keypair, authorize on node2, place on node1
for root to use during cluster-init, clean up afterward
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
pve_check() always uses sudo, so "sudo test -w /nix" passes as root
regardless of whether the SSH user can actually write there. Use a
direct non-sudo SSH command for the writability probe so the check
reflects wayne's own access, not root's.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
When create-proxmox-resource.sh clones the repo to pve1, it stays on
whatever branch was checked out. Add a pre-build phase that detects
branch mismatch and switches the remote clone to the current local
branch before building, so the Proxmox node always builds from the
same commits we're deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
pve1 has a pre-existing /nix store owned by a different UID; wayne's IPA
UID (50002) can't write to it. Add a pre-phase-3 check that uses wayne's
passwordless sudo to chown -R the store before create-proxmox-resource.sh
runs codex-setup.sh on the node.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Add root SSH key + passwordless sudo to ha-server nodes (needed for
deploy script to run cluster-init.sh via SSH as root on node1)
- Fix cluster-init.sh: correct default IPs (228/227/229 per variables.nix),
use \${VAR:-default} for all config so deploy.sh can override via env
- Fix acceptance-tests.sh: same IP corrections, add -i flag to SSH calls,
use \${VAR:-default} pattern
- Fix deploy.sh dry-run bugs: pve_check() always runs SSH for read-only
probes so bridge existence check is accurate; wait_for_ssh skips in
dry-run instead of timing out
- Fix cluster-init invocation: upload script via scp and run via SSH as
root on node1 (was incorrectly trying to run the script locally)
- Fix acceptance-tests invocation: pass IP env vars from deploy.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Run sync-host-keys.sh for proxmox-ha-server-{1,2}: generates SSH host
key pairs in vars/per-machine/, registers age anchors in .sops.yaml,
adds both hosts as recipients for common.yaml, ha-corosync-authkey,
and per-host secrets/keytab files
- Re-encrypt secrets/common.yaml with both new host keys
- Convert all stub secrets to real sops-encrypted files:
secrets/ha-server-{1,2}.yaml (YAML, beszel-token = PLACEHOLDER)
secrets/ha-server-{1,2}.keytab (binary, stub text encrypted)
secrets/ha-corosync-authkey (binary, stub text encrypted)
- Add scripts/ha/deploy.sh: full lifecycle script (bridge setup, VM
creation, DRBD disk + storage NIC attachment, boot wait, cluster-init,
acceptance tests, --destroy)
Bootstrap order (operator runs these before first deploy):
1. bash scripts/ha/deploy.sh # deploys, tests
# Post-deploy secret replacement:
2. sops secrets/ha-server-{1,2}.yaml (set real beszel-token)
3. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
5. Set services.beszel.agent.environment.KEY in host.nix after hub pairing
6. nixos-rebuild switch on both nodes to pick up real secrets
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Passing a FQDN like "nixos.sweet.home" instead of the short hostname
"nixos" caused the script to create a double-FQDN IPA host account
(nixos.sweet.home.sweet.home). Add an early check that rejects any
TARGET containing a dot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Script fixes:
- Rename HOSTNAME variable to TARGET (shadowed the bash builtin)
- Fix ipa-getkeytab -s to always use IPA_SERVER, not DC_HOST (diverge if
--dc is overridden to a jump host)
- Remove dead REALM variable
- Add EXIT trap to delete the plaintext keytab if the script aborts before
sops encryption completes; cleared after successful encrypt
- Distinguish real ipa host-add failures from "already exists" instead of
swallowing all errors with || true
- Warn explicitly when no platform age keys exist for the target (keytab
would be admin-only and the host couldn't decrypt it at boot)
- Fix sops fallback from pinned nixos-25.11 channel to nixpkgs (uses the
repo's own flake.lock)
- Expand "next steps" output to include networking.domain and nameservers
lines that host.nix requires for IPA membership
Module docs:
- Point to the script as the primary setup path; move manual steps to a
fallback section
- Note that certs/ipa-ca.crt is already committed (no need to re-fetch)
- Document the networking.domain and nameservers requirements in the header
- Add sync-host-keys.sh as explicit step 0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ipa-getkeytab runs as root via sudo so the temp file is root-owned;
scp as wayne gets Permission denied. Pipe through `sudo cat` over SSH
instead, which reads as root but writes locally as the invoking user.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single command to enroll a NixOS host in FreeIPA and produce a
sops-encrypted keytab at secrets/<hostname>.keytab:
- Adds the .sops.yaml creation rule automatically (with all registered
platform-variant age keys as recipients)
- SSHes to the domain controller to run ipa host-add + ipa-getkeytab
- Refreshes the admin Kerberos ticket via `ssh -t ... kinit admin` if
missing or expired, so no manual kinit step is needed
- SCPs the keytab and encrypts it in-place with sops (file must be at
secrets/<hostname>.keytab before encryption so the path-based creation
rule matches — the common failure point when doing this manually)
Also adds HOME_DOMAIN and IPA_SERVER to scripts/env.sh, matching
variables.nix's homeDomain/ipaServer (same manual-sync pattern as
NIX_CACHE_HOST/LAN_DOMAIN).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds proxmox-ha-server-1 and proxmox-ha-server-2 as real mkTarget entries
alongside the existing proxmox-server, backed by a new ha-server build type.
New modules
modules/ha/cluster-config.nix — DRBD resource + corosync nodelist sourced
from vars (haServer1Host/Ip, haServer2Host/Ip); resource-only fencing for
production STONITH; HA port firewall rules for DRBD, iSCSI, Corosync, pcsd
modules/build-types/ha-server.nix — imports pacemaker-stack + iscsi-target
+ cluster-config + beszel; NFS exports from vars.haStorageRoot (XFS-over-DRBD
mount); nfs-server.service.wantedBy force-cleared so Pacemaker controls
start/stop on the Active node only
New hosts
hosts/ha-server-{1,2}/host.nix — static IP from vars, unique hostId; sops
secrets (beszel, corosync authkey) are TODOs pending sync-host-keys.sh
variables.nix
haServer1/2Host, haServer1/2Ip, haServerVip, haStorageRoot, haIscsiIqn
ports.haServerDrbd/Iscsi/Corosync{1,2,Crypto}/PacemakerRemoted/Pcsd
scripts/ha/ (migrated + updated from test-lab/ha/)
cluster-init.sh — generates corosync authkey, initialises DRBD/XFS/iSCSI,
creates NFS dataset dirs, configures Pacemaker with DRBD + XFS + iSCSI
+ nfs-server + VIP; STONITH disabled initially (enable separately)
cluster-enable-stonith.sh — enables fence_pve_ssh STONITH after key deploy
fence-pve-ssh.py — Proxmox SSH fence agent (node names updated to ha-server-1/2)
acceptance-tests.sh — T1–T7 production acceptance tests
test-lab/ha/ removed — all Nix config moved to modules/ha/ and
modules/build-types/; scripts moved to scripts/ha/
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
SOPS_AGE_KEY_FILE was set in hosts/nixos/home.nix sessionVariables with a
literal ~ that Home Manager injects as-is into the environment. In bash,
tilde expansion does not happen inside double-quoted variable references, so
DEFAULT_SOPS_AGE_KEY_FILE resolved to ~/... literally and the -s file-existence
check in ensure_admin_decrypt_key silently failed. The script then generated
a brand-new age key (to ~/... relative to the repo root) while the real admin
key at ~/.config/sops/age/keys.txt went untouched -- making it appear the key
was lost when it was actually still intact.
Fix the home.nix root cause by using config.home.homeDirectory so the path
is fully resolved. Add tilde expansion in ensure_admin_decrypt_key as a
belt-and-suspenders guard for any caller whose environment has the same issue.
Also replace the auto-generate-a-new-key fallback with a hard failure: auto-
generating a new admin key is never useful (it cannot decrypt existing secrets)
and created serious confusion about whether the original key was lost.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the hardcoded hostname check (docker, pxe-boot) with a check
on config.fileSystems: any lxc-* host whose NixOS config declares an
NFS fileSystem entry is automatically made privileged. The script
already reads proxmoxLXC.privileged dynamically via
flake_target_lxc_privileged, so no logic change is needed there —
only the comment is updated to describe the new derivation.
Result: lxc-docker and lxc-pxe-boot (the two with NFS mounts) evaluate
as privileged=true; lxc-nix-cache, lxc-minimal, lxc-server,
lxc-tailscale-router, lxc-tor-relay evaluate as privileged=false.
Any future lxc-* host that declares an NFS mount gets the correct
privilege level for free without a separate manual edit.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
--pre-format-files placed the key on the QEMU builder VM's rootfs, not the
target disk. nixos-install chroots into the target and runs sshd-keygen, which
found no key in the chroot and generated a fresh (unregistered) one. sops then
could not decrypt on first boot because the key didn't match .sops.yaml, leaving
both root and nixos with '!' in /etc/shadow even after mutableUsers = false was
set (hashedPasswordFile pointed to paths sops never wrote).
Fix modules/platforms/proxmox.nix to embed the clan SSH host key in
environment.etc via NIXOS_HOST_KEYS_DIR at eval time -- the same pattern
lxc.nix uses. nixos-install's own activation places the key on the target disk,
sshd-keygen finds it already present and skips generation, and sops decrypts
correctly on first boot. Includes the same preserveSshHostKey/restoreSshHostKey
activation scripts as lxc.nix so subsequent nixos-rebuild switch calls (without
NIXOS_HOST_KEYS_DIR) don't remove the key as "obsolete" from environment.etc.
Update create-proxmox-resource.sh: switch VM builds from
./result-<target> --pre-format-files ... --build-memory 2048
to
NIXOS_HOST_KEYS_DIR=$(pwd)/host-keys nix build --impure ... diskoImagesScript
./result-<target> --build-memory 2048
matching the LXC build path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
Three bugs combined to leave every VM build with a shell but no boot disk:
1. The remote build script moved the raw image to /var/lib/vz/import/ before
qm importdisk could use it. If the mv failed (cross-filesystem copy, sudo
path, or any other reason) the remote script exited non-zero -- but the
local script's set -e handling of the SSH heredoc was inconsistent, so
qm create sometimes ran anyway, leaving a diskless VM shell.
Fix: skip the mv entirely. The diskoImagesScript writes <hostname>.raw into
its CWD (the remote repo dir, $out = $PWD at invocation). Import directly
from that path; clean it up after a successful import.
2. The qm importdisk output regex expected "Successfully imported disk as '...'"
but current Proxmox emits "unusedN: successfully imported disk '...'"
(lowercase, no "as"). The grep returned no match and exited 1.
3. The disk_id assignment used $(... | grep ...) without || true inside the
substitution. With set -euo pipefail, a non-zero grep exit aborts the
script before the fallback could run -- so the VM was always left with an
unattached unused0 disk.
Fix: update the primary regex to match the actual PVE format; add || true
inside the substitution so set -e never fires on a grep miss; add a qm
config fallback (scan for unusedN: lines) that works regardless of PVE
output format changes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011uRcikkTp3D5VbXj2DwNpQ
qm importdisk in QEMU 11.x outputs lowercase "successfully imported disk
as '...'" rather than the capitalised form the original grep expected.
The case mismatch made disk_id always empty, which caused the script to
exit 1 after qm create had already run -- leaving the VM with only an
EFI disk, no scsi0, and boot order still set to net0.
Fix by adding -i (case-insensitive) to the grep. Both the old capitalised
format (where the disk id had an "unused0:" prefix inside the quotes) and
the new lowercase format are handled correctly: the sed strip of unused0:
is preserved for backward compatibility, and the regex result is identical
either way.
Also add an early warning when --disk-size is passed for --type vm: the
flag is LXC-only for create mode and was silently ignored, leaving users
expecting a different size than the proxmoxImageSize in variables.nix.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
After the clan vars migration all keys are in vars/per-machine/, not
host-keys/. Update:
- File header: "existing clan var is never overwritten" (not host-keys/ file)
- Header --remove/--regenerate description: mention clan vars as primary
- usage() --remove, --regenerate-all-keys, --dry-run text
- cmd_remove/cmd_regenerate_all empty-guard messages
- README.md vars/per-machine/ row: "all deployed hosts" (not "LXC hosts")
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
locally_managed_hosts() only scanned host-keys/ (now empty for all
current targets), so --remove and --regenerate-all-keys silently did
nothing. Fix:
- locally_managed_hosts(): also yields targets from
vars/per-machine/*/openssh/ssh_host_ed25519_key/secret, deduped
- cmd_remove: shows [clan-vars] or [host-keys/] label per entry;
deletes vars/per-machine/<target>/openssh/ in addition to host-keys/
- cmd_regenerate_all: same -- removes clan vars dirs before regenerating
Also update CLAUDE.md and README.md to reflect that all flake targets
now use clan vars (not just lxc-*); host-keys/ is only for the
auto-installer's own pre-seeding path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
All actively deployed lxc-* hosts now have clan vars. Remove the legacy
scp -pr host-keys/ fallback in sync_remote_host_keys(): instead of
silently copying the gitignored directory, error clearly if no clan var
exists for the target and tell the operator how to generate one.
Also extend the uncommitted-changes check to cover vars/per-machine/ in
addition to .sops.yaml and secrets/, since clan vars must be committed
before the remote build git-pulls them.
Update the script header and sync_remote_host_keys comment to reflect
the new clan-only key flow.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Replaces the gitignored host-keys/ directory with clan vars as the
authoritative storage for SSH host keys. Keys are now generated as
sops-binary-encrypted clan var files (admin-key only) and checked into
vars/per-machine/<target>/openssh/, eliminating the plaintext private
key that previously had to live outside the repo.
Changes:
- modules/clan/ssh-host-key.nix: clan vars generator for the ed25519
SSH host key pair (neededFor="activation" — not mapped to sops.secrets,
delivered via tarball baking for LXC or --pre-format-files for VMs)
- flake.nix: add clanCore module + required settings to every mkTarget;
deduplicate bundled disko/sops-nix via follows; all 27 hosts eval clean
- flake.lock: updated to reflect the new follows constraints
- scripts/lib/clan-vars.sh: new helper library with
clan_ssh_key_exists / clan_ssh_pubkey_path / clan_decrypt_ssh_key /
clan_generate_ssh_key for use by the provisioning and sync scripts
- scripts/secrets/sync-host-keys.sh: queue_host_sync() now checks clan
vars first; generates via clan_generate_ssh_key if no key exists;
derives age fingerprint from clan pub key for .sops.yaml registration
- scripts/proxmox/create-proxmox-resource.sh: key management simplified
(sync-host-keys.sh now generates the key if missing, so the inline
prepare-host-key.sh call is gone); sync_remote_host_keys() decrypts
the clan key into a temp dir and scps just the two files to the node
when a clan key exists, falling back to the old host-keys/ scp for
any remaining legacy entries
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx
Three ordering-related fixes to the Proxmox provisioning flow:
1. prepare-host-key.sh: make idempotent -- if the key already exists, print
a note and exit 0 instead of erroring. The caller (create-proxmox-resource.sh)
already guards standalone calls, but the script itself should be safe to
run directly on a host that was already keyed.
2. create-proxmox-resource.sh: after sync-host-keys.sh updates .sops.yaml /
secrets/, detect uncommitted changes and block with a prompt until the
operator confirms they've committed and pushed. The PVE node's git pull
only picks up committed+pushed state; without this gate, a new host's sops
recipient is missing from the secrets files the image build uses, so the
host can't decrypt secrets on first boot.
3. create-proxmox-resource.sh: add an explicit existence check for the host
key in both the LXC and VM remote build heredocs, before it's passed as
--pre-format-files / NIXOS_HOST_KEYS_DIR input. Gives a clear error
pointing at sync-host-keys.sh instead of a raw `cp: cannot stat` from
disko deep in the build.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The disko images script does `cd "$TMPDIR"` before parsing its arguments,
so relative paths passed to --pre-format-files resolve against the temp
dir instead of the repo root. Use $(pwd) to capture the absolute repo
path before the disko script changes directory.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
If host-keys/<target>_ssh_host_ed25519_key doesn't exist, run
prepare-host-key.sh to generate it before sync-host-keys.sh runs.
Prevents sync-host-keys.sh from hitting its SKIP/exit-1 path (anchor
in .sops.yaml but no local key) and the downstream disko build failure
when --pre-format-files can't find the key file.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
read exits non-zero when stdin is not a terminal (set -e killed the
script silently). Catch that and emit a clear error pointing to the
SUDO_PASS environment variable rather than crashing with no output.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/dev/tty exists as a device node even without a controlling terminal,
so -r/-w tests pass but opening it fails. Plain 'read -r -s' from stdin
is enough: works interactively from a real terminal, and from a non-tty
context the caller should set SUDO_PASS in the environment instead.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Environments without a controlling terminal (containers, CI agents)
don't have /dev/tty. Try it first for the sudo password prompt, fall
back to plain stdin so the script works in both contexts.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Instead of ssh -t (requires PTY on both sides), prompt for the sudo
password once at startup and pipe it to each remote invocation via
sudo -S. This works from any context -- interactive terminal, background
agent, or script -- with no PTY needed on either end.
Also accepts SUDO_PASS from the environment for non-interactive callers.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ssh -t won't allocate a PTY when its own stdin is redirected (by a
heredoc). Replaced the heredoc-fed 'sudo bash -s' with commands passed
as an argument string so stdin stays free and -t can properly allocate
a PTY for the sudo password prompt.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
New script: scripts/secrets/push-host-keys.sh
- Pushes newly-generated SSH host keys from host-keys/ to already-running
NixOS hosts after sync-host-keys.sh --regenerate-all-keys.
- Before pushing any key, checks that .sops.yaml and secrets/*.yaml are
committed and pushed to the remote Gitea flake (hosts rebuild from there,
so recipient changes must land first); offers to auto-commit/push if not.
- Reads /etc/flake-target from each host to confirm which key to install,
handling the case where multiple flake targets share a hostname.
- Deduplicates by hostname in --all mode; skips hand-registered targets
that have no host-keys/ entry.
- --dry-run, --skip-git-check, SSH_USER override (default: nixos).
sync-host-keys.sh --regenerate-all-keys:
- Updated pre-confirmation warning to distinguish already-running hosts
(need push-host-keys.sh) from not-yet-deployed hosts (need installer
image rebuild).
- Added next-steps block after regeneration completes pointing to
push-host-keys.sh --all.
recover-hosts.sh:
- Header and SSH host key mismatch warn now cross-reference
push-host-keys.sh as the proactive (pre-drift) alternative.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NixOS's etc activation removes files that were in a previous generation's
environment.etc but absent from the current one -- even real copies, not
only symlinks. LXC tarballs bake the host key into environment.etc (via
NIXOS_HOST_KEYS_DIR), but every subsequent nixos-rebuild switch lacks that
env var, so the key is removed as "obsolete". sops-nix derives its age
decryption key from /etc/ssh/ssh_host_ed25519_key, so deletion cascades
into "Error getting data key: 0 successful groups required, got 0" for
every sops secret on the host.
Fix: two activation scripts bracket the etc step.
preserveSshHostKey (no deps, runs before etc): copies the live key to
/run (tmpfs) before etc can delete it.
restoreSshHostKey (deps=[etc], runs after etc): reinstalls via `install`
if etc removed the key. The resulting file is not tracked in either
generation's environment.etc, so subsequent rebuilds leave it alone.
scripts/recover-hosts.sh: restore both private and public key files (not
just the private key), use install(1) for atomic mode setting, and add a
post-rebuild sops-nix verification step to confirm success.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_014zT1L6hmsq6i1evAEH7dmi
The post-build cp/mv into /var/lib/vz/template/cache (LXC) and
/var/lib/vz/import (VM) are Proxmox-owned root directories -- they need
sudo_pfx just like pct/qm/pvesh do. nix build writes to the nix store
as the SSH user, but staging into /var/lib/vz/ requires root.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The script was root-only and hard-coded /etc/nix/nix.conf and
/etc/ssh/ssh_known_hosts, making it always fail (non-fatally) when
called as a non-root SSH user from create-proxmox-resource.sh.
Add dual-mode detection based on EUID:
- root (multi-user/daemon): existing behavior unchanged -- writes
/etc/nix/nix.conf, /etc/ssh/ssh_known_hosts, restarts nix-daemon
- non-root (single-user): writes ~/.config/nix/nix.conf and
~/.ssh/known_hosts, creates the config file if missing, skips the
daemon restart (single-user has no daemon), defaults REMOTE_BUILDER_KEY
to ~/.ssh/id_ed25519 instead of /root/.ssh/id_ed25519
create-proxmox-resource.sh already calls the script without sudo (as the
SSH user), so no change is needed there -- the script now handles both
cases on its own.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nix was installed as root on pve1 (the codex-setup.sh root path, meant
for container/Codex environments), making nix build require sudo there.
After cleaning up the root install and reinstalling as the SSH user
(wayne), nix is owned by that user and runs directly without sudo.
create-proxmox-resource.sh: drop sudo_pfx from nix build in both
remote scripts. The SSH user owns the store after reinstall; nix build
goes through the nix daemon-or-store directly. sudo stays on pct/qm/pvesh
(cluster IPC) and the disko image-writer script (writes to disk).
codex-setup.sh: add build-users-group = (empty) to the user nix.conf
written by the non-root install path. Guards against a stale
/etc/nix/nix.conf from a prior root install (which sets
build-users-group = nixbld) silently breaking single-user builds.
Manual cleanup required once on each Proxmox node that had root's nix:
sudo rm -rf /nix /etc/nix
sudo rm -f /etc/profile.d/nix.sh /etc/profile.d/nix-daemon.sh
for i in $(seq 1 10); do sudo userdel nixbld$i 2>/dev/null||true; done
sudo groupdel nixbld 2>/dev/null || true
After that, the next create-proxmox-resource.sh run auto-reinstalls
nix as the SSH user via codex-setup.sh.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nix on pve1 was installed as root (single-user), so wayne can't access
/nix/var/nix/db/big-lock without root -- nix build genuinely needs sudo
there. The previous fix to drop sudo_pfx was wrong.
The real fix is node config: add nix to wayne's NOPASSWD rules in
sudoers on pve1 (see below). pct/qm/pvesh already have NOPASSWD and
work fine in non-interactive SSH heredocs; nix was just missing from
that list.
On pve1 as root:
echo 'wayne ALL=(root) NOPASSWD: ALL' | tee /etc/sudoers.d/wayne-nopasswd
chmod 440 /etc/sudoers.d/wayne-nopasswd
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
nix build runs through the nix daemon and doesn't need root; the
tooling-check step already confirms the SSH user can run nix directly
(ensure_nix_profile + command -v nix). sudo without a TTY blocks
non-interactive SSH heredoc sessions with "a terminal is required".
Keep sudo on pct/qm/pvesh (cluster IPC) and the disko image-writer
script (writes to block devices) -- those actually require root.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single-user Nix installations are owned by root. When PROXMOX_SSH_USER
is not root, prefix the remote nix build command with sudo_prefix, same
as the Proxmox tool invocations. Passes sudo_prefix as an extra arg to
both the LXC tarball and VM disko image build heredocs.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
/root/nixos was only correct when PROXMOX_SSH_USER=root. Now that it
defaults to wayne, use /home/${PROXMOX_SSH_USER}/nixos so git clone
goes somewhere the SSH user can actually write to.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PROXMOX_SSH_USER was changed from root to wayne, but all remote pvesh/qm/pct
invocations assumed root. When run as a non-root user these commands fail with
ipcc_send_rec errors because they can't reach the pve-cluster IPC socket.
Adds a global sudo_prefix (empty when PROXMOX_SSH_USER=root, "sudo" otherwise)
and applies it to every remote Proxmox command in the script, including the
duplicate-host heredoc check, pvesh nextid, vmid existence checks, resource
destruction, and all create/start commands. Removes the now-redundant local
sudo_prefix definition that was previously only in the VM image build branch.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The previous fix (#42) only unmounted /mnt/{dev,proc,sys,run}, but disko
also mounts the ESP at /mnt/boot (modules/disko/baremetal.nix) -- another
nested mount blocking ZFS from unmounting its own root dataset at /mnt
the same way. Confirmed live: zpool export still failed with "cannot
unmount '/mnt': pool or dataset busy" after the chroot-only fix.
Replace the manual dev/proc/sys/run list with a single recursive
`umount -R /mnt`, which clears everything nested under /mnt -- current
and future mountpoints alike -- rather than needing to keep enumerating
whatever nixos-install/disko happen to leave mounted.
nixos-install bind-mounts /dev, /proc, /sys (and usually /run) into
/mnt to run the target's activation script in a chroot, and doesn't
unmount them again afterward. Left in place, those nested mounts made
ZFS refuse to unmount its own root dataset at /mnt: zpool export
failed with "cannot unmount '/mnt': pool or dataset busy", and because
of this script's set -e, that killed the script before it ever reached
reboot -- silently defeating the export-before-reboot fix from #40 on
every real run, which is why the ZFS-import stall kept recurring.
disko's --mode ...,mount leaves the pool imported (needed for
nixos-install to write into /mnt), and the script rebooted straight
into the newly-installed system without exporting it. That pool is
still stamped with the live installer's own hostid, which never
matches the target host's declared networking.hostId, and since
boot.zfs.forceImportRoot is false (the recommended setting, not a bug),
the first real boot refuses to force-import an unexported pool from a
different hostid -- which is exactly the ZFS-import stall baremetal-gui
was hitting after install. Exporting all pools right before reboot (a
no-op for non-ZFS hosts) clears the in-use state so import succeeds
regardless of hostid.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
