feat(provision): Phase 0-2 + fix — clan-core SSH host keys, activation ordering, boot-time sops #56

2026-07-25T09:14:23Z

beatzaplenty commented

2026-07-25 09:14:23 +00:00

Summary

Phase 0: Guard create-proxmox-resource.sh against missing host keys and uncommitted sops changes
Phase 1: Add clan-core 26.05 as a flake input (disko/sops-nix follows to avoid duplicate module conflicts)
Phase 2: Migrate SSH host keys to clan vars — sops binary-encrypted, admin-key-only, committed to repo instead of gitignored host-keys/
Fix: Two bugs in modules/platforms/lxc.nix confirmed live on deployed lxc-tor-relay:
1. Activation ordering: preserveSshHostKey ran after etc (position 7 vs 5), so etc deleted the key before it could be saved. Added deps to enforce chain: preserveSshHostKey -> etc -> restoreSshHostKey -> setupSecrets.
2. No boot-time secrets: sops-nix does not generate a boot service in this config (/run/secrets is tmpfs). Added nixos-lxc-sops-reinstall.service (wantedBy/before sysinit.target, DefaultDependencies=false) to reinstall secrets on every non-first boot.

Test plan

All 27 hosts + packages eval clean (codex-maintenance.sh passes)
Redeploy lxc-tor-relay on pve1 with --allow-duplicate-host --force-rebuild to restore correct SSH key and apply lxc.nix fix
Verify nixos-rebuild switch on tor-relay no longer fails with removing obsolete file / sops key error
Verify beszel-agent.service starts successfully after reboot

## Summary - **Phase 0**: Guard create-proxmox-resource.sh against missing host keys and uncommitted sops changes - **Phase 1**: Add clan-core 26.05 as a flake input (disko/sops-nix follows to avoid duplicate module conflicts) - **Phase 2**: Migrate SSH host keys to clan vars — sops binary-encrypted, admin-key-only, committed to repo instead of gitignored host-keys/ - **Fix**: Two bugs in modules/platforms/lxc.nix confirmed live on deployed lxc-tor-relay: 1. Activation ordering: preserveSshHostKey ran after etc (position 7 vs 5), so etc deleted the key before it could be saved. Added deps to enforce chain: preserveSshHostKey -> etc -> restoreSshHostKey -> setupSecrets. 2. No boot-time secrets: sops-nix does not generate a boot service in this config (/run/secrets is tmpfs). Added nixos-lxc-sops-reinstall.service (wantedBy/before sysinit.target, DefaultDependencies=false) to reinstall secrets on every non-first boot. ## Test plan - [ ] All 27 hosts + packages eval clean (codex-maintenance.sh passes) - [ ] Redeploy lxc-tor-relay on pve1 with --allow-duplicate-host --force-rebuild to restore correct SSH key and apply lxc.nix fix - [ ] Verify nixos-rebuild switch on tor-relay no longer fails with removing obsolete file / sops key error - [ ] Verify beszel-agent.service starts successfully after reboot

beatzaplenty added 4 commits 2026-07-25 09:14:24 +00:00

fix(provision): block build until sops changes are committed, guard missing host keys 78bb784265

Three ordering-related fixes to the Proxmox provisioning flow:

1. prepare-host-key.sh: make idempotent -- if the key already exists, print
   a note and exit 0 instead of erroring. The caller (create-proxmox-resource.sh)
   already guards standalone calls, but the script itself should be safe to
   run directly on a host that was already keyed.

2. create-proxmox-resource.sh: after sync-host-keys.sh updates .sops.yaml /
   secrets/, detect uncommitted changes and block with a prompt until the
   operator confirms they've committed and pushed. The PVE node's git pull
   only picks up committed+pushed state; without this gate, a new host's sops
   recipient is missing from the secrets files the image build uses, so the
   host can't decrypt secrets on first boot.

3. create-proxmox-resource.sh: add an explicit existence check for the host
   key in both the LXC and VM remote build heredocs, before it's passed as
   --pre-format-files / NIXOS_HOST_KEYS_DIR input. Gives a clear error
   pointing at sync-host-keys.sh instead of a raw `cp: cannot stat` from
   disko deep in the build.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(flake): add clan-core 26.05 as a flake input (Phase 1, no behavior change) a9745b594b

Introduces clan-core pinned to its 26.05 release alongside the existing nixpkgs
26.05 input. No host configuration is changed — this is a pure dependency
addition so Phase 2 (per-host vars/secret management migration) has the input
available without a separate flake.lock bump.

clan-core.inputs.nixpkgs.follows = "nixpkgs" keeps a single nixpkgs closure.
sops-nix remains as a flake input; vars layers on top of it rather than
replacing it (clan's sops storage backend still needs sops-nix).

All hosts evaluate cleanly (codex-maintenance.sh --full-check equivalent
triggered by the flake.nix change).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(provision): Phase 2 — migrate SSH host keys to clan vars a63e1c70c3

Replaces the gitignored host-keys/ directory with clan vars as the
authoritative storage for SSH host keys. Keys are now generated as
sops-binary-encrypted clan var files (admin-key only) and checked into
vars/per-machine/<target>/openssh/, eliminating the plaintext private
key that previously had to live outside the repo.

Changes:
- modules/clan/ssh-host-key.nix: clan vars generator for the ed25519
  SSH host key pair (neededFor="activation" — not mapped to sops.secrets,
  delivered via tarball baking for LXC or --pre-format-files for VMs)
- flake.nix: add clanCore module + required settings to every mkTarget;
  deduplicate bundled disko/sops-nix via follows; all 27 hosts eval clean
- flake.lock: updated to reflect the new follows constraints
- scripts/lib/clan-vars.sh: new helper library with
  clan_ssh_key_exists / clan_ssh_pubkey_path / clan_decrypt_ssh_key /
  clan_generate_ssh_key for use by the provisioning and sync scripts
- scripts/secrets/sync-host-keys.sh: queue_host_sync() now checks clan
  vars first; generates via clan_generate_ssh_key if no key exists;
  derives age fingerprint from clan pub key for .sops.yaml registration
- scripts/proxmox/create-proxmox-resource.sh: key management simplified
  (sync-host-keys.sh now generates the key if missing, so the inline
  prepare-host-key.sh call is gone); sync_remote_host_keys() decrypts
  the clan key into a temp dir and scps just the two files to the node
  when a clan key exists, falling back to the old host-keys/ scp for
  any remaining legacy entries

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx

fix(lxc): fix activation ordering and add boot-time sops reinstall

Check NixOS configurations / eval-hosts (pull_request) Successful in 10m33s

Details

055577ee91

Two bugs prevented nixos-rebuild switch from working on lxc-* hosts after
first boot, both confirmed live on a deployed lxc-tor-relay container:

1. Ordering bug: preserveSshHostKey had no explicit deps, so the topological
   sort placed it at position 7 — after etc at position 5. By the time it
   tried to save the SSH key, etc had already removed it as "obsolete"
   (absent from the current generation's environment.etc when built without
   NIXOS_HOST_KEYS_DIR). Consolidate all four system.activationScripts entries
   into one block and add etc = { deps = ["preserveSshHostKey"]; } and
   setupSecrets = { deps = ["restoreSshHostKey"]; } to enforce the correct
   save→etc→restore→sops chain.

2. No boot-time secrets: /run/secrets is a tmpfs cleared on every reboot, and
   sops-nix does NOT generate a boot-time service in this configuration
   (confirmed live: no sops-nix.service in systemctl list-unit-files).
   Add nixos-lxc-sops-reinstall.service, modelled after sops-nix's own service
   placement (wantedBy/before sysinit.target, DefaultDependencies=false), so
   secrets are reinstalled before basic.target on every non-first boot.
   ConditionPathExists skips it on first boot; nixos-lxc-first-boot-activate
   handles that case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01B2EJ4qTsM5KUqhS5c3GAwx

beatzaplenty merged commit 0853952269 into main

2026-07-25 09:16:43 +00:00

This repo is archived. You cannot comment on pull requests.

1 Participants

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: beatzaplenty/nixos#56