Root cause of recurring sops failures on new VM boots: disko builds raw
disk images, and create-proxmox-resource.sh only syncs the clan-var SSH
host key to the Proxmox node (for proxmox.nix to bake into the image) when
it actually builds — reusing a cached image skips sync_remote_host_keys, so
destroy+recreate reuses a stale image with the wrong or randomly-generated
key baked in. On first boot the VM gets a different key than what .sops.yaml
was encrypted for, and sops fails permanently.
Fix 1 — deploy.sh Phase 3: always pass --force-rebuild so every VM creation
rebuilds the disko image fresh with the current clan-var key baked in via
NIXOS_HOST_KEYS_DIR (proxmox.nix already reads this under --impure).
Fix 2 — deploy.sh Phase 5.5: after VMs boot, scan their actual ed25519 host
keys and, if they drift from clan vars, update the clan var pub-key files,
rewrite the .sops.yaml age anchors, and re-encrypt all affected sops files.
Defence-in-depth: normally a no-op after Fix 1, but catches any residual
mismatch (e.g. --skip-create-vms reuse of an older image).
Fix 3 — cluster-init.sh: add crm_standby -v on for both nodes before DRBD
metadata init. Without this, Pacemaker's OCF DRBD agent races: it sees
drbdadm down as a failure and immediately calls drbdadm up again, leaving
/dev/sdb busy when create-md / write-dev-uuid runs (drbdmeta exits 20 with
"stdin not a TTY, not waiting for confirmation"). Standby suppresses
resource scheduling during init; crm_standby -v off restores it after DRBD
is up on both nodes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
drbd.service runs drbdadm up all at activation time, but Pacemaker's
OCF drbd agent manages the resource lifecycle (up/down/promote/demote).
When both run simultaneously the device is busy, causing drbdmeta
apply-al to abort with exit 20.
Set wantedBy = [] so the kernel module and config are still present
(via services.drbd.enable = true) but systemd doesn't auto-start the
service. Pacemaker's OCF agent calls drbdadm directly.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
At first boot, pacemaker fails with 'Dependency failed' because ipa-activation
fails (IPA not enrolled yet) before corosync/authkey are ready, causing a
systemd boot-ordering race. The service recovers fine when started manually.
Add an explicit 'systemctl start pacemaker' on both nodes immediately after
restarting corosync so cluster-init doesn't time out waiting for it.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
The _patch_targetctl runtime dropin was a workaround for the wrong targetctl
binary path in the original iscsi-target.nix (used pkgs.targetcli-fb, but
targetctl is in rtslib-fb's python3 env). The module is now fixed to use
\${python3}/bin/targetctl, so new VM images have the correct ExecStart/ExecStop
baked in and the patch finds nothing — failing with exit 1 and killing the deploy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Adds nextcloud-client, chromium, git, vim, claude-code, fish, sops,
the Nextcloud tray service, SOPS_AGE_KEY_FILE, and Proxmox/PBS desktop
shortcuts to wayne's HM config on GUI hosts. Uses lib.mkDefault on the
EDITOR setting in ipa/client.nix so the GUI layer can override to vim.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Both secrets/ha-server-{1,2}.keytab had unresolved conflict markers from
the stash/merge interaction. Keep the upstream version (2 age recipients:
admin + the host's own key) so each host can decrypt its own keytab on boot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
The HA branch merge left conflict markers in the ha-server-{1,2}.keytab
path_regex entries. Keep the upstream side (adds *proxmox-ha-server-{1,2}
as recipients) so each host can decrypt its own IPA keytab on boot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Deploy/init fixes:
- iscsi-target.nix: targetctl binary is in rtslib-fb (python3 env), not
targetcli-fb — fixes ExecStart and ExecStop for the targetctl.service
- deploy.sh: _patch_targetctl() applies runtime dropin to both nodes before
cluster-init so Pacemaker can manage the iSCSI target from first start
- cluster-init.sh: replace crm configure heredoc with cibadmin --replace XML
(pacemaker-4.0 schema: globally-unique in meta_attributes, promoted-max/
promoted-node-max, Promoted role in constraints); force_unmount=true on
xfs-data; DRBD promote timeout 240s
- cluster-config.nix: add crm-fence-peer.sh/crm-unfence-peer.sh handlers;
update fencing comment to reflect resource-only + Pacemaker-aware handler
replacing STONITH during testing phase
- ha-server.nix: add openiscsi to systemPackages for T4 iscsiadm availability
Acceptance test fixes:
- acceptance-tests.sh: fix ((PASS++)) set -e bug → PASS=$((PASS+1));
detect Active/Standby dynamically via drbdadm role (Pacemaker can promote
either node); T4 bash TCP probe instead of iscsiadm; T5 timeout 120s;
T6 echo|sudo tee for root-owned XFS write (bash -c redirect runs as nixos
not sudo — permission denied); use ns cat / ns rm for root-owned reads
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
NixOS's group activation uses plain groupmod, which silently skips the
GID change when SSSD exposes the same GID (50010) via the IPA
docker-access group through NSS. Add an activation script that runs
after the normal 'groups' step and applies groupmod --non-unique so the
local docker group can share GID 50010 with the SSSD-provided IPA group.
If the GID actually changes on a live system the script also restarts
docker.socket + docker.service so the socket is recreated with the new
GID.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
cibadmin raw XML fails schema validation under pacemaker-4.0: globally-unique
is not a valid direct <clone> attribute, and master-max/master-node-max are
renamed. Switch to crm configure commands which are schema-version-aware:
- promotable clone with promoted-max/promoted-node-max
- order/colocation constraints using Promoted role (Pacemaker 4.0 naming)
- crm configure handles schema differences automatically
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
clearconfig does not reliably clear kernel LIO configfs state — the kernel
still holds backing-file references, blocking umount. Replace clearconfig
with explicit targetcli delete commands (/iscsi delete, /backstores/fileio
delete) which do release kernel state. Also make DRBD secondary demote
idempotent (skip if already Secondary).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Remove VIP-specific portal binding — the VIP doesn't exist until Pacemaker
assigns it; the default all-IPs portal (::0:3260) is correct for Pacemaker
to manage
- Clear existing LIO targets before re-running targetcli (idempotent on
partial failures)
- Tear down LIO kernel objects after saveconfig so umount succeeds (LIO holds
the backing file open otherwise)
- Guard mount with mountpoint check so re-runs don't fail when already mounted
- Use --replace for cibadmin constraints (idempotent vs --create)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- ha-server.nix: add xfsprogs to systemPackages so mkfs.xfs is on PATH for
root (needed by cluster-init.sh during initial setup)
- cluster-config.nix: create /var/lib/drbd via tmpfiles to silence
lk_bdev_save warnings from drbd-utils
- cluster-init.sh: dynamically find xfsprogs in /nix/store if not on PATH
(fallback for running VMs before xfsprogs is in the system profile)
- cluster-init.sh: fix DRBD metadata check on node2 — broken regex now uses
grep -E for ERE alternation to correctly skip create-md when DRBD is already
set up (previous regex would have triggered create-md on a live secondary)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Root SSH was failing because only the RSA admin key was authorized but the
local dev box only has an ed25519 key. Fix:
- cluster-config.nix: add ed25519 keys to root (same set as nixos user) so
future deployments work without the temp-key workaround
- deploy.sh/acceptance-tests.sh: SSH as nixos user with sudo instead of root@
- cluster-init.sh: HA_USER/HA_KEY env vars + n2_ssh()/n2_scp() helpers so
inter-node SSH works regardless of whether root-to-root is available
- deploy.sh Phase 6: generate temp keypair, authorize on node2, place on node1
for root to use during cluster-init, clean up afterward
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Two independent fixes:
1. hashedPassword = "!" on the IPA user stub
Without any shadow entry, pam_unix returns PAM_AUTHINFO_UNAVAIL
before prompting, so PAM_AUTHTOK is never set. The "!" marker
(account locked, not a real hash) ensures a shadow entry exists so
pam_unix prompts and sets PAM_AUTHTOK — which the subsequent
pam_sss module can then use.
2. pam_sss try_first_pass instead of use_first_pass (login + su)
use_first_pass silently fails when PAM_AUTHTOK is unset.
try_first_pass prompts independently in that case, making IPA
password auth work even if pam_unix returns early for any reason.
Root cause found during incident: wayne (uid 50002) was outside IPA's
auto-SID range (163800000+), so sidgen never assigned him an
ipaNTSecurityIdentifier. Without it, ipadb's handle_authdata step fails
with "Generic error" after SPAKE pre-auth succeeds. Fixed by manually
adding objectClass: ipaNTUserAttrs + ipaNTSecurityIdentifier RID 550002
to wayne's LDAP entry on domain-controller. Any future IPA user with a
manually-assigned uid outside the auto-range needs the same treatment.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pve_check() always uses sudo, so "sudo test -w /nix" passes as root
regardless of whether the SSH user can actually write there. Use a
direct non-sudo SSH command for the writability probe so the check
reflects wayne's own access, not root's.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
When create-proxmox-resource.sh clones the repo to pve1, it stays on
whatever branch was checked out. Add a pre-build phase that detects
branch mismatch and switches the remote clone to the current local
branch before building, so the Proxmox node always builds from the
same commits we're deploying.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
pve1 has a pre-existing /nix store owned by a different UID; wayne's IPA
UID (50002) can't write to it. Add a pre-phase-3 check that uses wayne's
passwordless sudo to chown -R the store before create-proxmox-resource.sh
runs codex-setup.sh on the node.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Add root SSH key + passwordless sudo to ha-server nodes (needed for
deploy script to run cluster-init.sh via SSH as root on node1)
- Fix cluster-init.sh: correct default IPs (228/227/229 per variables.nix),
use \${VAR:-default} for all config so deploy.sh can override via env
- Fix acceptance-tests.sh: same IP corrections, add -i flag to SSH calls,
use \${VAR:-default} pattern
- Fix deploy.sh dry-run bugs: pve_check() always runs SSH for read-only
probes so bridge existence check is accurate; wait_for_ssh skips in
dry-run instead of timing out
- Fix cluster-init invocation: upload script via scp and run via SSH as
root on node1 (was incorrectly trying to run the script locally)
- Fix acceptance-tests invocation: pass IP env vars from deploy.sh
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Run sync-host-keys.sh for proxmox-ha-server-{1,2}: generates SSH host
key pairs in vars/per-machine/, registers age anchors in .sops.yaml,
adds both hosts as recipients for common.yaml, ha-corosync-authkey,
and per-host secrets/keytab files
- Re-encrypt secrets/common.yaml with both new host keys
- Convert all stub secrets to real sops-encrypted files:
secrets/ha-server-{1,2}.yaml (YAML, beszel-token = PLACEHOLDER)
secrets/ha-server-{1,2}.keytab (binary, stub text encrypted)
secrets/ha-corosync-authkey (binary, stub text encrypted)
- Add scripts/ha/deploy.sh: full lifecycle script (bridge setup, VM
creation, DRBD disk + storage NIC attachment, boot wait, cluster-init,
acceptance tests, --destroy)
Bootstrap order (operator runs these before first deploy):
1. bash scripts/ha/deploy.sh # deploys, tests
# Post-deploy secret replacement:
2. sops secrets/ha-server-{1,2}.yaml (set real beszel-token)
3. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
5. Set services.beszel.agent.environment.KEY in host.nix after hub pairing
6. nixos-rebuild switch on both nodes to pick up real secrets
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
- Add stub keytab files (secrets/ha-server-{1,2}.keytab) so modules/ipa/client.nix
activates for both nodes — IPA security.ipa.enable = true confirmed by eval
- Add .sops.yaml creation rules for both keytabs (admin-only until
sync-host-keys.sh registers the host age keys)
Eval: both proxmox-ha-server-1 and proxmox-ha-server-2 build cleanly
with IPA enabled, beszel agent enabled, corosync authkey + beszel-token
+ ipa-host-keytab all wired via sops-nix.
Bootstrap order before first deploy:
1. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-1
2. bash scripts/secrets/sync-host-keys.sh proxmox-ha-server-2
3. sops updatekeys secrets/common.yaml
4. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.228 ha-server-1
5. bash scripts/ipa/create-nixos-ipa-host-account.sh --ip 192.168.2.227 ha-server-2
6. sops secrets/ha-server-1.yaml (set beszel-token)
7. sops secrets/ha-server-2.yaml (set beszel-token)
8. On node1 after first boot: corosync-keygen, then
sops -e --input-type binary /etc/corosync/authkey > secrets/ha-corosync-authkey
9. Set services.beszel.agent.environment.KEY in each host.nix after hub pairing
10. git add/commit the real secrets, then nixos-rebuild switch on both nodes
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HaH1cSGvhogRP5ExoF6nD8
Documentation fixes:
- README/AGENTS: rename tailscale-exit-node → tailscale-router, add ha-server
build type and proxmox-ha-server-{1,2} host table rows, add baremetal to
platform list, remove references to non-existent flake-target-refactor-spec.md
and remove-sensetive-info-refactor.md
- docs/auto-installer.md: fix lxc-tailscale-exit-node → lxc-tailscale-router,
add pxe-minimal to the flake outputs list
- variables.nix: fix domainControllerIp comment — IPA is the authoritative DNS
at .253 (Pi-hole is gone), not a forwarding intermediary
Code deduplication:
- Extract duplicate SSH host-key preservation activation scripts from
modules/platforms/lxc.nix and modules/platforms/proxmox.nix into a shared
modules/common/preserve-ssh-host-key.nix; both platforms now import it
- Replace 8-line hand-enumerated NFS export lists in server.nix and ha-server.nix
with a mkNfsExports helper that generates exports from vars.nfsShares — adding
a share to variables.nix now propagates to both exporters automatically
Dead code removal:
- modules/common/configuration.nix: remove leftover NixOS skeleton comments
(hardware-configuration import, grub lines) that were never used
- modules/docker/enable-service.nix: remove commented-out listenOptions and
daemon.settings blocks
- hosts/server/host.nix, hosts/nix-cache/host.nix: remove #DOCKER_HOST comments
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Create an IPA group "docker-access" (GID 50010) and pin the local
"docker" group to that GID on all Docker hosts. Any IPA user in the
docker-access group automatically gains docker socket access through
SSSD supplementary-group resolution — no per-host docker.members
entry needed.
Specific changes:
- variables.nix: add dockerAccessGid = 50010
- modules/docker/enable-service.nix: lib.mkForce docker GID to
dockerAccessGid, removing the need to name individual IPA users
- modules/build-types/docker.nix: remove direct wayne docker.members
entry (access now comes from IPA group)
- modules/ipa/client.nix: refactor repeated security.* / systemd.*
top-level keys into merged attribute sets (fixes statix W20); add
security.pam.services.lightdm.makeHomeDir so the GUI login path
also creates the home dir on first login
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Grants the domain wayne account docker socket access on docker.sweet.home
without needing sudo, alongside the existing nixos local user membership.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
SSSD's dyndns_iface defaults to "*" which registers every interface's IP in
IPA DNS, including all Docker bridge networks (172.x.x.x). This caused
docker.sweet.home to resolve to Docker bridge IPs instead of 192.168.2.225.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
On first enrollment /home/wayne doesn't exist until the IPA user's first
login (pam_mkhomedir creates it). home-manager-<user>.service cd's into
the home dir immediately and fails with ENOENT, causing the whole rebuild
activation to return exit code 4.
Add ConditionPathExists so systemd skips the service (condition not met,
no failure) instead. After first login the dir exists and subsequent
rebuilds activate Home Manager normally.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Passing a FQDN like "nixos.sweet.home" instead of the short hostname
"nixos" caused the script to create a double-FQDN IPA host account
(nixos.sweet.home.sweet.home). Add an early check that rejects any
TARGET containing a dot.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HM's useUserPackages creates a users.users stub for every configured HM
user, which lands wayne in /etc/passwd. NixOS adds pam_sss.so with the
"localusers" flag to the sudo PAM stack when SSSD is enabled; that flag
causes pam_sss to skip SSSD for any user found in local /etc/passwd,
falling through to pam_unix which has no shadow password for the stub.
Result: sudo auth always fails for the IPA user despite being in wheel.
Use NOPASSWD for the IPA user in sudoers instead. The IPA user already
authenticated to reach a shell (SSH key from IPA or Kerberos), so
re-prompting via a broken PAM path is security theater on a homelab.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
initgroups() uses NSS (groups: files sss) to build the supplemental
group list at login. Adding wheel to the local users.users stub means
the IPA user gets wheel membership from /etc/group, satisfying sudo's
group check.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Any enrolled host now automatically gets a Home Manager profile for the
IPA primary user (vars.ipaUser = "wayne"), covering what IPA doesn't:
dotfiles, user-scoped packages (tmux, sshfs), and EDITOR variable.
The home directory is pre-created by systemd-tmpfiles so HM activation
succeeds on steady-state systems before first login; pam_mkhomedir remains
as a fallback for fresh deploys where SSSD hasn't cached the user yet.
A minimal users.users stub satisfies NixOS's assertion requirements
(isNormalUser + group) that arise because home-manager.useUserPackages
creates a users.users entry to install packages to /etc/profiles/per-user/.
The stub is shadowed by SSSD at runtime (security.ipa sets passwd: sss files).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
IPA users (e.g. wayne@) don't get Home Manager so the aliases defined in
aliases.nix were invisible to them. Move Switch-nix, Test-nix, and
buildImage into programs.bash in configuration.nix so every user on every
host gets them via /etc/bashrc. Stub out aliases.nix for future per-user
HM-only additions.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The NixOS IPA module adds ConditionPathExists=/etc/krb5.keytab drop-ins
for auth-rpcgss-module.service and rpc-gssd.service via systemd.units.
In LXC containers with keytabs those conditions pass, the services start,
and then fail because auth_rpcgss can't be loaded and rpc_pipefs doesn't
exist in the container namespace.
Use lib.mkForce on our systemd.units text to win the conflict with NixOS's
existing definitions, and include ConditionVirtualization=!container
alongside the ConditionPathExists conditions so the services are skipped
(inactive, not failed) in containers that have a keytab.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
modules/ipa/client.nix is now a self-contained NixOS module (no longer a
parameterized function): it checks builtins.pathExists for
secrets/<hostname>.keytab at eval time and enables itself automatically if
found, making it a no-op for hosts without a keytab.
modules/common/configuration.nix imports it so every host in the flake is
a candidate for IPA enrollment — no per-host wiring needed. Adding a
keytab (via scripts/ipa/create-nixos-ipa-host-account.sh) is now the
only step required to enroll a host.
The module also sets networking.domain and networking.nameservers via
mkDefault when active, so new hosts don't need those set explicitly.
Also:
- Remove explicit IPA imports from hosts/nix-cache and hosts/tailscale-router
- Add secrets/pxe-boot.keytab + creation rule; remove incorrect
secrets/nixos.sweet.home.keytab and its creation rule
- Add .sops.yaml creation rules for all remaining host keytabs
(server, docker, tor-relay, nix-minimal, nixos) so the creation script
can target them without manual .sops.yaml edits
- Fix duplicate tailscale-router.keytab rule and corrupted gui.yaml comment
block in .sops.yaml
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Script fixes:
- Rename HOSTNAME variable to TARGET (shadowed the bash builtin)
- Fix ipa-getkeytab -s to always use IPA_SERVER, not DC_HOST (diverge if
--dc is overridden to a jump host)
- Remove dead REALM variable
- Add EXIT trap to delete the plaintext keytab if the script aborts before
sops encryption completes; cleared after successful encrypt
- Distinguish real ipa host-add failures from "already exists" instead of
swallowing all errors with || true
- Warn explicitly when no platform age keys exist for the target (keytab
would be admin-only and the host couldn't decrypt it at boot)
- Fix sops fallback from pinned nixos-25.11 channel to nixpkgs (uses the
repo's own flake.lock)
- Expand "next steps" output to include networking.domain and nameservers
lines that host.nix requires for IPA membership
Module docs:
- Point to the script as the primary setup path; move manual steps to a
fallback section
- Note that certs/ipa-ca.crt is already committed (no need to re-fetch)
- Document the networking.domain and nameservers requirements in the header
- Add sync-host-keys.sh as explicit step 0
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
OpenSSH 10.0 tightened AuthorizedKeysCommand security by checking every
path component of the command binary for group/world-write permission.
/nix/store is 1775 (group-writable by nixbld), so sshd silently skips
the command for any binary in the Nix store — causing IPA pubkey auth to
silently fail with no diagnostic.
Fix: copy sss_ssh_authorizedkeys to /usr/local/bin via systemd tmpfiles
(C+ copies the file rather than symlinking, so the path at runtime is
root-owned/755 throughout), and point AuthorizedKeysCommand at the copy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Imports docker/enable-service.nix and pins virtualisation.docker to
docker_29 via a nixpkgs overlay. Fixes indentation on `imports` block
and `programs.direnv.enable` line so nixpkgs-fmt passes.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- security.pam.services.sshd.makeHomeDir: IPA users have no pre-created
home directory on the host; without this, sshd opens a session to a
missing directory and resets the connection immediately after auth
- AuthorizedKeysCommand was already added in previous commit
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- modules/ipa/client.nix: add AuthorizedKeysCommand so sshd fetches SSH
public keys from IPA via sss_ssh_authorizedkeys, enabling pubkey login
without per-host authorized_keys files
- hosts/tailscale-router/host.nix: add IPA client module + networking.domain
so SSSD runs and wayne can authenticate on this host
- secrets/tailscale-router.keytab: sops-encrypted keytab for
tailscale-router.sweet.home (generated by create-nixos-ipa-host-account.sh)
- .sops.yaml: creation rule for secrets/tailscale-router.keytab
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
NixOS's dnsmasq module defaults resolveLocalQueries to true, which adds
127.0.0.1 to networking.nameservers and binds dnsmasq to listen-address=127.0.0.1.
This made the host route all its own DNS through dnsmasq, which had
no-resolv=true and no upstream for anything outside the tailnet domain —
so every non-tailscale DNS query from the host itself (including SSSD
resolving the IPA server FQDN after the IPA client module was added) failed.
Setting resolveLocalQueries=false limits dnsmasq to its intended role: a
forwarding proxy reachable on the LAN interface for IPA's conditional
forwarder. The host uses domainControllerIp directly (already set in
networking.nameservers in host.nix).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ipa-getkeytab runs as root via sudo so the temp file is root-owned;
scp as wayne gets Permission denied. Pipe through `sudo cat` over SSH
instead, which reads as root but writes locally as the invoking user.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Single command to enroll a NixOS host in FreeIPA and produce a
sops-encrypted keytab at secrets/<hostname>.keytab:
- Adds the .sops.yaml creation rule automatically (with all registered
platform-variant age keys as recipients)
- SSHes to the domain controller to run ipa host-add + ipa-getkeytab
- Refreshes the admin Kerberos ticket via `ssh -t ... kinit admin` if
missing or expired, so no manual kinit step is needed
- SCPs the keytab and encrypts it in-place with sops (file must be at
secrets/<hostname>.keytab before encryption so the path-based creation
rule matches — the common failure point when doing this manually)
Also adds HOME_DOMAIN and IPA_SERVER to scripts/env.sh, matching
variables.nix's homeDomain/ipaServer (same manual-sync pattern as
NIX_CACHE_HOST/LAN_DOMAIN).
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sops matches creation rules against the input file path, so encrypting
/tmp/<host>.keytab directly with stdout redirect fails to find the rule.
Copy to secrets/ first, then use -i to encrypt in-place.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
ipa-getkeytab confirmed ipa.sweet.home doesn't respond to LDAP;
domain-controller.sweet.home is the actual IPA server FQDN.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds modules/ipa/client.nix — a parameterized module that joins a NixOS host
to the sweet.home FreeIPA domain without ipa-client-install. It configures
security.ipa (SSSD, Kerberos, PAM, NSSwitch) and places a pre-provisioned host
keytab via sops-nix binary secret so enrollment is fully reproducible from the
flake.
- variables.nix: adds ipaServer (FQDN of the FreeIPA KDC; security.ipa.server
requires a hostname, not an IP, for Kerberos/TLS)
- certs/ipa-ca.crt: placeholder for the IPA CA public certificate (operator
replaces with: curl http://<ipa-server>/ipa/config/ca.crt)
- secrets/nix-cache.keytab: placeholder binary sops file (operator replaces
with the encrypted keytab after ipa host-add + ipa-getkeytab)
- .sops.yaml: adds creation rule for secrets/nix-cache.keytab (same recipients
as secrets/nix-cache.yaml)
- hosts/nix-cache/host.nix: imports the IPA client module; adds
networking.domain so the host's FQDN resolves correctly
Module header documents the three operator steps needed per host before deploy.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
zpool create -f was called if `zpool import -d /dev/disk/by-id` failed,
which could happen due to a race with systemd-udev-settle. The disk
would then be visible by the time zpool create ran, silently destroying
all data on an otherwise-intact pool.
Fix: locate the data disk first, retry the import directly against it
as a fallback, then check zdb -l for existing ZFS label metadata before
concluding the disk is blank. Remove -f so zpool create refuses rather
than overwrites if a pool is present.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
IPA refuses to create a forward zone for ts.net because it's a real
public TLD with DNSimple nameservers. The forward zone must use the
tailnet-specific subdomain (vars.tailnetDomain, e.g. tail13f623.ts.net)
instead. Update dnsmasq server selector and comments to match.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
FreeIPA (the new authoritative DNS) cannot reach 100.100.100.100
(Tailscale's internal MagicDNS resolver) directly because the DC is not
a Tailscale node. The tailscale-router IS a Tailscale node and can
reach 100.100.100.100 via tailscale0, so it now runs a dnsmasq
instance on its LAN interface that forwards all ts.net queries to
Tailscale's resolver.
After deploying this host, configure FreeIPA with:
kinit admin
ipa dnsforwardzone-add ts.net \
--forwarder=192.168.2.222 \
--forward-policy=only
This replaces Pi-hole's conditional forwarder for ts.net and restores
resolution of Tailscale MagicDNS names (e.g. raspberrypi.tail13f623.ts.net)
for all LAN hosts using FreeIPA as their DNS server.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
Adds static IP configuration to every NixOS host in the flake that
has a fixed LAN address, and centralises all network primitives
(IPs, gateway, prefix length, interface names) in variables.nix so
there is one place to update if any of them change.
variables.nix additions:
- lanGateway / lanPrefixLength — LAN gateway and /24 prefix, replacing
every hardcoded 192.168.2.254 / 24 across host files
- lxcLanInterface / vmLanInterface / vmStorageInterface — NIC names for
LXC containers (eth0), Proxmox VMs (ens18), and the HA storage NIC
(ens19), used as attribute keys so changing the name is a one-line edit
- haStoragePrefixLength — /29 for the storage subnet, mirrors haStorageCidr
- Per-host IP variables: nixCacheIp (.224), tailscaleRouterIp (.222),
torRelayIp (.221), serverIp (.226), dockerIp (.225)
host.nix changes:
- tailscale-router, tor-relay, nix-cache, pxe-boot: useDHCP = false,
static address on eth0 (lxcLanInterface), struct-form defaultGateway
(required when using systemd-networkd which LXC containers use)
- server, docker: useDHCP = false, static address on ens18 (vmLanInterface),
struct-form defaultGateway (works for both scripted networking and networkd)
- ha-server-1, ha-server-2: replace hardcoded 192.168.2.254 / 24 / 29
with the new variables; no functional change for these hosts
modules/build-types/pxe-boot.nix:
- Domain-controller kickstart template: replace hardcoded 192.168.2.138
and 192.168.2.254 with vars.domainControllerIp / vars.lanGateway /
vars.lanPrefixLength / vars.homeDomain so the template stays correct
if the DC IP or domain is ever changed again
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ULXzafSDwGhmFGnn3LtDSQ
vars.pxeServerIp was already set to the post-renumber target (.223) but the
pxe-boot CT is still at .247, so dnsmasq was advertising .223 as the TFTP server
and the VM couldn't reach it. Update to .247 so PXE boot works now; the comment
reminds us to flip it back to .223 when Stage 5 step 6 renumbers the CT.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
beatzaplentyandClaude Sonnet 4.6