Compare commits

...
Author SHA1 Message Date
beatzaplentyandClaude Sonnet 5 20f9475a7d Fix nix_extra_opts falsely reporting nix-cache's SSH remote builder down
The reachability check used `cat < /dev/tcp/${NIX_CACHE_HOST}/22`, which
blocks forever reading for EOF that never comes -- sshd sends its banner
and then holds the connection open waiting for the client to speak next.
Every single check hit the 3s timeout and reported "unreachable"
unconditionally, regardless of whether the remote builder was actually up.
Confirmed live: a plain TCP connect (`exec 3<>/dev/tcp/...`, no read)
returns in ~60ms against a healthy nix-cache instead of always timing out.

Fixing that exposed a second, previously-dormant bug: `printf -v
NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"` on a genuinely empty NIX_OPTS array
still runs one format pass and yields the literal `'' ` rather than an
empty string. A subprocess (e.g. sync-host-keys.sh) reusing this
process's decision via `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` then rebuilt
a 1-element array holding an empty string instead of a 0-element array,
which broke `nix-shell "${NIX_OPTS[@]}" -p <pkg>` with a bogus positional
argument the moment NIX_OPTS was legitimately empty (nix-cache reachable)
-- something the first bug had made impossible to ever hit before.

Also adds a couple of retries (1s apart) to both checks as a secondary
safety net against genuine multi-second blips, on top of fixing the
checks themselves.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:57:43 +10:00
beatzaplenty 6babb3eec5 Merge pull request 'Fix create-proxmox-resource.sh --dry-run hiding nix-cache probe results' (#1) from worktree-starry-painting-whistle into main
Reviewed-on: #1
2026-07-20 06:50:40 +00:00
beatzaplentyandClaude Sonnet 5 33730e6ccf Fix create-proxmox-resource.sh --dry-run hiding nix-cache probe results
The tarball/disko-image build previews were hardcoded strings that never
included ${NIX_OPTS[@]}, so --dry-run always showed the same "would build"
command whether nix-cache's substituter/remote-builder got disabled by
nix_extra_opts's reachability probe or not -- the actual (non-dry-run)
build commands already applied it correctly, only the preview lied.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 16:36:14 +10:00
beatzaplentyandClaude Sonnet 5 65f89806cb Fix beszel-agent losing its hub-pairing fingerprint on every restart
services.beszel.agent runs under DynamicUser=true with ProtectSystem =
"strict" and no StateDirectory, so /var/lib/beszel-agent -- where the
agent persists the fingerprint that locks its hub pairing to this
machine (github.com/henrygd/beszel/discussions/1542) -- was never
actually writable. Every restart silently failed to persist it and
regenerated a fresh one in memory, permanently desyncing from whatever
the hub had on record after the very first successful pairing. Affects
every host importing modules/beszel/enable-agent.nix (nix-cache, server),
not just full container rebuilds.

Found via nix-cache showing "fingerprint mismatch" after being rebuilt
post-outage; confirmed server was silently exposed to the same bug, just
hadn't restarted since its first pairing. Fixed by declaring
StateDirectory so systemd gives the dynamic user real persistent storage.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:59:24 +10:00
beatzaplentyandClaude Sonnet 5 c3007097a6 Fix create-proxmox-resource.sh defaulting hostname to the flake target
--name (used as pct/qm create's --hostname/--name) defaulted to
$flake_target (e.g. "lxc-nix-cache"), not $host (e.g. "nix-cache"). Since
proxmoxLXC.manageHostName pulls the guest's real networking.hostName
straight from Proxmox's own container config, this silently overrode
host.nix's hostName with a build-type-specific name. Default --name to
--host instead, so the guest's identity matches host.nix regardless of
which platform variant built it.

Found by spinning up a fresh lxc-nix-cache test container and noticing its
hostname was "lxc-nix-cache" instead of "nix-cache".

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 15:19:35 +10:00
beatzaplentyandClaude Sonnet 5 9724babcea Add tailscale-exit-node build type across all three platforms
New build type dedicated to Tailscale exit-node capability, wired up for
linode/proxmox/lxc like every other build type (the lxc variant is the one
actually intended for deployment). Kept separate from the "server" host
rather than bundling exit-node capability onto it.

Trimmed modules/tailscale/exit-node.nix down to pure exit-node behavior:
dropped the old --advertise-routes=${vars.lanCidr} bundling (meaningless
for a Linode-hosted VPS with no path to the LAN), and switched
extraUpFlags -> extraSetFlags. Confirmed against nixpkgs' tailscale.nix
that extraUpFlags is only applied by tailscaled-autoconnect, which itself
only runs when services.tailscale.authKeyFile is set -- nothing in this
repo sets one, so the old flags would never have actually been applied.
extraSetFlags runs unconditionally via tailscaled-set on every boot, so
--advertise-exit-node self-reapplies once the operator has done the
one-time manual `tailscale up` auth.

Verified: all three new targets eval cleanly, nixpkgs-fmt/statix clean,
and a dry-run build of lxc-tailscale-exit-node's tarball resolves its full
closure including tailscaled-set.service.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 13:38:39 +10:00
beatzaplentyandClaude Sonnet 5 7055bcdb97 Fix lxc-* hosts never completing first-boot user/secrets activation
virtualisation/proxmox-lxc.nix registers the Nix store DB via a systemd
service, never an activation script -- so neededForUsers sops secrets
(password hashes) and the user-creation step that consumes them never ran
on a real first boot, leaving /etc/shadow stuck with build-time placeholder
entries. boot.postBootCommands looked like the right hook (stage-2-init.sh
does invoke it) but switch-to-configuration behaves unreliably that early,
before systemd itself is up. Fixed with a genuine oneshot systemd service,
gated by ConditionPathExists so it only ever runs once.

Confirmed live via a from-scratch destroy+rebuild+redeploy of the
lxc-nix-cache test container: real password hashes applied automatically,
systemctl is-system-running -> running, zero failed units.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 13:38:21 +10:00
beatzaplentyandClaude Sonnet 5 d973da487c Fix lxc-* hosts having no host-key pre-seeding mechanism at all
The real root cause behind the original nix-cache 502, traced all the way
through: modules/installer/host-keys.nix (which NIXOS_HOST_KEYS_DIR=...
--impure actually wires up) is only ever imported by the installer's own
modules/installer/common.nix -- modules/platforms/lxc.nix, which every
real lxc-* host build actually uses, never imported anything like it.
docs/auto-installer.md previously claimed NIXOS_HOST_KEYS_DIR bakes a key
into lxc-* tarballs "the same way it does for the ISO/PXE installer
images" -- that was never actually true; I wrote it without verifying the
mechanism existed for lxc.nix specifically.

In practice this meant every lxc-* container booted with a freshly
self-generated SSH host key that could never match whatever .sops.yaml
actually trusts for that target, so *every* secret -- not just
cache-priv-key -- silently failed to decrypt. No error surfaces in the
boot log for this: the activation step that installs secrets only runs
on a genuinely fresh first activation and silently no-ops once
/run/current-system already exists, so by the time anyone looks the
window has closed. Found by manually invoking sops-install-secrets
directly: "Error getting data key: 0 successful groups required, got 0".

Fixed by giving modules/platforms/lxc.nix the same key-baking mechanism
the installer has, but keyed to its own exact flake target and placing
the key directly at /etc/ssh/ssh_host_ed25519_key (no copy step to stage
for, unlike the installer's /etc/host-keys/ staging area -- an lxc-*
tarball has no install step). The target name comes in via
specialArgs.flakeTarget (new, set by flake.nix's mkTarget) rather than
being read back from config.environment.etc."flake-target" -- reading
that back from within a module that also contributes to
environment.etc is circular (confirmed: "infinite recursion
encountered").

Verified live end-to-end against the real test container (lxc-nix-cache,
VMID 100 on pve.sweet.home): destroyed it, rebuilt the tarball fresh with
the fix, recreated it, and confirmed /run/secrets/ now has all three
secrets this host needs (beszel-token, cache-priv-key, nix-github-token),
nix-serve is active (running), and curl http://localhost/nix-cache-info
succeeds both directly and through nginx.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot
2026-07-20 12:29:01 +10:00
11 changed files with 288 additions and 24 deletions
+4 -3
View File
@@ -133,9 +133,10 @@ nixosSystem {
```
Platforms: `linode`, `proxmox`, `lxc`. Build types: `minimal`, `nix-cache`,
`server`, `docker`, `gui`, `pxe-boot`. Not every combination is built — e.g.
`pxe-boot` has no `linode` variant (PXE/DHCP/TFTP need LAN L2 adjacency a
Linode VPS doesn't have). Treat `flake.nix`'s `generatedTargets` as the source
`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`. Not every
combination is built — e.g. `pxe-boot` has no `linode` variant (PXE/DHCP/TFTP
need LAN L2 adjacency a Linode VPS doesn't have). Treat `flake.nix`'s
`generatedTargets` as the source
of truth for which hosts exist — `README.md`, `AGENTS.md`,
`docs/flake-lock-automation.md`, and the CI eval workflows
(`.github/workflows/check-nixos.yml`, `.gitea/workflows/check-nixos.yml`) list
+2 -1
View File
@@ -10,7 +10,7 @@ pieces composed in `flake.nix`:
- **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`
- **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
`gui`, `pxe-boot`
`gui`, `pxe-boot`, `tailscale-exit-node`
Not every combination exists — `pxe-boot` has no `linode` variant, since
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have. The full
@@ -26,6 +26,7 @@ list:
| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack (`proxmox-docker` is the real, deployed one — previously the flat `docker` target) |
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation (`proxmox-gui` is the real, deployed one — previously the flat `nixos` target) |
| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host (`proxmox-pxe-boot` is the real, deployed one — previously the flat `pxe-boot` target) |
| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node (no deployed target yet; `lxc-tailscale-exit-node` is the one planned for actual use) |
The "(real, deployed)" targets above are also tracked machine-readably in
`variables.nix`'s `deployedTargets` — keep both in sync when a deployment
+27 -5
View File
@@ -77,11 +77,33 @@ system profile) — there's no separate activation step to run yourself.
of this (build, host-key handling, upload, `pct create` with the flags
above) — see its `--help`.
Host keys still need pre-seeding the same way as any other host (see "Host
keys" below) — the sops-nix activation-vs-first-boot race is identical
regardless of how the image reaches the machine. `NIXOS_HOST_KEYS_DIR=...
nix build ... --impure` bakes the matching key into the tarball the same way
it does for the ISO/PXE installer images.
Host keys still need pre-seeding the same way as any other host — the
sops-nix activation-vs-first-boot race is identical regardless of how the
image reaches the machine. Unlike the ISO/PXE installer (where
`modules/installer/host-keys.nix` bakes *every* `host-keys/` entry into
`/etc/host-keys/` for `auto-install.sh` to pick from and copy at install
time — see "Host keys" below), an `lxc-*` tarball has no install step to
copy anything during, so `modules/platforms/lxc.nix` bakes this *one*
target's key straight into `/etc/ssh/ssh_host_ed25519_key(.pub)` directly,
keyed by its own exact flake target name (`config.environment.etc` can't
be read back from within a module still contributing to it, so this comes
in via `specialArgs.flakeTarget`, set by `flake.nix`'s `mkTarget`):
```sh
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure
```
Confirmed the hard way: without this, the tarball's own built-in system
just generates a fresh host key at first boot like any host would, which
can never match whatever `.sops.yaml` actually trusts for that target —
`sops-install-secrets` fails with `Error getting data key: 0 successful
groups required, got 0`, and *every* secret (including this host's own
login) permanently fails to decrypt, silently — no error in the boot log
at all, since the activation step that would install secrets only runs on
a from-scratch first activation and skips silently once `/run/current-system`
already exists. `scripts/create-proxmox-resource.sh` always builds with
`NIXOS_HOST_KEYS_DIR` set for this reason.
## Layout
+15 -2
View File
@@ -33,6 +33,9 @@
# nix-cache itself consumes the nix-cache substituter and remote
# builder.
mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix }:
let
flakeTarget = "${platform}-${buildType}";
in
nixpkgs.lib.nixosSystem {
inherit system;
modules = [
@@ -42,7 +45,7 @@
./modules/platforms/${platform}.nix
./modules/build-types/${buildType}.nix
hostPath
{ environment.etc."flake-target".text = "${platform}-${buildType}"; }
{ environment.etc."flake-target".text = flakeTarget; }
home-manager.nixosModules.home-manager
{
home-manager = {
@@ -56,7 +59,13 @@
./modules/nix-cache/client.nix
./modules/nix-cache/remote-builder-client.nix
];
specialArgs = { inherit inputs vars netbootSystem; };
# flakeTarget is passed via specialArgs (not read back from
# config.environment.etc."flake-target" above) specifically so
# modules/platforms/lxc.nix can use it to select its own host key
# file without a same-option circular dependency (a module
# contributing to environment.etc can't read the merged
# environment.etc it's itself contributing to).
specialArgs = { inherit inputs vars netbootSystem flakeTarget; };
};
# Generated platform x build-type matrix. pxe-boot has no linode
@@ -85,6 +94,10 @@
proxmox-pxe-boot = mkTarget { platform = "proxmox"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
lxc-pxe-boot = mkTarget { platform = "lxc"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
linode-tailscale-exit-node = mkTarget { platform = "linode"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
proxmox-tailscale-exit-node = mkTarget { platform = "proxmox"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
lxc-tailscale-exit-node = mkTarget { platform = "lxc"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
};
# Auto-install environments (migrated from the former nix-auto-installer
+12
View File
@@ -0,0 +1,12 @@
_:
{
networking.hostName = "exit-node";
# No networking.hostId: only ZFS-touching hosts (server, docker) need one
# for pool-import safety, and this host does neither.
# A genuinely new host (not a pre-refactor carry-over), so it tracks the
# flake's current nixpkgs release rather than being pinned to an older one.
system.stateVersion = "26.05";
}
+9
View File
@@ -6,4 +6,13 @@
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
};
# The upstream module runs beszel-agent under DynamicUser with
# ProtectSystem = "strict" and no StateDirectory, so /var/lib/beszel-agent
# (where the agent persists its hub-pairing fingerprint, per
# https://github.com/henrygd/beszel/discussions/1542) isn't writable --
# every restart silently fails to save it and regenerates a fresh one in
# memory, permanently desyncing from whatever the hub has on record after
# the very first successful pairing. Give it real persistent storage.
systemd.services.beszel-agent.serviceConfig.StateDirectory = "beszel-agent";
}
@@ -0,0 +1,21 @@
{ ... }:
{
imports = [
../tailscale/exit-node.nix
];
# "server", not "both": this build type only ever advertises itself as an
# exit node (see ../tailscale/exit-node.nix) -- it doesn't advertise LAN
# subnet routes, so it doesn't need the "client"-side loose reverse-path
# filtering that "both" would also turn on. Deliberately left unbundled
# from LAN-subnet-route advertisement so this build type stays valid on
# every platform, including linode (a remote VPS with no network path to
# the home LAN at all).
services.tailscale.useRoutingFeatures = "server";
# Forwarded exit-node traffic arrives on tailscale0 already
# tailscale-authenticated -- the firewall's normal per-port allow-list
# would otherwise drop it. Standard NixOS/Tailscale exit-node guidance.
networking.firewall.trustedInterfaces = [ "tailscale0" ];
}
+111 -1
View File
@@ -1,5 +1,44 @@
{ lib, modulesPath, ... }:
{ lib, modulesPath, flakeTarget, ... }:
let
# Bakes this exact flake target's pre-generated SSH host key straight
# into /etc/ssh/ -- mirrors modules/installer/host-keys.nix's
# builtins.getEnv pattern (impure and empty under normal `nix
# build`/`nix eval`, so this is a no-op unless explicitly opted into
# with NIXOS_HOST_KEYS_DIR=... --impure), but places the key directly
# rather than staging it under /etc/host-keys/ for a later manual copy
# -- this is the whole system for a `lxc-*` host, built straight to a
# pct-restorable tarball with no install step, so there's no later copy
# step to stage for.
#
# Without this, config.system.build.tarball's built-in system just
# generates a fresh host key at first boot like any other host would --
# but sops-nix derives its decryption key from *this* file, and
# .sops.yaml only trusts whatever key scripts/sync-host-keys.sh already
# registered for this exact target name. A freshly-generated key can
# never match that, so every secret (including this host's own login)
# permanently fails to decrypt. Confirmed live: sops-install-secrets
# errored with "Error getting data key: 0 successful groups required,
# got 0" -- the container's actual host key's age fingerprint didn't
# match the one registered in .sops.yaml at all.
hostKeysDirStr = builtins.getEnv "NIXOS_HOST_KEYS_DIR";
hasHostKeysDir = hostKeysDirStr != "" && builtins.pathExists hostKeysDirStr;
hostKeysDir = /. + hostKeysDirStr;
# flakeTarget ("${platform}-${buildType}") comes in via specialArgs from
# flake.nix's mkTarget -- exactly the name scripts/sync-host-keys.sh
# registers keys under. Deliberately not read back from
# config.environment.etc."flake-target" (which is set to the same value)
# -- this module also *contributes* to environment.etc below, and a
# module reading the merged value of an option it's still defining is a
# circular dependency (confirmed: "infinite recursion encountered").
privKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key";
pubKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key.pub";
hasKeyForThisTarget =
hasHostKeysDir
&& builtins.pathExists privKeyFile
&& builtins.pathExists pubKeyFile;
in
{
# LXC containers share the host kernel — Proxmox starts them by exec'ing
# /sbin/init directly, no bootloader/initrd involved — and Proxmox has its
@@ -35,4 +74,75 @@
# for the same reason; it just doesn't disable NetworkManager itself,
# which modules/common/configuration.nix enables for every host.
networking.networkmanager.enable = lib.mkForce false;
environment.etc = lib.mkIf hasKeyForThisTarget {
"ssh/ssh_host_ed25519_key" = {
source = privKeyFile;
mode = "0600";
};
"ssh/ssh_host_ed25519_key.pub" = {
source = pubKeyFile;
mode = "0644";
};
};
# virtualisation/proxmox-lxc.nix (imported above) registers the Nix
# store DB via a systemd service (register-nix-paths) -- it never runs
# an activation script at all. Confirmed live this means neither
# sops-nix's "for users" secrets (password hashes -- installed by the
# activation script itself, not a systemd service, since they need to
# exist *before* user creation) nor the user-creation step that
# consumes them ever run on a real lxc-* boot. Regular secrets
# (nix-serve's key, beszel's token, etc.) work anyway because sops-nix
# provides its own systemd service for those.
#
# A systemd service, not boot.postBootCommands: tried that first (it's
# a genuine, generally-invoked hook -- nixos/modules/system/boot/stage-2-init.sh,
# which becomes this container's actual /sbin/init, unconditionally
# runs it) but switch-to-configuration behaves differently that early in
# boot (raw stage-2-init.sh, before systemd itself has even started) --
# confirmed live it silently failed to rewrite /etc/shadow from there
# even in "test" mode, despite the exact same command working reliably
# every time when run post-boot (i.e. as a normal systemd service, which
# is what this is). Not fully root-caused why the early context
# specifically breaks it; a real systemd service sidesteps needing to.
#
# /etc/shadow already has PLACEHOLDER entries for every declared user
# baked in at build time (part of constructing the system closure).
# update-users-groups.pl deliberately never overwrites an *existing*
# shadow entry -- a correct safety property in general (don't clobber a
# real user's real password on a config rebuild) -- but on a genuine
# first boot that only means the real hashedPasswordFile-derived hash
# never gets the chance to be applied either, since the placeholder is
# already "seen". Safe to clear here specifically: there is no real
# password yet to protect on a first boot.
#
# "test" mode, not "boot": confirmed live "boot" mode aborts partway
# through (before rewriting /etc/shadow) on a warning that "/boot" is on
# a different filesystem -- a real check for a host with a bootloader to
# update, meaningless for a container that has none
# (boot.loader.{grub,systemd-boot}.enable are both false above), but it
# still aborts the script. "test" runs every activation step without
# touching boot-loader state at all.
#
# ConditionPathExists (systemd-native, not a bash-level check) means
# this only ever runs once, on the genuine first boot -- systemd itself
# skips even starting it on every later boot once the marker exists.
# switch-to-configuration is otherwise the operator's call per this
# repo's own safety rules, not something to run on every boot.
systemd.services.nixos-lxc-first-boot-activate = {
description = "Complete first-boot NixOS activation (users, secrets) for this LXC container";
wantedBy = [ "multi-user.target" ];
unitConfig.ConditionPathExists = "!/var/lib/nixos-lxc-first-boot-activated";
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
};
script = ''
rm -f /etc/shadow
/run/current-system/bin/switch-to-configuration test
mkdir -p /var/lib
touch /var/lib/nixos-lxc-first-boot-activated
'';
};
}
+10 -3
View File
@@ -1,12 +1,19 @@
{ vars, ... }:
_:
{
services.tailscale = {
enable = true;
extraUpFlags = [
# extraSetFlags (tailscale set, via the always-on tailscaled-set
# service), not extraUpFlags -- extraUpFlags is only ever applied by
# tailscaled-autoconnect, which itself only runs when
# services.tailscale.authKeyFile is set (nothing in this repo sets one,
# so tailscale up is a manual, one-time operator step on every host that
# uses this service). extraSetFlags has no such gate, so
# --advertise-exit-node self-reapplies on every boot once the operator
# has authenticated the node once.
extraSetFlags = [
"--advertise-exit-node"
"--advertise-routes=${vars.lanCidr}"
];
};
}
+25 -4
View File
@@ -40,8 +40,12 @@ Create mode (default):
config.networking.hostName (server, docker,
nix-cache, nixos, pxe-boot, nix-minimal). Use
--list to see what's available for --type.
--name <name> Proxmox display name/hostname (default: the flake
target name, e.g. lxc-server)
--name <name> Proxmox display name/hostname (default: --host's
value, e.g. nix-cache -- for lxc this becomes the
guest's real networking.hostName too, since
proxmoxLXC.manageHostName pulls it from Proxmox's
own container config, so it must match host.nix
regardless of build type)
--vmid <n> Numeric VMID (default: next free, via
\`pvesh get /cluster/nextid\` on the node).
Refuses to run if this ID already exists.
@@ -274,7 +278,13 @@ if [[ -z "$flake_target" ]]; then
exit 1
fi
[[ -z "$name" ]] && name="$flake_target"
# The container/VM's real identity is --host (e.g. "nix-cache"), validated
# above against config.networking.hostName -- not the flake target name
# (e.g. "lxc-nix-cache"), which is build-type-specific and only exists to
# pick which platform variant to build. Defaulting --name to the flake
# target would make lxc's --hostname (which proxmoxLXC.manageHostName
# feeds straight into the guest's real hostname) disagree with host.nix.
[[ -z "$name" ]] && name="$host"
# --- refuse to duplicate a host that's already really deployed ----------
# Checked by hostName, not exact flake target: proxmox-server being
@@ -383,9 +393,19 @@ else
fi
if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
# Mirrors the real build commands' "${NIX_OPTS[@]}" below -- nix_extra_opts
# (called earlier, once) has already decided whether nix-cache is in play,
# and the dry-run preview needs to reflect that decision instead of always
# printing the same command regardless of outcome.
nix_opts_display=""
if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
printf -v nix_opts_display '%q ' "${NIX_OPTS[@]}"
nix_opts_display=" ${nix_opts_display% }"
fi
if [[ "$type" == "lxc" ]]; then
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would build: NIXOS_HOST_KEYS_DIR=${repo_root}/host-keys nix build --impure \\"
echo "[dry-run] --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.tarball"
local_image="<built-tarball>"
else
@@ -399,7 +419,8 @@ if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
fi
else
if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would build: nix build .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
echo "[dry-run] would build: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
echo "[dry-run] would run: sudo ./result-${flake_target} \\"
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\"
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\"
+49 -2
View File
@@ -88,13 +88,60 @@ nix_extra_opts() {
fi
export NIX_EXTRA_OPTS_DECIDED=1
NIX_OPTS=()
if ! curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
# Retry a couple of times, 1s apart, before believing either check --
# belt-and-suspenders against a genuine multi-second blip (nix-cache
# restarting), on top of the fix below. Worst case (~11s total, host
# genuinely gone) is still nowhere near the 15s+ *per lookup* nix's own
# substituter retries would cost if this check didn't exist at all.
local attempt cache_up=0 builder_up=0
for attempt in 1 2 3; do
if curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
cache_up=1
break
fi
[[ "$attempt" -lt 3 ]] && sleep 1
done
if [[ "$cache_up" -eq 0 ]]; then
echo "nix-cache (http://${NIX_CACHE_HOST}) is unreachable -- skipping it (substituter + remote builder) for the rest of this run." >&2
NIX_OPTS=(--option substituters "https://cache.nixos.org/" --builders "")
elif ! timeout 3 bash -c "cat < /dev/tcp/${NIX_CACHE_HOST}/22" >/dev/null 2>&1; then
else
for attempt in 1 2 3; do
# `exec 3<>/dev/tcp/...` just opens the fd and returns -- it does NOT
# read from it. Confirmed live this is load-bearing, not stylistic:
# the previous `cat < /dev/tcp/.../22` blocked forever and always hit
# the timeout even against a perfectly healthy nix-cache, because
# sshd sends its banner and then holds the connection open waiting
# for the client to speak next -- `cat` never sees EOF, so this
# check reported "unreachable" unconditionally, 100% of the time,
# regardless of whether the remote builder was actually up.
if timeout 3 bash -c "exec 3<>/dev/tcp/${NIX_CACHE_HOST}/22" 2>/dev/null; then
builder_up=1
break
fi
[[ "$attempt" -lt 3 ]] && sleep 1
done
if [[ "$builder_up" -eq 0 ]]; then
echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
NIX_OPTS=(--builders "")
fi
fi
# `printf '%q '` with a genuinely empty NIX_OPTS still runs one format
# pass over a missing argument and yields the literal `'' ` rather than
# an empty string (confirmed live) -- a subprocess that later does
# `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` (the branch above, for e.g.
# sync-host-keys.sh reusing this process's decision) would then rebuild
# a 1-element array holding an empty string instead of a 0-element
# array, and `nix-shell "${NIX_OPTS[@]}" -p <pkg>` chokes on that stray
# element as a bogus positional argument. Guard the empty case
# explicitly so nix-cache being reachable (NIX_OPTS legitimately empty)
# round-trips as truly empty instead.
if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
else
NIX_EXTRA_OPTS=""
fi
export NIX_EXTRA_OPTS
}