Compare commits

..
Author SHA1 Message Date
beatzaplentyandClaude Sonnet 5 c939454983 Fix tailscale exit-node module: enable routing, compose on enable-service
Check NixOS configurations / eval-hosts (pull_request) Failing after 11m52s
exit-node.nix advertised --advertise-exit-node/--advertise-routes but
never enabled IP forwarding, so it wouldn't actually route traffic.
services.tailscale.useRoutingFeatures = "server" is the built-in way to
get the sysctls right without duplicating modules/networking/enable-ip-forwarding.nix.
Also opens the firewall for direct (non-DERP-relayed) peer connections,
and now imports enable-service.nix instead of duplicating `enable = true`,
so "plain tailscale" and "tailscale exit node" are two composable modules.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-07-20 12:24:18 +10:00
22 changed files with 144 additions and 548 deletions

No files matched your search

+6 -8
View File
@@ -93,10 +93,9 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
(`--force-rebuild` to skip that and always rebuild), and probes (`--force-rebuild` to skip that and always rebuild), and probes
nix-cache's substituter/remote-builder reachability once up front rather nix-cache's substituter/remote-builder reachability once up front rather
than letting every `nix build` call retry against it individually. than letting every `nix build` call retry against it individually.
Refuses to create a target whose host identity already exists live on Refuses to create a target whose host identity already has a real
the node (checked directly via `qm`/`pct`, not any file in this repo) deployment elsewhere (`variables.nix`'s `deployedTargets`) unless
unless `--allow-duplicate-host` is passed. `--dry-run` throughout both `--allow-duplicate-host` is passed. `--dry-run` throughout both modes.
modes.
- `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge, - `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
default cores/memory) sourced by `create-proxmox-resource.sh`. Add new default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
cross-script config here instead of duplicating it per-script. cross-script config here instead of duplicating it per-script.
@@ -134,10 +133,9 @@ nixosSystem {
``` ```
Platforms: `linode`, `proxmox`, `lxc`. Build types: `minimal`, `nix-cache`, Platforms: `linode`, `proxmox`, `lxc`. Build types: `minimal`, `nix-cache`,
`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`. Not every `server`, `docker`, `gui`, `pxe-boot`. Not every combination is built — e.g.
combination is built — e.g. `pxe-boot` has no `linode` variant (PXE/DHCP/TFTP `pxe-boot` has no `linode` variant (PXE/DHCP/TFTP need LAN L2 adjacency a
need LAN L2 adjacency a Linode VPS doesn't have). Treat `flake.nix`'s Linode VPS doesn't have). Treat `flake.nix`'s `generatedTargets` as the source
`generatedTargets` as the source
of truth for which hosts exist — `README.md`, `AGENTS.md`, of truth for which hosts exist — `README.md`, `AGENTS.md`,
`docs/flake-lock-automation.md`, and the CI eval workflows `docs/flake-lock-automation.md`, and the CI eval workflows
(`.github/workflows/check-nixos.yml`, `.gitea/workflows/check-nixos.yml`) list (`.github/workflows/check-nixos.yml`, `.gitea/workflows/check-nixos.yml`) list
+6 -9
View File
@@ -10,7 +10,7 @@ pieces composed in `flake.nix`:
- **Platforms** (what it runs on): `linode`, `proxmox`, `lxc` - **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`
- **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`, - **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
`gui`, `pxe-boot`, `tailscale-exit-node` `gui`, `pxe-boot`
Not every combination exists — `pxe-boot` has no `linode` variant, since Not every combination exists — `pxe-boot` has no `linode` variant, since
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have. The full PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have. The full
@@ -21,19 +21,16 @@ list:
| `linode-minimal` | Minimal NixOS host profile on a Linode VPS (real, deployed) | | `linode-minimal` | Minimal NixOS host profile on a Linode VPS (real, deployed) |
| `proxmox-minimal` | Minimal NixOS host profile on Proxmox (real, deployed — previously the flat `nix-minimal` target) | | `proxmox-minimal` | Minimal NixOS host profile on Proxmox (real, deployed — previously the flat `nix-minimal` target) |
| `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container | | `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container |
| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder (`lxc-nix-cache` is the real, deployed one — previously `proxmox-nix-cache`, itself previously the flat `nix-cache` target) | | `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder (`proxmox-nix-cache` is the real, deployed one — previously the flat `nix-cache` target) |
| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host (`proxmox-server` is the real, deployed one — previously the flat `server` target) | | `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host (`proxmox-server` is the real, deployed one — previously the flat `server` target) |
| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack (`proxmox-docker` is the real, deployed one — previously the flat `docker` target) | | `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack (`proxmox-docker` is the real, deployed one — previously the flat `docker` target) |
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation (`proxmox-gui` is the real, deployed one — previously the flat `nixos` target) | | `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation (`proxmox-gui` is the real, deployed one — previously the flat `nixos` target) |
| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host (`proxmox-pxe-boot` is the real, deployed one — previously the flat `pxe-boot` target) | | `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host (`proxmox-pxe-boot` is the real, deployed one — previously the flat `pxe-boot` target) |
| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node (no deployed target yet; `lxc-tailscale-exit-node` is the one planned for actual use) |
This table is the only place "(real, deployed)" status is tracked — there's The "(real, deployed)" targets above are also tracked machine-readably in
no separate machine-readable copy to keep in sync. `scripts/create-proxmox-resource.sh` `variables.nix`'s `deployedTargets` — keep both in sync when a deployment
guards against creating a same-identity duplicate of an already-deployed host changes. `scripts/create-proxmox-resource.sh` reads that list to refuse
by checking the Proxmox node itself (live `qm`/`pct` state) rather than any creating a same-identity duplicate of an already-deployed host by accident.
file in this repo, since a static list can't track whether a resource still
actually exists.
Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
(hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay (hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay
+5 -27
View File
@@ -77,33 +77,11 @@ system profile) — there's no separate activation step to run yourself.
of this (build, host-key handling, upload, `pct create` with the flags of this (build, host-key handling, upload, `pct create` with the flags
above) — see its `--help`. above) — see its `--help`.
Host keys still need pre-seeding the same way as any other host — the Host keys still need pre-seeding the same way as any other host (see "Host
sops-nix activation-vs-first-boot race is identical regardless of how the keys" below) — the sops-nix activation-vs-first-boot race is identical
image reaches the machine. Unlike the ISO/PXE installer (where regardless of how the image reaches the machine. `NIXOS_HOST_KEYS_DIR=...
`modules/installer/host-keys.nix` bakes *every* `host-keys/` entry into nix build ... --impure` bakes the matching key into the tarball the same way
`/etc/host-keys/` for `auto-install.sh` to pick from and copy at install it does for the ISO/PXE installer images.
time — see "Host keys" below), an `lxc-*` tarball has no install step to
copy anything during, so `modules/platforms/lxc.nix` bakes this *one*
target's key straight into `/etc/ssh/ssh_host_ed25519_key(.pub)` directly,
keyed by its own exact flake target name (`config.environment.etc` can't
be read back from within a module still contributing to it, so this comes
in via `specialArgs.flakeTarget`, set by `flake.nix`'s `mkTarget`):
```sh
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure
```
Confirmed the hard way: without this, the tarball's own built-in system
just generates a fresh host key at first boot like any host would, which
can never match whatever `.sops.yaml` actually trusts for that target —
`sops-install-secrets` fails with `Error getting data key: 0 successful
groups required, got 0`, and *every* secret (including this host's own
login) permanently fails to decrypt, silently — no error in the boot log
at all, since the activation step that would install secrets only runs on
a from-scratch first activation and skips silently once `/run/current-system`
already exists. `scripts/create-proxmox-resource.sh` always builds with
`NIXOS_HOST_KEYS_DIR` set for this reason.
## Layout ## Layout
-9
View File
@@ -59,15 +59,6 @@ On `nix-cache`, install the matching public key used by `nixremote` authorized k
The committed `nixremote` authorized keys are public SSH keys only. Keep the The committed `nixremote` authorized keys are public SSH keys only. Keep the
matching private keys on client hosts and out of the repository. matching private keys on client hosts and out of the repository.
nix-cache's own SSH *host* key is trusted declaratively via
`programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
sourced from `vars.nixCacheHostKey` (`variables.nix`) — every client rebuild
picks it up automatically, so distributed builds don't fail with "Host key
verification failed" on a client that has never manually SSH'd to nix-cache
before. If nix-cache's host key is ever rotated or the host rebuilt from
scratch, update `vars.nixCacheHostKey` to match its new
`/etc/ssh/ssh_host_ed25519_key.pub`.
## Manual verification ## Manual verification
After deployment: After deployment:
+2 -15
View File
@@ -33,9 +33,6 @@
# nix-cache itself consumes the nix-cache substituter and remote # nix-cache itself consumes the nix-cache substituter and remote
# builder. # builder.
mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix }: mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix }:
let
flakeTarget = "${platform}-${buildType}";
in
nixpkgs.lib.nixosSystem { nixpkgs.lib.nixosSystem {
inherit system; inherit system;
modules = [ modules = [
@@ -45,7 +42,7 @@
./modules/platforms/${platform}.nix ./modules/platforms/${platform}.nix
./modules/build-types/${buildType}.nix ./modules/build-types/${buildType}.nix
hostPath hostPath
{ environment.etc."flake-target".text = flakeTarget; } { environment.etc."flake-target".text = "${platform}-${buildType}"; }
home-manager.nixosModules.home-manager home-manager.nixosModules.home-manager
{ {
home-manager = { home-manager = {
@@ -59,13 +56,7 @@
./modules/nix-cache/client.nix ./modules/nix-cache/client.nix
./modules/nix-cache/remote-builder-client.nix ./modules/nix-cache/remote-builder-client.nix
]; ];
# flakeTarget is passed via specialArgs (not read back from specialArgs = { inherit inputs vars netbootSystem; };
# config.environment.etc."flake-target" above) specifically so
# modules/platforms/lxc.nix can use it to select its own host key
# file without a same-option circular dependency (a module
# contributing to environment.etc can't read the merged
# environment.etc it's itself contributing to).
specialArgs = { inherit inputs vars netbootSystem flakeTarget; };
}; };
# Generated platform x build-type matrix. pxe-boot has no linode # Generated platform x build-type matrix. pxe-boot has no linode
@@ -94,10 +85,6 @@
proxmox-pxe-boot = mkTarget { platform = "proxmox"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; }; proxmox-pxe-boot = mkTarget { platform = "proxmox"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
lxc-pxe-boot = mkTarget { platform = "lxc"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; }; lxc-pxe-boot = mkTarget { platform = "lxc"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
linode-tailscale-exit-node = mkTarget { platform = "linode"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
proxmox-tailscale-exit-node = mkTarget { platform = "proxmox"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
lxc-tailscale-exit-node = mkTarget { platform = "lxc"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
}; };
# Auto-install environments (migrated from the former nix-auto-installer # Auto-install environments (migrated from the former nix-auto-installer
-12
View File
@@ -1,12 +0,0 @@
_:
{
networking.hostName = "exit-node";
# No networking.hostId: only ZFS-touching hosts (server, docker) need one
# for pool-import safety, and this host does neither.
# A genuinely new host (not a pre-refactor carry-over), so it tracks the
# flake's current nixpkgs release rather than being pinned to an older one.
system.stateVersion = "26.05";
}
-9
View File
@@ -6,13 +6,4 @@
#DOCKER_HOST = "tcp://docker-socket-proxy:2375"; #DOCKER_HOST = "tcp://docker-socket-proxy:2375";
HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}"; HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
}; };
# The upstream module runs beszel-agent under DynamicUser with
# ProtectSystem = "strict" and no StateDirectory, so /var/lib/beszel-agent
# (where the agent persists its hub-pairing fingerprint, per
# https://github.com/henrygd/beszel/discussions/1542) isn't writable --
# every restart silently fails to save it and regenerates a fresh one in
# memory, permanently desyncing from whatever the hub has on record after
# the very first successful pairing. Give it real persistent storage.
systemd.services.beszel-agent.serviceConfig.StateDirectory = "beszel-agent";
} }
@@ -1,21 +0,0 @@
{ ... }:
{
imports = [
../tailscale/exit-node.nix
];
# "server", not "both": this build type only ever advertises itself as an
# exit node (see ../tailscale/exit-node.nix) -- it doesn't advertise LAN
# subnet routes, so it doesn't need the "client"-side loose reverse-path
# filtering that "both" would also turn on. Deliberately left unbundled
# from LAN-subnet-route advertisement so this build type stays valid on
# every platform, including linode (a remote VPS with no network path to
# the home LAN at all).
services.tailscale.useRoutingFeatures = "server";
# Forwarded exit-node traffic arrives on tailscale0 already
# tailscale-authenticated -- the firewall's normal per-port allow-list
# would otherwise drop it. Standard NixOS/Tailscale exit-node guidance.
networking.firewall.trustedInterfaces = [ "tailscale0" ];
}
-21
View File
@@ -17,26 +17,6 @@ let
--refresh \ --refresh \
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target) --flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
''; '';
# lxc-* hosts pre-seed their SSH host key at build time (see
# modules/platforms/lxc.nix) so sops-nix's .sops.yaml recipient matches on
# first boot -- without it, secrets permanently fail to decrypt (see that
# file's comment for the confirmed failure). That requires --impure plus
# NIXOS_HOST_KEYS_DIR pointing at the repo's host-keys/ dir, same pattern
# docs/auto-installer.md uses for the installer ISO. A function, not a
# shellAlias, since the target name has to interpolate into the middle of
# the flake attribute path, not just append after it. Must be run from the
# repo root, same as every other host-keys/ command in this repo.
buildImageFn = ''
buildImage() {
if [ -z "$1" ]; then
echo "usage: buildImage <flake-target> (e.g. lxc-docker)" >&2
return 1
fi
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
".#nixosConfigurations.$1.config.system.build.tarball"
}
'';
in in
{ {
programs.bash = { programs.bash = {
@@ -45,6 +25,5 @@ in
"Switch-nix" = mySwitchCmd; "Switch-nix" = mySwitchCmd;
"Test-nix" = myTestCmd; "Test-nix" = myTestCmd;
}; };
initExtra = buildImageFn;
}; };
} }
@@ -5,14 +5,6 @@
# sudo install -d -m 0700 /root/.ssh # sudo install -d -m 0700 /root/.ssh
# sudo install -m 0600 ./nixremote /root/.ssh/nixremote # sudo install -m 0600 ./nixremote /root/.ssh/nixremote
# sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version # sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
# Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
# can connect the first time without a manual ssh-keyscan/known_hosts
# step on every new client.
programs.ssh.knownHosts.${vars.nixCacheHost} = {
hostNames = [ vars.nixCacheHost ];
publicKey = vars.nixCacheHostKey;
};
nix = { nix = {
distributedBuilds = true; distributedBuilds = true;
+1 -111
View File
@@ -1,44 +1,5 @@
{ lib, modulesPath, flakeTarget, ... }: { lib, modulesPath, ... }:
let
# Bakes this exact flake target's pre-generated SSH host key straight
# into /etc/ssh/ -- mirrors modules/installer/host-keys.nix's
# builtins.getEnv pattern (impure and empty under normal `nix
# build`/`nix eval`, so this is a no-op unless explicitly opted into
# with NIXOS_HOST_KEYS_DIR=... --impure), but places the key directly
# rather than staging it under /etc/host-keys/ for a later manual copy
# -- this is the whole system for a `lxc-*` host, built straight to a
# pct-restorable tarball with no install step, so there's no later copy
# step to stage for.
#
# Without this, config.system.build.tarball's built-in system just
# generates a fresh host key at first boot like any other host would --
# but sops-nix derives its decryption key from *this* file, and
# .sops.yaml only trusts whatever key scripts/sync-host-keys.sh already
# registered for this exact target name. A freshly-generated key can
# never match that, so every secret (including this host's own login)
# permanently fails to decrypt. Confirmed live: sops-install-secrets
# errored with "Error getting data key: 0 successful groups required,
# got 0" -- the container's actual host key's age fingerprint didn't
# match the one registered in .sops.yaml at all.
hostKeysDirStr = builtins.getEnv "NIXOS_HOST_KEYS_DIR";
hasHostKeysDir = hostKeysDirStr != "" && builtins.pathExists hostKeysDirStr;
hostKeysDir = /. + hostKeysDirStr;
# flakeTarget ("${platform}-${buildType}") comes in via specialArgs from
# flake.nix's mkTarget -- exactly the name scripts/sync-host-keys.sh
# registers keys under. Deliberately not read back from
# config.environment.etc."flake-target" (which is set to the same value)
# -- this module also *contributes* to environment.etc below, and a
# module reading the merged value of an option it's still defining is a
# circular dependency (confirmed: "infinite recursion encountered").
privKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key";
pubKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key.pub";
hasKeyForThisTarget =
hasHostKeysDir
&& builtins.pathExists privKeyFile
&& builtins.pathExists pubKeyFile;
in
{ {
# LXC containers share the host kernel — Proxmox starts them by exec'ing # LXC containers share the host kernel — Proxmox starts them by exec'ing
# /sbin/init directly, no bootloader/initrd involved — and Proxmox has its # /sbin/init directly, no bootloader/initrd involved — and Proxmox has its
@@ -74,75 +35,4 @@ in
# for the same reason; it just doesn't disable NetworkManager itself, # for the same reason; it just doesn't disable NetworkManager itself,
# which modules/common/configuration.nix enables for every host. # which modules/common/configuration.nix enables for every host.
networking.networkmanager.enable = lib.mkForce false; networking.networkmanager.enable = lib.mkForce false;
environment.etc = lib.mkIf hasKeyForThisTarget {
"ssh/ssh_host_ed25519_key" = {
source = privKeyFile;
mode = "0600";
};
"ssh/ssh_host_ed25519_key.pub" = {
source = pubKeyFile;
mode = "0644";
};
};
# virtualisation/proxmox-lxc.nix (imported above) registers the Nix
# store DB via a systemd service (register-nix-paths) -- it never runs
# an activation script at all. Confirmed live this means neither
# sops-nix's "for users" secrets (password hashes -- installed by the
# activation script itself, not a systemd service, since they need to
# exist *before* user creation) nor the user-creation step that
# consumes them ever run on a real lxc-* boot. Regular secrets
# (nix-serve's key, beszel's token, etc.) work anyway because sops-nix
# provides its own systemd service for those.
#
# A systemd service, not boot.postBootCommands: tried that first (it's
# a genuine, generally-invoked hook -- nixos/modules/system/boot/stage-2-init.sh,
# which becomes this container's actual /sbin/init, unconditionally
# runs it) but switch-to-configuration behaves differently that early in
# boot (raw stage-2-init.sh, before systemd itself has even started) --
# confirmed live it silently failed to rewrite /etc/shadow from there
# even in "test" mode, despite the exact same command working reliably
# every time when run post-boot (i.e. as a normal systemd service, which
# is what this is). Not fully root-caused why the early context
# specifically breaks it; a real systemd service sidesteps needing to.
#
# /etc/shadow already has PLACEHOLDER entries for every declared user
# baked in at build time (part of constructing the system closure).
# update-users-groups.pl deliberately never overwrites an *existing*
# shadow entry -- a correct safety property in general (don't clobber a
# real user's real password on a config rebuild) -- but on a genuine
# first boot that only means the real hashedPasswordFile-derived hash
# never gets the chance to be applied either, since the placeholder is
# already "seen". Safe to clear here specifically: there is no real
# password yet to protect on a first boot.
#
# "test" mode, not "boot": confirmed live "boot" mode aborts partway
# through (before rewriting /etc/shadow) on a warning that "/boot" is on
# a different filesystem -- a real check for a host with a bootloader to
# update, meaningless for a container that has none
# (boot.loader.{grub,systemd-boot}.enable are both false above), but it
# still aborts the script. "test" runs every activation step without
# touching boot-loader state at all.
#
# ConditionPathExists (systemd-native, not a bash-level check) means
# this only ever runs once, on the genuine first boot -- systemd itself
# skips even starting it on every later boot once the marker exists.
# switch-to-configuration is otherwise the operator's call per this
# repo's own safety rules, not something to run on every boot.
systemd.services.nixos-lxc-first-boot-activate = {
description = "Complete first-boot NixOS activation (users, secrets) for this LXC container";
wantedBy = [ "multi-user.target" ];
unitConfig.ConditionPathExists = "!/var/lib/nixos-lxc-first-boot-activated";
serviceConfig = {
Type = "oneshot";
RemainAfterExit = true;
};
script = ''
rm -f /etc/shadow
/run/current-system/bin/switch-to-configuration test
mkdir -p /var/lib
touch /var/lib/nixos-lxc-first-boot-activated
'';
};
} }
+13 -12
View File
@@ -1,19 +1,20 @@
_: { vars, ... }:
{ {
services.tailscale = { imports = [ ./enable-service.nix ];
enable = true;
# extraSetFlags (tailscale set, via the always-on tailscaled-set services.tailscale = {
# service), not extraUpFlags -- extraUpFlags is only ever applied by # Enables the sysctl forwarding settings exit nodes/subnet routers need;
# tailscaled-autoconnect, which itself only runs when # without this, --advertise-exit-node has no effect.
# services.tailscale.authKeyFile is set (nothing in this repo sets one, useRoutingFeatures = "server";
# so tailscale up is a manual, one-time operator step on every host that
# uses this service). extraSetFlags has no such gate, so # Lets peers reach this node directly over the tailscale UDP port
# --advertise-exit-node self-reapplies on every boot once the operator # instead of relaying through DERP.
# has authenticated the node once. openFirewall = true;
extraSetFlags = [
extraUpFlags = [
"--advertise-exit-node" "--advertise-exit-node"
"--advertise-routes=${vars.lanCidr}"
]; ];
}; };
} }
+27 -17
View File
@@ -1,14 +1,22 @@
#!/usr/bin/env bash #!/usr/bin/env bash
set -euo pipefail set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" export NIX_CONFIG="${NIX_CONFIG:-}
# shellcheck source=lib/nix-bootstrap.sh experimental-features = nix-command flakes
source "${script_dir}/lib/nix-bootstrap.sh" accept-flake-config = false
# shellcheck source=lib/nix-eval.sh warn-dirty = false
source "${script_dir}/lib/nix-eval.sh" "
MODE="${1:-validate}" MODE="${1:-validate}"
ensure_nix_profile() {
if [ -f /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh ]; then
. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
elif [ -f "$HOME/.nix-profile/etc/profile.d/nix.sh" ]; then
. "$HOME/.nix-profile/etc/profile.d/nix.sh"
fi
}
ensure_nix_profile ensure_nix_profile
if ! command -v nix >/dev/null 2>&1; then if ! command -v nix >/dev/null 2>&1; then
@@ -16,7 +24,8 @@ if ! command -v nix >/dev/null 2>&1; then
exit 127 exit 127
fi fi
hosts="$(list_flake_targets .)" hosts_json="$(nix eval --json --no-use-registries --no-accept-flake-config .#nixosConfigurations --apply builtins.attrNames)"
hosts="$(echo "$hosts_json" | jq -r '.[]')"
echo "Hosts:" echo "Hosts:"
echo "$hosts" echo "$hosts"
@@ -35,17 +44,17 @@ fi
echo echo
echo "Checking Nix formatting with nixpkgs-fmt..." echo "Checking Nix formatting with nixpkgs-fmt..."
nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check . nix run --no-use-registries --no-accept-flake-config github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check .
echo echo
echo "Running statix lint..." echo "Running statix lint..."
nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check . nix run --no-use-registries --no-accept-flake-config github:NixOS/nixpkgs/nixos-25.11#statix -- check .
echo echo
echo "Evaluating host toplevel derivations..." echo "Evaluating host toplevel derivations..."
for host in $hosts; do for host in $hosts; do
echo "==> $host" echo "==> $host"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath" nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
# lxc-* hosts deploy via a directly pct-restore-able tarball instead of # lxc-* hosts deploy via a directly pct-restore-able tarball instead of
# nixos-install (see docs/auto-installer.md); proxmox-* hosts can # nixos-install (see docs/auto-installer.md); proxmox-* hosts can
@@ -55,21 +64,22 @@ for host in $hosts; do
case "$host" in case "$host" in
lxc-*) lxc-*)
echo "==> $host (tarball)" echo "==> $host (tarball)"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.tarball.drvPath" nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.tarball.drvPath"
;; ;;
proxmox-*) proxmox-*)
echo "==> $host (diskoImagesScript)" echo "==> $host (diskoImagesScript)"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript.drvPath" nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript.drvPath"
;; ;;
esac esac
done done
echo echo
echo "Evaluating buildable packages..." echo "Evaluating buildable packages..."
packages="$(nix eval --json "${NIX_EVAL_FLAGS[@]}" .#packages.x86_64-linux --apply builtins.attrNames | jq -r '.[]')" packages_json="$(nix eval --json --no-use-registries --no-accept-flake-config .#packages.x86_64-linux --apply builtins.attrNames)"
packages="$(echo "$packages_json" | jq -r '.[]')"
for pkg in $packages; do for pkg in $packages; do
echo "==> packages.x86_64-linux.${pkg}" echo "==> packages.x86_64-linux.${pkg}"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}" nix eval --raw --no-use-registries --no-accept-flake-config ".#packages.x86_64-linux.${pkg}"
done done
if [[ "$MODE" == "dry-run" ]]; then if [[ "$MODE" == "dry-run" ]]; then
@@ -77,16 +87,16 @@ if [[ "$MODE" == "dry-run" ]]; then
echo "Running dry-run builds for all hosts. This will not create result symlinks." echo "Running dry-run builds for all hosts. This will not create result symlinks."
for host in $hosts; do for host in $hosts; do
echo "==> Dry-run build: $host" echo "==> Dry-run build: $host"
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel" nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.toplevel"
case "$host" in case "$host" in
lxc-*) lxc-*)
echo "==> Dry-run build: $host (tarball)" echo "==> Dry-run build: $host (tarball)"
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.tarball" nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.tarball"
;; ;;
proxmox-*) proxmox-*)
echo "==> Dry-run build: $host (diskoImagesScript)" echo "==> Dry-run build: $host (diskoImagesScript)"
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript" nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript"
;; ;;
esac esac
done done
@@ -95,7 +105,7 @@ if [[ "$MODE" == "dry-run" ]]; then
echo "Running dry-run builds for all packages." echo "Running dry-run builds for all packages."
for pkg in $packages; do for pkg in $packages; do
echo "==> Dry-run build: packages.x86_64-linux.${pkg}" echo "==> Dry-run build: packages.x86_64-linux.${pkg}"
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}" nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#packages.x86_64-linux.${pkg}"
done done
fi fi
+15 -7
View File
@@ -1,11 +1,19 @@
#!/usr/bin/env bash #!/usr/bin/env bash
set -euo pipefail set -euo pipefail
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" export NIX_CONFIG="${NIX_CONFIG:-}
# shellcheck source=lib/nix-bootstrap.sh experimental-features = nix-command flakes
source "${script_dir}/lib/nix-bootstrap.sh" accept-flake-config = false
# shellcheck source=lib/nix-eval.sh warn-dirty = false
source "${script_dir}/lib/nix-eval.sh" "
ensure_nix_profile() {
if [ -f /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh ]; then
. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
elif [ -f "$HOME/.nix-profile/etc/profile.d/nix.sh" ]; then
. "$HOME/.nix-profile/etc/profile.d/nix.sh"
fi
}
install_nix_if_missing() { install_nix_if_missing() {
if command -v nix >/dev/null 2>&1; then if command -v nix >/dev/null 2>&1; then
@@ -71,13 +79,13 @@ if ! command -v jq >/dev/null 2>&1; then
fi fi
echo "Available NixOS hosts:" echo "Available NixOS hosts:"
hosts="$(list_flake_targets .)" hosts="$(nix eval --json --no-use-registries --no-accept-flake-config .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]')"
echo "$hosts" echo "$hosts"
echo "Evaluating all host toplevel derivations..." echo "Evaluating all host toplevel derivations..."
for host in $hosts; do for host in $hosts; do
echo "==> Evaluating $host" echo "==> Evaluating $host"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath" nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
done done
echo "Codex setup complete." echo "Codex setup complete."
+41 -101
View File
@@ -10,9 +10,7 @@
# #
# SAFETY: # SAFETY:
# - The default (create) mode only ever creates a NEW resource -- it # - The default (create) mode only ever creates a NEW resource -- it
# refuses to run if the target VMID already exists on the node, or if # refuses to run if the target VMID already exists on the node.
# a VM/CT identified as --host already exists under any other VMID
# (checked live against the node; --allow-duplicate-host overrides).
# - --modify only ever touches a resource you name explicitly via # - --modify only ever touches a resource you name explicitly via
# --vmid, shows exactly what will change first, and (outside # --vmid, shows exactly what will change first, and (outside
# --dry-run) always requires typing that VMID back to confirm before # --dry-run) always requires typing that VMID back to confirm before
@@ -26,8 +24,6 @@ set -euo pipefail
repo_root="$(cd "$(dirname "$0")/.." && pwd)" repo_root="$(cd "$(dirname "$0")/.." && pwd)"
# shellcheck source=env.sh # shellcheck source=env.sh
source "${repo_root}/scripts/env.sh" source "${repo_root}/scripts/env.sh"
# shellcheck source=lib/nix-eval.sh
source "${repo_root}/scripts/lib/nix-eval.sh"
sync_keys="${repo_root}/scripts/sync-host-keys.sh" sync_keys="${repo_root}/scripts/sync-host-keys.sh"
@@ -44,12 +40,8 @@ Create mode (default):
config.networking.hostName (server, docker, config.networking.hostName (server, docker,
nix-cache, nixos, pxe-boot, nix-minimal). Use nix-cache, nixos, pxe-boot, nix-minimal). Use
--list to see what's available for --type. --list to see what's available for --type.
--name <name> Proxmox display name/hostname (default: --host's --name <name> Proxmox display name/hostname (default: the flake
value, e.g. nix-cache -- for lxc this becomes the target name, e.g. lxc-server)
guest's real networking.hostName too, since
proxmoxLXC.manageHostName pulls it from Proxmox's
own container config, so it must match host.nix
regardless of build type)
--vmid <n> Numeric VMID (default: next free, via --vmid <n> Numeric VMID (default: next free, via
\`pvesh get /cluster/nextid\` on the node). \`pvesh get /cluster/nextid\` on the node).
Refuses to run if this ID already exists. Refuses to run if this ID already exists.
@@ -60,11 +52,10 @@ Create mode (default):
--force-rebuild Skip the "does the node already have this --force-rebuild Skip the "does the node already have this
image" check -- always build fresh and image" check -- always build fresh and
overwrite what's there. overwrite what's there.
--allow-duplicate-host Required if a VM/CT identified as --host --allow-duplicate-host Required if --host already has a real
already exists on the node (checked live via deployment elsewhere (variables.nix's
qm/pct, not any file in this repo) -- deployedTargets) -- otherwise refused, since
otherwise refused, since it'd share that it'd share that host's hostName/hostId.
host's hostName/hostId.
Modify mode (reconfigure an EXISTING resource -- requires --modify): Modify mode (reconfigure an EXISTING resource -- requires --modify):
--modify Switch to modify mode. --modify Switch to modify mode.
@@ -242,21 +233,15 @@ platform_prefix="lxc"
[[ -z "$memory" ]] && memory="$PROXMOX_DEFAULT_MEMORY_MB" [[ -z "$memory" ]] && memory="$PROXMOX_DEFAULT_MEMORY_MB"
# --- discover / resolve the flake target from --host -------------------- # --- discover / resolve the flake target from --host --------------------
# Emits "<target>\t<hostName>" pairs for every ${platform_prefix}-* flake
# target -- the one source both --list and the --host lookup below read
# from, so they can never see a different set of targets from each other.
targets_for_platform() {
local target
for target in $(list_flake_targets "$repo_root" 2>/dev/null | grep -- "^${platform_prefix}-"); do
printf '%s\t%s\n' "$target" "$(flake_target_hostname "$repo_root" "$target")"
done
}
list_hosts() { list_hosts() {
local target hostname local target hostname
while IFS=$'\t' read -r target hostname; do for target in $(nix eval --json --no-use-registries --no-accept-flake-config \
"${repo_root}#nixosConfigurations" --apply builtins.attrNames 2>/dev/null \
| jq -r --arg p "${platform_prefix}-" '.[] | select(startswith($p))'); do
hostname="$(nix eval --raw --no-use-registries --no-accept-flake-config \
"${repo_root}#nixosConfigurations.${target}.config.networking.hostName" 2>/dev/null)"
printf ' %-12s -> %s\n' "$hostname" "$target" printf ' %-12s -> %s\n' "$hostname" "$target"
done < <(targets_for_platform) done
} }
if [[ "$do_list" -eq 1 ]]; then if [[ "$do_list" -eq 1 ]]; then
@@ -271,12 +256,16 @@ if [[ -z "$host" ]]; then
fi fi
flake_target="" flake_target=""
while IFS=$'\t' read -r target hostname; do for target in $(nix eval --json --no-use-registries --no-accept-flake-config \
if [[ "$hostname" == "$host" ]]; then "${repo_root}#nixosConfigurations" --apply builtins.attrNames \
| jq -r --arg p "${platform_prefix}-" '.[] | select(startswith($p))'); do
hn="$(nix eval --raw --no-use-registries --no-accept-flake-config \
"${repo_root}#nixosConfigurations.${target}.config.networking.hostName")"
if [[ "$hn" == "$host" ]]; then
flake_target="$target" flake_target="$target"
break break
fi fi
done < <(targets_for_platform) done
if [[ -z "$flake_target" ]]; then if [[ -z "$flake_target" ]]; then
echo "ERROR: no ${platform_prefix}-* target has hostName '${host}'." >&2 echo "ERROR: no ${platform_prefix}-* target has hostName '${host}'." >&2
@@ -285,65 +274,27 @@ if [[ -z "$flake_target" ]]; then
exit 1 exit 1
fi fi
# The container/VM's real identity is --host (e.g. "nix-cache"), validated [[ -z "$name" ]] && name="$flake_target"
# above against config.networking.hostName -- not the flake target name
# (e.g. "lxc-nix-cache"), which is build-type-specific and only exists to
# pick which platform variant to build. Defaulting --name to the flake
# target would make lxc's --hostname (which proxmoxLXC.manageHostName
# feeds straight into the guest's real hostname) disagree with host.nix.
[[ -z "$name" ]] && name="$host"
# --- refuse to duplicate a host that's already live on the node --------- # --- refuse to duplicate a host that's already really deployed ----------
# Queries the node itself (qm/pct's own name/hostname config), not any # Checked by hostName, not exact flake target: proxmox-server being
# static list in this repo -- a file can't track whether a resource still # deployed also blocks --type lxc --host server, since both would carry
# actually exists, and this used to be checked against variables.nix's # the same hosts/server/host.nix identity (hostName, hostId).
# deployedTargets, which drifted stale (it kept naming a VM as "the real if [[ "$allow_duplicate_host" -eq 0 ]]; then
# deployment" well after that VM had been destroyed, blocking its own deployed_targets_json="$(nix eval --json --no-use-registries --no-accept-flake-config \
# redeploy) until that list was dropped in favour of this live check. This --file "${repo_root}/variables.nix" deployedTargets)"
# only catches guests identified with the default --name (== --host, what for dt in $(echo "$deployed_targets_json" | jq -r '.[]'); do
# this script itself always uses unless --name is overridden) -- a guest dt_hostname="$(nix eval --raw --no-use-registries --no-accept-flake-config \
# manually renamed on the node afterwards wouldn't match, but nothing here "${repo_root}#nixosConfigurations.${dt}.config.networking.hostName" 2>/dev/null || true)"
# creates guests that way. if [[ "$dt_hostname" == "$host" ]]; then
if [[ "$allow_duplicate_host" -eq 1 ]]; then echo "ERROR: '${host}' already has a real deployment (${dt}, per variables.nix's" >&2
echo echo "deployedTargets). Creating ${flake_target} would share its hostName/hostId --" >&2
echo "--allow-duplicate-host: skipping the check for an existing '${host}' on ${node}." echo "refusing by default. Pass --allow-duplicate-host if you really mean to spin" >&2
elif [[ "$dry_run" -eq 1 ]]; then echo "up a separate test instance of this host (it'll still get its own distinct" >&2
echo echo "sops key and VMID, never touching ${dt})." >&2
echo "[dry-run] would check ${node} for an existing VM/CT identified as '${host}'" exit 1
else fi
echo done
echo "==> Checking ${node} for an existing VM/CT identified as '${host}'..."
ssh_check_status=0
existing="$(ssh "$ssh_target" bash -s -- "$host" <<'REMOTE_SCRIPT'
target="$1"
for id in $(qm list 2>/dev/null | awk 'NR>1{print $1}'); do
n="$(qm config "$id" 2>/dev/null | grep -oP '^name:\s*\K\S+' || true)"
[[ "$n" == "$target" ]] && echo "vm ${id} ${n}"
done
for id in $(pct list 2>/dev/null | awk 'NR>1{print $1}'); do
n="$(pct config "$id" 2>/dev/null | grep -oP '^hostname:\s*\K\S+' || true)"
[[ "$n" == "$target" ]] && echo "lxc ${id} ${n}"
done
REMOTE_SCRIPT
)" || ssh_check_status=$?
if [[ "$ssh_check_status" -ne 0 ]]; then
echo "ERROR: couldn't reach ${node} (ssh exited ${ssh_check_status}) to check for an" >&2
echo "existing '${host}' resource -- refusing to guess. Fix connectivity and retry," >&2
echo "or pass --allow-duplicate-host if you're sure none exists (this skips the" >&2
echo "check entirely)." >&2
exit 1
fi
if [[ -n "$existing" ]]; then
echo "ERROR: '${host}' already exists on ${node}:" >&2
echo "$existing" | while read -r kind id n; do
echo " - ${kind} VMID ${id} (${n})" >&2
done
echo "Refusing to create a second resource sharing this identity. Pass" >&2
echo "--allow-duplicate-host to create one anyway (it gets its own distinct" >&2
echo "sops key and VMID -- the existing resource(s) above are left untouched)," >&2
echo "or use --modify to reconfigure the existing one instead." >&2
exit 1
fi
fi fi
echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '${name}'" echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '${name}'"
@@ -432,19 +383,9 @@ else
fi fi
if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
# Mirrors the real build commands' "${NIX_OPTS[@]}" below -- nix_extra_opts
# (called earlier, once) has already decided whether nix-cache is in play,
# and the dry-run preview needs to reflect that decision instead of always
# printing the same command regardless of outcome.
nix_opts_display=""
if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
printf -v nix_opts_display '%q ' "${NIX_OPTS[@]}"
nix_opts_display=" ${nix_opts_display% }"
fi
if [[ "$type" == "lxc" ]]; then if [[ "$type" == "lxc" ]]; then
if [[ "$dry_run" -eq 1 ]]; then if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would build: NIXOS_HOST_KEYS_DIR=${repo_root}/host-keys nix build --impure \\" echo "[dry-run] would build: NIXOS_HOST_KEYS_DIR=${repo_root}/host-keys nix build --impure \\"
echo "[dry-run] --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.tarball" echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.tarball"
local_image="<built-tarball>" local_image="<built-tarball>"
else else
@@ -458,8 +399,7 @@ if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
fi fi
else else
if [[ "$dry_run" -eq 1 ]]; then if [[ "$dry_run" -eq 1 ]]; then
echo "[dry-run] would build: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\" echo "[dry-run] would build: nix build .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
echo "[dry-run] would run: sudo ./result-${flake_target} \\" echo "[dry-run] would run: sudo ./result-${flake_target} \\"
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\" echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\"
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\" echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\"
+5 -52
View File
@@ -88,60 +88,13 @@ nix_extra_opts() {
fi fi
export NIX_EXTRA_OPTS_DECIDED=1 export NIX_EXTRA_OPTS_DECIDED=1
NIX_OPTS=() NIX_OPTS=()
if ! curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
# Retry a couple of times, 1s apart, before believing either check --
# belt-and-suspenders against a genuine multi-second blip (nix-cache
# restarting), on top of the fix below. Worst case (~11s total, host
# genuinely gone) is still nowhere near the 15s+ *per lookup* nix's own
# substituter retries would cost if this check didn't exist at all.
local attempt cache_up=0 builder_up=0
for attempt in 1 2 3; do
if curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
cache_up=1
break
fi
[[ "$attempt" -lt 3 ]] && sleep 1
done
if [[ "$cache_up" -eq 0 ]]; then
echo "nix-cache (http://${NIX_CACHE_HOST}) is unreachable -- skipping it (substituter + remote builder) for the rest of this run." >&2 echo "nix-cache (http://${NIX_CACHE_HOST}) is unreachable -- skipping it (substituter + remote builder) for the rest of this run." >&2
NIX_OPTS=(--option substituters "https://cache.nixos.org/" --builders "") NIX_OPTS=(--option substituters "https://cache.nixos.org/" --builders "")
else elif ! timeout 3 bash -c "cat < /dev/tcp/${NIX_CACHE_HOST}/22" >/dev/null 2>&1; then
for attempt in 1 2 3; do echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
# `exec 3<>/dev/tcp/...` just opens the fd and returns -- it does NOT NIX_OPTS=(--builders "")
# read from it. Confirmed live this is load-bearing, not stylistic:
# the previous `cat < /dev/tcp/.../22` blocked forever and always hit
# the timeout even against a perfectly healthy nix-cache, because
# sshd sends its banner and then holds the connection open waiting
# for the client to speak next -- `cat` never sees EOF, so this
# check reported "unreachable" unconditionally, 100% of the time,
# regardless of whether the remote builder was actually up.
if timeout 3 bash -c "exec 3<>/dev/tcp/${NIX_CACHE_HOST}/22" 2>/dev/null; then
builder_up=1
break
fi
[[ "$attempt" -lt 3 ]] && sleep 1
done
if [[ "$builder_up" -eq 0 ]]; then
echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
NIX_OPTS=(--builders "")
fi
fi
# `printf '%q '` with a genuinely empty NIX_OPTS still runs one format
# pass over a missing argument and yields the literal `'' ` rather than
# an empty string (confirmed live) -- a subprocess that later does
# `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` (the branch above, for e.g.
# sync-host-keys.sh reusing this process's decision) would then rebuild
# a 1-element array holding an empty string instead of a 0-element
# array, and `nix-shell "${NIX_OPTS[@]}" -p <pkg>` chokes on that stray
# element as a bogus positional argument. Guard the empty case
# explicitly so nix-cache being reachable (NIX_OPTS legitimately empty)
# round-trips as truly empty instead.
if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
else
NIX_EXTRA_OPTS=""
fi fi
printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
export NIX_EXTRA_OPTS export NIX_EXTRA_OPTS
} }
-20
View File
@@ -1,20 +0,0 @@
#!/usr/bin/env bash
# Shared Nix bootstrap for scripts/codex-setup.sh and
# scripts/codex-maintenance.sh: the nix.conf settings both need in effect
# before a single `nix` command runs (flakes enabled, never honor a flake
# input's own nixConfig, no "dirty tree" warning spam), plus a helper to
# pull an already-installed Nix's daemon/profile script onto PATH if it
# isn't there yet. Source this instead of copying it -- see CLAUDE.md.
export NIX_CONFIG="${NIX_CONFIG:-}
experimental-features = nix-command flakes
accept-flake-config = false
warn-dirty = false
"
ensure_nix_profile() {
if [ -f /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh ]; then
. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
elif [ -f "$HOME/.nix-profile/etc/profile.d/nix.sh" ]; then
. "$HOME/.nix-profile/etc/profile.d/nix.sh"
fi
}
-36
View File
@@ -1,36 +0,0 @@
#!/usr/bin/env bash
# Shared flake-introspection helpers for scripts/*.sh. Source alongside
# env.sh:
# source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/nix-eval.sh"
#
# NIX_EVAL_FLAGS: --no-use-registries so a call here never resolves through
# the user's global flake registry (every call targets this repo's own
# flake, or an explicit github: ref, not a registry alias); --no-accept-flake-config
# so a flake input's own nixConfig (e.g. a dependency's substituters) is
# never honored -- matches accept-flake-config = false already set repo-wide
# (see lib/nix-bootstrap.sh / CLAUDE.md). Reuse this array rather than
# retyping the two flags at each call site.
declare -a NIX_EVAL_FLAGS=(--no-use-registries --no-accept-flake-config)
# list_flake_targets <flake_ref>
# Prints the attribute names under <flake_ref>#nixosConfigurations, one per
# line, e.g.:
# list_flake_targets . # from inside the repo
# list_flake_targets "$repo_root" # from anywhere
list_flake_targets() {
local flake_ref="$1"
nix eval --json "${NIX_EVAL_FLAGS[@]}" \
"${flake_ref}#nixosConfigurations" --apply builtins.attrNames \
| jq -r '.[]'
}
# flake_target_hostname <flake_ref> <target>
# Prints one nixosConfigurations target's config.networking.hostName.
# Empty (not an error under set -e) if the target doesn't exist or the
# eval otherwise fails -- callers that need to distinguish "empty" from
# "eval failed" should check $? themselves instead of relying on this.
flake_target_hostname() {
local flake_ref="$1" target="$2"
nix eval --raw "${NIX_EVAL_FLAGS[@]}" \
"${flake_ref}#nixosConfigurations.${target}.config.networking.hostName" 2>/dev/null
}
-30
View File
@@ -1,30 +0,0 @@
#!/usr/bin/env bash
# Shared SSH-host-key / age-conversion helpers for scripts/sync-host-keys.sh
# and scripts/prepare-host-key.sh -- both generate the same kind of key
# (ed25519, no passphrase, the sops-nix age-derivation input) and convert it
# to an age recipient the same way; kept in one place so the two can't
# drift apart.
#
# Uses NIX_OPTS (an array of extra `nix-shell` options -- see env.sh's
# nix_extra_opts) if the caller has already set it, so a decision to avoid
# an unreachable nix-cache is reused here instead of probed again. Falls
# back to no extra options if the caller never sourced env.sh.
if ! declare -p NIX_OPTS >/dev/null 2>&1; then
declare -a NIX_OPTS=()
fi
# generate_host_ed25519_key <hostname> <keyfile>
# Writes <keyfile> and <keyfile>.pub. Caller is responsible for refusing to
# overwrite an existing keyfile -- this always runs ssh-keygen fresh.
generate_host_ed25519_key() {
local hostname="$1" keyfile="$2"
nix-shell "${NIX_OPTS[@]}" -p openssh --run \
"ssh-keygen -t ed25519 -N '' -C '${hostname}' -f '${keyfile}'" >/dev/null
}
# ssh_pubkey_to_age <pubkeyfile>
# Prints the age public key derived from an ed25519 SSH public key file.
ssh_pubkey_to_age() {
local pubkeyfile="$1"
nix-shell "${NIX_OPTS[@]}" -p ssh-to-age --run "ssh-to-age -i '${pubkeyfile}'"
}
+2 -7
View File
@@ -23,10 +23,6 @@
set -euo pipefail set -euo pipefail
repo_root="$(cd "$(dirname "$0")/.." && pwd)" repo_root="$(cd "$(dirname "$0")/.." && pwd)"
# shellcheck source=env.sh
source "${repo_root}/scripts/env.sh"
# shellcheck source=lib/ssh-host-keys.sh
source "${repo_root}/scripts/lib/ssh-host-keys.sh"
hostname="${1:?usage: scripts/prepare-host-key.sh <hostname>}" hostname="${1:?usage: scripts/prepare-host-key.sh <hostname>}"
sops_yaml="${repo_root}/.sops.yaml" sops_yaml="${repo_root}/.sops.yaml"
@@ -45,10 +41,9 @@ if [[ -f "$keyfile" ]]; then
exit 1 exit 1
fi fi
nix_extra_opts nix-shell -p openssh --run "ssh-keygen -t ed25519 -N '' -C '${hostname}' -f '${keyfile}'" >/dev/null
generate_host_ed25519_key "$hostname" "$keyfile"
age_pub="$(ssh_pubkey_to_age "${keyfile}.pub")" age_pub="$(nix-shell -p ssh-to-age --run "ssh-to-age -i '${keyfile}.pub'")"
cat <<EOF cat <<EOF
+5 -7
View File
@@ -31,10 +31,6 @@ editor="${repo_root}/scripts/lib/sync-host-keys-edit-sops.py"
# shellcheck source=env.sh # shellcheck source=env.sh
source "${repo_root}/scripts/env.sh" source "${repo_root}/scripts/env.sh"
# shellcheck source=lib/nix-eval.sh
source "${repo_root}/scripts/lib/nix-eval.sh"
# shellcheck source=lib/ssh-host-keys.sh
source "${repo_root}/scripts/lib/ssh-host-keys.sh"
mkdir -p "$keydir" mkdir -p "$keydir"
@@ -122,10 +118,12 @@ EOF
} }
discover_targets() { discover_targets() {
nix eval --json --no-use-registries --no-accept-flake-config \
"${repo_root}#nixosConfigurations" --apply builtins.attrNames \
| jq -r '.[] | select(. != "installer")'
# installer is the one nixosConfigurations target that doesn't import # installer is the one nixosConfigurations target that doesn't import
# sops-nix at all (see CLAUDE.md's "Security Notes" -- hardcoded login # sops-nix at all (see CLAUDE.md's "Security Notes" -- hardcoded login
# password instead) -- config.sops.secrets doesn't exist for it. # password instead) -- config.sops.secrets doesn't exist for it.
list_flake_targets "$repo_root" | grep -v '^installer$'
} }
locally_managed_hosts() { locally_managed_hosts() {
@@ -161,7 +159,7 @@ queue_host_sync() {
echo "[dry-run] ${host}: would generate host key" echo "[dry-run] ${host}: would generate host key"
else else
echo "==> ${host}: generating host key" echo "==> ${host}: generating host key"
generate_host_ed25519_key "$host" "$keyfile" nix-shell "${NIX_OPTS[@]}" -p openssh --run "ssh-keygen -t ed25519 -N '' -C '${host}' -f '${keyfile}'" >/dev/null
fi fi
else else
echo "==> ${host}: host key already present" echo "==> ${host}: host key already present"
@@ -172,7 +170,7 @@ queue_host_sync() {
if [[ "$dry_run" -eq 1 ]]; then if [[ "$dry_run" -eq 1 ]]; then
age_pub="dry-run-placeholder-not-a-real-key" age_pub="dry-run-placeholder-not-a-real-key"
else else
age_pub="$(ssh_pubkey_to_age "${keyfile}.pub")" age_pub="$(nix-shell "${NIX_OPTS[@]}" -p ssh-to-age --run "ssh-to-age -i '${keyfile}.pub'")"
fi fi
add_keys_json="$(jq --arg host "$host" --arg key "$age_pub" \ add_keys_json="$(jq --arg host "$host" --arg key "$age_pub" \
'. + [{host: $host, age_key: $key}]' <<<"$add_keys_json")" '. + [{host: $host, age_key: $key}]' <<<"$add_keys_json")"
+16 -9
View File
@@ -19,15 +19,6 @@
remoteBuilderUser = "nixremote"; # remote builder SSH user remoteBuilderUser = "nixremote"; # remote builder SSH user
# nix-cache's own SSH host public key (not a secret — the private half
# never leaves the host). Wired into every client's
# programs.ssh.knownHosts by modules/nix-cache/remote-builder-client.nix
# so distributed builds don't hit "Host key verification failed" on a
# fresh client that has never manually ssh'd to nix-cache before. Update
# this if nix-cache's host key is ever rotated or the host is rebuilt
# from scratch.
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
# Public keys authorized to SSH in as remoteBuilderUser on the nix-cache # Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
# host (modules/nix-cache/server.nix) — one per client host that's allowed # host (modules/nix-cache/server.nix) — one per client host that's allowed
# to use it as a distributed builder. # to use it as a distributed builder.
@@ -156,4 +147,20 @@
keep = 20; # number of rotated logs to retain before deleting the oldest keep = 20; # number of rotated logs to retain before deleting the oldest
}; };
# Flake targets with a real, currently-running deployment somewhere —
# matches README.md's Hosts table "(real, deployed)" annotations; update
# both together. Not consumed by any NixOS module (nothing in the actual
# system config should behave differently because of this) — it's read
# by scripts/create-proxmox-resource.sh to refuse creating a same-identity
# duplicate of an already-deployed host (shared hostName/hostId) unless
# you explicitly pass --allow-duplicate-host.
deployedTargets = [
"linode-minimal"
"proxmox-minimal"
"proxmox-nix-cache"
"proxmox-server"
"proxmox-docker"
"proxmox-gui"
"proxmox-pxe-boot"
];
} }