Archived
Compare commits
19
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
7e51168d1b | ||
|
|
ab5206b1c7 | ||
|
|
2041557ab3 | ||
|
|
2fd483697b | ||
|
|
89186b0dee | ||
|
|
8e3606cbd3 | ||
|
|
a18dfb0127 | ||
|
|
75f1342339 | ||
|
|
36ba99c9a1 | ||
|
|
0cd8f15b48 | ||
|
|
8613b93fa8 | ||
|
|
20f9475a7d | ||
|
|
6babb3eec5 | ||
|
|
33730e6ccf | ||
|
|
65f89806cb | ||
|
|
c3007097a6 | ||
|
|
9724babcea | ||
|
|
7055bcdb97 | ||
|
|
d973da487c |
@@ -93,9 +93,10 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
|
||||
(`--force-rebuild` to skip that and always rebuild), and probes
|
||||
nix-cache's substituter/remote-builder reachability once up front rather
|
||||
than letting every `nix build` call retry against it individually.
|
||||
Refuses to create a target whose host identity already has a real
|
||||
deployment elsewhere (`variables.nix`'s `deployedTargets`) unless
|
||||
`--allow-duplicate-host` is passed. `--dry-run` throughout both modes.
|
||||
Refuses to create a target whose host identity already exists live on
|
||||
the node (checked directly via `qm`/`pct`, not any file in this repo)
|
||||
unless `--allow-duplicate-host` is passed. `--dry-run` throughout both
|
||||
modes.
|
||||
- `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
|
||||
default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
|
||||
cross-script config here instead of duplicating it per-script.
|
||||
@@ -133,9 +134,10 @@ nixosSystem {
|
||||
```
|
||||
|
||||
Platforms: `linode`, `proxmox`, `lxc`. Build types: `minimal`, `nix-cache`,
|
||||
`server`, `docker`, `gui`, `pxe-boot`. Not every combination is built — e.g.
|
||||
`pxe-boot` has no `linode` variant (PXE/DHCP/TFTP need LAN L2 adjacency a
|
||||
Linode VPS doesn't have). Treat `flake.nix`'s `generatedTargets` as the source
|
||||
`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`. Not every
|
||||
combination is built — e.g. `pxe-boot` has no `linode` variant (PXE/DHCP/TFTP
|
||||
need LAN L2 adjacency a Linode VPS doesn't have). Treat `flake.nix`'s
|
||||
`generatedTargets` as the source
|
||||
of truth for which hosts exist — `README.md`, `AGENTS.md`,
|
||||
`docs/flake-lock-automation.md`, and the CI eval workflows
|
||||
(`.github/workflows/check-nixos.yml`, `.gitea/workflows/check-nixos.yml`) list
|
||||
|
||||
@@ -10,7 +10,7 @@ pieces composed in `flake.nix`:
|
||||
|
||||
- **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`
|
||||
- **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
|
||||
`gui`, `pxe-boot`
|
||||
`gui`, `pxe-boot`, `tailscale-exit-node`
|
||||
|
||||
Not every combination exists — `pxe-boot` has no `linode` variant, since
|
||||
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have. The full
|
||||
@@ -21,16 +21,19 @@ list:
|
||||
| `linode-minimal` | Minimal NixOS host profile on a Linode VPS (real, deployed) |
|
||||
| `proxmox-minimal` | Minimal NixOS host profile on Proxmox (real, deployed — previously the flat `nix-minimal` target) |
|
||||
| `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container |
|
||||
| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder (`proxmox-nix-cache` is the real, deployed one — previously the flat `nix-cache` target) |
|
||||
| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder (`lxc-nix-cache` is the real, deployed one — previously `proxmox-nix-cache`, itself previously the flat `nix-cache` target) |
|
||||
| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host (`proxmox-server` is the real, deployed one — previously the flat `server` target) |
|
||||
| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack (`proxmox-docker` is the real, deployed one — previously the flat `docker` target) |
|
||||
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation (`proxmox-gui` is the real, deployed one — previously the flat `nixos` target) |
|
||||
| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host (`proxmox-pxe-boot` is the real, deployed one — previously the flat `pxe-boot` target) |
|
||||
| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node (no deployed target yet; `lxc-tailscale-exit-node` is the one planned for actual use) |
|
||||
|
||||
The "(real, deployed)" targets above are also tracked machine-readably in
|
||||
`variables.nix`'s `deployedTargets` — keep both in sync when a deployment
|
||||
changes. `scripts/create-proxmox-resource.sh` reads that list to refuse
|
||||
creating a same-identity duplicate of an already-deployed host by accident.
|
||||
This table is the only place "(real, deployed)" status is tracked — there's
|
||||
no separate machine-readable copy to keep in sync. `scripts/create-proxmox-resource.sh`
|
||||
guards against creating a same-identity duplicate of an already-deployed host
|
||||
by checking the Proxmox node itself (live `qm`/`pct` state) rather than any
|
||||
file in this repo, since a static list can't track whether a resource still
|
||||
actually exists.
|
||||
|
||||
Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
|
||||
(hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay
|
||||
|
||||
+27
-5
@@ -77,11 +77,33 @@ system profile) — there's no separate activation step to run yourself.
|
||||
of this (build, host-key handling, upload, `pct create` with the flags
|
||||
above) — see its `--help`.
|
||||
|
||||
Host keys still need pre-seeding the same way as any other host (see "Host
|
||||
keys" below) — the sops-nix activation-vs-first-boot race is identical
|
||||
regardless of how the image reaches the machine. `NIXOS_HOST_KEYS_DIR=...
|
||||
nix build ... --impure` bakes the matching key into the tarball the same way
|
||||
it does for the ISO/PXE installer images.
|
||||
Host keys still need pre-seeding the same way as any other host — the
|
||||
sops-nix activation-vs-first-boot race is identical regardless of how the
|
||||
image reaches the machine. Unlike the ISO/PXE installer (where
|
||||
`modules/installer/host-keys.nix` bakes *every* `host-keys/` entry into
|
||||
`/etc/host-keys/` for `auto-install.sh` to pick from and copy at install
|
||||
time — see "Host keys" below), an `lxc-*` tarball has no install step to
|
||||
copy anything during, so `modules/platforms/lxc.nix` bakes this *one*
|
||||
target's key straight into `/etc/ssh/ssh_host_ed25519_key(.pub)` directly,
|
||||
keyed by its own exact flake target name (`config.environment.etc` can't
|
||||
be read back from within a module still contributing to it, so this comes
|
||||
in via `specialArgs.flakeTarget`, set by `flake.nix`'s `mkTarget`):
|
||||
|
||||
```sh
|
||||
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
|
||||
nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure
|
||||
```
|
||||
|
||||
Confirmed the hard way: without this, the tarball's own built-in system
|
||||
just generates a fresh host key at first boot like any host would, which
|
||||
can never match whatever `.sops.yaml` actually trusts for that target —
|
||||
`sops-install-secrets` fails with `Error getting data key: 0 successful
|
||||
groups required, got 0`, and *every* secret (including this host's own
|
||||
login) permanently fails to decrypt, silently — no error in the boot log
|
||||
at all, since the activation step that would install secrets only runs on
|
||||
a from-scratch first activation and skips silently once `/run/current-system`
|
||||
already exists. `scripts/create-proxmox-resource.sh` always builds with
|
||||
`NIXOS_HOST_KEYS_DIR` set for this reason.
|
||||
|
||||
## Layout
|
||||
|
||||
|
||||
@@ -59,6 +59,15 @@ On `nix-cache`, install the matching public key used by `nixremote` authorized k
|
||||
The committed `nixremote` authorized keys are public SSH keys only. Keep the
|
||||
matching private keys on client hosts and out of the repository.
|
||||
|
||||
nix-cache's own SSH *host* key is trusted declaratively via
|
||||
`programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
|
||||
sourced from `vars.nixCacheHostKey` (`variables.nix`) — every client rebuild
|
||||
picks it up automatically, so distributed builds don't fail with "Host key
|
||||
verification failed" on a client that has never manually SSH'd to nix-cache
|
||||
before. If nix-cache's host key is ever rotated or the host rebuilt from
|
||||
scratch, update `vars.nixCacheHostKey` to match its new
|
||||
`/etc/ssh/ssh_host_ed25519_key.pub`.
|
||||
|
||||
## Manual verification
|
||||
|
||||
After deployment:
|
||||
|
||||
@@ -33,6 +33,9 @@
|
||||
# nix-cache itself consumes the nix-cache substituter and remote
|
||||
# builder.
|
||||
mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix }:
|
||||
let
|
||||
flakeTarget = "${platform}-${buildType}";
|
||||
in
|
||||
nixpkgs.lib.nixosSystem {
|
||||
inherit system;
|
||||
modules = [
|
||||
@@ -42,7 +45,7 @@
|
||||
./modules/platforms/${platform}.nix
|
||||
./modules/build-types/${buildType}.nix
|
||||
hostPath
|
||||
{ environment.etc."flake-target".text = "${platform}-${buildType}"; }
|
||||
{ environment.etc."flake-target".text = flakeTarget; }
|
||||
home-manager.nixosModules.home-manager
|
||||
{
|
||||
home-manager = {
|
||||
@@ -56,7 +59,13 @@
|
||||
./modules/nix-cache/client.nix
|
||||
./modules/nix-cache/remote-builder-client.nix
|
||||
];
|
||||
specialArgs = { inherit inputs vars netbootSystem; };
|
||||
# flakeTarget is passed via specialArgs (not read back from
|
||||
# config.environment.etc."flake-target" above) specifically so
|
||||
# modules/platforms/lxc.nix can use it to select its own host key
|
||||
# file without a same-option circular dependency (a module
|
||||
# contributing to environment.etc can't read the merged
|
||||
# environment.etc it's itself contributing to).
|
||||
specialArgs = { inherit inputs vars netbootSystem flakeTarget; };
|
||||
};
|
||||
|
||||
# Generated platform x build-type matrix. pxe-boot has no linode
|
||||
@@ -85,6 +94,10 @@
|
||||
|
||||
proxmox-pxe-boot = mkTarget { platform = "proxmox"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
|
||||
lxc-pxe-boot = mkTarget { platform = "lxc"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
|
||||
|
||||
linode-tailscale-exit-node = mkTarget { platform = "linode"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
|
||||
proxmox-tailscale-exit-node = mkTarget { platform = "proxmox"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
|
||||
lxc-tailscale-exit-node = mkTarget { platform = "lxc"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
|
||||
};
|
||||
|
||||
# Auto-install environments (migrated from the former nix-auto-installer
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
_:
|
||||
|
||||
{
|
||||
networking.hostName = "exit-node";
|
||||
|
||||
# No networking.hostId: only ZFS-touching hosts (server, docker) need one
|
||||
# for pool-import safety, and this host does neither.
|
||||
|
||||
# A genuinely new host (not a pre-refactor carry-over), so it tracks the
|
||||
# flake's current nixpkgs release rather than being pinned to an older one.
|
||||
system.stateVersion = "26.05";
|
||||
}
|
||||
@@ -6,4 +6,13 @@
|
||||
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
|
||||
HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
|
||||
};
|
||||
|
||||
# The upstream module runs beszel-agent under DynamicUser with
|
||||
# ProtectSystem = "strict" and no StateDirectory, so /var/lib/beszel-agent
|
||||
# (where the agent persists its hub-pairing fingerprint, per
|
||||
# https://github.com/henrygd/beszel/discussions/1542) isn't writable --
|
||||
# every restart silently fails to save it and regenerates a fresh one in
|
||||
# memory, permanently desyncing from whatever the hub has on record after
|
||||
# the very first successful pairing. Give it real persistent storage.
|
||||
systemd.services.beszel-agent.serviceConfig.StateDirectory = "beszel-agent";
|
||||
}
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
{ ... }:
|
||||
|
||||
{
|
||||
imports = [
|
||||
../tailscale/exit-node.nix
|
||||
];
|
||||
|
||||
# "server", not "both": this build type only ever advertises itself as an
|
||||
# exit node (see ../tailscale/exit-node.nix) -- it doesn't advertise LAN
|
||||
# subnet routes, so it doesn't need the "client"-side loose reverse-path
|
||||
# filtering that "both" would also turn on. Deliberately left unbundled
|
||||
# from LAN-subnet-route advertisement so this build type stays valid on
|
||||
# every platform, including linode (a remote VPS with no network path to
|
||||
# the home LAN at all).
|
||||
services.tailscale.useRoutingFeatures = "server";
|
||||
|
||||
# Forwarded exit-node traffic arrives on tailscale0 already
|
||||
# tailscale-authenticated -- the firewall's normal per-port allow-list
|
||||
# would otherwise drop it. Standard NixOS/Tailscale exit-node guidance.
|
||||
networking.firewall.trustedInterfaces = [ "tailscale0" ];
|
||||
}
|
||||
@@ -17,6 +17,26 @@ let
|
||||
--refresh \
|
||||
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
|
||||
'';
|
||||
|
||||
# lxc-* hosts pre-seed their SSH host key at build time (see
|
||||
# modules/platforms/lxc.nix) so sops-nix's .sops.yaml recipient matches on
|
||||
# first boot -- without it, secrets permanently fail to decrypt (see that
|
||||
# file's comment for the confirmed failure). That requires --impure plus
|
||||
# NIXOS_HOST_KEYS_DIR pointing at the repo's host-keys/ dir, same pattern
|
||||
# docs/auto-installer.md uses for the installer ISO. A function, not a
|
||||
# shellAlias, since the target name has to interpolate into the middle of
|
||||
# the flake attribute path, not just append after it. Must be run from the
|
||||
# repo root, same as every other host-keys/ command in this repo.
|
||||
buildImageFn = ''
|
||||
buildImage() {
|
||||
if [ -z "$1" ]; then
|
||||
echo "usage: buildImage <flake-target> (e.g. lxc-docker)" >&2
|
||||
return 1
|
||||
fi
|
||||
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
|
||||
".#nixosConfigurations.$1.config.system.build.tarball"
|
||||
}
|
||||
'';
|
||||
in
|
||||
{
|
||||
programs.bash = {
|
||||
@@ -25,5 +45,6 @@ in
|
||||
"Switch-nix" = mySwitchCmd;
|
||||
"Test-nix" = myTestCmd;
|
||||
};
|
||||
initExtra = buildImageFn;
|
||||
};
|
||||
}
|
||||
|
||||
@@ -5,6 +5,14 @@
|
||||
# sudo install -d -m 0700 /root/.ssh
|
||||
# sudo install -m 0600 ./nixremote /root/.ssh/nixremote
|
||||
# sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
|
||||
# Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
|
||||
# can connect the first time without a manual ssh-keyscan/known_hosts
|
||||
# step on every new client.
|
||||
programs.ssh.knownHosts.${vars.nixCacheHost} = {
|
||||
hostNames = [ vars.nixCacheHost ];
|
||||
publicKey = vars.nixCacheHostKey;
|
||||
};
|
||||
|
||||
nix = {
|
||||
distributedBuilds = true;
|
||||
|
||||
|
||||
+111
-1
@@ -1,5 +1,44 @@
|
||||
{ lib, modulesPath, ... }:
|
||||
{ lib, modulesPath, flakeTarget, ... }:
|
||||
|
||||
let
|
||||
# Bakes this exact flake target's pre-generated SSH host key straight
|
||||
# into /etc/ssh/ -- mirrors modules/installer/host-keys.nix's
|
||||
# builtins.getEnv pattern (impure and empty under normal `nix
|
||||
# build`/`nix eval`, so this is a no-op unless explicitly opted into
|
||||
# with NIXOS_HOST_KEYS_DIR=... --impure), but places the key directly
|
||||
# rather than staging it under /etc/host-keys/ for a later manual copy
|
||||
# -- this is the whole system for a `lxc-*` host, built straight to a
|
||||
# pct-restorable tarball with no install step, so there's no later copy
|
||||
# step to stage for.
|
||||
#
|
||||
# Without this, config.system.build.tarball's built-in system just
|
||||
# generates a fresh host key at first boot like any other host would --
|
||||
# but sops-nix derives its decryption key from *this* file, and
|
||||
# .sops.yaml only trusts whatever key scripts/sync-host-keys.sh already
|
||||
# registered for this exact target name. A freshly-generated key can
|
||||
# never match that, so every secret (including this host's own login)
|
||||
# permanently fails to decrypt. Confirmed live: sops-install-secrets
|
||||
# errored with "Error getting data key: 0 successful groups required,
|
||||
# got 0" -- the container's actual host key's age fingerprint didn't
|
||||
# match the one registered in .sops.yaml at all.
|
||||
hostKeysDirStr = builtins.getEnv "NIXOS_HOST_KEYS_DIR";
|
||||
hasHostKeysDir = hostKeysDirStr != "" && builtins.pathExists hostKeysDirStr;
|
||||
hostKeysDir = /. + hostKeysDirStr;
|
||||
|
||||
# flakeTarget ("${platform}-${buildType}") comes in via specialArgs from
|
||||
# flake.nix's mkTarget -- exactly the name scripts/sync-host-keys.sh
|
||||
# registers keys under. Deliberately not read back from
|
||||
# config.environment.etc."flake-target" (which is set to the same value)
|
||||
# -- this module also *contributes* to environment.etc below, and a
|
||||
# module reading the merged value of an option it's still defining is a
|
||||
# circular dependency (confirmed: "infinite recursion encountered").
|
||||
privKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key";
|
||||
pubKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key.pub";
|
||||
hasKeyForThisTarget =
|
||||
hasHostKeysDir
|
||||
&& builtins.pathExists privKeyFile
|
||||
&& builtins.pathExists pubKeyFile;
|
||||
in
|
||||
{
|
||||
# LXC containers share the host kernel — Proxmox starts them by exec'ing
|
||||
# /sbin/init directly, no bootloader/initrd involved — and Proxmox has its
|
||||
@@ -35,4 +74,75 @@
|
||||
# for the same reason; it just doesn't disable NetworkManager itself,
|
||||
# which modules/common/configuration.nix enables for every host.
|
||||
networking.networkmanager.enable = lib.mkForce false;
|
||||
|
||||
environment.etc = lib.mkIf hasKeyForThisTarget {
|
||||
"ssh/ssh_host_ed25519_key" = {
|
||||
source = privKeyFile;
|
||||
mode = "0600";
|
||||
};
|
||||
"ssh/ssh_host_ed25519_key.pub" = {
|
||||
source = pubKeyFile;
|
||||
mode = "0644";
|
||||
};
|
||||
};
|
||||
|
||||
# virtualisation/proxmox-lxc.nix (imported above) registers the Nix
|
||||
# store DB via a systemd service (register-nix-paths) -- it never runs
|
||||
# an activation script at all. Confirmed live this means neither
|
||||
# sops-nix's "for users" secrets (password hashes -- installed by the
|
||||
# activation script itself, not a systemd service, since they need to
|
||||
# exist *before* user creation) nor the user-creation step that
|
||||
# consumes them ever run on a real lxc-* boot. Regular secrets
|
||||
# (nix-serve's key, beszel's token, etc.) work anyway because sops-nix
|
||||
# provides its own systemd service for those.
|
||||
#
|
||||
# A systemd service, not boot.postBootCommands: tried that first (it's
|
||||
# a genuine, generally-invoked hook -- nixos/modules/system/boot/stage-2-init.sh,
|
||||
# which becomes this container's actual /sbin/init, unconditionally
|
||||
# runs it) but switch-to-configuration behaves differently that early in
|
||||
# boot (raw stage-2-init.sh, before systemd itself has even started) --
|
||||
# confirmed live it silently failed to rewrite /etc/shadow from there
|
||||
# even in "test" mode, despite the exact same command working reliably
|
||||
# every time when run post-boot (i.e. as a normal systemd service, which
|
||||
# is what this is). Not fully root-caused why the early context
|
||||
# specifically breaks it; a real systemd service sidesteps needing to.
|
||||
#
|
||||
# /etc/shadow already has PLACEHOLDER entries for every declared user
|
||||
# baked in at build time (part of constructing the system closure).
|
||||
# update-users-groups.pl deliberately never overwrites an *existing*
|
||||
# shadow entry -- a correct safety property in general (don't clobber a
|
||||
# real user's real password on a config rebuild) -- but on a genuine
|
||||
# first boot that only means the real hashedPasswordFile-derived hash
|
||||
# never gets the chance to be applied either, since the placeholder is
|
||||
# already "seen". Safe to clear here specifically: there is no real
|
||||
# password yet to protect on a first boot.
|
||||
#
|
||||
# "test" mode, not "boot": confirmed live "boot" mode aborts partway
|
||||
# through (before rewriting /etc/shadow) on a warning that "/boot" is on
|
||||
# a different filesystem -- a real check for a host with a bootloader to
|
||||
# update, meaningless for a container that has none
|
||||
# (boot.loader.{grub,systemd-boot}.enable are both false above), but it
|
||||
# still aborts the script. "test" runs every activation step without
|
||||
# touching boot-loader state at all.
|
||||
#
|
||||
# ConditionPathExists (systemd-native, not a bash-level check) means
|
||||
# this only ever runs once, on the genuine first boot -- systemd itself
|
||||
# skips even starting it on every later boot once the marker exists.
|
||||
# switch-to-configuration is otherwise the operator's call per this
|
||||
# repo's own safety rules, not something to run on every boot.
|
||||
systemd.services.nixos-lxc-first-boot-activate = {
|
||||
description = "Complete first-boot NixOS activation (users, secrets) for this LXC container";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
unitConfig.ConditionPathExists = "!/var/lib/nixos-lxc-first-boot-activated";
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
};
|
||||
script = ''
|
||||
rm -f /etc/shadow
|
||||
/run/current-system/bin/switch-to-configuration test
|
||||
mkdir -p /var/lib
|
||||
touch /var/lib/nixos-lxc-first-boot-activated
|
||||
'';
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1,12 +1,19 @@
|
||||
{ vars, ... }:
|
||||
_:
|
||||
|
||||
{
|
||||
services.tailscale = {
|
||||
enable = true;
|
||||
|
||||
extraUpFlags = [
|
||||
# extraSetFlags (tailscale set, via the always-on tailscaled-set
|
||||
# service), not extraUpFlags -- extraUpFlags is only ever applied by
|
||||
# tailscaled-autoconnect, which itself only runs when
|
||||
# services.tailscale.authKeyFile is set (nothing in this repo sets one,
|
||||
# so tailscale up is a manual, one-time operator step on every host that
|
||||
# uses this service). extraSetFlags has no such gate, so
|
||||
# --advertise-exit-node self-reapplies on every boot once the operator
|
||||
# has authenticated the node once.
|
||||
extraSetFlags = [
|
||||
"--advertise-exit-node"
|
||||
"--advertise-routes=${vars.lanCidr}"
|
||||
];
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1,22 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
export NIX_CONFIG="${NIX_CONFIG:-}
|
||||
experimental-features = nix-command flakes
|
||||
accept-flake-config = false
|
||||
warn-dirty = false
|
||||
"
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=lib/nix-bootstrap.sh
|
||||
source "${script_dir}/lib/nix-bootstrap.sh"
|
||||
# shellcheck source=lib/nix-eval.sh
|
||||
source "${script_dir}/lib/nix-eval.sh"
|
||||
|
||||
MODE="${1:-validate}"
|
||||
|
||||
ensure_nix_profile() {
|
||||
if [ -f /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh ]; then
|
||||
. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
|
||||
elif [ -f "$HOME/.nix-profile/etc/profile.d/nix.sh" ]; then
|
||||
. "$HOME/.nix-profile/etc/profile.d/nix.sh"
|
||||
fi
|
||||
}
|
||||
|
||||
ensure_nix_profile
|
||||
|
||||
if ! command -v nix >/dev/null 2>&1; then
|
||||
@@ -24,8 +16,7 @@ if ! command -v nix >/dev/null 2>&1; then
|
||||
exit 127
|
||||
fi
|
||||
|
||||
hosts_json="$(nix eval --json --no-use-registries --no-accept-flake-config .#nixosConfigurations --apply builtins.attrNames)"
|
||||
hosts="$(echo "$hosts_json" | jq -r '.[]')"
|
||||
hosts="$(list_flake_targets .)"
|
||||
|
||||
echo "Hosts:"
|
||||
echo "$hosts"
|
||||
@@ -44,17 +35,17 @@ fi
|
||||
|
||||
echo
|
||||
echo "Checking Nix formatting with nixpkgs-fmt..."
|
||||
nix run --no-use-registries --no-accept-flake-config github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check .
|
||||
nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#nixpkgs-fmt -- --check .
|
||||
|
||||
echo
|
||||
echo "Running statix lint..."
|
||||
nix run --no-use-registries --no-accept-flake-config github:NixOS/nixpkgs/nixos-25.11#statix -- check .
|
||||
nix run "${NIX_EVAL_FLAGS[@]}" github:NixOS/nixpkgs/nixos-25.11#statix -- check .
|
||||
|
||||
echo
|
||||
echo "Evaluating host toplevel derivations..."
|
||||
for host in $hosts; do
|
||||
echo "==> $host"
|
||||
nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
|
||||
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
|
||||
|
||||
# lxc-* hosts deploy via a directly pct-restore-able tarball instead of
|
||||
# nixos-install (see docs/auto-installer.md); proxmox-* hosts can
|
||||
@@ -64,22 +55,21 @@ for host in $hosts; do
|
||||
case "$host" in
|
||||
lxc-*)
|
||||
echo "==> $host (tarball)"
|
||||
nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.tarball.drvPath"
|
||||
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.tarball.drvPath"
|
||||
;;
|
||||
proxmox-*)
|
||||
echo "==> $host (diskoImagesScript)"
|
||||
nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript.drvPath"
|
||||
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript.drvPath"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
echo
|
||||
echo "Evaluating buildable packages..."
|
||||
packages_json="$(nix eval --json --no-use-registries --no-accept-flake-config .#packages.x86_64-linux --apply builtins.attrNames)"
|
||||
packages="$(echo "$packages_json" | jq -r '.[]')"
|
||||
packages="$(nix eval --json "${NIX_EVAL_FLAGS[@]}" .#packages.x86_64-linux --apply builtins.attrNames | jq -r '.[]')"
|
||||
for pkg in $packages; do
|
||||
echo "==> packages.x86_64-linux.${pkg}"
|
||||
nix eval --raw --no-use-registries --no-accept-flake-config ".#packages.x86_64-linux.${pkg}"
|
||||
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
|
||||
done
|
||||
|
||||
if [[ "$MODE" == "dry-run" ]]; then
|
||||
@@ -87,16 +77,16 @@ if [[ "$MODE" == "dry-run" ]]; then
|
||||
echo "Running dry-run builds for all hosts. This will not create result symlinks."
|
||||
for host in $hosts; do
|
||||
echo "==> Dry-run build: $host"
|
||||
nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.toplevel"
|
||||
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel"
|
||||
|
||||
case "$host" in
|
||||
lxc-*)
|
||||
echo "==> Dry-run build: $host (tarball)"
|
||||
nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.tarball"
|
||||
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.tarball"
|
||||
;;
|
||||
proxmox-*)
|
||||
echo "==> Dry-run build: $host (diskoImagesScript)"
|
||||
nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript"
|
||||
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.diskoImagesScript"
|
||||
;;
|
||||
esac
|
||||
done
|
||||
@@ -105,7 +95,7 @@ if [[ "$MODE" == "dry-run" ]]; then
|
||||
echo "Running dry-run builds for all packages."
|
||||
for pkg in $packages; do
|
||||
echo "==> Dry-run build: packages.x86_64-linux.${pkg}"
|
||||
nix build --dry-run --no-link --no-use-registries --no-accept-flake-config ".#packages.x86_64-linux.${pkg}"
|
||||
nix build --dry-run --no-link "${NIX_EVAL_FLAGS[@]}" ".#packages.x86_64-linux.${pkg}"
|
||||
done
|
||||
fi
|
||||
|
||||
|
||||
+7
-15
@@ -1,19 +1,11 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
export NIX_CONFIG="${NIX_CONFIG:-}
|
||||
experimental-features = nix-command flakes
|
||||
accept-flake-config = false
|
||||
warn-dirty = false
|
||||
"
|
||||
|
||||
ensure_nix_profile() {
|
||||
if [ -f /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh ]; then
|
||||
. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
|
||||
elif [ -f "$HOME/.nix-profile/etc/profile.d/nix.sh" ]; then
|
||||
. "$HOME/.nix-profile/etc/profile.d/nix.sh"
|
||||
fi
|
||||
}
|
||||
script_dir="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
# shellcheck source=lib/nix-bootstrap.sh
|
||||
source "${script_dir}/lib/nix-bootstrap.sh"
|
||||
# shellcheck source=lib/nix-eval.sh
|
||||
source "${script_dir}/lib/nix-eval.sh"
|
||||
|
||||
install_nix_if_missing() {
|
||||
if command -v nix >/dev/null 2>&1; then
|
||||
@@ -79,13 +71,13 @@ if ! command -v jq >/dev/null 2>&1; then
|
||||
fi
|
||||
|
||||
echo "Available NixOS hosts:"
|
||||
hosts="$(nix eval --json --no-use-registries --no-accept-flake-config .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]')"
|
||||
hosts="$(list_flake_targets .)"
|
||||
echo "$hosts"
|
||||
|
||||
echo "Evaluating all host toplevel derivations..."
|
||||
for host in $hosts; do
|
||||
echo "==> Evaluating $host"
|
||||
nix eval --raw --no-use-registries --no-accept-flake-config ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
|
||||
nix eval --raw "${NIX_EVAL_FLAGS[@]}" ".#nixosConfigurations.${host}.config.system.build.toplevel.drvPath"
|
||||
done
|
||||
|
||||
echo "Codex setup complete."
|
||||
|
||||
@@ -10,7 +10,9 @@
|
||||
#
|
||||
# SAFETY:
|
||||
# - The default (create) mode only ever creates a NEW resource -- it
|
||||
# refuses to run if the target VMID already exists on the node.
|
||||
# refuses to run if the target VMID already exists on the node, or if
|
||||
# a VM/CT identified as --host already exists under any other VMID
|
||||
# (checked live against the node; --allow-duplicate-host overrides).
|
||||
# - --modify only ever touches a resource you name explicitly via
|
||||
# --vmid, shows exactly what will change first, and (outside
|
||||
# --dry-run) always requires typing that VMID back to confirm before
|
||||
@@ -24,6 +26,8 @@ set -euo pipefail
|
||||
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
# shellcheck source=env.sh
|
||||
source "${repo_root}/scripts/env.sh"
|
||||
# shellcheck source=lib/nix-eval.sh
|
||||
source "${repo_root}/scripts/lib/nix-eval.sh"
|
||||
|
||||
sync_keys="${repo_root}/scripts/sync-host-keys.sh"
|
||||
|
||||
@@ -40,8 +44,12 @@ Create mode (default):
|
||||
config.networking.hostName (server, docker,
|
||||
nix-cache, nixos, pxe-boot, nix-minimal). Use
|
||||
--list to see what's available for --type.
|
||||
--name <name> Proxmox display name/hostname (default: the flake
|
||||
target name, e.g. lxc-server)
|
||||
--name <name> Proxmox display name/hostname (default: --host's
|
||||
value, e.g. nix-cache -- for lxc this becomes the
|
||||
guest's real networking.hostName too, since
|
||||
proxmoxLXC.manageHostName pulls it from Proxmox's
|
||||
own container config, so it must match host.nix
|
||||
regardless of build type)
|
||||
--vmid <n> Numeric VMID (default: next free, via
|
||||
\`pvesh get /cluster/nextid\` on the node).
|
||||
Refuses to run if this ID already exists.
|
||||
@@ -52,10 +60,11 @@ Create mode (default):
|
||||
--force-rebuild Skip the "does the node already have this
|
||||
image" check -- always build fresh and
|
||||
overwrite what's there.
|
||||
--allow-duplicate-host Required if --host already has a real
|
||||
deployment elsewhere (variables.nix's
|
||||
deployedTargets) -- otherwise refused, since
|
||||
it'd share that host's hostName/hostId.
|
||||
--allow-duplicate-host Required if a VM/CT identified as --host
|
||||
already exists on the node (checked live via
|
||||
qm/pct, not any file in this repo) --
|
||||
otherwise refused, since it'd share that
|
||||
host's hostName/hostId.
|
||||
|
||||
Modify mode (reconfigure an EXISTING resource -- requires --modify):
|
||||
--modify Switch to modify mode.
|
||||
@@ -233,15 +242,21 @@ platform_prefix="lxc"
|
||||
[[ -z "$memory" ]] && memory="$PROXMOX_DEFAULT_MEMORY_MB"
|
||||
|
||||
# --- discover / resolve the flake target from --host --------------------
|
||||
# Emits "<target>\t<hostName>" pairs for every ${platform_prefix}-* flake
|
||||
# target -- the one source both --list and the --host lookup below read
|
||||
# from, so they can never see a different set of targets from each other.
|
||||
targets_for_platform() {
|
||||
local target
|
||||
for target in $(list_flake_targets "$repo_root" 2>/dev/null | grep -- "^${platform_prefix}-"); do
|
||||
printf '%s\t%s\n' "$target" "$(flake_target_hostname "$repo_root" "$target")"
|
||||
done
|
||||
}
|
||||
|
||||
list_hosts() {
|
||||
local target hostname
|
||||
for target in $(nix eval --json --no-use-registries --no-accept-flake-config \
|
||||
"${repo_root}#nixosConfigurations" --apply builtins.attrNames 2>/dev/null \
|
||||
| jq -r --arg p "${platform_prefix}-" '.[] | select(startswith($p))'); do
|
||||
hostname="$(nix eval --raw --no-use-registries --no-accept-flake-config \
|
||||
"${repo_root}#nixosConfigurations.${target}.config.networking.hostName" 2>/dev/null)"
|
||||
while IFS=$'\t' read -r target hostname; do
|
||||
printf ' %-12s -> %s\n' "$hostname" "$target"
|
||||
done
|
||||
done < <(targets_for_platform)
|
||||
}
|
||||
|
||||
if [[ "$do_list" -eq 1 ]]; then
|
||||
@@ -256,16 +271,12 @@ if [[ -z "$host" ]]; then
|
||||
fi
|
||||
|
||||
flake_target=""
|
||||
for target in $(nix eval --json --no-use-registries --no-accept-flake-config \
|
||||
"${repo_root}#nixosConfigurations" --apply builtins.attrNames \
|
||||
| jq -r --arg p "${platform_prefix}-" '.[] | select(startswith($p))'); do
|
||||
hn="$(nix eval --raw --no-use-registries --no-accept-flake-config \
|
||||
"${repo_root}#nixosConfigurations.${target}.config.networking.hostName")"
|
||||
if [[ "$hn" == "$host" ]]; then
|
||||
while IFS=$'\t' read -r target hostname; do
|
||||
if [[ "$hostname" == "$host" ]]; then
|
||||
flake_target="$target"
|
||||
break
|
||||
fi
|
||||
done
|
||||
done < <(targets_for_platform)
|
||||
|
||||
if [[ -z "$flake_target" ]]; then
|
||||
echo "ERROR: no ${platform_prefix}-* target has hostName '${host}'." >&2
|
||||
@@ -274,27 +285,65 @@ if [[ -z "$flake_target" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
[[ -z "$name" ]] && name="$flake_target"
|
||||
# The container/VM's real identity is --host (e.g. "nix-cache"), validated
|
||||
# above against config.networking.hostName -- not the flake target name
|
||||
# (e.g. "lxc-nix-cache"), which is build-type-specific and only exists to
|
||||
# pick which platform variant to build. Defaulting --name to the flake
|
||||
# target would make lxc's --hostname (which proxmoxLXC.manageHostName
|
||||
# feeds straight into the guest's real hostname) disagree with host.nix.
|
||||
[[ -z "$name" ]] && name="$host"
|
||||
|
||||
# --- refuse to duplicate a host that's already really deployed ----------
|
||||
# Checked by hostName, not exact flake target: proxmox-server being
|
||||
# deployed also blocks --type lxc --host server, since both would carry
|
||||
# the same hosts/server/host.nix identity (hostName, hostId).
|
||||
if [[ "$allow_duplicate_host" -eq 0 ]]; then
|
||||
deployed_targets_json="$(nix eval --json --no-use-registries --no-accept-flake-config \
|
||||
--file "${repo_root}/variables.nix" deployedTargets)"
|
||||
for dt in $(echo "$deployed_targets_json" | jq -r '.[]'); do
|
||||
dt_hostname="$(nix eval --raw --no-use-registries --no-accept-flake-config \
|
||||
"${repo_root}#nixosConfigurations.${dt}.config.networking.hostName" 2>/dev/null || true)"
|
||||
if [[ "$dt_hostname" == "$host" ]]; then
|
||||
echo "ERROR: '${host}' already has a real deployment (${dt}, per variables.nix's" >&2
|
||||
echo "deployedTargets). Creating ${flake_target} would share its hostName/hostId --" >&2
|
||||
echo "refusing by default. Pass --allow-duplicate-host if you really mean to spin" >&2
|
||||
echo "up a separate test instance of this host (it'll still get its own distinct" >&2
|
||||
echo "sops key and VMID, never touching ${dt})." >&2
|
||||
# --- refuse to duplicate a host that's already live on the node ---------
|
||||
# Queries the node itself (qm/pct's own name/hostname config), not any
|
||||
# static list in this repo -- a file can't track whether a resource still
|
||||
# actually exists, and this used to be checked against variables.nix's
|
||||
# deployedTargets, which drifted stale (it kept naming a VM as "the real
|
||||
# deployment" well after that VM had been destroyed, blocking its own
|
||||
# redeploy) until that list was dropped in favour of this live check. This
|
||||
# only catches guests identified with the default --name (== --host, what
|
||||
# this script itself always uses unless --name is overridden) -- a guest
|
||||
# manually renamed on the node afterwards wouldn't match, but nothing here
|
||||
# creates guests that way.
|
||||
if [[ "$allow_duplicate_host" -eq 1 ]]; then
|
||||
echo
|
||||
echo "--allow-duplicate-host: skipping the check for an existing '${host}' on ${node}."
|
||||
elif [[ "$dry_run" -eq 1 ]]; then
|
||||
echo
|
||||
echo "[dry-run] would check ${node} for an existing VM/CT identified as '${host}'"
|
||||
else
|
||||
echo
|
||||
echo "==> Checking ${node} for an existing VM/CT identified as '${host}'..."
|
||||
ssh_check_status=0
|
||||
existing="$(ssh "$ssh_target" bash -s -- "$host" <<'REMOTE_SCRIPT'
|
||||
target="$1"
|
||||
for id in $(qm list 2>/dev/null | awk 'NR>1{print $1}'); do
|
||||
n="$(qm config "$id" 2>/dev/null | grep -oP '^name:\s*\K\S+' || true)"
|
||||
[[ "$n" == "$target" ]] && echo "vm ${id} ${n}"
|
||||
done
|
||||
for id in $(pct list 2>/dev/null | awk 'NR>1{print $1}'); do
|
||||
n="$(pct config "$id" 2>/dev/null | grep -oP '^hostname:\s*\K\S+' || true)"
|
||||
[[ "$n" == "$target" ]] && echo "lxc ${id} ${n}"
|
||||
done
|
||||
REMOTE_SCRIPT
|
||||
)" || ssh_check_status=$?
|
||||
if [[ "$ssh_check_status" -ne 0 ]]; then
|
||||
echo "ERROR: couldn't reach ${node} (ssh exited ${ssh_check_status}) to check for an" >&2
|
||||
echo "existing '${host}' resource -- refusing to guess. Fix connectivity and retry," >&2
|
||||
echo "or pass --allow-duplicate-host if you're sure none exists (this skips the" >&2
|
||||
echo "check entirely)." >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -n "$existing" ]]; then
|
||||
echo "ERROR: '${host}' already exists on ${node}:" >&2
|
||||
echo "$existing" | while read -r kind id n; do
|
||||
echo " - ${kind} VMID ${id} (${n})" >&2
|
||||
done
|
||||
echo "Refusing to create a second resource sharing this identity. Pass" >&2
|
||||
echo "--allow-duplicate-host to create one anyway (it gets its own distinct" >&2
|
||||
echo "sops key and VMID -- the existing resource(s) above are left untouched)," >&2
|
||||
echo "or use --modify to reconfigure the existing one instead." >&2
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '${name}'"
|
||||
@@ -383,9 +432,19 @@ else
|
||||
fi
|
||||
|
||||
if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
|
||||
# Mirrors the real build commands' "${NIX_OPTS[@]}" below -- nix_extra_opts
|
||||
# (called earlier, once) has already decided whether nix-cache is in play,
|
||||
# and the dry-run preview needs to reflect that decision instead of always
|
||||
# printing the same command regardless of outcome.
|
||||
nix_opts_display=""
|
||||
if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
|
||||
printf -v nix_opts_display '%q ' "${NIX_OPTS[@]}"
|
||||
nix_opts_display=" ${nix_opts_display% }"
|
||||
fi
|
||||
if [[ "$type" == "lxc" ]]; then
|
||||
if [[ "$dry_run" -eq 1 ]]; then
|
||||
echo "[dry-run] would build: NIXOS_HOST_KEYS_DIR=${repo_root}/host-keys nix build --impure \\"
|
||||
echo "[dry-run] --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
|
||||
echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.tarball"
|
||||
local_image="<built-tarball>"
|
||||
else
|
||||
@@ -399,7 +458,8 @@ if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
|
||||
fi
|
||||
else
|
||||
if [[ "$dry_run" -eq 1 ]]; then
|
||||
echo "[dry-run] would build: nix build .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
|
||||
echo "[dry-run] would build: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
|
||||
echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
|
||||
echo "[dry-run] would run: sudo ./result-${flake_target} \\"
|
||||
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\"
|
||||
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\"
|
||||
|
||||
+49
-2
@@ -88,13 +88,60 @@ nix_extra_opts() {
|
||||
fi
|
||||
export NIX_EXTRA_OPTS_DECIDED=1
|
||||
NIX_OPTS=()
|
||||
if ! curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
|
||||
|
||||
# Retry a couple of times, 1s apart, before believing either check --
|
||||
# belt-and-suspenders against a genuine multi-second blip (nix-cache
|
||||
# restarting), on top of the fix below. Worst case (~11s total, host
|
||||
# genuinely gone) is still nowhere near the 15s+ *per lookup* nix's own
|
||||
# substituter retries would cost if this check didn't exist at all.
|
||||
local attempt cache_up=0 builder_up=0
|
||||
for attempt in 1 2 3; do
|
||||
if curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
|
||||
cache_up=1
|
||||
break
|
||||
fi
|
||||
[[ "$attempt" -lt 3 ]] && sleep 1
|
||||
done
|
||||
|
||||
if [[ "$cache_up" -eq 0 ]]; then
|
||||
echo "nix-cache (http://${NIX_CACHE_HOST}) is unreachable -- skipping it (substituter + remote builder) for the rest of this run." >&2
|
||||
NIX_OPTS=(--option substituters "https://cache.nixos.org/" --builders "")
|
||||
elif ! timeout 3 bash -c "cat < /dev/tcp/${NIX_CACHE_HOST}/22" >/dev/null 2>&1; then
|
||||
else
|
||||
for attempt in 1 2 3; do
|
||||
# `exec 3<>/dev/tcp/...` just opens the fd and returns -- it does NOT
|
||||
# read from it. Confirmed live this is load-bearing, not stylistic:
|
||||
# the previous `cat < /dev/tcp/.../22` blocked forever and always hit
|
||||
# the timeout even against a perfectly healthy nix-cache, because
|
||||
# sshd sends its banner and then holds the connection open waiting
|
||||
# for the client to speak next -- `cat` never sees EOF, so this
|
||||
# check reported "unreachable" unconditionally, 100% of the time,
|
||||
# regardless of whether the remote builder was actually up.
|
||||
if timeout 3 bash -c "exec 3<>/dev/tcp/${NIX_CACHE_HOST}/22" 2>/dev/null; then
|
||||
builder_up=1
|
||||
break
|
||||
fi
|
||||
[[ "$attempt" -lt 3 ]] && sleep 1
|
||||
done
|
||||
if [[ "$builder_up" -eq 0 ]]; then
|
||||
echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
|
||||
NIX_OPTS=(--builders "")
|
||||
fi
|
||||
fi
|
||||
|
||||
# `printf '%q '` with a genuinely empty NIX_OPTS still runs one format
|
||||
# pass over a missing argument and yields the literal `'' ` rather than
|
||||
# an empty string (confirmed live) -- a subprocess that later does
|
||||
# `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` (the branch above, for e.g.
|
||||
# sync-host-keys.sh reusing this process's decision) would then rebuild
|
||||
# a 1-element array holding an empty string instead of a 0-element
|
||||
# array, and `nix-shell "${NIX_OPTS[@]}" -p <pkg>` chokes on that stray
|
||||
# element as a bogus positional argument. Guard the empty case
|
||||
# explicitly so nix-cache being reachable (NIX_OPTS legitimately empty)
|
||||
# round-trips as truly empty instead.
|
||||
if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
|
||||
printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
|
||||
else
|
||||
NIX_EXTRA_OPTS=""
|
||||
fi
|
||||
export NIX_EXTRA_OPTS
|
||||
}
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared Nix bootstrap for scripts/codex-setup.sh and
|
||||
# scripts/codex-maintenance.sh: the nix.conf settings both need in effect
|
||||
# before a single `nix` command runs (flakes enabled, never honor a flake
|
||||
# input's own nixConfig, no "dirty tree" warning spam), plus a helper to
|
||||
# pull an already-installed Nix's daemon/profile script onto PATH if it
|
||||
# isn't there yet. Source this instead of copying it -- see CLAUDE.md.
|
||||
export NIX_CONFIG="${NIX_CONFIG:-}
|
||||
experimental-features = nix-command flakes
|
||||
accept-flake-config = false
|
||||
warn-dirty = false
|
||||
"
|
||||
|
||||
ensure_nix_profile() {
|
||||
if [ -f /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh ]; then
|
||||
. /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
|
||||
elif [ -f "$HOME/.nix-profile/etc/profile.d/nix.sh" ]; then
|
||||
. "$HOME/.nix-profile/etc/profile.d/nix.sh"
|
||||
fi
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared flake-introspection helpers for scripts/*.sh. Source alongside
|
||||
# env.sh:
|
||||
# source "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)/lib/nix-eval.sh"
|
||||
#
|
||||
# NIX_EVAL_FLAGS: --no-use-registries so a call here never resolves through
|
||||
# the user's global flake registry (every call targets this repo's own
|
||||
# flake, or an explicit github: ref, not a registry alias); --no-accept-flake-config
|
||||
# so a flake input's own nixConfig (e.g. a dependency's substituters) is
|
||||
# never honored -- matches accept-flake-config = false already set repo-wide
|
||||
# (see lib/nix-bootstrap.sh / CLAUDE.md). Reuse this array rather than
|
||||
# retyping the two flags at each call site.
|
||||
declare -a NIX_EVAL_FLAGS=(--no-use-registries --no-accept-flake-config)
|
||||
|
||||
# list_flake_targets <flake_ref>
|
||||
# Prints the attribute names under <flake_ref>#nixosConfigurations, one per
|
||||
# line, e.g.:
|
||||
# list_flake_targets . # from inside the repo
|
||||
# list_flake_targets "$repo_root" # from anywhere
|
||||
list_flake_targets() {
|
||||
local flake_ref="$1"
|
||||
nix eval --json "${NIX_EVAL_FLAGS[@]}" \
|
||||
"${flake_ref}#nixosConfigurations" --apply builtins.attrNames \
|
||||
| jq -r '.[]'
|
||||
}
|
||||
|
||||
# flake_target_hostname <flake_ref> <target>
|
||||
# Prints one nixosConfigurations target's config.networking.hostName.
|
||||
# Empty (not an error under set -e) if the target doesn't exist or the
|
||||
# eval otherwise fails -- callers that need to distinguish "empty" from
|
||||
# "eval failed" should check $? themselves instead of relying on this.
|
||||
flake_target_hostname() {
|
||||
local flake_ref="$1" target="$2"
|
||||
nix eval --raw "${NIX_EVAL_FLAGS[@]}" \
|
||||
"${flake_ref}#nixosConfigurations.${target}.config.networking.hostName" 2>/dev/null
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared SSH-host-key / age-conversion helpers for scripts/sync-host-keys.sh
|
||||
# and scripts/prepare-host-key.sh -- both generate the same kind of key
|
||||
# (ed25519, no passphrase, the sops-nix age-derivation input) and convert it
|
||||
# to an age recipient the same way; kept in one place so the two can't
|
||||
# drift apart.
|
||||
#
|
||||
# Uses NIX_OPTS (an array of extra `nix-shell` options -- see env.sh's
|
||||
# nix_extra_opts) if the caller has already set it, so a decision to avoid
|
||||
# an unreachable nix-cache is reused here instead of probed again. Falls
|
||||
# back to no extra options if the caller never sourced env.sh.
|
||||
if ! declare -p NIX_OPTS >/dev/null 2>&1; then
|
||||
declare -a NIX_OPTS=()
|
||||
fi
|
||||
|
||||
# generate_host_ed25519_key <hostname> <keyfile>
|
||||
# Writes <keyfile> and <keyfile>.pub. Caller is responsible for refusing to
|
||||
# overwrite an existing keyfile -- this always runs ssh-keygen fresh.
|
||||
generate_host_ed25519_key() {
|
||||
local hostname="$1" keyfile="$2"
|
||||
nix-shell "${NIX_OPTS[@]}" -p openssh --run \
|
||||
"ssh-keygen -t ed25519 -N '' -C '${hostname}' -f '${keyfile}'" >/dev/null
|
||||
}
|
||||
|
||||
# ssh_pubkey_to_age <pubkeyfile>
|
||||
# Prints the age public key derived from an ed25519 SSH public key file.
|
||||
ssh_pubkey_to_age() {
|
||||
local pubkeyfile="$1"
|
||||
nix-shell "${NIX_OPTS[@]}" -p ssh-to-age --run "ssh-to-age -i '${pubkeyfile}'"
|
||||
}
|
||||
@@ -23,6 +23,10 @@
|
||||
set -euo pipefail
|
||||
|
||||
repo_root="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
# shellcheck source=env.sh
|
||||
source "${repo_root}/scripts/env.sh"
|
||||
# shellcheck source=lib/ssh-host-keys.sh
|
||||
source "${repo_root}/scripts/lib/ssh-host-keys.sh"
|
||||
|
||||
hostname="${1:?usage: scripts/prepare-host-key.sh <hostname>}"
|
||||
sops_yaml="${repo_root}/.sops.yaml"
|
||||
@@ -41,9 +45,10 @@ if [[ -f "$keyfile" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
nix-shell -p openssh --run "ssh-keygen -t ed25519 -N '' -C '${hostname}' -f '${keyfile}'" >/dev/null
|
||||
nix_extra_opts
|
||||
generate_host_ed25519_key "$hostname" "$keyfile"
|
||||
|
||||
age_pub="$(nix-shell -p ssh-to-age --run "ssh-to-age -i '${keyfile}.pub'")"
|
||||
age_pub="$(ssh_pubkey_to_age "${keyfile}.pub")"
|
||||
|
||||
cat <<EOF
|
||||
|
||||
|
||||
@@ -31,6 +31,10 @@ editor="${repo_root}/scripts/lib/sync-host-keys-edit-sops.py"
|
||||
|
||||
# shellcheck source=env.sh
|
||||
source "${repo_root}/scripts/env.sh"
|
||||
# shellcheck source=lib/nix-eval.sh
|
||||
source "${repo_root}/scripts/lib/nix-eval.sh"
|
||||
# shellcheck source=lib/ssh-host-keys.sh
|
||||
source "${repo_root}/scripts/lib/ssh-host-keys.sh"
|
||||
|
||||
mkdir -p "$keydir"
|
||||
|
||||
@@ -118,12 +122,10 @@ EOF
|
||||
}
|
||||
|
||||
discover_targets() {
|
||||
nix eval --json --no-use-registries --no-accept-flake-config \
|
||||
"${repo_root}#nixosConfigurations" --apply builtins.attrNames \
|
||||
| jq -r '.[] | select(. != "installer")'
|
||||
# installer is the one nixosConfigurations target that doesn't import
|
||||
# sops-nix at all (see CLAUDE.md's "Security Notes" -- hardcoded login
|
||||
# password instead) -- config.sops.secrets doesn't exist for it.
|
||||
list_flake_targets "$repo_root" | grep -v '^installer$'
|
||||
}
|
||||
|
||||
locally_managed_hosts() {
|
||||
@@ -159,7 +161,7 @@ queue_host_sync() {
|
||||
echo "[dry-run] ${host}: would generate host key"
|
||||
else
|
||||
echo "==> ${host}: generating host key"
|
||||
nix-shell "${NIX_OPTS[@]}" -p openssh --run "ssh-keygen -t ed25519 -N '' -C '${host}' -f '${keyfile}'" >/dev/null
|
||||
generate_host_ed25519_key "$host" "$keyfile"
|
||||
fi
|
||||
else
|
||||
echo "==> ${host}: host key already present"
|
||||
@@ -170,7 +172,7 @@ queue_host_sync() {
|
||||
if [[ "$dry_run" -eq 1 ]]; then
|
||||
age_pub="dry-run-placeholder-not-a-real-key"
|
||||
else
|
||||
age_pub="$(nix-shell "${NIX_OPTS[@]}" -p ssh-to-age --run "ssh-to-age -i '${keyfile}.pub'")"
|
||||
age_pub="$(ssh_pubkey_to_age "${keyfile}.pub")"
|
||||
fi
|
||||
add_keys_json="$(jq --arg host "$host" --arg key "$age_pub" \
|
||||
'. + [{host: $host, age_key: $key}]' <<<"$add_keys_json")"
|
||||
|
||||
+9
-16
@@ -19,6 +19,15 @@
|
||||
|
||||
remoteBuilderUser = "nixremote"; # remote builder SSH user
|
||||
|
||||
# nix-cache's own SSH host public key (not a secret — the private half
|
||||
# never leaves the host). Wired into every client's
|
||||
# programs.ssh.knownHosts by modules/nix-cache/remote-builder-client.nix
|
||||
# so distributed builds don't hit "Host key verification failed" on a
|
||||
# fresh client that has never manually ssh'd to nix-cache before. Update
|
||||
# this if nix-cache's host key is ever rotated or the host is rebuilt
|
||||
# from scratch.
|
||||
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
|
||||
|
||||
# Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
|
||||
# host (modules/nix-cache/server.nix) — one per client host that's allowed
|
||||
# to use it as a distributed builder.
|
||||
@@ -147,20 +156,4 @@
|
||||
keep = 20; # number of rotated logs to retain before deleting the oldest
|
||||
};
|
||||
|
||||
# Flake targets with a real, currently-running deployment somewhere —
|
||||
# matches README.md's Hosts table "(real, deployed)" annotations; update
|
||||
# both together. Not consumed by any NixOS module (nothing in the actual
|
||||
# system config should behave differently because of this) — it's read
|
||||
# by scripts/create-proxmox-resource.sh to refuse creating a same-identity
|
||||
# duplicate of an already-deployed host (shared hostName/hostId) unless
|
||||
# you explicitly pass --allow-duplicate-host.
|
||||
deployedTargets = [
|
||||
"linode-minimal"
|
||||
"proxmox-minimal"
|
||||
"proxmox-nix-cache"
|
||||
"proxmox-server"
|
||||
"proxmox-docker"
|
||||
"proxmox-gui"
|
||||
"proxmox-pxe-boot"
|
||||
];
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user