Archived
Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
  beatzaplentyandClaude Sonnet 5
|
c939454983 |
@@ -93,10 +93,9 @@ Beyond `codex-setup.sh`/`codex-maintenance.sh` above, `scripts/` also has:
|
||||
(`--force-rebuild` to skip that and always rebuild), and probes
|
||||
nix-cache's substituter/remote-builder reachability once up front rather
|
||||
than letting every `nix build` call retry against it individually.
|
||||
Refuses to create a target whose host identity already exists live on
|
||||
the node (checked directly via `qm`/`pct`, not any file in this repo)
|
||||
unless `--allow-duplicate-host` is passed. `--dry-run` throughout both
|
||||
modes.
|
||||
Refuses to create a target whose host identity already has a real
|
||||
deployment elsewhere (`variables.nix`'s `deployedTargets`) unless
|
||||
`--allow-duplicate-host` is passed. `--dry-run` throughout both modes.
|
||||
- `scripts/env.sh` — shared config (`PROXMOX_HOST`, storage pool, bridge,
|
||||
default cores/memory) sourced by `create-proxmox-resource.sh`. Add new
|
||||
cross-script config here instead of duplicating it per-script.
|
||||
@@ -134,10 +133,9 @@ nixosSystem {
|
||||
```
|
||||
|
||||
Platforms: `linode`, `proxmox`, `lxc`. Build types: `minimal`, `nix-cache`,
|
||||
`server`, `docker`, `gui`, `pxe-boot`, `tailscale-exit-node`. Not every
|
||||
combination is built — e.g. `pxe-boot` has no `linode` variant (PXE/DHCP/TFTP
|
||||
need LAN L2 adjacency a Linode VPS doesn't have). Treat `flake.nix`'s
|
||||
`generatedTargets` as the source
|
||||
`server`, `docker`, `gui`, `pxe-boot`. Not every combination is built — e.g.
|
||||
`pxe-boot` has no `linode` variant (PXE/DHCP/TFTP need LAN L2 adjacency a
|
||||
Linode VPS doesn't have). Treat `flake.nix`'s `generatedTargets` as the source
|
||||
of truth for which hosts exist — `README.md`, `AGENTS.md`,
|
||||
`docs/flake-lock-automation.md`, and the CI eval workflows
|
||||
(`.github/workflows/check-nixos.yml`, `.gitea/workflows/check-nixos.yml`) list
|
||||
|
||||
@@ -10,7 +10,7 @@ pieces composed in `flake.nix`:
|
||||
|
||||
- **Platforms** (what it runs on): `linode`, `proxmox`, `lxc`
|
||||
- **Build types** (what it's for): `minimal`, `nix-cache`, `server`, `docker`,
|
||||
`gui`, `pxe-boot`, `tailscale-exit-node`
|
||||
`gui`, `pxe-boot`
|
||||
|
||||
Not every combination exists — `pxe-boot` has no `linode` variant, since
|
||||
PXE/DHCP/TFTP need LAN L2 adjacency that a Linode VPS doesn't have. The full
|
||||
@@ -18,23 +18,19 @@ list:
|
||||
|
||||
| Target | Purpose |
|
||||
| --- | --- |
|
||||
| `linode-minimal` | Minimal NixOS host profile on a Linode VPS |
|
||||
| `proxmox-minimal` | Minimal NixOS host profile on Proxmox — previously the flat `nix-minimal` target |
|
||||
| `linode-minimal` | Minimal NixOS host profile on a Linode VPS (real, deployed) |
|
||||
| `proxmox-minimal` | Minimal NixOS host profile on Proxmox (real, deployed — previously the flat `nix-minimal` target) |
|
||||
| `lxc-minimal` | Minimal NixOS host profile in a Proxmox LXC container |
|
||||
| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder — previously the flat `nix-cache` target |
|
||||
| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host — previously the flat `server` target |
|
||||
| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack — previously the flat `docker` target |
|
||||
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation — previously the flat `nixos` target |
|
||||
| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host — previously the flat `pxe-boot` target |
|
||||
| `linode-tailscale-exit-node` / `proxmox-tailscale-exit-node` / `lxc-tailscale-exit-node` | Tailscale exit node |
|
||||
| `linode-nix-cache` / `proxmox-nix-cache` / `lxc-nix-cache` | Local Nix binary cache and remote builder (`proxmox-nix-cache` is the real, deployed one — previously the flat `nix-cache` target) |
|
||||
| `linode-server` / `proxmox-server` / `lxc-server` | Storage, NFS, backup, and monitoring exporter host (`proxmox-server` is the real, deployed one — previously the flat `server` target) |
|
||||
| `linode-docker` / `proxmox-docker` / `lxc-docker` | Docker host for the main container stack (`proxmox-docker` is the real, deployed one — previously the flat `docker` target) |
|
||||
| `linode-gui` / `proxmox-gui` / `lxc-gui` | Cinnamon desktop workstation (`proxmox-gui` is the real, deployed one — previously the flat `nixos` target) |
|
||||
| `proxmox-pxe-boot` / `lxc-pxe-boot` | HTTP/iPXE boot asset host (`proxmox-pxe-boot` is the real, deployed one — previously the flat `pxe-boot` target) |
|
||||
|
||||
Which variant of a given buildtype is actually deployed isn't tracked
|
||||
anywhere in this repo — that's live infrastructure state, not something a
|
||||
committed file can keep accurate, and it changes independently of the code.
|
||||
Check the Proxmox node itself, or `/etc/flake-target` on a running host (see
|
||||
below), if you need to know what's really out there right now.
|
||||
`scripts/create-proxmox-resource.sh`'s duplicate-host guard works the same
|
||||
way: it checks the Proxmox node directly rather than any file here.
|
||||
The "(real, deployed)" targets above are also tracked machine-readably in
|
||||
`variables.nix`'s `deployedTargets` — keep both in sync when a deployment
|
||||
changes. `scripts/create-proxmox-resource.sh` reads that list to refuse
|
||||
creating a same-identity duplicate of an already-deployed host by accident.
|
||||
|
||||
Each buildtype's `hosts/<name>/host.nix` carries the per-machine identity
|
||||
(hostname, hostId, per-machine secrets, `system.stateVersion`) that must stay
|
||||
|
||||
+5
-27
@@ -77,33 +77,11 @@ system profile) — there's no separate activation step to run yourself.
|
||||
of this (build, host-key handling, upload, `pct create` with the flags
|
||||
above) — see its `--help`.
|
||||
|
||||
Host keys still need pre-seeding the same way as any other host — the
|
||||
sops-nix activation-vs-first-boot race is identical regardless of how the
|
||||
image reaches the machine. Unlike the ISO/PXE installer (where
|
||||
`modules/installer/host-keys.nix` bakes *every* `host-keys/` entry into
|
||||
`/etc/host-keys/` for `auto-install.sh` to pick from and copy at install
|
||||
time — see "Host keys" below), an `lxc-*` tarball has no install step to
|
||||
copy anything during, so `modules/platforms/lxc.nix` bakes this *one*
|
||||
target's key straight into `/etc/ssh/ssh_host_ed25519_key(.pub)` directly,
|
||||
keyed by its own exact flake target name (`config.environment.etc` can't
|
||||
be read back from within a module still contributing to it, so this comes
|
||||
in via `specialArgs.flakeTarget`, set by `flake.nix`'s `mkTarget`):
|
||||
|
||||
```sh
|
||||
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" \
|
||||
nix build .#nixosConfigurations.lxc-nix-cache.config.system.build.tarball --impure
|
||||
```
|
||||
|
||||
Confirmed the hard way: without this, the tarball's own built-in system
|
||||
just generates a fresh host key at first boot like any host would, which
|
||||
can never match whatever `.sops.yaml` actually trusts for that target —
|
||||
`sops-install-secrets` fails with `Error getting data key: 0 successful
|
||||
groups required, got 0`, and *every* secret (including this host's own
|
||||
login) permanently fails to decrypt, silently — no error in the boot log
|
||||
at all, since the activation step that would install secrets only runs on
|
||||
a from-scratch first activation and skips silently once `/run/current-system`
|
||||
already exists. `scripts/create-proxmox-resource.sh` always builds with
|
||||
`NIXOS_HOST_KEYS_DIR` set for this reason.
|
||||
Host keys still need pre-seeding the same way as any other host (see "Host
|
||||
keys" below) — the sops-nix activation-vs-first-boot race is identical
|
||||
regardless of how the image reaches the machine. `NIXOS_HOST_KEYS_DIR=...
|
||||
nix build ... --impure` bakes the matching key into the tarball the same way
|
||||
it does for the ISO/PXE installer images.
|
||||
|
||||
## Layout
|
||||
|
||||
|
||||
@@ -59,15 +59,6 @@ On `nix-cache`, install the matching public key used by `nixremote` authorized k
|
||||
The committed `nixremote` authorized keys are public SSH keys only. Keep the
|
||||
matching private keys on client hosts and out of the repository.
|
||||
|
||||
nix-cache's own SSH *host* key is trusted declaratively via
|
||||
`programs.ssh.knownHosts` in `modules/nix-cache/remote-builder-client.nix`,
|
||||
sourced from `vars.nixCacheHostKey` (`variables.nix`) — every client rebuild
|
||||
picks it up automatically, so distributed builds don't fail with "Host key
|
||||
verification failed" on a client that has never manually SSH'd to nix-cache
|
||||
before. If nix-cache's host key is ever rotated or the host rebuilt from
|
||||
scratch, update `vars.nixCacheHostKey` to match its new
|
||||
`/etc/ssh/ssh_host_ed25519_key.pub`.
|
||||
|
||||
## Manual verification
|
||||
|
||||
After deployment:
|
||||
|
||||
@@ -33,9 +33,6 @@
|
||||
# nix-cache itself consumes the nix-cache substituter and remote
|
||||
# builder.
|
||||
mkTarget = { platform, buildType, hostPath, homeFile ? ./modules/common/home.nix }:
|
||||
let
|
||||
flakeTarget = "${platform}-${buildType}";
|
||||
in
|
||||
nixpkgs.lib.nixosSystem {
|
||||
inherit system;
|
||||
modules = [
|
||||
@@ -45,7 +42,7 @@
|
||||
./modules/platforms/${platform}.nix
|
||||
./modules/build-types/${buildType}.nix
|
||||
hostPath
|
||||
{ environment.etc."flake-target".text = flakeTarget; }
|
||||
{ environment.etc."flake-target".text = "${platform}-${buildType}"; }
|
||||
home-manager.nixosModules.home-manager
|
||||
{
|
||||
home-manager = {
|
||||
@@ -59,13 +56,7 @@
|
||||
./modules/nix-cache/client.nix
|
||||
./modules/nix-cache/remote-builder-client.nix
|
||||
];
|
||||
# flakeTarget is passed via specialArgs (not read back from
|
||||
# config.environment.etc."flake-target" above) specifically so
|
||||
# modules/platforms/lxc.nix can use it to select its own host key
|
||||
# file without a same-option circular dependency (a module
|
||||
# contributing to environment.etc can't read the merged
|
||||
# environment.etc it's itself contributing to).
|
||||
specialArgs = { inherit inputs vars netbootSystem flakeTarget; };
|
||||
specialArgs = { inherit inputs vars netbootSystem; };
|
||||
};
|
||||
|
||||
# Generated platform x build-type matrix. pxe-boot has no linode
|
||||
@@ -94,10 +85,6 @@
|
||||
|
||||
proxmox-pxe-boot = mkTarget { platform = "proxmox"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
|
||||
lxc-pxe-boot = mkTarget { platform = "lxc"; buildType = "pxe-boot"; hostPath = ./hosts/pxe-boot/host.nix; };
|
||||
|
||||
linode-tailscale-exit-node = mkTarget { platform = "linode"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
|
||||
proxmox-tailscale-exit-node = mkTarget { platform = "proxmox"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
|
||||
lxc-tailscale-exit-node = mkTarget { platform = "lxc"; buildType = "tailscale-exit-node"; hostPath = ./hosts/tailscale-exit-node/host.nix; };
|
||||
};
|
||||
|
||||
# Auto-install environments (migrated from the former nix-auto-installer
|
||||
|
||||
@@ -1,12 +0,0 @@
|
||||
_:
|
||||
|
||||
{
|
||||
networking.hostName = "exit-node";
|
||||
|
||||
# No networking.hostId: only ZFS-touching hosts (server, docker) need one
|
||||
# for pool-import safety, and this host does neither.
|
||||
|
||||
# A genuinely new host (not a pre-refactor carry-over), so it tracks the
|
||||
# flake's current nixpkgs release rather than being pinned to an older one.
|
||||
system.stateVersion = "26.05";
|
||||
}
|
||||
@@ -6,13 +6,4 @@
|
||||
#DOCKER_HOST = "tcp://docker-socket-proxy:2375";
|
||||
HUB_URL = "http://${vars.dockerHost}.${vars.homeDomain}:${toString vars.ports.beszelHub}";
|
||||
};
|
||||
|
||||
# The upstream module runs beszel-agent under DynamicUser with
|
||||
# ProtectSystem = "strict" and no StateDirectory, so /var/lib/beszel-agent
|
||||
# (where the agent persists its hub-pairing fingerprint, per
|
||||
# https://github.com/henrygd/beszel/discussions/1542) isn't writable --
|
||||
# every restart silently fails to save it and regenerates a fresh one in
|
||||
# memory, permanently desyncing from whatever the hub has on record after
|
||||
# the very first successful pairing. Give it real persistent storage.
|
||||
systemd.services.beszel-agent.serviceConfig.StateDirectory = "beszel-agent";
|
||||
}
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
{ ... }:
|
||||
|
||||
{
|
||||
imports = [
|
||||
../tailscale/exit-node.nix
|
||||
];
|
||||
|
||||
# "server", not "both": this build type only ever advertises itself as an
|
||||
# exit node (see ../tailscale/exit-node.nix) -- it doesn't advertise LAN
|
||||
# subnet routes, so it doesn't need the "client"-side loose reverse-path
|
||||
# filtering that "both" would also turn on. Deliberately left unbundled
|
||||
# from LAN-subnet-route advertisement so this build type stays valid on
|
||||
# every platform, including linode (a remote VPS with no network path to
|
||||
# the home LAN at all).
|
||||
services.tailscale.useRoutingFeatures = "server";
|
||||
|
||||
# Forwarded exit-node traffic arrives on tailscale0 already
|
||||
# tailscale-authenticated -- the firewall's normal per-port allow-list
|
||||
# would otherwise drop it. Standard NixOS/Tailscale exit-node guidance.
|
||||
networking.firewall.trustedInterfaces = [ "tailscale0" ];
|
||||
}
|
||||
@@ -17,26 +17,6 @@ let
|
||||
--refresh \
|
||||
--flake git+https://${vars.lanDomain}/beatzaplenty/nixos.git#$(cat /etc/flake-target)
|
||||
'';
|
||||
|
||||
# lxc-* hosts pre-seed their SSH host key at build time (see
|
||||
# modules/platforms/lxc.nix) so sops-nix's .sops.yaml recipient matches on
|
||||
# first boot -- without it, secrets permanently fail to decrypt (see that
|
||||
# file's comment for the confirmed failure). That requires --impure plus
|
||||
# NIXOS_HOST_KEYS_DIR pointing at the repo's host-keys/ dir, same pattern
|
||||
# docs/auto-installer.md uses for the installer ISO. A function, not a
|
||||
# shellAlias, since the target name has to interpolate into the middle of
|
||||
# the flake attribute path, not just append after it. Must be run from the
|
||||
# repo root, same as every other host-keys/ command in this repo.
|
||||
buildImageFn = ''
|
||||
buildImage() {
|
||||
if [ -z "$1" ]; then
|
||||
echo "usage: buildImage <flake-target> (e.g. lxc-docker)" >&2
|
||||
return 1
|
||||
fi
|
||||
NIXOS_HOST_KEYS_DIR="$(pwd)/host-keys" nix build --impure \
|
||||
".#nixosConfigurations.$1.config.system.build.tarball"
|
||||
}
|
||||
'';
|
||||
in
|
||||
{
|
||||
programs.bash = {
|
||||
@@ -45,6 +25,5 @@ in
|
||||
"Switch-nix" = mySwitchCmd;
|
||||
"Test-nix" = myTestCmd;
|
||||
};
|
||||
initExtra = buildImageFn;
|
||||
};
|
||||
}
|
||||
|
||||
@@ -5,14 +5,6 @@
|
||||
# sudo install -d -m 0700 /root/.ssh
|
||||
# sudo install -m 0600 ./nixremote /root/.ssh/nixremote
|
||||
# sudo ssh -i /root/.ssh/nixremote nixremote@nix-cache nix-store --version
|
||||
# Trust nix-cache's SSH host key declaratively so the nix-daemon (root)
|
||||
# can connect the first time without a manual ssh-keyscan/known_hosts
|
||||
# step on every new client.
|
||||
programs.ssh.knownHosts.${vars.nixCacheHost} = {
|
||||
hostNames = [ vars.nixCacheHost ];
|
||||
publicKey = vars.nixCacheHostKey;
|
||||
};
|
||||
|
||||
nix = {
|
||||
distributedBuilds = true;
|
||||
|
||||
|
||||
+1
-111
@@ -1,44 +1,5 @@
|
||||
{ lib, modulesPath, flakeTarget, ... }:
|
||||
{ lib, modulesPath, ... }:
|
||||
|
||||
let
|
||||
# Bakes this exact flake target's pre-generated SSH host key straight
|
||||
# into /etc/ssh/ -- mirrors modules/installer/host-keys.nix's
|
||||
# builtins.getEnv pattern (impure and empty under normal `nix
|
||||
# build`/`nix eval`, so this is a no-op unless explicitly opted into
|
||||
# with NIXOS_HOST_KEYS_DIR=... --impure), but places the key directly
|
||||
# rather than staging it under /etc/host-keys/ for a later manual copy
|
||||
# -- this is the whole system for a `lxc-*` host, built straight to a
|
||||
# pct-restorable tarball with no install step, so there's no later copy
|
||||
# step to stage for.
|
||||
#
|
||||
# Without this, config.system.build.tarball's built-in system just
|
||||
# generates a fresh host key at first boot like any other host would --
|
||||
# but sops-nix derives its decryption key from *this* file, and
|
||||
# .sops.yaml only trusts whatever key scripts/sync-host-keys.sh already
|
||||
# registered for this exact target name. A freshly-generated key can
|
||||
# never match that, so every secret (including this host's own login)
|
||||
# permanently fails to decrypt. Confirmed live: sops-install-secrets
|
||||
# errored with "Error getting data key: 0 successful groups required,
|
||||
# got 0" -- the container's actual host key's age fingerprint didn't
|
||||
# match the one registered in .sops.yaml at all.
|
||||
hostKeysDirStr = builtins.getEnv "NIXOS_HOST_KEYS_DIR";
|
||||
hasHostKeysDir = hostKeysDirStr != "" && builtins.pathExists hostKeysDirStr;
|
||||
hostKeysDir = /. + hostKeysDirStr;
|
||||
|
||||
# flakeTarget ("${platform}-${buildType}") comes in via specialArgs from
|
||||
# flake.nix's mkTarget -- exactly the name scripts/sync-host-keys.sh
|
||||
# registers keys under. Deliberately not read back from
|
||||
# config.environment.etc."flake-target" (which is set to the same value)
|
||||
# -- this module also *contributes* to environment.etc below, and a
|
||||
# module reading the merged value of an option it's still defining is a
|
||||
# circular dependency (confirmed: "infinite recursion encountered").
|
||||
privKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key";
|
||||
pubKeyFile = hostKeysDir + "/${flakeTarget}_ssh_host_ed25519_key.pub";
|
||||
hasKeyForThisTarget =
|
||||
hasHostKeysDir
|
||||
&& builtins.pathExists privKeyFile
|
||||
&& builtins.pathExists pubKeyFile;
|
||||
in
|
||||
{
|
||||
# LXC containers share the host kernel — Proxmox starts them by exec'ing
|
||||
# /sbin/init directly, no bootloader/initrd involved — and Proxmox has its
|
||||
@@ -74,75 +35,4 @@ in
|
||||
# for the same reason; it just doesn't disable NetworkManager itself,
|
||||
# which modules/common/configuration.nix enables for every host.
|
||||
networking.networkmanager.enable = lib.mkForce false;
|
||||
|
||||
environment.etc = lib.mkIf hasKeyForThisTarget {
|
||||
"ssh/ssh_host_ed25519_key" = {
|
||||
source = privKeyFile;
|
||||
mode = "0600";
|
||||
};
|
||||
"ssh/ssh_host_ed25519_key.pub" = {
|
||||
source = pubKeyFile;
|
||||
mode = "0644";
|
||||
};
|
||||
};
|
||||
|
||||
# virtualisation/proxmox-lxc.nix (imported above) registers the Nix
|
||||
# store DB via a systemd service (register-nix-paths) -- it never runs
|
||||
# an activation script at all. Confirmed live this means neither
|
||||
# sops-nix's "for users" secrets (password hashes -- installed by the
|
||||
# activation script itself, not a systemd service, since they need to
|
||||
# exist *before* user creation) nor the user-creation step that
|
||||
# consumes them ever run on a real lxc-* boot. Regular secrets
|
||||
# (nix-serve's key, beszel's token, etc.) work anyway because sops-nix
|
||||
# provides its own systemd service for those.
|
||||
#
|
||||
# A systemd service, not boot.postBootCommands: tried that first (it's
|
||||
# a genuine, generally-invoked hook -- nixos/modules/system/boot/stage-2-init.sh,
|
||||
# which becomes this container's actual /sbin/init, unconditionally
|
||||
# runs it) but switch-to-configuration behaves differently that early in
|
||||
# boot (raw stage-2-init.sh, before systemd itself has even started) --
|
||||
# confirmed live it silently failed to rewrite /etc/shadow from there
|
||||
# even in "test" mode, despite the exact same command working reliably
|
||||
# every time when run post-boot (i.e. as a normal systemd service, which
|
||||
# is what this is). Not fully root-caused why the early context
|
||||
# specifically breaks it; a real systemd service sidesteps needing to.
|
||||
#
|
||||
# /etc/shadow already has PLACEHOLDER entries for every declared user
|
||||
# baked in at build time (part of constructing the system closure).
|
||||
# update-users-groups.pl deliberately never overwrites an *existing*
|
||||
# shadow entry -- a correct safety property in general (don't clobber a
|
||||
# real user's real password on a config rebuild) -- but on a genuine
|
||||
# first boot that only means the real hashedPasswordFile-derived hash
|
||||
# never gets the chance to be applied either, since the placeholder is
|
||||
# already "seen". Safe to clear here specifically: there is no real
|
||||
# password yet to protect on a first boot.
|
||||
#
|
||||
# "test" mode, not "boot": confirmed live "boot" mode aborts partway
|
||||
# through (before rewriting /etc/shadow) on a warning that "/boot" is on
|
||||
# a different filesystem -- a real check for a host with a bootloader to
|
||||
# update, meaningless for a container that has none
|
||||
# (boot.loader.{grub,systemd-boot}.enable are both false above), but it
|
||||
# still aborts the script. "test" runs every activation step without
|
||||
# touching boot-loader state at all.
|
||||
#
|
||||
# ConditionPathExists (systemd-native, not a bash-level check) means
|
||||
# this only ever runs once, on the genuine first boot -- systemd itself
|
||||
# skips even starting it on every later boot once the marker exists.
|
||||
# switch-to-configuration is otherwise the operator's call per this
|
||||
# repo's own safety rules, not something to run on every boot.
|
||||
systemd.services.nixos-lxc-first-boot-activate = {
|
||||
description = "Complete first-boot NixOS activation (users, secrets) for this LXC container";
|
||||
wantedBy = [ "multi-user.target" ];
|
||||
unitConfig.ConditionPathExists = "!/var/lib/nixos-lxc-first-boot-activated";
|
||||
serviceConfig = {
|
||||
Type = "oneshot";
|
||||
RemainAfterExit = true;
|
||||
};
|
||||
script = ''
|
||||
rm -f /etc/shadow
|
||||
/run/current-system/bin/switch-to-configuration test
|
||||
mkdir -p /var/lib
|
||||
touch /var/lib/nixos-lxc-first-boot-activated
|
||||
'';
|
||||
};
|
||||
}
|
||||
|
||||
@@ -1,19 +1,20 @@
|
||||
_:
|
||||
{ vars, ... }:
|
||||
|
||||
{
|
||||
services.tailscale = {
|
||||
enable = true;
|
||||
imports = [ ./enable-service.nix ];
|
||||
|
||||
# extraSetFlags (tailscale set, via the always-on tailscaled-set
|
||||
# service), not extraUpFlags -- extraUpFlags is only ever applied by
|
||||
# tailscaled-autoconnect, which itself only runs when
|
||||
# services.tailscale.authKeyFile is set (nothing in this repo sets one,
|
||||
# so tailscale up is a manual, one-time operator step on every host that
|
||||
# uses this service). extraSetFlags has no such gate, so
|
||||
# --advertise-exit-node self-reapplies on every boot once the operator
|
||||
# has authenticated the node once.
|
||||
extraSetFlags = [
|
||||
services.tailscale = {
|
||||
# Enables the sysctl forwarding settings exit nodes/subnet routers need;
|
||||
# without this, --advertise-exit-node has no effect.
|
||||
useRoutingFeatures = "server";
|
||||
|
||||
# Lets peers reach this node directly over the tailscale UDP port
|
||||
# instead of relaying through DERP.
|
||||
openFirewall = true;
|
||||
|
||||
extraUpFlags = [
|
||||
"--advertise-exit-node"
|
||||
"--advertise-routes=${vars.lanCidr}"
|
||||
];
|
||||
};
|
||||
}
|
||||
|
||||
@@ -10,9 +10,7 @@
|
||||
#
|
||||
# SAFETY:
|
||||
# - The default (create) mode only ever creates a NEW resource -- it
|
||||
# refuses to run if the target VMID already exists on the node, or if
|
||||
# a VM/CT identified as --host already exists under any other VMID
|
||||
# (checked live against the node; --allow-duplicate-host overrides).
|
||||
# refuses to run if the target VMID already exists on the node.
|
||||
# - --modify only ever touches a resource you name explicitly via
|
||||
# --vmid, shows exactly what will change first, and (outside
|
||||
# --dry-run) always requires typing that VMID back to confirm before
|
||||
@@ -42,12 +40,8 @@ Create mode (default):
|
||||
config.networking.hostName (server, docker,
|
||||
nix-cache, nixos, pxe-boot, nix-minimal). Use
|
||||
--list to see what's available for --type.
|
||||
--name <name> Proxmox display name/hostname (default: --host's
|
||||
value, e.g. nix-cache -- for lxc this becomes the
|
||||
guest's real networking.hostName too, since
|
||||
proxmoxLXC.manageHostName pulls it from Proxmox's
|
||||
own container config, so it must match host.nix
|
||||
regardless of build type)
|
||||
--name <name> Proxmox display name/hostname (default: the flake
|
||||
target name, e.g. lxc-server)
|
||||
--vmid <n> Numeric VMID (default: next free, via
|
||||
\`pvesh get /cluster/nextid\` on the node).
|
||||
Refuses to run if this ID already exists.
|
||||
@@ -58,11 +52,10 @@ Create mode (default):
|
||||
--force-rebuild Skip the "does the node already have this
|
||||
image" check -- always build fresh and
|
||||
overwrite what's there.
|
||||
--allow-duplicate-host Required if a VM/CT identified as --host
|
||||
already exists on the node (checked live via
|
||||
qm/pct, not any file in this repo) --
|
||||
otherwise refused, since it'd share that
|
||||
host's hostName/hostId.
|
||||
--allow-duplicate-host Required if --host already has a real
|
||||
deployment elsewhere (variables.nix's
|
||||
deployedTargets) -- otherwise refused, since
|
||||
it'd share that host's hostName/hostId.
|
||||
|
||||
Modify mode (reconfigure an EXISTING resource -- requires --modify):
|
||||
--modify Switch to modify mode.
|
||||
@@ -281,66 +274,27 @@ if [[ -z "$flake_target" ]]; then
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# The container/VM's real identity is --host (e.g. "nix-cache"), validated
|
||||
# above against config.networking.hostName -- not the flake target name
|
||||
# (e.g. "lxc-nix-cache"), which is build-type-specific and only exists to
|
||||
# pick which platform variant to build. Defaulting --name to the flake
|
||||
# target would make lxc's --hostname (which proxmoxLXC.manageHostName
|
||||
# feeds straight into the guest's real hostname) disagree with host.nix.
|
||||
[[ -z "$name" ]] && name="$host"
|
||||
[[ -z "$name" ]] && name="$flake_target"
|
||||
|
||||
# --- refuse to duplicate a host that's already live on the node ---------
|
||||
# Queries the node itself (qm/pct's own name/hostname config), not any
|
||||
# static list in this repo -- a file can't track whether a resource still
|
||||
# actually exists, and this used to be checked against variables.nix's
|
||||
# deployedTargets, which drifted stale (it kept naming a VM as "the real
|
||||
# deployment" well after that VM had been destroyed, blocking its own
|
||||
# redeploy) until that list was dropped in favour of this live check. This
|
||||
# only catches guests identified with the default --name (== --host, what
|
||||
# this script itself always uses unless --name is overridden) -- a guest
|
||||
# manually renamed on the node afterwards wouldn't match, but nothing here
|
||||
# creates guests that way.
|
||||
if [[ "$allow_duplicate_host" -eq 1 ]]; then
|
||||
echo
|
||||
echo "--allow-duplicate-host: skipping the check for an existing '${host}' on ${node}."
|
||||
elif [[ "$dry_run" -eq 1 ]]; then
|
||||
echo
|
||||
echo "[dry-run] would check ${node} for an existing VM/CT identified as '${host}'"
|
||||
else
|
||||
echo
|
||||
echo "==> Checking ${node} for an existing VM/CT identified as '${host}'..."
|
||||
ssh_check_status=0
|
||||
existing="$(ssh "$ssh_target" bash -s -- "$host" <<'REMOTE_SCRIPT'
|
||||
target="$1"
|
||||
for id in $(qm list 2>/dev/null | awk 'NR>1{print $1}'); do
|
||||
n="$(qm config "$id" 2>/dev/null | grep -oP '^name:\s*\K\S+' || true)"
|
||||
[[ "$n" == "$target" ]] && echo "vm ${id} ${n}"
|
||||
done
|
||||
for id in $(pct list 2>/dev/null | awk 'NR>1{print $1}'); do
|
||||
n="$(pct config "$id" 2>/dev/null | grep -oP '^hostname:\s*\K\S+' || true)"
|
||||
[[ "$n" == "$target" ]] && echo "lxc ${id} ${n}"
|
||||
done
|
||||
exit 0
|
||||
REMOTE_SCRIPT
|
||||
)" || ssh_check_status=$?
|
||||
if [[ "$ssh_check_status" -ne 0 ]]; then
|
||||
echo "ERROR: couldn't reach ${node} (ssh exited ${ssh_check_status}) to check for an" >&2
|
||||
echo "existing '${host}' resource -- refusing to guess. Fix connectivity and retry," >&2
|
||||
echo "or pass --allow-duplicate-host if you're sure none exists (this skips the" >&2
|
||||
echo "check entirely)." >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -n "$existing" ]]; then
|
||||
echo "ERROR: '${host}' already exists on ${node}:" >&2
|
||||
echo "$existing" | while read -r kind id n; do
|
||||
echo " - ${kind} VMID ${id} (${n})" >&2
|
||||
done
|
||||
echo "Refusing to create a second resource sharing this identity. Pass" >&2
|
||||
echo "--allow-duplicate-host to create one anyway (it gets its own distinct" >&2
|
||||
echo "sops key and VMID -- the existing resource(s) above are left untouched)," >&2
|
||||
echo "or use --modify to reconfigure the existing one instead." >&2
|
||||
exit 1
|
||||
fi
|
||||
# --- refuse to duplicate a host that's already really deployed ----------
|
||||
# Checked by hostName, not exact flake target: proxmox-server being
|
||||
# deployed also blocks --type lxc --host server, since both would carry
|
||||
# the same hosts/server/host.nix identity (hostName, hostId).
|
||||
if [[ "$allow_duplicate_host" -eq 0 ]]; then
|
||||
deployed_targets_json="$(nix eval --json --no-use-registries --no-accept-flake-config \
|
||||
--file "${repo_root}/variables.nix" deployedTargets)"
|
||||
for dt in $(echo "$deployed_targets_json" | jq -r '.[]'); do
|
||||
dt_hostname="$(nix eval --raw --no-use-registries --no-accept-flake-config \
|
||||
"${repo_root}#nixosConfigurations.${dt}.config.networking.hostName" 2>/dev/null || true)"
|
||||
if [[ "$dt_hostname" == "$host" ]]; then
|
||||
echo "ERROR: '${host}' already has a real deployment (${dt}, per variables.nix's" >&2
|
||||
echo "deployedTargets). Creating ${flake_target} would share its hostName/hostId --" >&2
|
||||
echo "refusing by default. Pass --allow-duplicate-host if you really mean to spin" >&2
|
||||
echo "up a separate test instance of this host (it'll still get its own distinct" >&2
|
||||
echo "sops key and VMID, never touching ${dt})." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
fi
|
||||
|
||||
echo "Target: ${flake_target} (host=${host}, type=${type}) -> Proxmox resource '${name}'"
|
||||
@@ -429,19 +383,9 @@ else
|
||||
fi
|
||||
|
||||
if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
|
||||
# Mirrors the real build commands' "${NIX_OPTS[@]}" below -- nix_extra_opts
|
||||
# (called earlier, once) has already decided whether nix-cache is in play,
|
||||
# and the dry-run preview needs to reflect that decision instead of always
|
||||
# printing the same command regardless of outcome.
|
||||
nix_opts_display=""
|
||||
if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
|
||||
printf -v nix_opts_display '%q ' "${NIX_OPTS[@]}"
|
||||
nix_opts_display=" ${nix_opts_display% }"
|
||||
fi
|
||||
if [[ "$type" == "lxc" ]]; then
|
||||
if [[ "$dry_run" -eq 1 ]]; then
|
||||
echo "[dry-run] would build: NIXOS_HOST_KEYS_DIR=${repo_root}/host-keys nix build --impure \\"
|
||||
echo "[dry-run] --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
|
||||
echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.tarball"
|
||||
local_image="<built-tarball>"
|
||||
else
|
||||
@@ -455,8 +399,7 @@ if [[ "$image_already_remote" -eq 0 && -z "$local_image" ]]; then
|
||||
fi
|
||||
else
|
||||
if [[ "$dry_run" -eq 1 ]]; then
|
||||
echo "[dry-run] would build: nix build --no-use-registries --no-accept-flake-config${nix_opts_display} \\"
|
||||
echo "[dry-run] .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
|
||||
echo "[dry-run] would build: nix build .#nixosConfigurations.${flake_target}.config.system.build.diskoImagesScript"
|
||||
echo "[dry-run] would run: sudo ./result-${flake_target} \\"
|
||||
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key /etc/ssh/ssh_host_ed25519_key \\"
|
||||
echo "[dry-run] --pre-format-files host-keys/${flake_target}_ssh_host_ed25519_key.pub /etc/ssh/ssh_host_ed25519_key.pub \\"
|
||||
|
||||
+5
-52
@@ -88,60 +88,13 @@ nix_extra_opts() {
|
||||
fi
|
||||
export NIX_EXTRA_OPTS_DECIDED=1
|
||||
NIX_OPTS=()
|
||||
|
||||
# Retry a couple of times, 1s apart, before believing either check --
|
||||
# belt-and-suspenders against a genuine multi-second blip (nix-cache
|
||||
# restarting), on top of the fix below. Worst case (~11s total, host
|
||||
# genuinely gone) is still nowhere near the 15s+ *per lookup* nix's own
|
||||
# substituter retries would cost if this check didn't exist at all.
|
||||
local attempt cache_up=0 builder_up=0
|
||||
for attempt in 1 2 3; do
|
||||
if curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
|
||||
cache_up=1
|
||||
break
|
||||
fi
|
||||
[[ "$attempt" -lt 3 ]] && sleep 1
|
||||
done
|
||||
|
||||
if [[ "$cache_up" -eq 0 ]]; then
|
||||
if ! curl --silent --fail --max-time 3 "http://${NIX_CACHE_HOST}/nix-cache-info" >/dev/null 2>&1; then
|
||||
echo "nix-cache (http://${NIX_CACHE_HOST}) is unreachable -- skipping it (substituter + remote builder) for the rest of this run." >&2
|
||||
NIX_OPTS=(--option substituters "https://cache.nixos.org/" --builders "")
|
||||
else
|
||||
for attempt in 1 2 3; do
|
||||
# `exec 3<>/dev/tcp/...` just opens the fd and returns -- it does NOT
|
||||
# read from it. Confirmed live this is load-bearing, not stylistic:
|
||||
# the previous `cat < /dev/tcp/.../22` blocked forever and always hit
|
||||
# the timeout even against a perfectly healthy nix-cache, because
|
||||
# sshd sends its banner and then holds the connection open waiting
|
||||
# for the client to speak next -- `cat` never sees EOF, so this
|
||||
# check reported "unreachable" unconditionally, 100% of the time,
|
||||
# regardless of whether the remote builder was actually up.
|
||||
if timeout 3 bash -c "exec 3<>/dev/tcp/${NIX_CACHE_HOST}/22" 2>/dev/null; then
|
||||
builder_up=1
|
||||
break
|
||||
fi
|
||||
[[ "$attempt" -lt 3 ]] && sleep 1
|
||||
done
|
||||
if [[ "$builder_up" -eq 0 ]]; then
|
||||
echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
|
||||
NIX_OPTS=(--builders "")
|
||||
fi
|
||||
fi
|
||||
|
||||
# `printf '%q '` with a genuinely empty NIX_OPTS still runs one format
|
||||
# pass over a missing argument and yields the literal `'' ` rather than
|
||||
# an empty string (confirmed live) -- a subprocess that later does
|
||||
# `eval "NIX_OPTS=(${NIX_EXTRA_OPTS})"` (the branch above, for e.g.
|
||||
# sync-host-keys.sh reusing this process's decision) would then rebuild
|
||||
# a 1-element array holding an empty string instead of a 0-element
|
||||
# array, and `nix-shell "${NIX_OPTS[@]}" -p <pkg>` chokes on that stray
|
||||
# element as a bogus positional argument. Guard the empty case
|
||||
# explicitly so nix-cache being reachable (NIX_OPTS legitimately empty)
|
||||
# round-trips as truly empty instead.
|
||||
if [[ ${#NIX_OPTS[@]} -gt 0 ]]; then
|
||||
printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
|
||||
else
|
||||
NIX_EXTRA_OPTS=""
|
||||
elif ! timeout 3 bash -c "cat < /dev/tcp/${NIX_CACHE_HOST}/22" >/dev/null 2>&1; then
|
||||
echo "nix-cache's SSH remote builder (nixremote@${NIX_CACHE_HOST}:22) is unreachable -- disabling remote builds for the rest of this run." >&2
|
||||
NIX_OPTS=(--builders "")
|
||||
fi
|
||||
printf -v NIX_EXTRA_OPTS '%q ' "${NIX_OPTS[@]}"
|
||||
export NIX_EXTRA_OPTS
|
||||
}
|
||||
|
||||
+16
-9
@@ -19,15 +19,6 @@
|
||||
|
||||
remoteBuilderUser = "nixremote"; # remote builder SSH user
|
||||
|
||||
# nix-cache's own SSH host public key (not a secret — the private half
|
||||
# never leaves the host). Wired into every client's
|
||||
# programs.ssh.knownHosts by modules/nix-cache/remote-builder-client.nix
|
||||
# so distributed builds don't hit "Host key verification failed" on a
|
||||
# fresh client that has never manually ssh'd to nix-cache before. Update
|
||||
# this if nix-cache's host key is ever rotated or the host is rebuilt
|
||||
# from scratch.
|
||||
nixCacheHostKey = "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHrMKZlIGUd3pH9G3AqbsruqUGjxIXMAZw52u9MwiBCn lxc-nix-cache";
|
||||
|
||||
# Public keys authorized to SSH in as remoteBuilderUser on the nix-cache
|
||||
# host (modules/nix-cache/server.nix) — one per client host that's allowed
|
||||
# to use it as a distributed builder.
|
||||
@@ -156,4 +147,20 @@
|
||||
keep = 20; # number of rotated logs to retain before deleting the oldest
|
||||
};
|
||||
|
||||
# Flake targets with a real, currently-running deployment somewhere —
|
||||
# matches README.md's Hosts table "(real, deployed)" annotations; update
|
||||
# both together. Not consumed by any NixOS module (nothing in the actual
|
||||
# system config should behave differently because of this) — it's read
|
||||
# by scripts/create-proxmox-resource.sh to refuse creating a same-identity
|
||||
# duplicate of an already-deployed host (shared hostName/hostId) unless
|
||||
# you explicitly pass --allow-duplicate-host.
|
||||
deployedTargets = [
|
||||
"linode-minimal"
|
||||
"proxmox-minimal"
|
||||
"proxmox-nix-cache"
|
||||
"proxmox-server"
|
||||
"proxmox-docker"
|
||||
"proxmox-gui"
|
||||
"proxmox-pxe-boot"
|
||||
];
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user