Archived

This repository has been archived on 2026-07-30. You can view files and clone it. You cannot open issues or pull requests or push a commit.

Files

T

beatzaplentyandClaude Sonnet 5 a3e85b5079

Check NixOS configurations / eval-hosts (push) Failing after 10m55s

Details

Fix nix-cache retry storms and lxc creation in create-proxmox-resource.sh

Two independent problems found while actually running the script:

1. nix build/nix-shell retry each unreachable substituter/builder up to
   5x with backoff, per store path -- with nix-cache down this compounds
   into minutes of noise. scripts/env.sh gains nix_extra_opts(), which
   probes http://nix-cache and nixremote@nix-cache:22 once via plain
   curl/TCP (bypassing Nix's own retry logic entirely -- confirmed
   nix store ping still retries 5x even with a short connect-timeout)
   and exports the decision so create-proxmox-resource.sh and the
   sync-host-keys.sh subprocess it shells out to both reuse it instead
   of probing independently.

2. The actual failure: "archive contains no configuration file". pct
   restore expects a vzdump backup archive with embedded config;
   config.system.build.tarball is a plain CT template tarball -- wrong
   Proxmox mechanism entirely. Fixed to pct create against it as a vztmpl
   template instead, uploaded to /var/lib/vz/template/cache/ rather than
   /var/lib/vz/dump/. This same wrong claim had propagated into
   docs/auto-installer.md, README.md, and CLAUDE.md from when the script
   was first written -- corrected everywhere.

Also: checks for an already-uploaded image on the node (fixed
<flake_target>.tar.xz/.raw naming) before building, skipping build+upload
entirely if found (--force-rebuild to always rebuild).

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot

2026-07-20 11:05:26 +10:00

12 KiB

Raw Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Repo purpose

Flake-based NixOS configuration for Wayne's LAN servers and workstation. There is no application code here — changes are Nix module edits that affect real machines when deployed.

Safety rules (read before touching anything)

Never run nixos-rebuild switch|boot|test, nixos-install, parted, mkfs, mkswap, swapon, mount, or any other destructive disk/deploy command from an agent session, even if asked indirectly. Deployment is done manually by the operator on the target host.
Validation is limited to evaluation, linting, formatting checks, and nix build --dry-run --no-link.
Do not add secrets, tokens, private keys, or new password hashes to the repo.
This repo currently contains committed password hashes in modules/installer/common.nix (the auto-installer's own root/nixos login — a deliberate, documented choice, see docs/auto-installer.md, not accidental tech debt) and SSH public keys in variables.nix (vars.adminSshKey, vars.remoteBuilderAuthorizedKeys) plus a couple of per-host KEY values for beszel-agent auth (hosts/server/host.nix, hosts/nix-cache/host.nix). Don't use the installer's hardcoded hash as a template for a real host — every other host uses sops-nix (hashedPasswordFile, see "Security Notes" in README.md). Flag any new secret-like string you encounter instead of committing it.
host-keys/ is gitignored — locally-generated private SSH host keys for the auto-installer (see docs/auto-installer.md). Never commit its contents; if git status ever shows it as trackable, something is wrong.

Commands

# One-time environment bootstrap (installs Nix if missing, prints hosts)
bash scripts/codex-setup.sh

# Full validation: secret grep, nixpkgs-fmt --check, statix lint, eval all hosts
bash scripts/codex-maintenance.sh

# Same, plus a dry-run build (no result symlink) of every host's toplevel
bash scripts/codex-maintenance.sh dry-run

# List the hosts the flake currently exposes
nix eval --json .#nixosConfigurations --apply builtins.attrNames | jq -r '.[]'

# Evaluate a single host without building (fast sanity check)
nix eval .#nixosConfigurations.<host>.config.system.build.toplevel.drvPath --raw

# Dry-run build a single host
nix build --dry-run --no-link .#nixosConfigurations.<host>.config.system.build.toplevel

Formatting/lint tools (nixpkgs-fmt, statix) are not installed locally; the maintenance script pulls them via nix run github:NixOS/nixpkgs/nixos-25.11#<tool>. There is no test suite — "correctness" here means the flake evaluates and nixpkgs-fmt/statix are clean.

In an interactive agent session, prefer targeted checks over full-repo sweeps: after editing one or two hosts/modules, evaluate just the nixosConfigurations.<host> you touched (plus any config.system.build.tarball /diskoImagesScript/package output affected) rather than looping over every host — codex-maintenance.sh evaluates 18 hosts plus every package/tarball/ image variant now and is slow to run after each small change. Reserve a full codex-maintenance.sh run for changes that plausibly affect every host (modules/common/*, flake.nix, variables.nix) or as a final check before committing. This is a session-workflow preference only — it does not apply to CI, which should keep running the full script on every push/PR regardless of diff size; that's the point of it.

Scripts

Beyond codex-setup.sh/codex-maintenance.sh above, scripts/ also has:

scripts/sync-host-keys.sh — generates/registers SSH host keys and their .sops.yaml/secrets/*.yaml recipients for flake targets, idempotently (--all, <target>, --remove, --regenerate-all-keys, all with --dry-run). The primary tool for provisioning a new host's secrets access — see "Creating a new machine" in docs/auto-installer.md.
scripts/prepare-host-key.sh — narrower predecessor: generates a key by an arbitrary name without touching .sops.yaml. Still useful to pre-generate a key before its flake target exists yet, since sync-host-keys.sh can only act on targets nixosConfigurations already has.
scripts/create-proxmox-resource.sh — builds a lxc-*/proxmox-* target's tarball/disk image and creates it on a real Proxmox node (pct create against the tarball as a CT template / qm create+ importdisk), or reconfigures an existing resource's cores/memory/disk size (--modify, always requires typing the VMID back to confirm). Checks for an already-uploaded image on the node before building (--force-rebuild to skip that and always rebuild), and probes nix-cache's substituter/remote-builder reachability once up front rather than letting every nix build call retry against it individually. Refuses to create a target whose host identity already has a real deployment elsewhere (variables.nix's deployedTargets) unless --allow-duplicate-host is passed. --dry-run throughout both modes.
scripts/env.sh — shared config (PROXMOX_HOST, storage pool, bridge, default cores/memory) sourced by create-proxmox-resource.sh. Add new cross-script config here instead of duplicating it per-script.
scripts/bump-nixpkgs-release.sh — bumps flake.nix's nixpkgs.url/ home-manager.url in place. Exists because flake input URLs can't reference variables.nix (confirmed empirically — nix flake metadata errors on it), so this is the closest equivalent to a single source of truth for the tracked release.

sync-host-keys.sh and create-proxmox-resource.sh genuinely mutate real state when run for real (not --dry-run): real secrets/*.yaml recipients, real Proxmox VMs/containers. They require the operator's own SSH/sops access, which an agent session doesn't have — but don't suggest running either non-dry-run without the operator's explicit go-ahead even if it becomes technically reachable.

Architecture

flake.nix is the single entry point. It generates one nixosConfigurations.<platform>-<buildtype> attribute per target via the mkTarget function, composed from:

nixosSystem {
  modules = [
    disko.nixosModules.disko
    sops-nix.nixosModules.sops
    ./modules/common/configuration.nix
    ./modules/platforms/${platform}.nix        # what it runs on
    ./modules/build-types/${buildType}.nix     # what it's for
    hostPath                                   # hosts/<name>/host.nix — per-machine identity
    home-manager.nixosModules.home-manager { ... }
  ] ++ (client-only modules, for every buildType except "nix-cache" itself)
}

Platforms: linode, proxmox, lxc. Build types: minimal, nix-cache, server, docker, gui, pxe-boot. Not every combination is built — e.g. pxe-boot has no linode variant (PXE/DHCP/TFTP need LAN L2 adjacency a Linode VPS doesn't have). Treat flake.nix's generatedTargets as the source of truth for which hosts exist — README.md, AGENTS.md, docs/flake-lock-automation.md, and the CI eval workflows (.github/workflows/check-nixos.yml, .gitea/workflows/check-nixos.yml) list hosts by hand (or, for the CI workflows, evaluate the flake dynamically) and can drift from it, so re-check them against flake.nix when adding or removing a host.

Composition pattern

hosts/<name>/host.nix — per-machine identity only: hostname, hostId, per-machine secrets, system.stateVersion. These files carry no imports of their own beyond narrow parameterized helpers (see modules/beszel/host-token.nix below) — all shared behavior comes from the platform/build-type modules composed in flake.nix, not from the host file.
modules/platforms/{linode,proxmox,lxc}.nix — platform-specific config: boot method, guest tooling, and (for linode/proxmox) the hypervisor-specific hardware config, imported directly by the platform module itself (../hardware-configuration/vm/{proxmox,linode}.nix) — not wired in from flake.nix. lxc.nix has no hardware-configuration counterpart since containers share the host kernel; instead it imports nixpkgs' own virtualisation/proxmox-lxc.nix, which gives every lxc-* host a config.system.build.tarball output — a plain rootfs tarball, used as a pct create ... vztmpl CT template (not pct restore, which expects vzdump backup-archive metadata this doesn't have), no install step — see docs/auto-installer.md.
modules/build-types/*.nix — what a system is for: minimal/server/docker/gui/pxe-boot/nix-cache.
modules/common/configuration.nix — base NixOS config imported by every host: locale, users, nix settings, git.
modules/common/home.nix / hosts/nixos/home.nix — Home Manager config for the nixos user; the nixos workstation (gui build type) has its own, other hosts share modules/common/home.nix.
modules/disko/proxmox.nix — declarative disk layout (GPT: ESP + swap + ext4 root) via disko, used by all Proxmox-VM hosts (proxmox-*, not lxc-*). Also carries imageSize/imageName, letting every proxmox-* host be built as a standalone, qm importdisk-ready .raw image with no install step — see docs/proxmox-images.md.
modules/disko/linode.nix — linode-*'s disko config, deliberately different in kind from the Proxmox one: Linode provisions and sizes /dev/sda//dev/sdb itself as whole, unpartitioned devices before the OS boots, so this declares them with destroy = false (disko never wipes them) and a bare filesystem/swap content type instead of a partition table — idempotent against an already-provisioned disk, never destructive.
modules/boot/efi.nix — systemd-boot + EFI vars, paired with the disko module.
modules/installer/ — the auto-installer environment (ISO, also served as PXE netboot): common.nix (shared config + the generated auto-install.sh), iso.nix, host-keys.nix (optionally bakes host-keys/ into the image under --impure). See docs/auto-installer.md.
modules/pxe-boot/stage-installer-artifacts.nix — builds the installer's netboot image and stages it on the pxe-boot host so its iPXE menu can chain straight to it. See docs/pxe-boot.md.
modules/nix-cache/{client,server,remote-builder-client}.nix — binary cache substituter + SSH remote-builder wiring; see docs/nix-cache.md for the full design (per-host local stores, no shared /nix/store, and how the nixremote signing/SSH keys fit together).
modules/beszel/host-token.nix — parameterized helper module ({ name, sopsFile }) that wires a host's beszel-agent sops secret/template and environmentFile; used by hosts/server/host.nix and hosts/nix-cache/host.nix to avoid duplicating that boilerplate.
modules/tailscale/, modules/docker/, modules/networking/, modules/traefik/, modules/services/* — single-purpose, single-host feature modules (e.g. docker/enable-service.nix, services/zfs/enable-service.nix). Grep modules/build-types/*.nix for each build type's imports list to see which modules apply where.

New host = new hosts/<name>/host.nix + a matching mkTarget { platform; buildType; hostPath; } entry added to flake.nix's generatedTargets, composed from existing modules/* pieces rather than duplicating config.

12 KiB Raw Blame History