nixos/docs/pxe-boot.md

# pxe-boot

The `pxe-boot` host serves HTTP boot assets for iPXE clients — including
self-staged copies of both this flake's own auto-installer netboot image
(see `docs/auto-installer.md` for what that image actually is and does once
booted) and a vanilla, unmodified NixOS minimal netboot image for plain
rescue/inspection use.

## Host Role

- Hostname: `pxe-boot`
- Web service: nginx on TCP port 80
- PXE root: `/srv/pxe`
- HTTP root for scripts and images: `/srv/pxe/http`
- TFTP root for first-stage bootloaders: `/srv/pxe/tftp`
- iPXE entry script: `/srv/pxe/http/boot.ipxe`
- Generated iPXE menu: `/srv/pxe/http/menu.ipxe`
- Debian Minimal iPXE script: `/srv/pxe/http/debian.ipxe`
- SystemRescue iPXE script: `/srv/pxe/http/systemrescue.ipxe`
- TFTP fallback script: `/srv/pxe/tftp/autoexec.ipxe`
- Boot binaries copied from the Nix `ipxe` package:
  - `/srv/pxe/tftp/ipxe.efi`
  - `/srv/pxe/tftp/undionly.kpxe`

## Directory Layout

The host creates these directories with systemd tmpfiles:

```text
/srv/pxe
/srv/pxe/http
/srv/pxe/http/images  -> /mnt/pxe-images  (symlink to NFS share)
/srv/pxe/http/auto-installer
/srv/pxe/http/nixos-minimal
/srv/pxe/http/debian
/srv/pxe/http/systemrescue
/srv/pxe/http/ubuntu
/srv/pxe/http/rescue
/srv/pxe/tftp
```

`/srv/pxe/http/images` is a symlink to `/mnt/pxe-images`, which is an NFS
mount of `server.sweet.home:/tank/pxe-boot/images`
(`modules/pxe-boot/mount-pxe-images.nix`). Place large images there (ISOs,
disk images) rather than on the pxe-boot host's own root disk. For an LXC
pxe-boot container the mount uses NFSv3+nolock with `nofail` (eager,
non-blocking on server unavailability); for a Proxmox VM it uses NFSv4.2
with `x-systemd.automount` (lazy, triggered on first access).

When running as `lxc-pxe-boot`, the Proxmox container must have
`features: nesting=1,mount=nfs` (at minimum) in its Proxmox config. `nesting=1`
is required by systemd 260+ for credential isolation (user namespace creation
and internal move-mounts); without it, AppArmor denies both, and every
systemd service that uses `PrivateUsers`, `PrivateDevices`, or credential
passing fails on boot. `mount=nfs` allows the NFSv3 mount. Both are set
automatically by `scripts/proxmox/create-proxmox-resource.sh` (via
`PROXMOX_DEFAULT_LXC_FEATURES` in `scripts/env.sh` which defaults to
`nesting=1,keyctl=1,mount=nfs;nfs4`). If you ever change these features
manually via `pct set`, be sure to include both — `pct set` replaces the
entire features string, it does not append to it.

The HTTP iPXE chain is:

```text
undionly.kpxe or ipxe.efi
  -> autoexec.ipxe from the TFTP root, when iPXE requests it
  -> http://192.168.2.223/boot.ipxe
  -> http://192.168.2.223/menu.ipxe
```

The generated menu currently exposes entries for:

- NixOS Auto-Installer
- NixOS Minimal
- Debian Minimal
- FreeIPA Server (Rocky Linux 9)
- SystemRescue environment
- iPXE shell
- Reboot

Both NixOS entries chain-load a `netboot.ipxe` staged into their own
directory (`/srv/pxe/http/auto-installer/netboot.ipxe` and
`/srv/pxe/http/nixos-minimal/netboot.ipxe`), each nixpkgs' own generated
netboot iPXE script (correct `init=`/`initrd=` kernel parameters included)
rather than a hand-rolled boot line — that script in turn expects its
kernel/initrd siblings in the same directory. Each directory's three files
(`bzImage`, `initrd`, `netboot.ipxe`) are built from source and staged
automatically by `modules/pxe-boot/stage-installer-artifacts.nix` via
`systemd.tmpfiles.rules` — no manual operator step required:

- `auto-installer` is this flake's own `netbootSystem` (`flake.nix`) — the
  same auto-installer image `nix build .#pxe` produces. See
  `docs/auto-installer.md`.
- `nixos-minimal` is `netbootMinimalSystem` (`flake.nix`) — nixpkgs'
  `netboot-minimal.nix` composed on its own, with none of this flake's
  auto-installer wiring (no `common.nix`, no `auto-install.sh`, no baked
  host keys or custom users). Same `nix build .#pxe-minimal` mechanism as
  the auto-installer image, just a different module composition. Useful
  as a plain rescue/inspection shell that doesn't assume anything about
  this flake.

Both images set `networking.hostName` to match their menu entry/staged
directory name (`auto-installer` / `nixos-minimal`), so each one's
generated system name (`nixos-system-<name>-*`) is self-describing rather
than the nixpkgs default of `nixos-system-nixos-*` for both.

The Debian Minimal entry chains `http://<pxeServerIp>/debian.ipxe`, which loads
the Debian bookworm netboot kernel and initrd from `/srv/pxe/http/debian/`. The
`fetch-debian-netboot.service` oneshot downloads these files from
`deb.debian.org` on first boot (idempotent — skips if files are already
present):

```text
/srv/pxe/http/debian/linux      (Debian bookworm netboot kernel)
/srv/pxe/http/debian/initrd.gz  (Debian bookworm netboot initrd)
```

The service requires outbound internet access on the pxe-boot host. To
re-download (e.g. after a Debian point release), delete the files and restart
the service:

```bash
rm /srv/pxe/http/debian/linux /srv/pxe/http/debian/initrd.gz
systemctl restart fetch-debian-netboot.service
```

To update to a different Debian release, change `debianRelease` in
`modules/build-types/pxe-boot.nix` and redeploy.

The **FreeIPA Server (Rocky Linux 9)** entry chains
`http://<pxeServerIp>/rocky-freeipa.ipxe`, which boots the Rocky Linux 9
Anaconda installer with a Kickstart file (`rocky-freeipa.ks`) hosted on the
same server. The `fetch-rocky-pxeboot.service` oneshot downloads the pxeboot
kernel and initrd from the Rocky Linux mirror on first boot (idempotent):

```text
/srv/pxe/http/rocky/vmlinuz     (Rocky Linux 9 Anaconda pxeboot kernel)
/srv/pxe/http/rocky/initrd.img  (Rocky Linux 9 Anaconda pxeboot initrd)
```

The Kickstart file is generated from the NixOS module and staged at
`/srv/pxe/http/rocky-freeipa.ks`. It performs a fully unattended install:

1. Installs Rocky Linux 9 with `ipa-server` + `ipa-server-dns` packages
2. Configures static IP `192.168.2.138`, hostname `domain-controller.sweet.home`
3. Creates user `wayne` with the `adminSshKey` from `variables.nix`
4. Generates random IPA passwords and writes them to `/root/ipa-credentials.txt`
5. Creates a `freeipa-first-boot.service` oneshot that runs `ipa-server-install`
   on first reboot (~20 minutes)

After the install completes:
- SSH in as `wayne@domain-controller` using the admin key
- Monitor FreeIPA install progress: `sudo tail -f /root/freeipa-install.log`
- Retrieve credentials: `sudo cat /root/ipa-credentials.txt` (save to password manager)
- Configure Pi-hole: `server=/sweet.home/192.168.2.138` in dnsmasq

To refresh the pxeboot files (e.g. after a Rocky point release):

```bash
rm /srv/pxe/http/rocky/vmlinuz /srv/pxe/http/rocky/initrd.img
systemctl restart fetch-rocky-pxeboot.service
```

To update to a different Rocky release, change `rockyRelease` in
`modules/build-types/pxe-boot.nix` and redeploy.

The SystemRescue entry expects the source ISO at:

```text
/srv/pxe/http/images/systemrescue.iso
```

Since `/srv/pxe/http/images` is the NFS-backed symlink, place the ISO on the
NFS share at `server.sweet.home:/tank/pxe-boot/images/systemrescue.iso`.

The `stage-systemrescue.service` oneshot extracts that ISO into:

```text
/srv/pxe/http/systemrescue
```

The rescue menu entry then chains `http://192.168.2.223/systemrescue.ipxe`,
which loads the SystemRescue kernel and initramfs from the extracted tree and
uses `archiso_http_srv` to fetch the squashfs payload over HTTP.

## Validation

Safe evaluation check:

```bash
nix eval .#nixosConfigurations.proxmox-pxe-boot.config.system.build.toplevel.drvPath --raw
```

After deployment by an operator, basic service checks are:

```bash
curl http://pxe-boot/boot.ipxe
curl http://pxe-boot/menu.ipxe
curl http://pxe-boot/debian.ipxe
curl -I http://pxe-boot/debian/linux
curl -I http://pxe-boot/debian/initrd.gz
curl http://pxe-boot/rocky-freeipa.ipxe
curl http://pxe-boot/rocky-freeipa.ks
curl -I http://pxe-boot/rocky/vmlinuz
curl -I http://pxe-boot/rocky/initrd.img
curl http://pxe-boot/systemrescue.ipxe
curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/vmlinuz
curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/sysresccd.img
```

During a successful BIOS chainload, TFTP should deliver `undionly.kpxe` once,
then nginx should log requests for `/boot.ipxe` and `/menu.ipxe`. Repeated TFTP
downloads of `undionly.kpxe` indicate the iPXE stage is still not reaching the
HTTP chain.