Archived
beatzaplentyandClaude Sonnet 4.6
86e55ff954
- Correct the NFS path from /tank/proxmox/pxe-images to /tank/pxe-boot/images (matches variables.nix's proxmoxPxeImages.subpath) - Clarify that LXC uses NFSv3+nolock while VM uses NFSv4.2+automount - Add explicit note that lxc-pxe-boot needs features: nesting=1,mount=nfs and why: nesting=1 is required by systemd 260+ for userns/credential isolation (AppArmor denies userns_create without it), mount=nfs for NFSv3 access. pct set replaces the whole features string — include both or the container will fail to boot. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
173 lines
6.5 KiB
Markdown
173 lines
6.5 KiB
Markdown
# pxe-boot
|
|
|
|
The `pxe-boot` host serves HTTP boot assets for iPXE clients — including
|
|
self-staged copies of both this flake's own auto-installer netboot image
|
|
(see `docs/auto-installer.md` for what that image actually is and does once
|
|
booted) and a vanilla, unmodified NixOS minimal netboot image for plain
|
|
rescue/inspection use.
|
|
|
|
## Host Role
|
|
|
|
- Hostname: `pxe-boot`
|
|
- Web service: nginx on TCP port 80
|
|
- PXE root: `/srv/pxe`
|
|
- HTTP root for scripts and images: `/srv/pxe/http`
|
|
- TFTP root for first-stage bootloaders: `/srv/pxe/tftp`
|
|
- iPXE entry script: `/srv/pxe/http/boot.ipxe`
|
|
- Generated iPXE menu: `/srv/pxe/http/menu.ipxe`
|
|
- Debian Minimal iPXE script: `/srv/pxe/http/debian.ipxe`
|
|
- SystemRescue iPXE script: `/srv/pxe/http/systemrescue.ipxe`
|
|
- TFTP fallback script: `/srv/pxe/tftp/autoexec.ipxe`
|
|
- Boot binaries copied from the Nix `ipxe` package:
|
|
- `/srv/pxe/tftp/ipxe.efi`
|
|
- `/srv/pxe/tftp/undionly.kpxe`
|
|
|
|
## Directory Layout
|
|
|
|
The host creates these directories with systemd tmpfiles:
|
|
|
|
```text
|
|
/srv/pxe
|
|
/srv/pxe/http
|
|
/srv/pxe/http/images -> /mnt/pxe-images (symlink to NFS share)
|
|
/srv/pxe/http/auto-installer
|
|
/srv/pxe/http/nixos-minimal
|
|
/srv/pxe/http/debian
|
|
/srv/pxe/http/systemrescue
|
|
/srv/pxe/http/ubuntu
|
|
/srv/pxe/http/rescue
|
|
/srv/pxe/tftp
|
|
```
|
|
|
|
`/srv/pxe/http/images` is a symlink to `/mnt/pxe-images`, which is an NFS
|
|
mount of `server.sweet.home:/tank/pxe-boot/images`
|
|
(`modules/pxe-boot/mount-pxe-images.nix`). Place large images there (ISOs,
|
|
disk images) rather than on the pxe-boot host's own root disk. For an LXC
|
|
pxe-boot container the mount uses NFSv3+nolock with `nofail` (eager,
|
|
non-blocking on server unavailability); for a Proxmox VM it uses NFSv4.2
|
|
with `x-systemd.automount` (lazy, triggered on first access).
|
|
|
|
When running as `lxc-pxe-boot`, the Proxmox container must have
|
|
`features: nesting=1,mount=nfs` (at minimum) in its Proxmox config. `nesting=1`
|
|
is required by systemd 260+ for credential isolation (user namespace creation
|
|
and internal move-mounts); without it, AppArmor denies both, and every
|
|
systemd service that uses `PrivateUsers`, `PrivateDevices`, or credential
|
|
passing fails on boot. `mount=nfs` allows the NFSv3 mount. Both are set
|
|
automatically by `scripts/proxmox/create-proxmox-resource.sh` (via
|
|
`PROXMOX_DEFAULT_LXC_FEATURES` in `scripts/env.sh` which defaults to
|
|
`nesting=1,keyctl=1,mount=nfs;nfs4`). If you ever change these features
|
|
manually via `pct set`, be sure to include both — `pct set` replaces the
|
|
entire features string, it does not append to it.
|
|
|
|
The HTTP iPXE chain is:
|
|
|
|
```text
|
|
undionly.kpxe or ipxe.efi
|
|
-> autoexec.ipxe from the TFTP root, when iPXE requests it
|
|
-> http://192.168.2.247/boot.ipxe
|
|
-> http://192.168.2.247/menu.ipxe
|
|
```
|
|
|
|
The generated menu currently exposes entries for:
|
|
|
|
- NixOS Auto-Installer
|
|
- NixOS Minimal
|
|
- Debian Minimal
|
|
- SystemRescue environment
|
|
- iPXE shell
|
|
- Reboot
|
|
|
|
Both NixOS entries chain-load a `netboot.ipxe` staged into their own
|
|
directory (`/srv/pxe/http/auto-installer/netboot.ipxe` and
|
|
`/srv/pxe/http/nixos-minimal/netboot.ipxe`), each nixpkgs' own generated
|
|
netboot iPXE script (correct `init=`/`initrd=` kernel parameters included)
|
|
rather than a hand-rolled boot line — that script in turn expects its
|
|
kernel/initrd siblings in the same directory. Each directory's three files
|
|
(`bzImage`, `initrd`, `netboot.ipxe`) are built from source and staged
|
|
automatically by `modules/pxe-boot/stage-installer-artifacts.nix` via
|
|
`systemd.tmpfiles.rules` — no manual operator step required:
|
|
|
|
- `auto-installer` is this flake's own `netbootSystem` (`flake.nix`) — the
|
|
same auto-installer image `nix build .#pxe` produces. See
|
|
`docs/auto-installer.md`.
|
|
- `nixos-minimal` is `netbootMinimalSystem` (`flake.nix`) — nixpkgs'
|
|
`netboot-minimal.nix` composed on its own, with none of this flake's
|
|
auto-installer wiring (no `common.nix`, no `auto-install.sh`, no baked
|
|
host keys or custom users). Same `nix build .#pxe-minimal` mechanism as
|
|
the auto-installer image, just a different module composition. Useful
|
|
as a plain rescue/inspection shell that doesn't assume anything about
|
|
this flake.
|
|
|
|
Both images set `networking.hostName` to match their menu entry/staged
|
|
directory name (`auto-installer` / `nixos-minimal`), so each one's
|
|
generated system name (`nixos-system-<name>-*`) is self-describing rather
|
|
than the nixpkgs default of `nixos-system-nixos-*` for both.
|
|
|
|
The Debian Minimal entry chains `http://<pxeServerIp>/debian.ipxe`, which loads
|
|
the Debian bookworm netboot kernel and initrd from `/srv/pxe/http/debian/`. The
|
|
`fetch-debian-netboot.service` oneshot downloads these files from
|
|
`deb.debian.org` on first boot (idempotent — skips if files are already
|
|
present):
|
|
|
|
```text
|
|
/srv/pxe/http/debian/linux (Debian bookworm netboot kernel)
|
|
/srv/pxe/http/debian/initrd.gz (Debian bookworm netboot initrd)
|
|
```
|
|
|
|
The service requires outbound internet access on the pxe-boot host. To
|
|
re-download (e.g. after a Debian point release), delete the files and restart
|
|
the service:
|
|
|
|
```bash
|
|
rm /srv/pxe/http/debian/linux /srv/pxe/http/debian/initrd.gz
|
|
systemctl restart fetch-debian-netboot.service
|
|
```
|
|
|
|
To update to a different Debian release, change `debianRelease` in
|
|
`modules/build-types/pxe-boot.nix` and redeploy.
|
|
|
|
The SystemRescue entry expects the source ISO at:
|
|
|
|
```text
|
|
/srv/pxe/http/images/systemrescue.iso
|
|
```
|
|
|
|
Since `/srv/pxe/http/images` is the NFS-backed symlink, place the ISO on the
|
|
NFS share at `server.sweet.home:/tank/pxe-boot/images/systemrescue.iso`.
|
|
|
|
The `stage-systemrescue.service` oneshot extracts that ISO into:
|
|
|
|
```text
|
|
/srv/pxe/http/systemrescue
|
|
```
|
|
|
|
The rescue menu entry then chains `http://192.168.2.247/systemrescue.ipxe`,
|
|
which loads the SystemRescue kernel and initramfs from the extracted tree and
|
|
uses `archiso_http_srv` to fetch the squashfs payload over HTTP.
|
|
|
|
## Validation
|
|
|
|
Safe evaluation check:
|
|
|
|
```bash
|
|
nix eval .#nixosConfigurations.proxmox-pxe-boot.config.system.build.toplevel.drvPath --raw
|
|
```
|
|
|
|
After deployment by an operator, basic service checks are:
|
|
|
|
```bash
|
|
curl http://pxe-boot/boot.ipxe
|
|
curl http://pxe-boot/menu.ipxe
|
|
curl http://pxe-boot/debian.ipxe
|
|
curl -I http://pxe-boot/debian/linux
|
|
curl -I http://pxe-boot/debian/initrd.gz
|
|
curl http://pxe-boot/systemrescue.ipxe
|
|
curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/vmlinuz
|
|
curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/sysresccd.img
|
|
```
|
|
|
|
During a successful BIOS chainload, TFTP should deliver `undionly.kpxe` once,
|
|
then nginx should log requests for `/boot.ipxe` and `/menu.ipxe`. Repeated TFTP
|
|
downloads of `undionly.kpxe` indicate the iPXE stage is still not reaching the
|
|
HTTP chain.
|