beatzaplenty/nixos
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
This repository has been archived on 2026-07-30. You can view files and clone it. You cannot open issues or pull requests or push a commit.
Files
479444d26aca968df9dce048bc3d36fba966f0f8
nixos/docs/pxe-boot.md
T
beatzaplentyandClaude Sonnet 4.6 86e55ff954 docs(pxe-boot): fix NFS path, clarify NFSv3/NFSv4 split, document nesting=1 requirement 
- Correct the NFS path from /tank/proxmox/pxe-images to /tank/pxe-boot/images
  (matches variables.nix's proxmoxPxeImages.subpath)
- Clarify that LXC uses NFSv3+nolock while VM uses NFSv4.2+automount
- Add explicit note that lxc-pxe-boot needs features: nesting=1,mount=nfs and
  why: nesting=1 is required by systemd 260+ for userns/credential isolation
  (AppArmor denies userns_create without it), mount=nfs for NFSv3 access.
  pct set replaces the whole features string — include both or the container
  will fail to boot.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-27 05:13:21 +10:00

6.5 KiB
Raw Blame History

pxe-boot

The pxe-boot host serves HTTP boot assets for iPXE clients — including self-staged copies of both this flake's own auto-installer netboot image (see docs/auto-installer.md for what that image actually is and does once booted) and a vanilla, unmodified NixOS minimal netboot image for plain rescue/inspection use.

Host Role

  • Hostname: pxe-boot
  • Web service: nginx on TCP port 80
  • PXE root: /srv/pxe
  • HTTP root for scripts and images: /srv/pxe/http
  • TFTP root for first-stage bootloaders: /srv/pxe/tftp
  • iPXE entry script: /srv/pxe/http/boot.ipxe
  • Generated iPXE menu: /srv/pxe/http/menu.ipxe
  • Debian Minimal iPXE script: /srv/pxe/http/debian.ipxe
  • SystemRescue iPXE script: /srv/pxe/http/systemrescue.ipxe
  • TFTP fallback script: /srv/pxe/tftp/autoexec.ipxe
  • Boot binaries copied from the Nix ipxe package:
    • /srv/pxe/tftp/ipxe.efi
    • /srv/pxe/tftp/undionly.kpxe

Directory Layout

The host creates these directories with systemd tmpfiles:

/srv/pxe
/srv/pxe/http
/srv/pxe/http/images  -> /mnt/pxe-images  (symlink to NFS share)
/srv/pxe/http/auto-installer
/srv/pxe/http/nixos-minimal
/srv/pxe/http/debian
/srv/pxe/http/systemrescue
/srv/pxe/http/ubuntu
/srv/pxe/http/rescue
/srv/pxe/tftp

/srv/pxe/http/images is a symlink to /mnt/pxe-images, which is an NFS mount of server.sweet.home:/tank/pxe-boot/images (modules/pxe-boot/mount-pxe-images.nix). Place large images there (ISOs, disk images) rather than on the pxe-boot host's own root disk. For an LXC pxe-boot container the mount uses NFSv3+nolock with nofail (eager, non-blocking on server unavailability); for a Proxmox VM it uses NFSv4.2 with x-systemd.automount (lazy, triggered on first access).

When running as lxc-pxe-boot, the Proxmox container must have features: nesting=1,mount=nfs (at minimum) in its Proxmox config. nesting=1 is required by systemd 260+ for credential isolation (user namespace creation and internal move-mounts); without it, AppArmor denies both, and every systemd service that uses PrivateUsers, PrivateDevices, or credential passing fails on boot. mount=nfs allows the NFSv3 mount. Both are set automatically by scripts/proxmox/create-proxmox-resource.sh (via PROXMOX_DEFAULT_LXC_FEATURES in scripts/env.sh which defaults to nesting=1,keyctl=1,mount=nfs;nfs4). If you ever change these features manually via pct set, be sure to include both — pct set replaces the entire features string, it does not append to it.

The HTTP iPXE chain is:

undionly.kpxe or ipxe.efi
  -> autoexec.ipxe from the TFTP root, when iPXE requests it
  -> http://192.168.2.247/boot.ipxe
  -> http://192.168.2.247/menu.ipxe

The generated menu currently exposes entries for:

  • NixOS Auto-Installer
  • NixOS Minimal
  • Debian Minimal
  • SystemRescue environment
  • iPXE shell
  • Reboot

Both NixOS entries chain-load a netboot.ipxe staged into their own directory (/srv/pxe/http/auto-installer/netboot.ipxe and /srv/pxe/http/nixos-minimal/netboot.ipxe), each nixpkgs' own generated netboot iPXE script (correct init=/initrd= kernel parameters included) rather than a hand-rolled boot line — that script in turn expects its kernel/initrd siblings in the same directory. Each directory's three files (bzImage, initrd, netboot.ipxe) are built from source and staged automatically by modules/pxe-boot/stage-installer-artifacts.nix via systemd.tmpfiles.rules — no manual operator step required:

  • auto-installer is this flake's own netbootSystem (flake.nix) — the same auto-installer image nix build .#pxe produces. See docs/auto-installer.md.
  • nixos-minimal is netbootMinimalSystem (flake.nix) — nixpkgs' netboot-minimal.nix composed on its own, with none of this flake's auto-installer wiring (no common.nix, no auto-install.sh, no baked host keys or custom users). Same nix build .#pxe-minimal mechanism as the auto-installer image, just a different module composition. Useful as a plain rescue/inspection shell that doesn't assume anything about this flake.

Both images set networking.hostName to match their menu entry/staged directory name (auto-installer / nixos-minimal), so each one's generated system name (nixos-system-<name>-*) is self-describing rather than the nixpkgs default of nixos-system-nixos-* for both.

The Debian Minimal entry chains http://<pxeServerIp>/debian.ipxe, which loads the Debian bookworm netboot kernel and initrd from /srv/pxe/http/debian/. The fetch-debian-netboot.service oneshot downloads these files from deb.debian.org on first boot (idempotent — skips if files are already present):

/srv/pxe/http/debian/linux      (Debian bookworm netboot kernel)
/srv/pxe/http/debian/initrd.gz  (Debian bookworm netboot initrd)

The service requires outbound internet access on the pxe-boot host. To re-download (e.g. after a Debian point release), delete the files and restart the service:

rm /srv/pxe/http/debian/linux /srv/pxe/http/debian/initrd.gz
systemctl restart fetch-debian-netboot.service

To update to a different Debian release, change debianRelease in modules/build-types/pxe-boot.nix and redeploy.

The SystemRescue entry expects the source ISO at:

/srv/pxe/http/images/systemrescue.iso

Since /srv/pxe/http/images is the NFS-backed symlink, place the ISO on the NFS share at server.sweet.home:/tank/pxe-boot/images/systemrescue.iso.

The stage-systemrescue.service oneshot extracts that ISO into:

/srv/pxe/http/systemrescue

The rescue menu entry then chains http://192.168.2.247/systemrescue.ipxe, which loads the SystemRescue kernel and initramfs from the extracted tree and uses archiso_http_srv to fetch the squashfs payload over HTTP.

Validation

Safe evaluation check:

nix eval .#nixosConfigurations.proxmox-pxe-boot.config.system.build.toplevel.drvPath --raw

After deployment by an operator, basic service checks are:

curl http://pxe-boot/boot.ipxe
curl http://pxe-boot/menu.ipxe
curl http://pxe-boot/debian.ipxe
curl -I http://pxe-boot/debian/linux
curl -I http://pxe-boot/debian/initrd.gz
curl http://pxe-boot/systemrescue.ipxe
curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/vmlinuz
curl -I http://pxe-boot/systemrescue/sysresccd/boot/x86_64/sysresccd.img

During a successful BIOS chainload, TFTP should deliver undionly.kpxe once, then nginx should log requests for /boot.ipxe and /menu.ipxe. Repeated TFTP downloads of undionly.kpxe indicate the iPXE stage is still not reaching the HTTP chain.

Reference in New Issue View Git Blame Copy Permalink