From 274d54a774bbbc6cee5f6b8e6b120f3273ec2e09 Mon Sep 17 00:00:00 2001 From: beatzaplenty Date: Mon, 20 Jul 2026 11:33:53 +1000 Subject: [PATCH] Fix LXC container creation: unprivileged, nesting/keyctl, swap sizing Found and fixed live against a real test container (VMID 100, lxc-nix-cache on pve.sweet.home) after the previous pct-restore-to-pct-create fix still produced a container that booted into garbled console output: 1. pct create's own CLI default for --unprivileged is privileged (unlike the web UI, whose checkbox defaults the other way), but modules/platforms/lxc.nix sets proxmoxLXC.privileged = false, so the image assumes it's running unprivileged. Real mismatch -- now passes --unprivileged 1 explicitly. 2. The actual root cause of the garbled console: modern (v247+) systemd routinely uses nested user namespaces and credential mounts (even plain getty units, via LoadCredential=-style mechanisms), which AppArmor's default LXC confinement denies without --features nesting=1,keyctl=1. Confirmed via the host's kernel audit log: every getty unit was crash-looping on a denied /run/credentials/* mount every ~3s, and core services like nsncd failed userns_create the same way -- the system never finished activating. Fixed live (pct set + restart on the running test container) before committing the script change: systemctl is-system-running went from never completing to "running" with zero failed units. 3. --memory doesn't touch swap -- confirmed live it silently stayed at Proxmox's own 512M default with --memory 2048. Now defaults --swap to whatever --memory resolves to. docs/auto-installer.md's manual pct create walkthrough gets the same fixes, with the "why" for each flag, since a human following it by hand would hit the identical bugs. Co-Authored-By: Claude Sonnet 5 Claude-Session: https://claude.ai/code/session_01La55Nsss8jZ7ZuzUV9mfot --- docs/auto-installer.md | 26 ++++++++++++++++++++++++-- scripts/create-proxmox-resource.sh | 26 +++++++++++++++++++++++++- scripts/env.sh | 20 +++++++++++++++++++- 3 files changed, 68 insertions(+), 4 deletions(-) diff --git a/docs/auto-installer.md b/docs/auto-installer.md index cccc46b..19a4cb9 100644 --- a/docs/auto-installer.md +++ b/docs/auto-installer.md @@ -45,15 +45,37 @@ of its own: ```sh pct create local:vztmpl/.tar.xz \ - --rootfs local-lvm:8 --hostname --cores 2 --memory 2048 \ + --unprivileged 1 --features nesting=1,keyctl=1 \ + --rootfs local-lvm:8 --hostname --cores 2 --memory 2048 --swap 2048 \ --net0 name=eth0,bridge=vmbr0,ip=dhcp pct start ``` +Every one of those extra flags is load-bearing, confirmed by actually +booting one: + +- `--unprivileged 1` — `modules/platforms/lxc.nix` sets + `proxmoxLXC.privileged = false`, so the image assumes it's running + unprivileged. `pct create`'s own CLI default for this flag is + privileged (unlike the web UI, whose checkbox defaults the other way) + — omit it and you get a privileged container running a NixOS config + that assumes unprivileged, a real mismatch. +- `--features nesting=1,keyctl=1` — required for a modern (v247+) + systemd guest to boot unprivileged at all. Without it, AppArmor denies + the nested user namespaces and credential mounts systemd routinely + uses (even plain getty units) — every getty crash-loops on a denied + `/run/credentials/*` mount every ~3s (this is what garbage on the + console turns out to be) while core services like `nsncd` fail the + same way, and the system never finishes activating. +- `--swap 2048` — `--memory` doesn't touch swap; it silently stays at + Proxmox's own 512M default otherwise. Match it to `--memory` unless + you deliberately want otherwise. + First boot runs `boot.postBootCommands` (registers the Nix store DB and system profile) — there's no separate activation step to run yourself. `scripts/create-proxmox-resource.sh --type lxc --host ` automates all -of this (build, host-key handling, upload, `pct create`) — see its `--help`. +of this (build, host-key handling, upload, `pct create` with the flags +above) — see its `--help`. Host keys still need pre-seeding the same way as any other host (see "Host keys" below) — the sops-nix activation-vs-first-boot race is identical diff --git a/scripts/create-proxmox-resource.sh b/scripts/create-proxmox-resource.sh index 8a201dc..18c6171 100755 --- a/scripts/create-proxmox-resource.sh +++ b/scripts/create-proxmox-resource.sh @@ -73,6 +73,10 @@ Shared: modify: omit to leave unchanged. --memory create: default \$PROXMOX_DEFAULT_MEMORY_MB (${PROXMOX_DEFAULT_MEMORY_MB}). modify: omit to leave unchanged. + --swap lxc only, create time: \`--memory\` doesn't + touch swap -- it silently stays at Proxmox's + own 512M default otherwise. (default: matches + whatever --memory resolves to) --storage (default: \$PROXMOX_STORAGE, ${PROXMOX_STORAGE}) --iso-storage (default: \$PROXMOX_ISO_STORAGE, ${PROXMOX_ISO_STORAGE}) --bridge (default: \$PROXMOX_BRIDGE, ${PROXMOX_BRIDGE}) @@ -95,6 +99,7 @@ name="" vmid="" cores="" memory="" +swap="" disk_size="" grow_disk="" image="" @@ -114,6 +119,7 @@ while [[ $# -gt 0 ]]; do --vmid) vmid="$2"; shift 2 ;; --cores) cores="$2"; shift 2 ;; --memory) memory="$2"; shift 2 ;; + --swap) swap="$2"; shift 2 ;; --disk-size) disk_size="$2"; shift 2 ;; --grow-disk) grow_disk="$2"; shift 2 ;; --image) image="$2"; shift 2 ;; @@ -436,7 +442,25 @@ echo if [[ "$type" == "lxc" ]]; then echo "==> Creating LXC container ${vmid} (${name})..." local_disk_size="${disk_size:-$PROXMOX_DEFAULT_LXC_DISK_GB}" - create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --net0 name=eth0,bridge=${bridge},ip=dhcp" + # --memory doesn't touch swap -- it silently stays at Proxmox's own + # 512M default otherwise (confirmed live: --memory 2048 left swap at + # 512). Default to matching whatever --memory resolved to above. + local_swap="${swap:-$memory}" + # --unprivileged 1: modules/platforms/lxc.nix sets proxmoxLXC.privileged + # = false, so the NixOS config inside the image assumes it's running as + # an unprivileged container (cgroup/capability/mount expectations baked + # in at boot). `pct create`'s own CLI default for this flag is + # privileged (unlike the web UI, which defaults its checkbox the other + # way) -- leaving it unset creates a privileged container running a + # NixOS config that assumes unprivileged, a real mismatch. + # + # --features nesting=1,keyctl=1: required for a modern (v247+) systemd + # guest to actually boot unprivileged -- confirmed live: without this, + # AppArmor denies the nested user namespaces and credential mounts + # systemd routinely uses (even plain getty units), and every getty + # crash-loops on a denied mount every ~3s (visible as garbage on the + # console) while core services like nsncd fail the same way. + create_cmd="pct create ${vmid} ${iso_storage}:vztmpl/${remote_filename} --unprivileged 1 --features ${PROXMOX_DEFAULT_LXC_FEATURES} --rootfs ${storage}:${local_disk_size} --hostname ${name} --cores ${cores} --memory ${memory} --swap ${local_swap} --net0 name=eth0,bridge=${bridge},ip=dhcp" remote "$create_cmd" remote "pct start ${vmid}" else diff --git a/scripts/env.sh b/scripts/env.sh index b777d20..08456f9 100755 --- a/scripts/env.sh +++ b/scripts/env.sh @@ -30,8 +30,26 @@ # no backup metadata to infer it from. Matches Proxmox's own GUI default. : "${PROXMOX_DEFAULT_LXC_DISK_GB:=8}" +# `pct create --memory` only sets RAM -- swap is a wholly separate +# parameter that otherwise silently stays at Proxmox's own 512M default +# regardless of --memory (confirmed: creating with --memory 2048 left +# swap at 512). create-proxmox-resource.sh defaults --swap to whatever +# --memory resolves to at runtime rather than a static value here, so it +# tracks a --memory picked at the CLI too, not just the default above. + +# Required for a modern (v247+) systemd guest to actually boot as an +# unprivileged container: systemd's routine use of nested user namespaces +# and credential mounts (LoadCredential=, DynamicUser=, etc. -- used even +# by plain getty units) gets denied by AppArmor's default LXC confinement +# without these. Confirmed live: without them, every getty unit +# crash-loops on a denied `/run/credentials/*` mount every ~3s (visible +# as garbage on the console) and core services like nsncd fail the same +# way on userns_create; system.build.tarball never finishes activating. +: "${PROXMOX_DEFAULT_LXC_FEATURES:=nesting=1,keyctl=1}" + export PROXMOX_HOST PROXMOX_SSH_USER PROXMOX_STORAGE PROXMOX_ISO_STORAGE \ - PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES PROXMOX_DEFAULT_MEMORY_MB PROXMOX_DEFAULT_LXC_DISK_GB + PROXMOX_BRIDGE PROXMOX_DEFAULT_CORES PROXMOX_DEFAULT_MEMORY_MB \ + PROXMOX_DEFAULT_LXC_DISK_GB PROXMOX_DEFAULT_LXC_FEATURES # Matches variables.nix's nixCacheHost -- update both if it ever changes. : "${NIX_CACHE_HOST:=nix-cache}"