Merge pull request 'Stage 1: base Proxmox config/hardening toolset' (#1) from worktree-dapper-hopping-ritchie into main

Reviewed-on: beatzaplenty/proxmox-configuration#1
2026-07-21 08:53:29 +00:00
parent 24a0047fa8 2cca6a7dc0
commit 8c5b76cb15
17 changed files with 625 additions and 97 deletions
@@ -1,30 +1,48 @@
 # Proxmox Configuration

-Config, planning docs, and scripts for rebuilding my Proxmox VE homelab on new
-hardware, with the end goal of a 3-node HA cluster.
+Base configuration and hardening toolset for Proxmox VE hosts, plus planning
+docs for eventually growing this into a 3-node HA/Ceph cluster. See
+`docs/00-overview.md` for the staging: **Stage 1** (base config/hardening,
+applies to any host — active) vs. **Stage 2** (multi-node HA/Ceph — future,
+deferred).

 ## Goals

- Fresh install on new hardware (old box has a KVM/virtualization bug that
-  forced disabling hardware-accelerated virtualization — not expected to
-  recur on new hardware; verify VT-x/AMD-V + IOMMU in BIOS before assuming
-  otherwise).
- Move off LVM-thin to ZFS for local (boot + pre-cluster) storage.
- Build node 1 so its disk and network layout doesn't need rework when nodes
-  2 and 3 are added later.
- End state: 3-node cluster, quorum via corosync, HA-managed VMs backed by
-  Ceph. No intermediate ZFS-replication HA step — Ceph goes live as soon as
-  node 2/3 join.
+- Stage 1: a reusable, idempotent base-hardening toolset (`scripts/`) that
+  can be run against any new Proxmox host — repo/updates, SSH, firewall, PVE
+  user/access hardening — verified with `scripts/audit.sh`.
+- Stage 2 (future): 3-node cluster, quorum via corosync, HA-managed VMs
+  backed by Ceph. Needs dedicated hardware node 1 (`pve1`, an ASUS PN53 mini
+  PC) doesn't have — see `docs/01-hardware-node1.md`.

 ## Repo layout

 - `docs/` — planning docs: hardware layout, storage migration, networking,
  security hardening. Read `docs/00-overview.md` first.
- `scripts/` — scripts to apply configuration on a node (SSH hardening,
-  repo switch, etc.). Idempotent, safe to re-run.
+- `scripts/` — scripts to apply configuration on a node (SSH hardening, repo
+  switch, firewall, updates, etc). Idempotent, safe to re-run.
+  `scripts/bootstrap.sh` runs the full Stage 1 sequence end to end;
+  `scripts/audit.sh` verifies it (read-only). `scripts/lib/` holds shared
+  helpers (`common.sh`) sourced by the other scripts.
 - `config/` — reference config files/snippets to drop onto a node (firewall
  rules, sshd config, etc.).

+## Quick start (Stage 1, on a fresh node)
+
+```
+MGMT_CIDR=192.168.2.0/24 ./scripts/bootstrap.sh
+./scripts/create-admin-user.sh <username>
+# then enable 2FA for that user + root@pam via the web UI
+./scripts/audit.sh
+```
+
 ## Status

-Planning stage — node 1 not yet built.
+`pve1` built (ASUS PN53 mini PC, ZFS mirror boot+VM storage, single
+2.5GbE NIC) and already running production VMs/CTs. Stage 1 base
+hardening applied and verified (`scripts/audit.sh` all green): enterprise
+repos removed, SSH key-only + fail2ban, unattended security upgrades,
+PVE firewall (mgmt-only), subscription nag disabled, named admin user
+(`wayne@pve`) created. Remaining manual step: enable 2FA/TOTP for
+`wayne@pve` and `root@pam` via the web UI. Stage 2 (cluster/Ceph) not
+started — needs nodes 2/3 on hardware that can actually support it.
@@ -1,11 +1,17 @@
 # Example cluster-wide firewall rules for /etc/pve/firewall/cluster.fw
 #
-# Placeholders to fill in once the network is built:
-#   <MGMT_CIDR>    - management VLAN/subnet, e.g. 192.168.10.0/24
-#   <COROSYNC_CIDR> - corosync VLAN/subnet
-#   <CEPH_CIDR>    - Ceph public + backend VLAN/subnet (once Ceph is live)
+# Stage 1 (single host, current): only the mgmt IPSET applies. Applied
+# automatically by scripts/deploy-firewall.sh, which fills in <MGMT_CIDR>.
 #
-# Copy to /etc/pve/firewall/cluster.fw and edit before enabling.
+# Stage 2 (future cluster/Ceph): the corosync and Ceph rules below are
+# commented out placeholders. Uncomment and fill in <COROSYNC_CIDR> /
+# <CEPH_CIDR> when nodes 2/3 join and those networks actually exist -
+# leaving them active on a single node with no corosync/Ceph traffic is
+# just dead config, and a literal `<COROSYNC_CIDR>` is invalid syntax if
+# left uncommented and unfilled.
+#
+# Copy to /etc/pve/firewall/cluster.fw and edit before enabling (or use
+# scripts/deploy-firewall.sh).

 [OPTIONS]
 enable: 1
@@ -20,12 +26,12 @@ policy_out: ACCEPT
 IN ACCEPT -source +mgmt -p tcp -dport 8006 -log nolog
 IN ACCEPT -source +mgmt -p tcp -dport 22 -log nolog

-# Corosync (cluster quorum) - nodes only, restrict source to the
-# corosync subnet once it's provisioned
-IN ACCEPT -source <COROSYNC_CIDR> -p udp -dport 5404:5405 -log nolog
+# Stage 2: Corosync (cluster quorum) - uncomment once node 2/3 join and
+# the corosync network/VLAN exists.
+# IN ACCEPT -source <COROSYNC_CIDR> -p udp -dport 5404:5405 -log nolog

-# Ceph (uncomment once Ceph is live; ports: mon 3300,6789, osd/mgr/mds
-# 6800-7300)
+# Stage 2: Ceph (uncomment once Ceph is live; ports: mon 3300,6789,
+# osd/mgr/mds 6800-7300)
 # IN ACCEPT -source <CEPH_CIDR> -p tcp -dport 3300 -log nolog
 # IN ACCEPT -source <CEPH_CIDR> -p tcp -dport 6789 -log nolog
 # IN ACCEPT -source <CEPH_CIDR> -p tcp -dport 6800:7300 -log nolog
@@ -1,18 +1,65 @@
 # Overview & Roadmap

+## Staging
+
+This repo now targets two distinct stages, in order:
+
+- **Stage 1 (active)** — base configuration and hardening for a single
+  Proxmox host, applicable to *any* node regardless of eventual cluster
+  plans: repo/updates, SSH, firewall, PVE user/access hardening. This is
+  what `scripts/bootstrap.sh`, `scripts/audit.sh`, and
+  `04-security-hardening.md` cover, and what's being built out against
+  node 1 (`pve1`, an ASUS PN53 mini PC) right now.
+- **Stage 2 (future)** — the multi-node HA/Ceph cluster described below
+  and in `01-hardware-node1.md` / `02-storage-zfs-ceph.md` /
+  `03-networking.md`. Deliberately deferred: `pve1`'s hardware (2 NVMe
+  already merged into one ZFS mirror used for both boot and VM storage, a
+  single 2.5GbE NIC) can't support the separate boot/Ceph disks or
+  bonded/segregated networking those docs assume. Revisit once dedicated
+  cluster hardware (nodes 2/3) is actually being bought and provisioned;
+  until then treat the content below as a target design, not a
+  description of `pve1`.
+
 ## Background

-Current hardware requires disabling KVM hardware virtualization for VMs to
+Old hardware required disabling KVM hardware virtualization for VMs to
 start at all (falls back to software emulation — slow). This is a
-host/BIOS-level issue, not a Proxmox limitation, and shouldn't be needed on
-new hardware. Before reusing that workaround on the new box:
+host/BIOS-level issue, not a Proxmox limitation. Confirmed not present on
+`pve1` (ASUS PN53, Ryzen 7 7735HS): AMD-V and IOMMU both show enabled at
+boot (`dmesg | grep -i iommu`), no workaround needed. If this hardware line
+is reused for nodes 2/3, this should hold there too, but re-verify per
+node before assuming it.

- Confirm VT-x (Intel) / AMD-V (AMD) is enabled in BIOS/UEFI.
- Confirm IOMMU is enabled if passthrough is planned.
- Update BIOS/microcode first.
- Rule out running Proxmox nested inside another hypervisor.
+## Stage 1: base config & hardening (active)

-## End goal
+Applies to `pve1` now, and to every future node regardless of whether it
+ever joins the Stage 2 cluster. Covered by `04-security-hardening.md` and
+`scripts/bootstrap.sh` / `scripts/audit.sh`:
+
+1. Fresh PVE install; confirm VT-x/AMD-V + IOMMU per the note above.
+2. Switch off the enterprise repos, onto no-subscription
+   (`scripts/switch-to-no-subscription-repo.sh`).
+3. SSH hardening: key-only root login + fail2ban
+   (`scripts/harden-ssh.sh`).
+4. Unattended security upgrades, no auto-reboot
+   (`scripts/setup-unattended-upgrades.sh`).
+5. PVE datacenter firewall, default-deny, mgmt-only SSH/8006
+   (`scripts/deploy-firewall.sh`).
+6. Named PVE admin user (Administrator role) + 2FA, `root@pam` reserved
+   for emergencies (`scripts/create-admin-user.sh`, then manual TOTP
+   enrollment via the web UI).
+7. Verify with `scripts/audit.sh`.
+
+`pve1`'s actual disk/network layout (single ZFS mirror for boot + VMs, one
+2.5GbE NIC) is documented as-is in `01-hardware-node1.md` — Stage 1 doesn't
+require or assume the split-disk/multi-NIC layout Stage 2 wants.
+
+## Stage 2: HA cluster + Ceph (future, deferred)
+
+Everything below this point is the target design for when nodes 2 and 3
+are actually being provisioned. Not applicable to `pve1` as it stands.
+
+### End goal

 3-node Proxmox VE cluster with HA-managed VMs backed by **Ceph** — true
 distributed shared storage, sync replication, near-zero RPO on failover
@@ -25,7 +72,7 @@ goes live — as soon as nodes 2/3 join, VMs move onto Ceph rather than
 adopting ZFS replication as a stopgap. Simpler end state, one storage
 model to operate instead of two.

-## Cluster fundamentals (apply from node 1 onward)
+### Cluster fundamentals (apply from node 1 onward)

 - 3 nodes minimum for real quorum. If starting with 2, add a QDevice
  (small VM or Raspberry Pi) as tie-breaker.
@@ -38,16 +85,19 @@ model to operate instead of two.
  rather than `host` if nodes will ever have different CPUs — needed for
  clean live migration.

-## Rollout sequence
+### Rollout sequence (Stage 2, once dedicated cluster hardware exists)

-1. Build node 1 per `01-hardware-node1.md` — fresh PVE install on ZFS boot
-   mirror, Ceph-earmarked disks left idle (or as a temporary local ZFS
-   pool, to be wiped later — see `02-storage-zfs-ceph.md`).
-2. Migrate VMs off old hardware via `vzdump` → copy backups → `qmrestore`
-   onto the new ZFS storage (converts disks to ZVOLs). No HA yet — single
-   node.
-3. Apply hardening (`04-security-hardening.md`) and networking
-   (`03-networking.md`) before exposing the node beyond the LAN.
+1. Build node 1 per `01-hardware-node1.md`'s target design — fresh PVE
+   install on a dedicated ZFS boot mirror, Ceph-earmarked disks left idle
+   (or as a temporary local ZFS pool, to be wiped later — see
+   `02-storage-zfs-ceph.md`). Note: this assumes hardware with enough
+   disks/NICs to separate boot, Ceph, and network roles — `pve1` does not
+   have this and stays on Stage 1 only unless rebuilt on different
+   hardware.
+2. Migrate VMs onto the new node via `vzdump` → copy backups →
+   `qmrestore` (converts disks to ZVOLs). No HA yet — single node.
+3. Stage 1 base hardening already applied; layer on Stage 2 networking
+   (`03-networking.md`) before joining a cluster.
 4. Add nodes 2 and 3 identically (same disk/network layout).
 5. Join cluster, stand up dedicated corosync network.
 6. Wipe the Ceph-earmarked disks (if used as temporary ZFS) and
@@ -1,5 +1,31 @@
 # Node 1 Hardware Layout

+## pve1 as built (current reality)
+
+`pve1` is an ASUS PN53 mini PC (Ryzen 7 7735HS, 32GB RAM), not the
+dedicated-server hardware the target design below assumes:
+
+- **Disks**: 2x 2TB NVMe (Crucial CT2000E100SSD8), both in a single ZFS
+  mirror (`rpool`) that serves as both the boot pool and VM storage
+  (`local-zfs` = `rpool/data`). No spare disks to earmark for Ceph — the
+  chassis only has 2 NVMe slots.
+- **Network**: one physical NIC (Realtek RTL8125, 2.5GbE), bridged as
+  `vmbr0`. No second NIC for a dedicated corosync/Ceph link. (An unused
+  `nic1` stanza in `/etc/network/interfaces` is a leftover from the
+  installer template — there is no second NIC on this hardware.)
+
+This is sufficient and correct for **Stage 1** (see `00-overview.md`) —
+base config and hardening don't need split disks or multiple NICs. It is
+*not* sufficient for **Stage 2** (Ceph/HA) as designed below without
+either different hardware or a materially different plan (e.g.
+USB/Thunderbolt-attached OSD storage, which trades away the
+enterprise-SSD/PLP guidance below — not recommended, revisit when
+actually provisioning nodes 2/3). Treat everything from here down as the
+Stage 2 target design for purpose-built hardware, not a description of
+`pve1`.
+
+## Target design (Stage 2, future dedicated hardware)
+
 Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect
 disks or network when the cluster grows.

@@ -1,5 +1,10 @@
 # Storage: LVM-thin → ZFS → Ceph

+**Stage 2 (future).** Describes the target storage model once dedicated
+cluster hardware exists. `pve1`'s current single ZFS mirror (boot + VM
+storage combined, see `01-hardware-node1.md`) is the Stage 1 end state for
+now, not an intermediate step being actively migrated from.
+
 ## Why move off LVM-thin

 Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs
@@ -1,5 +1,18 @@
 # Networking

+## pve1 as built (Stage 1, current)
+
+Single NIC (2.5GbE), single bridge `vmbr0` on the flat LAN
+(`192.168.2.0/24`), no VLANs. There is no corosync or Ceph traffic to
+separate yet — this node isn't clustered. Segmentation for Stage 1 is
+done at the firewall, not the network: `scripts/deploy-firewall.sh`
+restricts SSH (22) and the web UI (8006) to the management CIDR via the
+PVE datacenter firewall (default-deny inbound otherwise). That's
+sufficient until Stage 2 needs actual separate physical/VLAN paths for
+corosync and Ceph traffic — see below.
+
+## Target design (Stage 2, future cluster)
+
 ## Required separation

 Keep these on logically separate networks/VLANs, ideally separate NICs:
@@ -1,14 +1,33 @@
 # Security Hardening

-Proxmox has no `sudo` out of the box — everything defaults to root. That's
-the install default, not the recommended end state. Two layers to harden
-separately.
+Stage 1 (see `00-overview.md`) — applies to any Proxmox host, independent
+of cluster plans. Proxmox has no `sudo` out of the box — everything
+defaults to root. That's the install default, not the recommended end
+state. Two layers to harden separately.
+
+## Checklist / script mapping
+
+Run `scripts/bootstrap.sh` for everything except the admin user (needs a
+username decision) and 2FA enrollment (must be done interactively via the
+web UI — there's no safe way to script TOTP secret generation over SSH).
+Then run `scripts/audit.sh` to verify. Order matters (matches
+`bootstrap.sh`):
+
+| # | Item | Script | Manual step required? |
+|---|------|--------|------------------------|
+| 1 | Remove enterprise repos, switch to no-subscription | `switch-to-no-subscription-repo.sh` | no |
+| 2 | SSH: key-only root login + fail2ban | `harden-ssh.sh` | no (requires an `authorized_keys` already in place — script warns if missing) |
+| 3 | Unattended security upgrades, no auto-reboot | `setup-unattended-upgrades.sh` | no |
+| 4 | PVE firewall, default-deny, mgmt-only SSH/8006 | `deploy-firewall.sh` | needs `MGMT_CIDR` set |
+| 5 | Disable subscription nag (cosmetic) | `disable-subscription-nag.sh` | no |
+| 6 | Named PVE admin user, Administrator role | `create-admin-user.sh <username>` | yes — pick the username, change the generated password on first login |
+| 7 | 2FA/TOTP on that user and `root@pam` | — | yes — web UI only: Datacenter → Permissions → Two Factor, or user menu → TFA |
+| 8 | Verify everything above | `audit.sh` | no |

 ## Linux/SSH layer

 - `PermitRootLogin prohibit-password` in `sshd_config` — root can only
  log in via SSH key, never password. Kills most brute-force attempts.
-  See `scripts/harden-ssh.sh`.
 - fail2ban jail for SSH on top of that.
 - Restrict SSH to the management VLAN/trusted IPs via the Proxmox
  firewall (see `03-networking.md`) rather than exposing broadly.
@@ -20,7 +39,8 @@ separately.

 - Keep `root@pam` for emergencies only.
 - Create a named user (e.g. `wayne@pve`) with the Administrator role for
-  routine cluster management: Datacenter → Permissions → Users.
+  routine cluster management — `create-admin-user.sh` does this, or
+  Datacenter → Permissions → Users manually.
 - Enable 2FA (TOTP or hardware key) on both that account and `root@pam`:
  Datacenter → Permissions → Realms/Users.
 - For API integrations (monitoring, automation, Terraform, etc.), issue
@@ -30,15 +50,29 @@ separately.
 ## Firewall

 Default-deny at datacenter/node level, whitelist only what's needed (see
-`03-networking.md` for the specifics). Config templates in
-`config/pve-firewall/`.
+`03-networking.md` for the specifics). Template in
+`config/pve-firewall/cluster.fw.example`, applied by
+`scripts/deploy-firewall.sh`.

 ## Repos and updates

 Fresh installs point at the enterprise repo, which fails on `apt update`
-without a subscription. Switch to the no-subscription repo (or pay for
-enterprise). See `scripts/switch-to-no-subscription-repo.sh`. Keep the
-host patched — hypervisor CVEs are high-value targets.
+without a subscription. `scripts/switch-to-no-subscription-repo.sh`
+removes the enterprise sources entirely (renamed `.disabled`, not just
+commented out) and switches to the no-subscription repo — handles both
+the legacy `.list` format and the deb822 `.sources` format current
+installers write. Keep the host patched — hypervisor CVEs are high-value
+targets; `scripts/setup-unattended-upgrades.sh` automates security
+patches (deliberately no auto-reboot on a hypervisor — check
+`/var/run/reboot-required` and reboot during a planned window).
+
+The web UI's "No valid subscription" popup and dashboard indicator are
+cosmetic upsell, not a security control, but with no subscription they'll
+nag on every login — `scripts/disable-subscription-nag.sh` patches
+`proxmox-widget-toolkit`'s JS to suppress them, and installs an apt
+`Post-Invoke` hook that reapplies the patch automatically after every
+`apt`/`dpkg` run, since a `proxmox-widget-toolkit` package upgrade
+overwrites the patched file.

 ## Misc

@@ -47,7 +81,7 @@ host patched — hypervisor CVEs are high-value targets.
 - If the web UI is ever needed outside the LAN, put it behind a VPN —
  don't port-forward 8006 directly.

-## Further reading
+## Further reading / not yet automated here

 - CIS Benchmark for Proxmox VE
 - Community PVE hardening guides (kernel parameters, audit logging,
@@ -0,0 +1,100 @@
+#!/bin/bash
+# Read-only Stage 1 base-hardening audit. Checks the current state of a PVE
+# host against the checklist in docs/04-security-hardening.md and prints
+# PASS/FAIL per item. Exits non-zero if anything fails, so it can gate CI or
+# be run periodically as a compliance check. Makes no changes.
+#
+# Usage: ./audit.sh   (run as root on the PVE host)
+set -uo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=lib/common.sh
+source "${SCRIPT_DIR}/lib/common.sh"
+require_root
+
+AUDIT_FAIL=0
+
+# --- apt repos: no enabled enterprise source ---
+ENTERPRISE_ENABLED=0
+for f in /etc/apt/sources.list.d/*.sources /etc/apt/sources.list.d/*.list; do
+    [ -f "$f" ] || continue
+    grep -qi 'enterprise.proxmox.com' "$f" 2>/dev/null && ENTERPRISE_ENABLED=1
+done
+if [ "$ENTERPRISE_ENABLED" -eq 0 ]; then
+    audit_pass "no enabled enterprise apt repo"
+else
+    audit_fail "an enterprise apt repo is still enabled (needs a subscription to update)"
+fi
+
+# --- SSH ---
+SSHD_T="$(sshd -T 2>/dev/null)"
+if echo "$SSHD_T" | grep -qiE '^permitrootlogin (prohibit-password|without-password)'; then
+    audit_pass "sshd: PermitRootLogin prohibit-password (key-only)"
+else
+    audit_fail "sshd: PermitRootLogin is not key-only (prohibit-password/without-password)"
+fi
+if echo "$SSHD_T" | grep -qi '^passwordauthentication no'; then
+    audit_pass "sshd: PasswordAuthentication no"
+else
+    audit_fail "sshd: PasswordAuthentication is not disabled"
+fi
+
+# --- fail2ban ---
+if systemctl is-active --quiet fail2ban 2>/dev/null; then
+    audit_pass "fail2ban is active"
+else
+    audit_fail "fail2ban is not active"
+fi
+
+# --- PVE firewall ---
+FW_STATUS="$(pve-firewall status 2>/dev/null || true)"
+if echo "$FW_STATUS" | grep -qi '^Status: enabled'; then
+    audit_pass "pve-firewall is enabled"
+else
+    audit_fail "pve-firewall is not enabled (status: ${FW_STATUS:-unknown})"
+fi
+if [ -f /etc/pve/firewall/cluster.fw ] && grep -qi '^policy_in:\s*DROP' /etc/pve/firewall/cluster.fw 2>/dev/null; then
+    audit_pass "cluster.fw has default-deny inbound policy"
+else
+    audit_fail "cluster.fw missing or does not default-deny inbound"
+fi
+
+# --- unattended-upgrades ---
+if dpkg -s unattended-upgrades >/dev/null 2>&1 && systemctl is-enabled --quiet unattended-upgrades 2>/dev/null; then
+    audit_pass "unattended-upgrades installed and enabled"
+else
+    audit_fail "unattended-upgrades not installed/enabled"
+fi
+if [ -f /var/run/reboot-required ]; then
+    audit_warn "a reboot is pending (/var/run/reboot-required) - schedule one"
+fi
+
+# --- named admin user (not just root@pam) ---
+if pveum user list --output-format json 2>/dev/null | grep -q '"userid":"[^"]*@pve"'; then
+    audit_pass "a named @pve admin user exists (root@pam is not the only account)"
+else
+    audit_fail "no named @pve user found - root@pam is the only account"
+fi
+
+# --- time sync ---
+if timedatectl show -p NTPSynchronized --value 2>/dev/null | grep -qx 'yes'; then
+    audit_pass "clock is NTP-synchronized"
+else
+    audit_fail "clock is not NTP-synchronized"
+fi
+
+# --- subscription nag (cosmetic - warn only, never fails the audit) ---
+JS_FILE="/usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js"
+if [ -f "$JS_FILE" ] && ! grep -qF "data.status.toLowerCase() !== 'active'" "$JS_FILE"; then
+    audit_pass "subscription nag patch applied"
+else
+    audit_warn "subscription nag patch not applied (cosmetic only, see scripts/disable-subscription-nag.sh)"
+fi
+
+echo
+if [ "$AUDIT_FAIL" -eq 0 ]; then
+    echo "All Stage 1 base-hardening checks passed."
+else
+    echo "One or more checks failed - see FAIL lines above."
+fi
+exit "$AUDIT_FAIL"
@@ -0,0 +1,45 @@
+#!/bin/bash
+# Stage 1 base config + hardening, end to end, for a single fresh PVE host.
+# Runs the individual scripts in order. Idempotent - safe to re-run.
+#
+# Does NOT create the named admin user (needs a username decision) - run
+# create-admin-user.sh separately afterwards. Run audit.sh at the end to
+# verify.
+#
+# Usage: MGMT_CIDR=192.168.2.0/24 ./bootstrap.sh
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=lib/common.sh
+source "${SCRIPT_DIR}/lib/common.sh"
+require_root
+
+if [ -z "${MGMT_CIDR:-}" ]; then
+    echo "MGMT_CIDR is not set. Example: MGMT_CIDR=192.168.2.0/24 $0" >&2
+    exit 1
+fi
+
+echo "=== 1/5: remove enterprise repos, switch to no-subscription ==="
+"${SCRIPT_DIR}/switch-to-no-subscription-repo.sh"
+
+echo
+echo "=== 2/5: SSH hardening (key-only root login + fail2ban) ==="
+"${SCRIPT_DIR}/harden-ssh.sh"
+
+echo
+echo "=== 3/5: unattended security upgrades ==="
+"${SCRIPT_DIR}/setup-unattended-upgrades.sh"
+
+echo
+echo "=== 4/5: PVE firewall (mgmt-only SSH/8006) ==="
+MGMT_CIDR="$MGMT_CIDR" "${SCRIPT_DIR}/deploy-firewall.sh"
+
+echo
+echo "=== 5/5: disable subscription nag (cosmetic) ==="
+"${SCRIPT_DIR}/disable-subscription-nag.sh"
+
+echo
+echo "=== Base hardening applied. Remaining manual/deliberate steps: ==="
+echo "  - ${SCRIPT_DIR}/create-admin-user.sh <username>"
+echo "  - Enable 2FA/TOTP for that user and root@pam via the web UI"
+echo "  - ${SCRIPT_DIR}/audit.sh   (verify everything above)"
@@ -0,0 +1,43 @@
+#!/bin/bash
+# Create a named PVE admin user (Administrator role) so root@pam can be
+# reserved for emergencies. Generates a random initial password, printed
+# once - change it and enable TOTP on first login (Datacenter -> Permissions
+# -> Two Factor, or the user icon menu in the top right).
+#
+# Idempotent - if the user already exists, does nothing (won't reset an
+# existing password). Run as root on the PVE host.
+#
+# Usage: ./create-admin-user.sh <username>   (realm is always @pve)
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=lib/common.sh
+source "${SCRIPT_DIR}/lib/common.sh"
+require_root
+
+USERNAME="${1:-}"
+if [ -z "$USERNAME" ]; then
+    echo "Usage: $0 <username>" >&2
+    exit 1
+fi
+
+USERID="${USERNAME}@pve"
+
+if pveum user list --output-format json 2>/dev/null | grep -q "\"${USERID}\""; then
+    echo "${USERID} already exists - not touching password or role. Skipping."
+    exit 0
+fi
+
+PASSWORD="$(openssl rand -base64 24)"
+
+pveum user add "$USERID" --password "$PASSWORD" --comment "Named admin account, created by create-admin-user.sh"
+pveum acl modify / --users "$USERID" --roles Administrator
+
+echo
+echo "Created ${USERID} with the Administrator role."
+echo "Initial password (shown once - not logged anywhere): ${PASSWORD}"
+echo
+echo "Next steps (do these before relying on this account):"
+echo "  1. Log in as ${USERID} and change the password."
+echo "  2. Enable TOTP/2FA for ${USERID} (and for root@pam)."
+echo "  3. Reserve root@pam for emergencies only from here on."
@@ -0,0 +1,48 @@
+#!/bin/bash
+# Deploy the Proxmox datacenter-level firewall from
+# config/pve-firewall/cluster.fw.example, with the management CIDR filled
+# in, and enable it. Default-deny inbound; allow SSH/8006 from mgmt only.
+#
+# Idempotent - safe to re-run. Run as root on the PVE host.
+#
+# Usage: MGMT_CIDR=192.168.2.0/24 ./deploy-firewall.sh
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=lib/common.sh
+source "${SCRIPT_DIR}/lib/common.sh"
+require_root
+
+if [ -z "${MGMT_CIDR:-}" ]; then
+    echo "MGMT_CIDR is not set. Example: MGMT_CIDR=192.168.2.0/24 $0" >&2
+    exit 1
+fi
+
+if ! [[ "$MGMT_CIDR" =~ ^([0-9]{1,3}\.){3}[0-9]{1,3}/[0-9]{1,2}$ ]]; then
+    echo "MGMT_CIDR '${MGMT_CIDR}' doesn't look like a CIDR (e.g. 192.168.2.0/24)." >&2
+    exit 1
+fi
+
+TEMPLATE="${SCRIPT_DIR}/../config/pve-firewall/cluster.fw.example"
+if [ ! -f "$TEMPLATE" ]; then
+    echo "Template not found: $TEMPLATE" >&2
+    exit 1
+fi
+
+# Corosync/Ceph rules stay commented placeholders until Stage 2 (cluster);
+# only the mgmt IPSET is real for a single Stage 1 node.
+mkdir -p /etc/pve/firewall
+write_if_changed "/etc/pve/firewall/cluster.fw" "$(sed "s|<MGMT_CIDR>|${MGMT_CIDR}|" "$TEMPLATE")"
+
+echo "Validating ruleset..."
+pve-firewall compile
+
+echo "Restarting pve-firewall..."
+pve-firewall restart
+sleep 1
+pve-firewall status
+
+echo
+echo "Firewall enabled. SSH (22) and the web UI (8006) are now only reachable"
+echo "from ${MGMT_CIDR}. If your current SSH session is NOT from that range,"
+echo "reconnect and verify access before closing this session."
@@ -0,0 +1,24 @@
+#!/bin/bash
+# Install the subscription-nag patch (lib/nag-patch.sh) as a persistent
+# standalone script under /usr/local/sbin, plus an apt Post-Invoke hook that
+# re-applies it after every dpkg run - a proxmox-widget-toolkit package
+# upgrade overwrites the patched file, so without the hook the patch would
+# silently revert on the next `apt upgrade`.
+#
+# Idempotent - safe to re-run. Run as root on the PVE host.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=lib/common.sh
+source "${SCRIPT_DIR}/lib/common.sh"
+require_root
+
+INSTALLED="/usr/local/sbin/pve-disable-subscription-nag.sh"
+write_if_changed "$INSTALLED" "$(cat "${SCRIPT_DIR}/lib/nag-patch.sh")"
+chmod +x "$INSTALLED"
+
+HOOK="/etc/apt/apt.conf.d/85pve-nosubnag"
+write_if_changed "$HOOK" 'DPkg::Post-Invoke { "test -x /usr/local/sbin/pve-disable-subscription-nag.sh && /usr/local/sbin/pve-disable-subscription-nag.sh || true"; };'
+
+"$INSTALLED"
+echo "Subscription nag patch installed; will reapply automatically after updates."
@@ -3,20 +3,10 @@
 # + fail2ban. Idempotent - safe to re-run. Run as root on the PVE host.
 set -euo pipefail

-DROPIN_DIR="/etc/ssh/sshd_config.d"
-DROPIN_FILE="${DROPIN_DIR}/99-hardening.conf"
-
-if [ "$(id -u)" -ne 0 ]; then
-    echo "Must run as root." >&2
-    exit 1
-fi
-
-mkdir -p "$DROPIN_DIR"
-cat > "$DROPIN_FILE" <<'EOF'
-PermitRootLogin prohibit-password
-PasswordAuthentication no
-EOF
-echo "Wrote $DROPIN_FILE"
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=lib/common.sh
+source "${SCRIPT_DIR}/lib/common.sh"
+require_root

 if ! authorized_keys_present=$(find /root/.ssh/authorized_keys /home/*/.ssh/authorized_keys -type f 2>/dev/null | head -n1); then
    authorized_keys_present=""
@@ -26,6 +16,10 @@ if [ -z "$authorized_keys_present" ]; then
    echo "Add your SSH public key before disconnecting, or you'll lock yourself out." >&2
 fi

+mkdir -p /etc/ssh/sshd_config.d
+write_if_changed "/etc/ssh/sshd_config.d/99-hardening.conf" "PermitRootLogin prohibit-password
+PasswordAuthentication no"
+
 sshd -t
 systemctl reload sshd
 echo "sshd reloaded with key-only root login."
@@ -36,15 +30,13 @@ if ! dpkg -s fail2ban >/dev/null 2>&1; then
 fi

 mkdir -p /etc/fail2ban/jail.d
-cat > /etc/fail2ban/jail.d/sshd.local <<'EOF'
-[sshd]
+write_if_changed "/etc/fail2ban/jail.d/sshd.local" "[sshd]
 enabled = true
 port = ssh
 backend = systemd
 maxretry = 5
 bantime = 1h
-findtime = 10m
-EOF
+findtime = 10m"

 systemctl enable --now fail2ban
 systemctl restart fail2ban
@@ -0,0 +1,47 @@
+#!/bin/bash
+# Shared helpers for proxmox-configuration scripts. Sourced, not executed
+# directly:
+#   SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+#   source "${SCRIPT_DIR}/lib/common.sh"
+
+require_root() {
+    if [ "$(id -u)" -ne 0 ]; then
+        echo "Must run as root." >&2
+        exit 1
+    fi
+}
+
+# Codename of the running Debian/PVE release, e.g. "trixie".
+pve_codename() {
+    (. /etc/os-release && echo "$VERSION_CODENAME")
+}
+
+# backup_file <path>
+# Copies an existing file to <path>.bak.<epoch>. No-op if it doesn't exist.
+backup_file() {
+    local path="$1"
+    if [ -f "$path" ]; then
+        cp "$path" "${path}.bak.$(date +%s)"
+        echo "Backed up ${path}"
+    fi
+}
+
+# write_if_changed <path> <content>
+# Writes content to path only if it differs from what's already there,
+# backing up the previous version first. Prints what happened.
+write_if_changed() {
+    local path="$1" content="$2"
+    if [ -f "$path" ] && [ "$(cat "$path")" = "$content" ]; then
+        echo "Already up to date: $path"
+        return 0
+    fi
+    backup_file "$path"
+    printf '%s\n' "$content" > "$path"
+    echo "Wrote $path"
+}
+
+# --- audit.sh status helpers ---
+# Callers should initialize: AUDIT_FAIL=0
+audit_pass() { echo "PASS  $1"; }
+audit_fail() { echo "FAIL  $1"; AUDIT_FAIL=1; }
+audit_warn() { echo "WARN  $1"; }
@@ -0,0 +1,22 @@
+#!/bin/bash
+# Neutralizes the Proxmox "No valid subscription" nag (login popup and the
+# dashboard subscription indicator) by patching proxmox-widget-toolkit's
+# proxmoxlib.js. Cosmetic only - doesn't create or spoof a subscription
+# anywhere except this UI check.
+#
+# This file is not run from the repo directly - disable-subscription-nag.sh
+# installs a copy of it to /usr/local/sbin and wires it into an apt
+# Post-Invoke hook, because a proxmox-widget-toolkit package upgrade
+# overwrites proxmoxlib.js and reverts the patch. Idempotent: exits quietly
+# if already patched or if the file isn't present.
+set -euo pipefail
+
+JS_FILE="/usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js"
+[ -f "$JS_FILE" ] || exit 0
+
+PATTERN="data.status.toLowerCase() !== 'active'"
+grep -qF "$PATTERN" "$JS_FILE" || exit 0
+
+cp "$JS_FILE" "${JS_FILE}.bak.$(date +%s)"
+sed -i "s/${PATTERN}/false/g" "$JS_FILE"
+echo "Patched subscription nag in $JS_FILE"
@@ -0,0 +1,46 @@
+#!/bin/bash
+# Install and configure unattended-upgrades for security patches. Deliberately
+# conservative for a hypervisor: security-only origins (Debian security +
+# the active PVE repo), no automatic reboot ever - a flag file is left at
+# /var/run/reboot-required for you to act on manually.
+#
+# Idempotent - safe to re-run. Run as root on the PVE host.
+set -euo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=lib/common.sh
+source "${SCRIPT_DIR}/lib/common.sh"
+require_root
+
+if ! dpkg -s unattended-upgrades >/dev/null 2>&1; then
+    apt-get update
+    apt-get install -y unattended-upgrades
+fi
+
+CODENAME="$(pve_codename)"
+
+write_if_changed "/etc/apt/apt.conf.d/51pve-unattended-upgrades.conf" "// Managed by proxmox-configuration/scripts/setup-unattended-upgrades.sh
+Unattended-Upgrade::Origins-Pattern {
+        \"origin=Debian,codename=${CODENAME},label=Debian-Security\";
+        \"origin=Debian,codename=${CODENAME}-security,label=Debian-Security\";
+        \"origin=Proxmox\";
+};
+
+// Never auto-reboot a hypervisor. Check /var/run/reboot-required manually
+// (or via scripts/audit.sh) and reboot during a planned maintenance window.
+Unattended-Upgrade::Automatic-Reboot \"false\";
+
+// Don't remove packages automatically; review before doing so by hand.
+Unattended-Upgrade::Remove-Unused-Dependencies \"false\";
+Unattended-Upgrade::Remove-Unused-Kernel-Packages \"false\";"
+
+write_if_changed "/etc/apt/apt.conf.d/20auto-upgrades" '// Managed by proxmox-configuration/scripts/setup-unattended-upgrades.sh
+APT::Periodic::Update-Package-Lists "1";
+APT::Periodic::Unattended-Upgrade "1";
+APT::Periodic::Download-Upgradeable-Packages "1";
+APT::Periodic::AutocleanInterval "7";'
+
+systemctl enable --now unattended-upgrades.service >/dev/null
+echo "unattended-upgrades enabled (security-only origins, no auto-reboot)."
+echo "Dry run:"
+unattended-upgrade --dry-run --debug 2>&1 | tail -20
@@ -1,33 +1,42 @@
 #!/bin/bash
-# Switch a fresh Proxmox VE install from the enterprise repo (which fails
-# on apt update without a paid subscription) to the no-subscription repo.
+# Remove the Proxmox/Ceph enterprise apt sources (which fail on apt update
+# without a paid subscription) and switch to the no-subscription repo.
+# Handles both the legacy one-line .list format and the deb822 .sources
+# format current PVE installers write.
 # Idempotent - safe to re-run. Run as root on the PVE host.
 set -euo pipefail

-if [ "$(id -u)" -ne 0 ]; then
-    echo "Must run as root." >&2
-    exit 1
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=lib/common.sh
+source "${SCRIPT_DIR}/lib/common.sh"
+require_root
+
+SOURCES_DIR="/etc/apt/sources.list.d"
+
+# Any sources file pointing at the enterprise host gets moved out of apt's
+# way entirely (renamed .disabled) rather than commented out in place -
+# apt only reads *.sources/*.list, so this fully removes it from
+# consideration while keeping a copy on disk for reference.
+for f in "${SOURCES_DIR}"/*.sources "${SOURCES_DIR}"/*.list; do
+    [ -f "$f" ] || continue
+    grep -qi 'enterprise.proxmox.com' "$f" 2>/dev/null || continue
+    mv "$f" "${f}.disabled"
+    echo "Removed enterprise source (renamed to .disabled): $f"
+done
+
+write_if_changed "${SOURCES_DIR}/pve-no-subscription.sources" "Types: deb
+URIs: http://download.proxmox.com/debian/pve
+Suites: $(pve_codename)
+Components: pve-no-subscription
+Signed-By: /usr/share/keyrings/proxmox-archive-keyring.gpg"
+
+# Clean up a legacy-format no-subscription file from a previous run of an
+# older version of this script, to avoid two sources for the same repo.
+LEGACY_NOSUB="${SOURCES_DIR}/pve-no-subscription.list"
+if [ -f "$LEGACY_NOSUB" ]; then
+    rm -f "$LEGACY_NOSUB"
+    echo "Removed superseded $LEGACY_NOSUB"
 fi

-CODENAME="$(. /etc/os-release && echo "$VERSION_CODENAME")"
-
-ENTERPRISE_LIST="/etc/apt/sources.list.d/pve-enterprise.list"
-if [ -f "$ENTERPRISE_LIST" ]; then
-    sed -i 's/^deb/#deb/' "$ENTERPRISE_LIST"
-    echo "Disabled $ENTERPRISE_LIST"
-fi
-
-CEPH_ENTERPRISE_LIST="/etc/apt/sources.list.d/ceph.list"
-if [ -f "$CEPH_ENTERPRISE_LIST" ] && grep -q enterprise "$CEPH_ENTERPRISE_LIST" 2>/dev/null; then
-    sed -i 's/^deb/#deb/' "$CEPH_ENTERPRISE_LIST"
-    echo "Disabled $CEPH_ENTERPRISE_LIST"
-fi
-
-NOSUB_LIST="/etc/apt/sources.list.d/pve-no-subscription.list"
-cat > "$NOSUB_LIST" <<EOF
-deb http://download.proxmox.com/debian/pve ${CODENAME} pve-no-subscription
-EOF
-echo "Wrote $NOSUB_LIST for codename '${CODENAME}'."
-
 apt-get update
 echo "Repo switched. Review 'apt list --upgradable' before upgrading."