Archived
Stage 1 base config/hardening toolset, applied and verified on pve1
Splits the repo into Stage 1 (base host config/hardening, applies to any node) and Stage 2 (future HA/Ceph cluster, deferred - pve1's mini-PC hardware can't support the assumed split-disk/multi-NIC layout). Adds the Stage 1 toolset: firewall deploy, named admin user creation, unattended security upgrades, subscription-nag removal (with an apt hook so the patch survives package updates), and a read-only audit script. Fixes switch-to-no-subscription-repo.sh, which only handled the legacy .list format and silently no-op'd against PVE 9's deb822 .sources files; it now removes enterprise sources outright rather than commenting them out. Shared logic (root check, idempotent file writes, backups) factored into scripts/lib/common.sh. Ran the full sequence against pve1 via scripts/bootstrap.sh + create-admin-user.sh; scripts/audit.sh confirms all checks pass.
This commit is contained in:
+68
-18
@@ -1,18 +1,65 @@
|
||||
# Overview & Roadmap
|
||||
|
||||
## Staging
|
||||
|
||||
This repo now targets two distinct stages, in order:
|
||||
|
||||
- **Stage 1 (active)** — base configuration and hardening for a single
|
||||
Proxmox host, applicable to *any* node regardless of eventual cluster
|
||||
plans: repo/updates, SSH, firewall, PVE user/access hardening. This is
|
||||
what `scripts/bootstrap.sh`, `scripts/audit.sh`, and
|
||||
`04-security-hardening.md` cover, and what's being built out against
|
||||
node 1 (`pve1`, an ASUS PN53 mini PC) right now.
|
||||
- **Stage 2 (future)** — the multi-node HA/Ceph cluster described below
|
||||
and in `01-hardware-node1.md` / `02-storage-zfs-ceph.md` /
|
||||
`03-networking.md`. Deliberately deferred: `pve1`'s hardware (2 NVMe
|
||||
already merged into one ZFS mirror used for both boot and VM storage, a
|
||||
single 2.5GbE NIC) can't support the separate boot/Ceph disks or
|
||||
bonded/segregated networking those docs assume. Revisit once dedicated
|
||||
cluster hardware (nodes 2/3) is actually being bought and provisioned;
|
||||
until then treat the content below as a target design, not a
|
||||
description of `pve1`.
|
||||
|
||||
## Background
|
||||
|
||||
Current hardware requires disabling KVM hardware virtualization for VMs to
|
||||
Old hardware required disabling KVM hardware virtualization for VMs to
|
||||
start at all (falls back to software emulation — slow). This is a
|
||||
host/BIOS-level issue, not a Proxmox limitation, and shouldn't be needed on
|
||||
new hardware. Before reusing that workaround on the new box:
|
||||
host/BIOS-level issue, not a Proxmox limitation. Confirmed not present on
|
||||
`pve1` (ASUS PN53, Ryzen 7 7735HS): AMD-V and IOMMU both show enabled at
|
||||
boot (`dmesg | grep -i iommu`), no workaround needed. If this hardware line
|
||||
is reused for nodes 2/3, this should hold there too, but re-verify per
|
||||
node before assuming it.
|
||||
|
||||
- Confirm VT-x (Intel) / AMD-V (AMD) is enabled in BIOS/UEFI.
|
||||
- Confirm IOMMU is enabled if passthrough is planned.
|
||||
- Update BIOS/microcode first.
|
||||
- Rule out running Proxmox nested inside another hypervisor.
|
||||
## Stage 1: base config & hardening (active)
|
||||
|
||||
## End goal
|
||||
Applies to `pve1` now, and to every future node regardless of whether it
|
||||
ever joins the Stage 2 cluster. Covered by `04-security-hardening.md` and
|
||||
`scripts/bootstrap.sh` / `scripts/audit.sh`:
|
||||
|
||||
1. Fresh PVE install; confirm VT-x/AMD-V + IOMMU per the note above.
|
||||
2. Switch off the enterprise repos, onto no-subscription
|
||||
(`scripts/switch-to-no-subscription-repo.sh`).
|
||||
3. SSH hardening: key-only root login + fail2ban
|
||||
(`scripts/harden-ssh.sh`).
|
||||
4. Unattended security upgrades, no auto-reboot
|
||||
(`scripts/setup-unattended-upgrades.sh`).
|
||||
5. PVE datacenter firewall, default-deny, mgmt-only SSH/8006
|
||||
(`scripts/deploy-firewall.sh`).
|
||||
6. Named PVE admin user (Administrator role) + 2FA, `root@pam` reserved
|
||||
for emergencies (`scripts/create-admin-user.sh`, then manual TOTP
|
||||
enrollment via the web UI).
|
||||
7. Verify with `scripts/audit.sh`.
|
||||
|
||||
`pve1`'s actual disk/network layout (single ZFS mirror for boot + VMs, one
|
||||
2.5GbE NIC) is documented as-is in `01-hardware-node1.md` — Stage 1 doesn't
|
||||
require or assume the split-disk/multi-NIC layout Stage 2 wants.
|
||||
|
||||
## Stage 2: HA cluster + Ceph (future, deferred)
|
||||
|
||||
Everything below this point is the target design for when nodes 2 and 3
|
||||
are actually being provisioned. Not applicable to `pve1` as it stands.
|
||||
|
||||
### End goal
|
||||
|
||||
3-node Proxmox VE cluster with HA-managed VMs backed by **Ceph** — true
|
||||
distributed shared storage, sync replication, near-zero RPO on failover
|
||||
@@ -25,7 +72,7 @@ goes live — as soon as nodes 2/3 join, VMs move onto Ceph rather than
|
||||
adopting ZFS replication as a stopgap. Simpler end state, one storage
|
||||
model to operate instead of two.
|
||||
|
||||
## Cluster fundamentals (apply from node 1 onward)
|
||||
### Cluster fundamentals (apply from node 1 onward)
|
||||
|
||||
- 3 nodes minimum for real quorum. If starting with 2, add a QDevice
|
||||
(small VM or Raspberry Pi) as tie-breaker.
|
||||
@@ -38,16 +85,19 @@ model to operate instead of two.
|
||||
rather than `host` if nodes will ever have different CPUs — needed for
|
||||
clean live migration.
|
||||
|
||||
## Rollout sequence
|
||||
### Rollout sequence (Stage 2, once dedicated cluster hardware exists)
|
||||
|
||||
1. Build node 1 per `01-hardware-node1.md` — fresh PVE install on ZFS boot
|
||||
mirror, Ceph-earmarked disks left idle (or as a temporary local ZFS
|
||||
pool, to be wiped later — see `02-storage-zfs-ceph.md`).
|
||||
2. Migrate VMs off old hardware via `vzdump` → copy backups → `qmrestore`
|
||||
onto the new ZFS storage (converts disks to ZVOLs). No HA yet — single
|
||||
node.
|
||||
3. Apply hardening (`04-security-hardening.md`) and networking
|
||||
(`03-networking.md`) before exposing the node beyond the LAN.
|
||||
1. Build node 1 per `01-hardware-node1.md`'s target design — fresh PVE
|
||||
install on a dedicated ZFS boot mirror, Ceph-earmarked disks left idle
|
||||
(or as a temporary local ZFS pool, to be wiped later — see
|
||||
`02-storage-zfs-ceph.md`). Note: this assumes hardware with enough
|
||||
disks/NICs to separate boot, Ceph, and network roles — `pve1` does not
|
||||
have this and stays on Stage 1 only unless rebuilt on different
|
||||
hardware.
|
||||
2. Migrate VMs onto the new node via `vzdump` → copy backups →
|
||||
`qmrestore` (converts disks to ZVOLs). No HA yet — single node.
|
||||
3. Stage 1 base hardening already applied; layer on Stage 2 networking
|
||||
(`03-networking.md`) before joining a cluster.
|
||||
4. Add nodes 2 and 3 identically (same disk/network layout).
|
||||
5. Join cluster, stand up dedicated corosync network.
|
||||
6. Wipe the Ceph-earmarked disks (if used as temporary ZFS) and
|
||||
|
||||
@@ -1,5 +1,31 @@
|
||||
# Node 1 Hardware Layout
|
||||
|
||||
## pve1 as built (current reality)
|
||||
|
||||
`pve1` is an ASUS PN53 mini PC (Ryzen 7 7735HS, 32GB RAM), not the
|
||||
dedicated-server hardware the target design below assumes:
|
||||
|
||||
- **Disks**: 2x 2TB NVMe (Crucial CT2000E100SSD8), both in a single ZFS
|
||||
mirror (`rpool`) that serves as both the boot pool and VM storage
|
||||
(`local-zfs` = `rpool/data`). No spare disks to earmark for Ceph — the
|
||||
chassis only has 2 NVMe slots.
|
||||
- **Network**: one physical NIC (Realtek RTL8125, 2.5GbE), bridged as
|
||||
`vmbr0`. No second NIC for a dedicated corosync/Ceph link. (An unused
|
||||
`nic1` stanza in `/etc/network/interfaces` is a leftover from the
|
||||
installer template — there is no second NIC on this hardware.)
|
||||
|
||||
This is sufficient and correct for **Stage 1** (see `00-overview.md`) —
|
||||
base config and hardening don't need split disks or multiple NICs. It is
|
||||
*not* sufficient for **Stage 2** (Ceph/HA) as designed below without
|
||||
either different hardware or a materially different plan (e.g.
|
||||
USB/Thunderbolt-attached OSD storage, which trades away the
|
||||
enterprise-SSD/PLP guidance below — not recommended, revisit when
|
||||
actually provisioning nodes 2/3). Treat everything from here down as the
|
||||
Stage 2 target design for purpose-built hardware, not a description of
|
||||
`pve1`.
|
||||
|
||||
## Target design (Stage 2, future dedicated hardware)
|
||||
|
||||
Build node 1 so nodes 2/3 are drop-in identical later — don't re-architect
|
||||
disks or network when the cluster grows.
|
||||
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
# Storage: LVM-thin → ZFS → Ceph
|
||||
|
||||
**Stage 2 (future).** Describes the target storage model once dedicated
|
||||
cluster hardware exists. `pve1`'s current single ZFS mirror (boot + VM
|
||||
storage combined, see `01-hardware-node1.md`) is the Stage 1 end state for
|
||||
now, not an intermediate step being actively migrated from.
|
||||
|
||||
## Why move off LVM-thin
|
||||
|
||||
Neither ZFS nor LVM-thin is shared storage — both are node-local. HA needs
|
||||
|
||||
@@ -1,5 +1,18 @@
|
||||
# Networking
|
||||
|
||||
## pve1 as built (Stage 1, current)
|
||||
|
||||
Single NIC (2.5GbE), single bridge `vmbr0` on the flat LAN
|
||||
(`192.168.2.0/24`), no VLANs. There is no corosync or Ceph traffic to
|
||||
separate yet — this node isn't clustered. Segmentation for Stage 1 is
|
||||
done at the firewall, not the network: `scripts/deploy-firewall.sh`
|
||||
restricts SSH (22) and the web UI (8006) to the management CIDR via the
|
||||
PVE datacenter firewall (default-deny inbound otherwise). That's
|
||||
sufficient until Stage 2 needs actual separate physical/VLAN paths for
|
||||
corosync and Ceph traffic — see below.
|
||||
|
||||
## Target design (Stage 2, future cluster)
|
||||
|
||||
## Required separation
|
||||
|
||||
Keep these on logically separate networks/VLANs, ideally separate NICs:
|
||||
|
||||
@@ -1,14 +1,33 @@
|
||||
# Security Hardening
|
||||
|
||||
Proxmox has no `sudo` out of the box — everything defaults to root. That's
|
||||
the install default, not the recommended end state. Two layers to harden
|
||||
separately.
|
||||
Stage 1 (see `00-overview.md`) — applies to any Proxmox host, independent
|
||||
of cluster plans. Proxmox has no `sudo` out of the box — everything
|
||||
defaults to root. That's the install default, not the recommended end
|
||||
state. Two layers to harden separately.
|
||||
|
||||
## Checklist / script mapping
|
||||
|
||||
Run `scripts/bootstrap.sh` for everything except the admin user (needs a
|
||||
username decision) and 2FA enrollment (must be done interactively via the
|
||||
web UI — there's no safe way to script TOTP secret generation over SSH).
|
||||
Then run `scripts/audit.sh` to verify. Order matters (matches
|
||||
`bootstrap.sh`):
|
||||
|
||||
| # | Item | Script | Manual step required? |
|
||||
|---|------|--------|------------------------|
|
||||
| 1 | Remove enterprise repos, switch to no-subscription | `switch-to-no-subscription-repo.sh` | no |
|
||||
| 2 | SSH: key-only root login + fail2ban | `harden-ssh.sh` | no (requires an `authorized_keys` already in place — script warns if missing) |
|
||||
| 3 | Unattended security upgrades, no auto-reboot | `setup-unattended-upgrades.sh` | no |
|
||||
| 4 | PVE firewall, default-deny, mgmt-only SSH/8006 | `deploy-firewall.sh` | needs `MGMT_CIDR` set |
|
||||
| 5 | Disable subscription nag (cosmetic) | `disable-subscription-nag.sh` | no |
|
||||
| 6 | Named PVE admin user, Administrator role | `create-admin-user.sh <username>` | yes — pick the username, change the generated password on first login |
|
||||
| 7 | 2FA/TOTP on that user and `root@pam` | — | yes — web UI only: Datacenter → Permissions → Two Factor, or user menu → TFA |
|
||||
| 8 | Verify everything above | `audit.sh` | no |
|
||||
|
||||
## Linux/SSH layer
|
||||
|
||||
- `PermitRootLogin prohibit-password` in `sshd_config` — root can only
|
||||
log in via SSH key, never password. Kills most brute-force attempts.
|
||||
See `scripts/harden-ssh.sh`.
|
||||
- fail2ban jail for SSH on top of that.
|
||||
- Restrict SSH to the management VLAN/trusted IPs via the Proxmox
|
||||
firewall (see `03-networking.md`) rather than exposing broadly.
|
||||
@@ -20,7 +39,8 @@ separately.
|
||||
|
||||
- Keep `root@pam` for emergencies only.
|
||||
- Create a named user (e.g. `wayne@pve`) with the Administrator role for
|
||||
routine cluster management: Datacenter → Permissions → Users.
|
||||
routine cluster management — `create-admin-user.sh` does this, or
|
||||
Datacenter → Permissions → Users manually.
|
||||
- Enable 2FA (TOTP or hardware key) on both that account and `root@pam`:
|
||||
Datacenter → Permissions → Realms/Users.
|
||||
- For API integrations (monitoring, automation, Terraform, etc.), issue
|
||||
@@ -30,15 +50,29 @@ separately.
|
||||
## Firewall
|
||||
|
||||
Default-deny at datacenter/node level, whitelist only what's needed (see
|
||||
`03-networking.md` for the specifics). Config templates in
|
||||
`config/pve-firewall/`.
|
||||
`03-networking.md` for the specifics). Template in
|
||||
`config/pve-firewall/cluster.fw.example`, applied by
|
||||
`scripts/deploy-firewall.sh`.
|
||||
|
||||
## Repos and updates
|
||||
|
||||
Fresh installs point at the enterprise repo, which fails on `apt update`
|
||||
without a subscription. Switch to the no-subscription repo (or pay for
|
||||
enterprise). See `scripts/switch-to-no-subscription-repo.sh`. Keep the
|
||||
host patched — hypervisor CVEs are high-value targets.
|
||||
without a subscription. `scripts/switch-to-no-subscription-repo.sh`
|
||||
removes the enterprise sources entirely (renamed `.disabled`, not just
|
||||
commented out) and switches to the no-subscription repo — handles both
|
||||
the legacy `.list` format and the deb822 `.sources` format current
|
||||
installers write. Keep the host patched — hypervisor CVEs are high-value
|
||||
targets; `scripts/setup-unattended-upgrades.sh` automates security
|
||||
patches (deliberately no auto-reboot on a hypervisor — check
|
||||
`/var/run/reboot-required` and reboot during a planned window).
|
||||
|
||||
The web UI's "No valid subscription" popup and dashboard indicator are
|
||||
cosmetic upsell, not a security control, but with no subscription they'll
|
||||
nag on every login — `scripts/disable-subscription-nag.sh` patches
|
||||
`proxmox-widget-toolkit`'s JS to suppress them, and installs an apt
|
||||
`Post-Invoke` hook that reapplies the patch automatically after every
|
||||
`apt`/`dpkg` run, since a `proxmox-widget-toolkit` package upgrade
|
||||
overwrites the patched file.
|
||||
|
||||
## Misc
|
||||
|
||||
@@ -47,7 +81,7 @@ host patched — hypervisor CVEs are high-value targets.
|
||||
- If the web UI is ever needed outside the LAN, put it behind a VPN —
|
||||
don't port-forward 8006 directly.
|
||||
|
||||
## Further reading
|
||||
## Further reading / not yet automated here
|
||||
|
||||
- CIS Benchmark for Proxmox VE
|
||||
- Community PVE hardening guides (kernel parameters, audit logging,
|
||||
|
||||
Reference in New Issue
Block a user