Archived
beatzaplenty
ba73420350
pve-test was briefly clustered with pve1 then deliberately de-clustered so it could move to wifi-primary networking (4addr bridge mode, bonded with a wired LAN backup) - a change not achievable while clustered given corosync's latency requirements. Captures that as a reproducible script plus docs: cluster separation procedure, the wifi network design and the live-cutover pitfalls hit along the way, and node-role/history context. Also adds CLAUDE.md guardrails for pve1 (production) vs pve-test (sandbox) - this repo had none before, despite scripts here being able to make real changes to both. Separately: both nodes' mgmt firewalls were dropping ICMP by default (TCP 22/8006 only), which looked like an outage mid-troubleshooting even though SSH/web UI were fine. Added an explicit ping-allow rule to the firewall template, applied it live on both nodes, and added an audit.sh check so it stays enforced.
106 lines
5.3 KiB
Markdown
106 lines
5.3 KiB
Markdown
# Security Hardening
|
|
|
|
Stage 1 (see `00-overview.md`) — applies to any Proxmox host, independent
|
|
of cluster plans. Proxmox has no `sudo` out of the box — everything
|
|
defaults to root. That's the install default, not the recommended end
|
|
state. Two layers to harden separately.
|
|
|
|
## Checklist / script mapping
|
|
|
|
Run `scripts/bootstrap.sh` for everything except the admin user (needs a
|
|
username decision) and 2FA enrollment (must be done interactively via the
|
|
web UI — there's no safe way to script TOTP secret generation over SSH).
|
|
Then run `scripts/audit.sh` to verify. Order matters (matches
|
|
`bootstrap.sh`):
|
|
|
|
| # | Item | Script | Manual step required? |
|
|
|---|------|--------|------------------------|
|
|
| 1 | Remove enterprise repos, switch to no-subscription | `switch-to-no-subscription-repo.sh` | no |
|
|
| 2 | SSH: key-only root login + fail2ban | `harden-ssh.sh` | no (requires an `authorized_keys` already in place — script warns if missing) |
|
|
| 3 | Unattended security upgrades, no auto-reboot | `setup-unattended-upgrades.sh` | no |
|
|
| 4 | PVE firewall, default-deny, mgmt-only SSH/8006 | `deploy-firewall.sh` | needs `MGMT_CIDR` set |
|
|
| 5 | Disable subscription nag (cosmetic) | `disable-subscription-nag.sh` | no |
|
|
| 6 | Named PVE admin user, Administrator role | `create-admin-user.sh <username>` | yes — pick the username, change the generated password on first login |
|
|
| 7 | 2FA/TOTP on that user and `root@pam` | — | yes — web UI only: Datacenter → Permissions → Two Factor, or user menu → TFA |
|
|
| 8 | Verify everything above | `audit.sh` | no |
|
|
|
|
## Linux/SSH layer
|
|
|
|
- `PermitRootLogin prohibit-password` in `sshd_config` — root can only
|
|
log in via SSH key, never password. Kills most brute-force attempts.
|
|
- fail2ban jail for SSH on top of that.
|
|
- Restrict SSH to the management VLAN/trusted IPs via the Proxmox
|
|
firewall (see `03-networking.md`) rather than exposing broadly.
|
|
- A separate Linux sudo user isn't strictly required for day-to-day PVE
|
|
admin (the PVE permission system below governs that), but worth adding
|
|
if multiple people SSH into the box directly, for accountability.
|
|
|
|
## PVE/web layer (the one that actually matters day-to-day)
|
|
|
|
- Keep `root@pam` for emergencies only.
|
|
- Create a named user (e.g. `wayne@pve`) with the Administrator role for
|
|
routine cluster management — `create-admin-user.sh` does this, or
|
|
Datacenter → Permissions → Users manually.
|
|
- Enable 2FA (TOTP or hardware key) on both that account and `root@pam`:
|
|
Datacenter → Permissions → Realms/Users.
|
|
- For API integrations (monitoring, automation, Terraform, etc.), issue
|
|
scoped API tokens with least-privilege roles (e.g. `PVEAuditor` or a
|
|
custom role) — never hand out root credentials.
|
|
|
|
## Firewall
|
|
|
|
Default-deny at datacenter/node level, whitelist only what's needed (see
|
|
`03-networking.md` for the specifics). Template in
|
|
`config/pve-firewall/cluster.fw.example`, applied by
|
|
`scripts/deploy-firewall.sh`.
|
|
|
|
**ICMP echo (ping) is explicitly allowed from the management network**,
|
|
alongside SSH/8006 — not required for anything to function, but without
|
|
it `policy_in: DROP` silently eats ping while SSH/web UI keep working.
|
|
That split (ping dead, everything else fine) reads exactly like a real
|
|
outage mid-troubleshooting; see `06-pve-test-wifi-network.md` for a case
|
|
this caused genuine confusion after a network change. If you ever debug
|
|
"can't ping but can SSH", check this firewall before assuming the
|
|
network itself is broken.
|
|
|
|
**This file is cluster-wide, not per-node**: `cluster.fw` lives in
|
|
`/etc/pve/firewall/` — shared via pmxcfs across every node in a cluster.
|
|
A node that joins a cluster inherits whatever's already there, and (per
|
|
`docs/05-node-roles.md`) keeps its own local copy after leaving. Don't
|
|
assume a node's firewall state matches what `deploy-firewall.sh` was
|
|
last run with directly against it — check `pve-firewall status` /
|
|
`/etc/pve/firewall/cluster.fw` on the actual node.
|
|
|
|
## Repos and updates
|
|
|
|
Fresh installs point at the enterprise repo, which fails on `apt update`
|
|
without a subscription. `scripts/switch-to-no-subscription-repo.sh`
|
|
removes the enterprise sources entirely (renamed `.disabled`, not just
|
|
commented out) and switches to the no-subscription repo — handles both
|
|
the legacy `.list` format and the deb822 `.sources` format current
|
|
installers write. Keep the host patched — hypervisor CVEs are high-value
|
|
targets; `scripts/setup-unattended-upgrades.sh` automates security
|
|
patches (deliberately no auto-reboot on a hypervisor — check
|
|
`/var/run/reboot-required` and reboot during a planned window).
|
|
|
|
The web UI's "No valid subscription" popup and dashboard indicator are
|
|
cosmetic upsell, not a security control, but with no subscription they'll
|
|
nag on every login — `scripts/disable-subscription-nag.sh` patches
|
|
`proxmox-widget-toolkit`'s JS to suppress them, and installs an apt
|
|
`Post-Invoke` hook that reapplies the patch automatically after every
|
|
`apt`/`dpkg` run, since a `proxmox-widget-toolkit` package upgrade
|
|
overwrites the patched file.
|
|
|
|
## Misc
|
|
|
|
- Management interface on a network you trust, not the same broadcast
|
|
domain as guest VM traffic.
|
|
- If the web UI is ever needed outside the LAN, put it behind a VPN —
|
|
don't port-forward 8006 directly.
|
|
|
|
## Further reading / not yet automated here
|
|
|
|
- CIS Benchmark for Proxmox VE
|
|
- Community PVE hardening guides (kernel parameters, audit logging,
|
|
storage encryption)
|