beatzaplenty/nixos
Archived
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial infrastructure mono-repo scaffold

Browse Source 
Consolidates nixos, docker, raspi, and debian-configuration into a single
infrastructure-as-code repo. Includes:

- ansible/: full inventory + proxmox-hardening, freeipa, and raspberrypi
  roles (converted from debian-configuration bash scripts)
- terraform/: Proxmox VMs, Dynu DNS, Pi-hole (decommissioned stub),
  Docker container catalog — migrated from docker/infrastructure/terraform/
- stacks/docker/, stacks/raspi/, nixos/: placeholder READMEs pending
  git subtree population (see implementation plan)
- docs/: internal MkDocs site with architecture, network topology, runbooks,
  and drift-detection guide; external sanitized site
- scripts/: drift-detect.sh, docs-build.sh, install-hooks.sh, check-secrets.sh
- CI: secret-scan (push/PR), drift-detect (daily), docs-build (on change)
- Pi-hole removed throughout — DNS is FreeIPA, DHCP is router

See docs/internal/implementation-plan.md for the phased rollout after
pushing to Gitea.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01UvNjoxTWEDkhXsd1Dq2ETP
This commit is contained in:
beatzaplenty
2026-07-30 07:07:47 +10:00
co-authored by Claude Sonnet 4.6
commit f85c65870f
125 changed files with 8186 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/internal/implementation-plan.md
+372
View File
@@ -0,0 +1,372 @@
# Implementation Plan
Phased rollout of the `infrastructure` mono-repo. Work through these phases in order
after the initial push to Gitea. Each phase is independently completeable — the repo
is usable after Phase 0 even if later phases aren't done yet.
---
## Phase 0: Initial push and git history migration
**Goal:** Get the repo to Gitea and bring in existing repos with their commit histories.
### 0.1 — Create Gitea repo and push
```bash
cd /home/wayne/repos/infrastructure
git add -A
git commit -m "Initial infrastructure mono-repo scaffold"
# Create the repo on Gitea first (via web UI), then:
git remote add origin git@gitea.lan.ddnsgeek.com:wayne/infrastructure.git
git push -u origin main
```
### 0.2 — Migrate existing repos via git subtree
`git subtree` merges another repo's history into a subdirectory. Run these in order.
`--squash` collapses history into one merge commit (cleaner). Remove `--squash` if you
want full per-commit history (makes the log noisier but preserves full blame).
```bash
# Main docker stack → stacks/docker/
git subtree add --prefix=stacks/docker /home/wayne/repos/docker main --squash
git push
# Raspberry Pi stack → stacks/raspi/
git subtree add --prefix=stacks/raspi /home/wayne/repos/raspi main --squash
git push
```
> **Note:** `debian-configuration` is NOT imported via subtree — it's being dissolved.
> Its content has been converted into Ansible roles in `ansible/roles/`. The original
> repo should be archived in Gitea (Settings → Danger Zone → Archive) after Phase 3
> is complete.
### 0.3 — Install pre-commit hook locally
```bash
./scripts/install-hooks.sh
```
### 0.4 — Configure Gitea CI secrets
In the Gitea repo → Settings → Secrets, add:
| Secret name | Value |
|-------------|-------|
| `PROXMOX_ENDPOINT` | `https://pve1.sweet.home:8006/` |
| `PROXMOX_API_TOKEN_ID` | Terraform API token ID (create in PVE if not exists) |
| `PROXMOX_API_TOKEN_SECRET` | Terraform API token secret |
| `DYNU_API_KEY` | Dynu API key |
| `ANSIBLE_SSH_KEY` | Private SSH key for Ansible connections (base64 or raw PEM) |
| `GOTIFY_URL` | `https://gotify.lan.ddnsgeek.com` |
| `GOTIFY_TOKEN` | Gotify app token |
**Creating a Proxmox API token for Terraform (if not already done):**
```bash
# On pve1 as root:
pveum user add terraform@pve --comment "Terraform service account"
pveum role add TerraformRole --privs "VM.Allocate VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Monitor VM.Audit VM.PowerMgmt Datastore.AllocateSpace Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify"
pveum aclmod / -user terraform@pve -role TerraformRole
pveum user token add terraform@pve tf --privsep=0
# Copy the token secret — it's shown only once
```
---
## Phase 1: Ansible — verify connectivity and run check mode
**Goal:** Confirm Ansible can reach all managed hosts and produce clean check-mode output.
### 1.1 — Install Ansible and collections
```bash
pip install ansible
cd ansible
ansible-galaxy collection install -r collections/requirements.yml
```
### 1.2 — Verify inventory and connectivity
```bash
cd ansible
# List all hosts
ansible all --list-hosts
# Ping everything
ansible-playbook playbooks/ping.yml
```
Fix any unreachable hosts (SSH key, hostname, `ansible_user`) before proceeding.
### 1.3 — Run check mode against each host group
```bash
# Proxmox nodes
ansible-playbook playbooks/proxmox.yml --check --diff --limit proxmox
# FreeIPA (check only — install tasks are gated by ipa_installed check)
ansible-playbook playbooks/freeipa.yml --check --diff
# Raspberry Pi
ansible-playbook playbooks/raspi.yml --check --diff
```
**Expected result:** `check` mode should show either "no changes" (already hardened) or
show exactly the diffs you'd expect. If it shows unexpected changes, review the task
and update the role defaults before applying.
### 1.4 — Apply to pve-test first (sandbox validation)
```bash
ansible-playbook playbooks/proxmox.yml --limit pve-test.sweet.home
```
Verify the node is still reachable and the PVE web UI works before applying to pve1.
### 1.5 — Apply to production hosts
```bash
# Confirm check mode looks clean, then:
ansible-playbook playbooks/proxmox.yml --limit pve1.sweet.home
ansible-playbook playbooks/raspi.yml
```
---
## Phase 2: Terraform — import existing infrastructure state
**Goal:** Bring existing Proxmox VMs and DNS records under Terraform state management,
enabling drift detection.
### 2.1 — Configure credentials
```bash
export TF_VAR_proxmox_endpoint="https://pve1.sweet.home:8006/"
export TF_VAR_proxmox_api_token_id="terraform@pve!tf"
export TF_VAR_proxmox_api_token_secret="<token>"
```
Or create `terraform/proxmox/terraform.tfvars` (this file is git-ignored):
```hcl
proxmox_endpoint = "https://pve1.sweet.home:8006/"
proxmox_api_token_id = "terraform@pve!tf"
proxmox_api_token_secret = "<token>"
```
### 2.2 — Initialise Terraform workspaces
```bash
cd terraform/proxmox && terraform init
cd ../dns && terraform init
```
### 2.3 — Verify existing resource blocks match live state
The `terraform/proxmox/` workspace already has resource blocks for existing VMs
(`docker.tf`, `nix-cache.tf`, `server-nixos.tf`, etc.) — these were generated from
live state previously. Verify they're accurate:
```bash
cd terraform/proxmox
# Check if state is already populated (it may not be on first run)
terraform state list
# If state is empty, import each existing VM:
terraform import proxmox_virtual_environment_vm.docker pve/qemu/103
terraform import proxmox_virtual_environment_vm.nix-cache pve/qemu/105
terraform import proxmox_virtual_environment_vm.server-nixos pve/qemu/104
# Add imports for any other VMs not yet in state
# VMID reference: check 'qm list' on pve1
```
### 2.4 — Run plan and reconcile any drift
```bash
terraform plan
```
If plan shows changes, the `.tf` resource block differs from live state.
Either update the `.tf` file to match (if the live state is correct) or apply
to enforce the declared state. **Do not apply to production VMs without reviewing
every planned change.**
### 2.5 — DNS workspace
```bash
cd terraform/dns
# Configure Dynu credentials if not already set:
export TF_VAR_dynu_api_key="<api_key>"
terraform plan
```
The DNS workspace already has the existing records imported. Verify no drift.
### 2.6 — Verify drift detection works end-to-end
Manually introduce a small change in the PVE UI (e.g. add a VM note) and run:
```bash
./scripts/drift-detect.sh --terraform
```
Confirm it detects the change. Revert it or update the `.tf` to match.
---
## Phase 3: NixOS flake migration
**Goal:** Move the `nixos` repo into this mono-repo while keeping the flake fully functional.
### 3.1 — Import with git subtree
```bash
git subtree add --prefix=nixos /home/wayne/repos/nixos main --squash
git push
```
### 3.2 — Update the remote flake URL on all NixOS hosts
The `Switch-nix` and `Test-nix` aliases on NixOS hosts reference the old Gitea URL.
Update `nixos/variables.nix` (the `flakeUrl` variable) to point at the new path:
```nix
# Old:
flakeUrl = "git+ssh://gitea.lan.ddnsgeek.com/wayne/nixos";
# New:
flakeUrl = "git+ssh://gitea.lan.ddnsgeek.com/wayne/infrastructure?dir=nixos";
```
Deploy the change to the `docker` LXC first (lowest risk), then remaining hosts.
### 3.3 — Validate flake builds from subdirectory
```bash
nix build ./nixos#proxmox-docker
nix build ./nixos#proxmox-nix-cache
```
### 3.4 — Archive the old nixos repo
Once all hosts are rebuilding successfully from the new URL, archive the old repo:
Gitea → `wayne/nixos` → Settings → Danger Zone → Archive Repository.
---
## Phase 4: Activate drift detection CI
**Goal:** Daily automated drift checks running in Gitea CI with Gotify notifications.
### 4.1 — Verify secrets are configured (from Phase 0.4)
Trigger the `drift-detect` workflow manually from Gitea CI → Actions → Drift Detection.
### 4.2 — Confirm first successful run
Review the step summary in Gitea. Expected outcome on a clean run:
- Terraform: "No changes. Infrastructure matches configuration."
- Ansible: `changed=0` on all hosts
### 4.3 — Test the notification path
Temporarily modify a `.tf` resource (without applying) and trigger a manual run.
Confirm the Gotify notification arrives.
---
## Phase 5: Documentation pipeline
**Goal:** Auto-generated service catalog and host inventory, with both internal and external sites building on every push.
### 5.1 — Install docs dependencies
```bash
pip install mkdocs mkdocs-material jinja2 pyyaml
```
### 5.2 — Test local build
```bash
./scripts/docs-build.sh
./scripts/docs-build.sh --serve # preview at http://localhost:8000
```
### 5.3 — Implement service catalog generator
Create `scripts/generate-service-catalog.py`:
- Parses all `docker-compose.yml` files under `stacks/`
- Extracts service name, image, Traefik domain (from labels), network memberships
- Outputs `docs/generated/service-catalog.md`
Template for the output:
```markdown
# Service Catalog
Auto-generated from compose files in stacks/.
| Service | Image | Domain | Stack |
|---------|-------|--------|-------|
| traefik | traefik:latest | — | stacks/docker/core/traefik |
| nextcloud-webapp | nextcloud:production | nextcloud.lan.ddnsgeek.com | stacks/docker/apps/nextcloud |
...
```
### 5.4 — Configure docs deploy target
Set these Gitea secrets (from Phase 0.4):
- `PAGES_SSH_KEY` — key for rsync to the docs server
- `PAGES_HOST` — user@host for rsync
- `PAGES_PATH_INTERNAL` — path on the docs server for internal site
Or use Gitea Pages if self-hosted Gitea supports it.
---
## Phase 6: Cleanup and archive
**Goal:** Retire the old repos cleanly.
### 6.1 — Verify nothing still references old repos
- All NixOS hosts rebuilding from `infrastructure?dir=nixos` ✓
- Docker stack deployed from `stacks/docker/` ✓
- Ansible roles replacing debian-configuration scripts ✓
### 6.2 — Archive deprecated repos
Archive these repos on Gitea (makes them read-only, preserves history):
- `wayne/debian-configuration` — content dissolved into `ansible/roles/`
- `wayne/docker` — content migrated to `stacks/docker/`
- `wayne/raspi` — content migrated to `stacks/raspi/`
- `wayne/nixos` — migrated to `infrastructure/nixos/`
Do NOT delete them — they have git history that may be useful for reference.
### 6.3 — Update any external references
- CLAUDE.md files in the old repos — add a note pointing to the new mono-repo
- Any README links, bookmarks, or documentation that references the old repo URLs
---
## Ongoing: Adding new infrastructure
### Adding a new Proxmox VM
1. Add a resource block in `terraform/proxmox/<name>.tf`
2. Run `terraform plan` → `terraform apply`
3. Add the host to `ansible/inventory/hosts.yml`
4. If NixOS: add to `nixos/hosts/<name>/host.nix`, run `./scripts/sync-host-keys.sh`
5. Deploy: `nixos-rebuild switch --flake ./nixos#<target> --target-host <hostname>`
### Adding a new Docker service
1. Add a `docker-compose.yml` under `stacks/docker/apps/<service>/` or `stacks/docker/monitoring/<service>/`
2. Add Traefik labels, networks, secrets (following existing service patterns)
3. Update `stacks/docker/default-environment.env` if new env vars are needed
4. Test: `./services-up.sh config` (validates interpolation), then `./services-up.sh up -d <service>`
### Rotating credentials
- Proxmox API token: create new, update Gitea secrets, delete old
- Ansible SSH key: replace `ANSIBLE_SSH_KEY` secret, re-add public key to all hosts
- Dynu API key: update `DYNU_API_KEY` secret in Gitea
- NixOS SOPS secrets: use `nixos/scripts/rotate-admin-key.sh`