nixos/docs/internal/implementation-plan.md

Implementation Plan

Phased rollout of the infrastructure mono-repo. Work through these phases in order after the initial push to Gitea. Each phase is independently completeable — the repo is usable after Phase 0 even if later phases aren't done yet.

---

Phase 0: Initial push and git history migration

Goal: Get the repo to Gitea and bring in existing repos with their commit histories.

0.1 — Create Gitea repo and push

cd /home/wayne/repos/infrastructure
git add -A
git commit -m "Initial infrastructure mono-repo scaffold"

# Create the repo on Gitea first (via web UI), then:
git remote add origin git@gitea.lan.ddnsgeek.com:wayne/infrastructure.git
git push -u origin main

0.2 — Migrate existing repos via git subtree

git subtree merges another repo's history into a subdirectory. Run these in order. --squash collapses history into one merge commit (cleaner). Remove --squash if you want full per-commit history (makes the log noisier but preserves full blame).

# Main docker stack → stacks/docker/
git subtree add --prefix=stacks/docker /home/wayne/repos/docker main --squash
git push

# Raspberry Pi stack → stacks/raspi/
git subtree add --prefix=stacks/raspi /home/wayne/repos/raspi main --squash
git push

Note: debian-configuration is NOT imported via subtree — it's being dissolved. Its content has been converted into Ansible roles in ansible/roles/. The original repo should be archived in Gitea (Settings → Danger Zone → Archive) after Phase 3 is complete.

0.3 — Install pre-commit hook locally

./scripts/install-hooks.sh

0.4 — Configure Gitea CI secrets

In the Gitea repo → Settings → Secrets, add:

Secret name	Value
PROXMOX_ENDPOINT	https://pve1.sweet.home:8006/
PROXMOX_API_TOKEN_ID	Terraform API token ID (create in PVE if not exists)
PROXMOX_API_TOKEN_SECRET	Terraform API token secret
DYNU_API_KEY	Dynu API key
ANSIBLE_SSH_KEY	Private SSH key for Ansible connections (base64 or raw PEM)
GOTIFY_URL	https://gotify.lan.ddnsgeek.com
GOTIFY_TOKEN	Gotify app token

Creating a Proxmox API token for Terraform (if not already done):

# On pve1 as root:
pveum user add terraform@pve --comment "Terraform service account"
pveum role add TerraformRole --privs "VM.Allocate VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Monitor VM.Audit VM.PowerMgmt Datastore.AllocateSpace Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify"
pveum aclmod / -user terraform@pve -role TerraformRole
pveum user token add terraform@pve tf --privsep=0
# Copy the token secret — it's shown only once

---

Phase 1: Ansible — verify connectivity and run check mode

Goal: Confirm Ansible can reach all managed hosts and produce clean check-mode output.

1.1 — Install Ansible and collections

pip install ansible
cd ansible
ansible-galaxy collection install -r collections/requirements.yml

1.2 — Verify inventory and connectivity

cd ansible
# List all hosts
ansible all --list-hosts

# Ping everything
ansible-playbook playbooks/ping.yml

Fix any unreachable hosts (SSH key, hostname, ansible_user) before proceeding.

1.3 — Run check mode against each host group

# Proxmox nodes
ansible-playbook playbooks/proxmox.yml --check --diff --limit proxmox

# FreeIPA (check only — install tasks are gated by ipa_installed check)
ansible-playbook playbooks/freeipa.yml --check --diff

# Raspberry Pi
ansible-playbook playbooks/raspi.yml --check --diff

Expected result: check mode should show either "no changes" (already hardened) or show exactly the diffs you'd expect. If it shows unexpected changes, review the task and update the role defaults before applying.

1.4 — Apply to pve-test first (sandbox validation)

ansible-playbook playbooks/proxmox.yml --limit pve-test.sweet.home

Verify the node is still reachable and the PVE web UI works before applying to pve1.

1.5 — Apply to production hosts

# Confirm check mode looks clean, then:
ansible-playbook playbooks/proxmox.yml --limit pve1.sweet.home
ansible-playbook playbooks/raspi.yml

---

Phase 2: Terraform — import existing infrastructure state

Goal: Bring existing Proxmox VMs and DNS records under Terraform state management, enabling drift detection.

2.1 — Configure credentials

export TF_VAR_proxmox_endpoint="https://pve1.sweet.home:8006/"
export TF_VAR_proxmox_api_token_id="terraform@pve!tf"
export TF_VAR_proxmox_api_token_secret="<token>"

Or create terraform/proxmox/terraform.tfvars (this file is git-ignored):

proxmox_endpoint           = "https://pve1.sweet.home:8006/"
proxmox_api_token_id       = "terraform@pve!tf"
proxmox_api_token_secret   = "<token>"

2.2 — Initialise Terraform workspaces

cd terraform/proxmox && terraform init
cd ../dns && terraform init

2.3 — Verify existing resource blocks match live state

The terraform/proxmox/ workspace already has resource blocks for existing VMs (docker.tf, nix-cache.tf, server-nixos.tf, etc.) — these were generated from live state previously. Verify they're accurate:

cd terraform/proxmox
# Check if state is already populated (it may not be on first run)
terraform state list

# If state is empty, import each existing VM:
terraform import proxmox_virtual_environment_vm.docker pve/qemu/103
terraform import proxmox_virtual_environment_vm.nix-cache pve/qemu/105
terraform import proxmox_virtual_environment_vm.server-nixos pve/qemu/104
# Add imports for any other VMs not yet in state
# VMID reference: check 'qm list' on pve1

2.4 — Run plan and reconcile any drift

terraform plan

If plan shows changes, the .tf resource block differs from live state. Either update the .tf file to match (if the live state is correct) or apply to enforce the declared state. Do not apply to production VMs without reviewing every planned change.

2.5 — DNS workspace

cd terraform/dns
# Configure Dynu credentials if not already set:
export TF_VAR_dynu_api_key="<api_key>"
terraform plan

The DNS workspace already has the existing records imported. Verify no drift.

2.6 — Verify drift detection works end-to-end

Manually introduce a small change in the PVE UI (e.g. add a VM note) and run:

./scripts/drift-detect.sh --terraform

Confirm it detects the change. Revert it or update the .tf to match.

---

Phase 3: NixOS flake migration

Goal: Move the nixos repo into this mono-repo while keeping the flake fully functional.

3.1 — Import with git subtree

git subtree add --prefix=nixos /home/wayne/repos/nixos main --squash
git push

3.2 — Update the remote flake URL on all NixOS hosts

The Switch-nix and Test-nix aliases on NixOS hosts reference the old Gitea URL. Update nixos/variables.nix (the flakeUrl variable) to point at the new path:

# Old:
flakeUrl = "git+ssh://gitea.lan.ddnsgeek.com/wayne/nixos";
# New:
flakeUrl = "git+ssh://gitea.lan.ddnsgeek.com/wayne/infrastructure?dir=nixos";

Deploy the change to the docker LXC first (lowest risk), then remaining hosts.

3.3 — Validate flake builds from subdirectory

nix build ./nixos#proxmox-docker
nix build ./nixos#proxmox-nix-cache

3.4 — Archive the old nixos repo

Once all hosts are rebuilding successfully from the new URL, archive the old repo: Gitea → wayne/nixos → Settings → Danger Zone → Archive Repository.

---

Phase 4: Activate drift detection CI

Goal: Daily automated drift checks running in Gitea CI with Gotify notifications.

4.1 — Verify secrets are configured (from Phase 0.4)

Trigger the drift-detect workflow manually from Gitea CI → Actions → Drift Detection.

4.2 — Confirm first successful run

Review the step summary in Gitea. Expected outcome on a clean run:

• Terraform: "No changes. Infrastructure matches configuration."
• Ansible: changed=0 on all hosts

4.3 — Test the notification path

Temporarily modify a .tf resource (without applying) and trigger a manual run. Confirm the Gotify notification arrives.

---

Phase 5: Documentation pipeline

Goal: Auto-generated service catalog and host inventory, with both internal and external sites building on every push.

5.1 — Install docs dependencies

pip install mkdocs mkdocs-material jinja2 pyyaml

5.2 — Test local build

./scripts/docs-build.sh
./scripts/docs-build.sh --serve  # preview at http://localhost:8000

5.3 — Implement service catalog generator

Create scripts/generate-service-catalog.py:

• Parses all docker-compose.yml files under stacks/
• Extracts service name, image, Traefik domain (from labels), network memberships
• Outputs docs/generated/service-catalog.md

Template for the output:

# Service Catalog
Auto-generated from compose files in stacks/.

| Service | Image | Domain | Stack |
|---------|-------|--------|-------|
| traefik | traefik:latest | — | stacks/docker/core/traefik |
| nextcloud-webapp | nextcloud:production | nextcloud.lan.ddnsgeek.com | stacks/docker/apps/nextcloud |
...

5.4 — Configure docs deploy target

Set these Gitea secrets (from Phase 0.4):

• PAGES_SSH_KEY — key for rsync to the docs server
• PAGES_HOST — user@host for rsync
• PAGES_PATH_INTERNAL — path on the docs server for internal site

Or use Gitea Pages if self-hosted Gitea supports it.

---

Phase 6: Cleanup and archive

Goal: Retire the old repos cleanly.

6.1 — Verify nothing still references old repos

• All NixOS hosts rebuilding from infrastructure?dir=nixos ✓
• Docker stack deployed from stacks/docker/ ✓
• Ansible roles replacing debian-configuration scripts ✓

6.2 — Archive deprecated repos

Archive these repos on Gitea (makes them read-only, preserves history):

• wayne/debian-configuration — content dissolved into ansible/roles/
• wayne/docker — content migrated to stacks/docker/
• wayne/raspi — content migrated to stacks/raspi/
• wayne/nixos — migrated to infrastructure/nixos/

Do NOT delete them — they have git history that may be useful for reference.

6.3 — Update any external references

• CLAUDE.md files in the old repos — add a note pointing to the new mono-repo
• Any README links, bookmarks, or documentation that references the old repo URLs

---

Ongoing: Adding new infrastructure

Adding a new Proxmox VM

1. Add a resource block in terraform/proxmox/<name>.tf
2. Run terraform plan → terraform apply
3. Add the host to ansible/inventory/hosts.yml
4. If NixOS: add to nixos/hosts/<name>/host.nix, run ./scripts/sync-host-keys.sh
5. Deploy: nixos-rebuild switch --flake ./nixos#<target> --target-host <hostname>

Adding a new Docker service

1. Add a docker-compose.yml under stacks/docker/apps/<service>/ or stacks/docker/monitoring/<service>/
2. Add Traefik labels, networks, secrets (following existing service patterns)
3. Update stacks/docker/default-environment.env if new env vars are needed
4. Test: ./services-up.sh config (validates interpolation), then ./services-up.sh up -d <service>

Rotating credentials

• Proxmox API token: create new, update Gitea secrets, delete old
• Ansible SSH key: replace ANSIBLE_SSH_KEY secret, re-add public key to all hosts
• Dynu API key: update DYNU_API_KEY secret in Gitea
• NixOS SOPS secrets: use nixos/scripts/rotate-admin-key.sh