nixos/docs/internal/implementation-plan.md

# Implementation Plan

Phased rollout of the `infrastructure` mono-repo. Work through these phases in order
after the initial push to Gitea. Each phase is independently completeable — the repo
is usable after Phase 0 even if later phases aren't done yet.

---

## Phase 0: Initial push and git history migration

**Goal:** Get the repo to Gitea and bring in existing repos with their commit histories.

### 0.1 — Create Gitea repo and push

```bash
cd /home/wayne/repos/infrastructure
git add -A
git commit -m "Initial infrastructure mono-repo scaffold"

# Create the repo on Gitea first (via web UI), then:
git remote add origin git@gitea.lan.ddnsgeek.com:wayne/infrastructure.git
git push -u origin main
```

### 0.2 — Migrate existing repos via git subtree

`git subtree` merges another repo's history into a subdirectory. Run these in order.
`--squash` collapses history into one merge commit (cleaner). Remove `--squash` if you
want full per-commit history (makes the log noisier but preserves full blame).

```bash
# Main docker stack → stacks/docker/
git subtree add --prefix=stacks/docker /home/wayne/repos/docker main --squash
git push

# Raspberry Pi stack → stacks/raspi/
git subtree add --prefix=stacks/raspi /home/wayne/repos/raspi main --squash
git push
```

> **Note:** `debian-configuration` is NOT imported via subtree — it's being dissolved.
> Its content has been converted into Ansible roles in `ansible/roles/`. The original
> repo should be archived in Gitea (Settings → Danger Zone → Archive) after Phase 3
> is complete.

### 0.3 — Install pre-commit hook locally

```bash
./scripts/install-hooks.sh
```

### 0.4 — Configure Gitea CI secrets

In the Gitea repo → Settings → Secrets, add:

| Secret name | Value |
|-------------|-------|
| `PROXMOX_ENDPOINT` | `https://pve1.sweet.home:8006/` |
| `PROXMOX_API_TOKEN_ID` | Terraform API token ID (create in PVE if not exists) |
| `PROXMOX_API_TOKEN_SECRET` | Terraform API token secret |
| `DYNU_API_KEY` | Dynu API key |
| `ANSIBLE_SSH_KEY` | Private SSH key for Ansible connections (base64 or raw PEM) |
| `GOTIFY_URL` | `https://gotify.lan.ddnsgeek.com` |
| `GOTIFY_TOKEN` | Gotify app token |

**Creating a Proxmox API token for Terraform (if not already done):**
```bash
# On pve1 as root:
pveum user add terraform@pve --comment "Terraform service account"
pveum role add TerraformRole --privs "VM.Allocate VM.Clone VM.Config.CDROM VM.Config.CPU VM.Config.Cloudinit VM.Config.Disk VM.Config.HWType VM.Config.Memory VM.Config.Network VM.Config.Options VM.Monitor VM.Audit VM.PowerMgmt Datastore.AllocateSpace Datastore.Audit Pool.Allocate Sys.Audit Sys.Console Sys.Modify"
pveum aclmod / -user terraform@pve -role TerraformRole
pveum user token add terraform@pve tf --privsep=0
# Copy the token secret — it's shown only once
```

---

## Phase 1: Ansible — verify connectivity and run check mode

**Goal:** Confirm Ansible can reach all managed hosts and produce clean check-mode output.

### 1.1 — Install Ansible and collections

```bash
pip install ansible
cd ansible
ansible-galaxy collection install -r collections/requirements.yml
```

### 1.2 — Verify inventory and connectivity

```bash
cd ansible
# List all hosts
ansible all --list-hosts

# Ping everything
ansible-playbook playbooks/ping.yml
```

Fix any unreachable hosts (SSH key, hostname, `ansible_user`) before proceeding.

### 1.3 — Run check mode against each host group

```bash
# Proxmox nodes
ansible-playbook playbooks/proxmox.yml --check --diff --limit proxmox

# FreeIPA (check only — install tasks are gated by ipa_installed check)
ansible-playbook playbooks/freeipa.yml --check --diff

# Raspberry Pi
ansible-playbook playbooks/raspi.yml --check --diff
```

**Expected result:** `check` mode should show either "no changes" (already hardened) or
show exactly the diffs you'd expect. If it shows unexpected changes, review the task
and update the role defaults before applying.

### 1.4 — Apply to pve-test first (sandbox validation)

```bash
ansible-playbook playbooks/proxmox.yml --limit pve-test.sweet.home
```

Verify the node is still reachable and the PVE web UI works before applying to pve1.

### 1.5 — Apply to production hosts

```bash
# Confirm check mode looks clean, then:
ansible-playbook playbooks/proxmox.yml --limit pve1.sweet.home
ansible-playbook playbooks/raspi.yml
```

---

## Phase 2: Terraform — import existing infrastructure state

**Goal:** Bring existing Proxmox VMs and DNS records under Terraform state management,
enabling drift detection.

### 2.1 — Configure credentials

```bash
export TF_VAR_proxmox_endpoint="https://pve1.sweet.home:8006/"
export TF_VAR_proxmox_api_token_id="terraform@pve!tf"
export TF_VAR_proxmox_api_token_secret="<token>"
```

Or create `terraform/proxmox/terraform.tfvars` (this file is git-ignored):
```hcl
proxmox_endpoint           = "https://pve1.sweet.home:8006/"
proxmox_api_token_id       = "terraform@pve!tf"
proxmox_api_token_secret   = "<token>"
```

### 2.2 — Initialise Terraform workspaces

```bash
cd terraform/proxmox && terraform init
cd ../dns && terraform init
```

### 2.3 — Verify existing resource blocks match live state

The `terraform/proxmox/` workspace already has resource blocks for existing VMs
(`docker.tf`, `nix-cache.tf`, `server-nixos.tf`, etc.) — these were generated from
live state previously. Verify they're accurate:

```bash
cd terraform/proxmox
# Check if state is already populated (it may not be on first run)
terraform state list

# If state is empty, import each existing VM:
terraform import proxmox_virtual_environment_vm.docker pve/qemu/103
terraform import proxmox_virtual_environment_vm.nix-cache pve/qemu/105
terraform import proxmox_virtual_environment_vm.server-nixos pve/qemu/104
# Add imports for any other VMs not yet in state
# VMID reference: check 'qm list' on pve1
```

### 2.4 — Run plan and reconcile any drift

```bash
terraform plan
```

If plan shows changes, the `.tf` resource block differs from live state.
Either update the `.tf` file to match (if the live state is correct) or apply
to enforce the declared state. **Do not apply to production VMs without reviewing
every planned change.**

### 2.5 — DNS workspace

```bash
cd terraform/dns
# Configure Dynu credentials if not already set:
export TF_VAR_dynu_api_key="<api_key>"
terraform plan
```

The DNS workspace already has the existing records imported. Verify no drift.

### 2.6 — Verify drift detection works end-to-end

Manually introduce a small change in the PVE UI (e.g. add a VM note) and run:
```bash
./scripts/drift-detect.sh --terraform
```
Confirm it detects the change. Revert it or update the `.tf` to match.

---

## Phase 3: NixOS flake migration

**Goal:** Move the `nixos` repo into this mono-repo while keeping the flake fully functional.

### 3.1 — Import with git subtree

```bash
git subtree add --prefix=nixos /home/wayne/repos/nixos main --squash
git push
```

### 3.2 — Update the remote flake URL on all NixOS hosts

The `Switch-nix` and `Test-nix` aliases on NixOS hosts reference the old Gitea URL.
Update `nixos/variables.nix` (the `flakeUrl` variable) to point at the new path:

```nix
# Old:
flakeUrl = "git+ssh://gitea.lan.ddnsgeek.com/wayne/nixos";
# New:
flakeUrl = "git+ssh://gitea.lan.ddnsgeek.com/wayne/infrastructure?dir=nixos";
```

Deploy the change to the `docker` LXC first (lowest risk), then remaining hosts.

### 3.3 — Validate flake builds from subdirectory

```bash
nix build ./nixos#proxmox-docker
nix build ./nixos#proxmox-nix-cache
```

### 3.4 — Archive the old nixos repo

Once all hosts are rebuilding successfully from the new URL, archive the old repo:
Gitea → `wayne/nixos` → Settings → Danger Zone → Archive Repository.

---

## Phase 4: Activate drift detection CI

**Goal:** Daily automated drift checks running in Gitea CI with Gotify notifications.

### 4.1 — Verify secrets are configured (from Phase 0.4)

Trigger the `drift-detect` workflow manually from Gitea CI → Actions → Drift Detection.

### 4.2 — Confirm first successful run

Review the step summary in Gitea. Expected outcome on a clean run:
- Terraform: "No changes. Infrastructure matches configuration."
- Ansible: `changed=0` on all hosts

### 4.3 — Test the notification path

Temporarily modify a `.tf` resource (without applying) and trigger a manual run.
Confirm the Gotify notification arrives.

---

## Phase 5: Documentation pipeline

**Goal:** Auto-generated service catalog and host inventory, with both internal and external sites building on every push.

### 5.1 — Install docs dependencies

```bash
pip install mkdocs mkdocs-material jinja2 pyyaml
```

### 5.2 — Test local build

```bash
./scripts/docs-build.sh
./scripts/docs-build.sh --serve  # preview at http://localhost:8000
```

### 5.3 — Implement service catalog generator

Create `scripts/generate-service-catalog.py`:
- Parses all `docker-compose.yml` files under `stacks/`
- Extracts service name, image, Traefik domain (from labels), network memberships
- Outputs `docs/generated/service-catalog.md`

Template for the output:
```markdown
# Service Catalog
Auto-generated from compose files in stacks/.

| Service | Image | Domain | Stack |
|---------|-------|--------|-------|
| traefik | traefik:latest | — | stacks/docker/core/traefik |
| nextcloud-webapp | nextcloud:production | nextcloud.lan.ddnsgeek.com | stacks/docker/apps/nextcloud |
...
```

### 5.4 — Configure docs deploy target

Set these Gitea secrets (from Phase 0.4):
- `PAGES_SSH_KEY` — key for rsync to the docs server
- `PAGES_HOST` — user@host for rsync
- `PAGES_PATH_INTERNAL` — path on the docs server for internal site

Or use Gitea Pages if self-hosted Gitea supports it.

---

## Phase 6: Cleanup and archive

**Goal:** Retire the old repos cleanly.

### 6.1 — Verify nothing still references old repos

- All NixOS hosts rebuilding from `infrastructure?dir=nixos` ✓
- Docker stack deployed from `stacks/docker/` ✓
- Ansible roles replacing debian-configuration scripts ✓

### 6.2 — Archive deprecated repos

Archive these repos on Gitea (makes them read-only, preserves history):
- `wayne/debian-configuration` — content dissolved into `ansible/roles/`
- `wayne/docker` — content migrated to `stacks/docker/`
- `wayne/raspi` — content migrated to `stacks/raspi/`
- `wayne/nixos` — migrated to `infrastructure/nixos/`

Do NOT delete them — they have git history that may be useful for reference.

### 6.3 — Update any external references

- CLAUDE.md files in the old repos — add a note pointing to the new mono-repo
- Any README links, bookmarks, or documentation that references the old repo URLs

---

## Ongoing: Adding new infrastructure

### Adding a new Proxmox VM

1. Add a resource block in `terraform/proxmox/<name>.tf`
2. Run `terraform plan` → `terraform apply`
3. Add the host to `ansible/inventory/hosts.yml`
4. If NixOS: add to `nixos/hosts/<name>/host.nix`, run `./scripts/sync-host-keys.sh`
5. Deploy: `nixos-rebuild switch --flake ./nixos#<target> --target-host <hostname>`

### Adding a new Docker service

1. Add a `docker-compose.yml` under `stacks/docker/apps/<service>/` or `stacks/docker/monitoring/<service>/`
2. Add Traefik labels, networks, secrets (following existing service patterns)
3. Update `stacks/docker/default-environment.env` if new env vars are needed
4. Test: `./services-up.sh config` (validates interpolation), then `./services-up.sh up -d <service>`

### Rotating credentials

- Proxmox API token: create new, update Gitea secrets, delete old
- Ansible SSH key: replace `ANSIBLE_SSH_KEY` secret, re-add public key to all hosts
- Dynu API key: update `DYNU_API_KEY` secret in Gitea
- NixOS SOPS secrets: use `nixos/scripts/rotate-admin-key.sh`