diff --git a/DEPLOYMENT.md b/DEPLOYMENT.md index f4b90d8..ce642cb 100644 --- a/DEPLOYMENT.md +++ b/DEPLOYMENT.md @@ -1,15 +1,17 @@ -# Deployment prerequisites (required) +# Deployment Prerequisites -Before running `docker compose up`, you **must** provision runtime secrets. +Before running compose operations, provision local secret material. -## 1) Create non-committed secret files +## 1) Create non-committed secret env file ```bash cp secrets/.env.secrets.example secrets/stack-secrets.env chmod 600 secrets/stack-secrets.env ``` -Create these Docker secret files (all ignored by git): +## 2) Create required Docker secret files + +All files below are expected locally and are gitignored: - `secrets/nextcloud_db_root_password.txt` - `secrets/nextcloud_db_password.txt` @@ -26,23 +28,22 @@ Recommended permissions: chmod 600 secrets/*.txt ``` -## 2) Rotate previously committed credentials +## 3) Validate composed configuration -These values were previously hardcoded and must be rotated in upstream systems immediately: - -- Database credentials (Nextcloud, Passbolt, InfluxDB). -- Nextcloud SMTP app password. -- Authelia reset JWT secret, session secret, storage encryption key. -- Traefik CrowdSec LAPI key. -- Gotify admin password. -- Prometheus Uptime Kuma basic-auth password. - -## 3) Start stack - -After secrets are provisioned: +Use the repository composition entrypoint: ```bash -docker compose -f core/docker-compose.yml up -d -docker compose -f monitoring/prometheus/docker-compose.yml up -d -docker compose -f apps/nextcloud/docker-compose.yml up -d +./services-up.sh --profile all config ``` + +This confirms compose rendering with shared env/network inputs before any runtime operation. + +## 4) Rotate previously committed credentials + +If migrating from older states where secrets were committed, rotate upstream values immediately (DB credentials, app passwords, auth keys, and API tokens). + +## Related docs + +- [SECURITY_SECRETS_INVENTORY.md](SECURITY_SECRETS_INVENTORY.md) +- [docs/docker-environment.md](docs/docker-environment.md) +- [docs/source-of-truth.md](docs/source-of-truth.md) diff --git a/README.md b/README.md index 07e8073..a47a406 100644 --- a/README.md +++ b/README.md @@ -1,8 +1,67 @@ -# Docker + Traefik Homelab Stack +# Homelab Docker + Terraform Inventory Repository -This repository defines a multi-compose Docker environment with Traefik as ingress, app workloads, and a monitoring/alerting plane. +This repository is both: -## High-Level Architecture +1. **operational** (Docker Compose application/runtime definition), and +2. **documentary/inventory-oriented** (Terraform capture of Proxmox VMs, host metadata, and selected Docker objects). + +If you only read one section, read **[Source-of-truth boundaries](docs/source-of-truth.md)** first. + +--- + +## Quick navigation + +- Architecture overview: [docs/architecture.md](docs/architecture.md) +- Repository layout: [docs/repo-structure.md](docs/repo-structure.md) +- Source-of-truth boundaries and guardrails: [docs/source-of-truth.md](docs/source-of-truth.md) +- Docker environment composition and `services-up.sh`: [docs/docker-environment.md](docs/docker-environment.md) +- Terraform workflows (brownfield import/reconciliation): [docs/terraform-workflows.md](docs/terraform-workflows.md) +- Infrastructure inventory intent and outputs: [docs/infrastructure-inventory.md](docs/infrastructure-inventory.md) +- Deployment prerequisites and secrets setup: [DEPLOYMENT.md](DEPLOYMENT.md) +- Secrets inventory: [SECURITY_SECRETS_INVENTORY.md](SECURITY_SECRETS_INVENTORY.md) + +Terraform subtrees: + +- Terraform root docs: [infrastructure/terraform/README.md](infrastructure/terraform/README.md) +- Terraform Docker mirror: [infrastructure/terraform/docker/README.md](infrastructure/terraform/docker/README.md) +- Terraform Proxmox inventory: [infrastructure/terraform/proxmox/README.md](infrastructure/terraform/proxmox/README.md) + +--- + +## Operating model + +### Docker Compose (runtime authority) + +- Compose files under `core/`, `apps/`, and `monitoring/` describe runtime services. +- `services-up.sh` composes the environment by discovering compose files and applying common env/network inputs. +- For service runtime behavior, start from Compose files and `services-up.sh` (not Terraform). + +### Terraform (inventory and reconciliation authority) + +- Terraform under `infrastructure/terraform/` is used to codify and reconcile existing infrastructure. +- Current repo usage emphasizes **brownfield import-first workflows** and safe reconciliation. +- Terraform captures: + - Proxmox VM configuration for existing VMs. + - Physical host metadata in locals/outputs. + - Documentation-oriented Docker container mirroring (limited, selective). + +Terraform here is **not** a replacement for Docker Compose deployment. + +--- + +## Guardrails + +- Do not run destructive Terraform commands casually. +- Do not treat generated Terraform config as final without manual review. +- Do not commit real secrets, credentials, or local state. +- Keep one-resource-per-file patterns where already established in Terraform subdirectories. +- Prefer shaping outputs for documentation/tooling consumption over dumping raw provider objects. + +See [docs/source-of-truth.md](docs/source-of-truth.md) and [docs/terraform-workflows.md](docs/terraform-workflows.md) for concrete do/don't guidance. + +--- + +## High-level architecture ```mermaid flowchart TB @@ -43,4 +102,4 @@ flowchart TB Prometheus --> Gotify ``` -For a request-flow/network view and architecture notes, see [docs/architecture.md](docs/architecture.md). +For request-flow and network detail, see [docs/architecture.md](docs/architecture.md). diff --git a/SECURITY_SECRETS_INVENTORY.md b/SECURITY_SECRETS_INVENTORY.md index 39b1ccd..7a7e4a0 100644 --- a/SECURITY_SECRETS_INVENTORY.md +++ b/SECURITY_SECRETS_INVENTORY.md @@ -1,6 +1,14 @@ # Security Secrets Inventory -This inventory is aligned with `secrets/.env.secrets.example` and documents only the values that are expected to be set in the non-committed secrets env file (`secrets/stack-secrets.env`). +This inventory tracks values expected in local secret material and where they are consumed. + +## Scope and authority + +- Canonical example template: `secrets/.env.secrets.example` +- Runtime-loaded secret env file: `secrets/stack-secrets.env` (local, non-committed) +- Docker secret files: `secrets/*.txt` (local, non-committed) + +For setup steps, see [DEPLOYMENT.md](DEPLOYMENT.md). ## Secrets expected in `secrets/.env.secrets.example` @@ -22,16 +30,26 @@ This inventory is aligned with `secrets/.env.secrets.example` and documents only | `INFLUXDB_INIT_USERNAME` | `monitoring/prometheus/docker-compose.yml` | InfluxDB initial username. | | `PIHOLE_PASSWORD` | `monitoring/prometheus/docker-compose.yml` | Exporter auth / Pi-hole integration password. | -## Managed outside `.env.secrets.example` +## Secrets managed outside `.env.secrets.example` -The following sensitive values are intentionally not duplicated in `secrets/.env.secrets.example` because they are provided via Docker secrets (`*_FILE`) or other mounted secret files: +The following sensitive values are intentionally provided via Docker secret files (`*_FILE`) or other mounted secret files instead of the shared `.env` template: -- Database/root passwords for Nextcloud, Passbolt, and supporting services that are wired through Docker secrets. -- Redis runtime password (`--requirepass`) loaded from a Docker secret. +- Database/root passwords for Nextcloud, Passbolt, and supporting services. +- Redis runtime password loaded from a Docker secret. - `DOCKER_INFLUXDB_INIT_PASSWORD` loaded from Docker secret in monitoring. -- Uptime Kuma basic auth password loaded via `password_file` in Prometheus config. -- Core stack secrets injected via env substitution in committed config files, such as: - - `AUTHELIA_JWT_SECRET` - - `AUTHELIA_SESSION_SECRET` - - `AUTHELIA_STORAGE_ENCRYPTION_KEY` - - `CROWDSEC_LAPI_KEY` +- Uptime Kuma basic-auth password loaded via `password_file` in Prometheus config. +- Core stack secret values injected via env substitution (for example Authelia and CrowdSec values). + +## Commit safety rules + +Never commit: + +- `secrets/stack-secrets.env` +- `secrets/*.txt` real secret files +- real Terraform `.tfvars` containing credentials +- Terraform state files with sensitive runtime metadata + +## Related docs + +- [DEPLOYMENT.md](DEPLOYMENT.md) +- [docs/source-of-truth.md](docs/source-of-truth.md) diff --git a/docs/architecture.md b/docs/architecture.md index 1d2c2b2..bae9f09 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -2,13 +2,9 @@ ## Overview -This stack uses **Traefik v3** as the internet-facing ingress for application and operations UIs. Service routing is primarily label-driven from Docker Compose files, with a shared `traefik` bridge network for reverse-proxied traffic and a `monitor` network for internal telemetry components. +This stack uses **Traefik v3** as internet-facing ingress for application and operations UIs. Service routing is label-driven from Docker Compose files, with shared Docker networks (`traefik`, `monitor`) connecting reverse-proxied and telemetry services. -TLS is terminated at Traefik using ACME HTTP challenge (`myresolver`), with additional hardening via: - -- a default middleware chain (security headers, CrowdSec bouncer, error pages), -- Authelia forward-auth middleware on selected routes, -- mTLS TLS options (`mtls-private-admin`) on private-admin endpoints. +TLS is terminated at Traefik (ACME HTTP challenge), with hardening via middleware chains, Authelia forward-auth for selected routes, CrowdSec integration, and mTLS options for private-admin paths. ## Network / Request Flow @@ -18,7 +14,7 @@ flowchart LR T -->|HTTP->HTTPS redirect| T T -->|ACME HTTP challenge| LE[Let's Encrypt ACME] - subgraph TraefikNet["Docker network: traefik (172.21.0.0 slash 16)"] + subgraph TraefikNet[Docker network: traefik] A[Authelia] CS[CrowdSec LAPI] EP[Error Pages] @@ -76,51 +72,23 @@ flowchart LR T --> DSP ``` -## Key Components +## Key components -- **Ingress & security plane:** Traefik, Authelia, CrowdSec, Error Pages. -- **User-facing applications:** Nextcloud, Passbolt, Gitea, Gramps Web (Family Tree), SearXNG. +- **Ingress/security plane:** Traefik, Authelia, CrowdSec, Error Pages. +- **User-facing apps:** Nextcloud, Passbolt, Gitea, Gramps Web, SearXNG. - **Monitoring/ops:** Prometheus, Grafana, InfluxDB, Node-RED, Uptime Kuma, Portainer, Gotify. -- **Support plane:** Docker Socket Proxy (shared Docker API gateway for Traefik/automation/ops tools). +- **Support plane:** Docker Socket Proxy for controlled Docker API access. -## Remote Hosts Observed +## Relationship to Terraform inventory -Prometheus scrape targets indicate additional infrastructure outside the local Compose deployment, including hostnames for: +Terraform in `infrastructure/terraform/` captures infrastructure inventory and reconciliation state for Proxmox VMs, physical host metadata, and selected Docker mirrors. -- `raspberrypi.tail13f623.ts.net` -- `pve.sweet.home` -- `pbs.sweet.home` -- `pihole` -- `server` -- `nix-cache` -- `kuma.lan.ddnsgeek.com` +Use architecture docs together with: -## Runtime Inventory Input +- [docs/source-of-truth.md](source-of-truth.md) +- [docs/terraform-workflows.md](terraform-workflows.md) +- [docs/infrastructure-inventory.md](infrastructure-inventory.md) -Prometheus runtime inventory snapshots are exported with `scripts/export_prometheus_inventory.py` and committed under `docs/runtime/`. The latest human-readable summary is in [docs/prometheus-inventory.md](prometheus-inventory.md). +## Notes on runtime vs declared state -These artifacts are an observed-runtime input for architecture diagrams/docs and should be combined with repository configuration, not treated as sole source of truth. - -## Assumptions / Unknowns - -The repository provides enough detail to infer **container-level architecture**, but not full **Proxmox host/VM topology**. - -Unknowns (left intentionally as placeholders): - -- **Proxmox physical hosts:** _unknown from repo contents._ -- **VM/LXC inventory and placement:** _unknown from repo contents._ -- **Which services run on which Proxmox node(s):** _unknown from repo contents._ -- **Inter-host VLAN/subnet layout beyond Docker bridges:** _unknown from repo contents._ - -If you want, this section can be replaced with a concrete Proxmox topology once you add an inventory source (e.g., Terraform, Ansible inventory, or a diagram export). - -### Data sources - -- Existing repository architecture docs for declared topology - -### Notes from inventory - -- The `up` query indicates scrape success from Prometheus perspective only. -- Use static repository architecture docs and deployment configs with this runtime export for complete diagrams. - - +Runtime scrape targets and health signals are useful observed-state inputs, but they do not replace declared config authority from Compose/Terraform sources. diff --git a/docs/docker-environment.md b/docs/docker-environment.md new file mode 100644 index 0000000..cb7aafb --- /dev/null +++ b/docs/docker-environment.md @@ -0,0 +1,61 @@ +# Docker Environment Composition + +This repo uses multi-file Docker Compose with a wrapper script as the composition entrypoint. + +## Composition source of truth + +`services-up.sh` is the composition authority for this repository. + +It: + +1. discovers compose files under `apps/`, `monitoring/`, and `core/`, +2. prepends shared baseline files, +3. applies `default-environment.env` and `secrets/stack-secrets.env`, +4. invokes `docker compose` with a stable project name. + +Because of this, when validating or understanding runtime composition, prefer running: + +```bash +./services-up.sh --profile all config +``` + +## Inputs used by `services-up.sh` + +- `default-network.yml` +- discovered `docker-compose.yml` / `docker-compose.yaml` files under `core/`, `apps/`, `monitoring/` +- `default-environment.env` +- `secrets/stack-secrets.env` (local, not committed) + +## Typical workflows + +### Validate final composed model + +```bash +./services-up.sh --profile all config +``` + +Use this to review merged services, networks, volumes, profiles, and environment substitution. + +### Validate one compose file directly + +```bash +docker compose -f apps/nextcloud/docker-compose.yml config +``` + +Use this when focused on one service family. + +### Deployment prerequisites + +Before runtime operations, follow [DEPLOYMENT.md](../DEPLOYMENT.md) to create required local secret files. + +## What not to do + +- Do not treat archived compose files in `archive/` as active runtime definitions. +- Do not hardcode secrets in committed compose files. +- Do not bypass `services-up.sh` when trying to understand full active composition. + +## Related docs + +- [docs/source-of-truth.md](source-of-truth.md) +- [docs/repo-structure.md](repo-structure.md) +- [docs/architecture.md](architecture.md) diff --git a/docs/documentation-strategy.md b/docs/documentation-strategy.md new file mode 100644 index 0000000..97e2742 --- /dev/null +++ b/docs/documentation-strategy.md @@ -0,0 +1,36 @@ +# Documentation Strategy + +This repository's documentation should help both humans and Codex agents make safe, accurate changes. + +## Principles + +1. **Authority first**: identify authoritative files clearly (Compose + `services-up.sh` for runtime; Terraform for structured inventory/reconciliation). +2. **Task-oriented docs**: include practical workflow steps, not only conceptual text. +3. **No speculation**: document implemented behavior, not aspirational designs without code evidence. +4. **Cross-linking**: root README and topic docs should point to each other for discoverability. +5. **Safety clarity**: explicitly note what should not be committed or applied casually. + +## Documentation quality checklist + +Before merging doc changes, check: + +- Is this statement verifiable from current repo files? +- Does this conflict with existing docs? +- Does this clarify source-of-truth boundaries? +- Does this improve a real workflow for contributors/Codex? +- Are sensitive details excluded? + +## Recommended update cadence + +Update docs when any of these change: + +- `services-up.sh` composition behavior, +- major Compose directory or profile structure, +- Terraform workflow conventions, +- inventory output shapes, +- secret handling conventions. + +## Audience-specific outcomes + +- Humans should quickly understand how to operate the repo safely. +- Future Codex runs should quickly identify authoritative files, guardrails, and reconciliation workflows. diff --git a/docs/infrastructure-inventory.md b/docs/infrastructure-inventory.md new file mode 100644 index 0000000..6cb62d7 --- /dev/null +++ b/docs/infrastructure-inventory.md @@ -0,0 +1,51 @@ +# Infrastructure Inventory Model + +This repository treats infrastructure inventory as first-class documentation. + +## Intent + +The goal is not only deployment configuration, but also a maintainable map of: + +- what hosts/VMs exist, +- how they are identified, +- what selected runtime objects are mirrored into Terraform, +- what outputs can be consumed by docs and future tooling. + +## Current inventory sources + +### 1) Terraform Proxmox layer + +`infrastructure/terraform/proxmox/` contains imported/reconciled VM resources and local metadata for physical hosts. + +This is currently the most structured host/VM inventory in the repo. + +### 2) Terraform Docker layer + +`infrastructure/terraform/docker/` contains selective Docker container resources used as documentation-oriented mirrors. + +These resources should match existing running containers, not redefine runtime composition strategy. + +### 3) Compose runtime definitions + +Compose files define intended service runtime composition, networking, labels, and integration. + +### 4) Architecture docs + +`docs/architecture.md` provides a human-readable topology view based on repository configuration and observed runtime signals. + +## Output shaping expectations + +When adding Terraform outputs for documentation/tooling: + +- prefer concise inventory maps/lists, +- include stable identifiers and roles, +- avoid raw giant provider objects where possible, +- include descriptions so future consumers understand intent. + +## Limitations today + +- No full generated inventory document pipeline is present yet. +- Some Terraform files still include generated boilerplate comments requiring ongoing cleanup. +- Ansible/NixOS operational layers are not yet implemented in a way that provides authoritative inventory in this repo. + +These limitations are expected for the current adoption stage. diff --git a/docs/repo-structure.md b/docs/repo-structure.md new file mode 100644 index 0000000..cc7cccd --- /dev/null +++ b/docs/repo-structure.md @@ -0,0 +1,37 @@ +# Repository Structure + +This page explains where to find authoritative files quickly. + +## Top-level directories + +- `core/` — core platform/security services (Traefik, Authelia, CrowdSec, error pages). +- `apps/` — user/business applications (Nextcloud, Passbolt, Gitea, Gramps, SearXNG). +- `monitoring/` — observability and operational tooling (Prometheus, Grafana, InfluxDB, Node-RED, etc.). +- `infrastructure/terraform/` — brownfield Terraform inventory/reconciliation layers. +- `docs/` — repository-level architecture and workflow documentation. +- `archive/` — historical compose/config artifacts not part of active runtime composition. +- `secrets/` — local secret material and templates; never commit real values. + +## Key top-level files + +- `services-up.sh` — runtime composition entrypoint for multi-compose environment. +- `default-network.yml` — shared docker network definitions used across compose files. +- `default-environment.env` — non-secret default env values for compose rendering. +- `DEPLOYMENT.md` — prerequisite setup before runtime operations. +- `SECURITY_SECRETS_INVENTORY.md` — tracked secret-variable inventory and usage map. + +## Terraform layout + +- `infrastructure/terraform/README.md` — Terraform purpose and boundaries in this repo. +- `infrastructure/terraform/proxmox/` — imported/reconciled Proxmox VM resources and host metadata. +- `infrastructure/terraform/docker/` — selective Docker container documentation mirrors. +- `infrastructure/terraform/bootstrap/` — backend/provider bootstrap scaffolding. +- `infrastructure/terraform/scripts/reconcile_from_plan.sh` — helper for `terraform plan -generate-config-out` reconciliation workflow. + +## Fast path for future Codex runs + +1. Read [README.md](../README.md). +2. Read [docs/source-of-truth.md](source-of-truth.md). +3. Read [docs/docker-environment.md](docker-environment.md). +4. Read [docs/terraform-workflows.md](terraform-workflows.md). +5. Only then edit Compose/Terraform files. diff --git a/docs/source-of-truth.md b/docs/source-of-truth.md new file mode 100644 index 0000000..5e39f1a --- /dev/null +++ b/docs/source-of-truth.md @@ -0,0 +1,47 @@ +# Source-of-Truth Boundaries + +This repository has multiple layers. Knowing the authority for each layer prevents accidental drift. + +## Boundary summary + +| Layer | Primary authority | Purpose | +|---|---|---| +| Application/runtime container composition | `services-up.sh` + Compose files under `core/`, `apps/`, `monitoring/` | What runs in the Docker environment and how services are wired. | +| Docker shared baseline inputs | `default-network.yml`, `default-environment.env`, `secrets/stack-secrets.env` | Shared network/env material applied during compose rendering. | +| Infrastructure inventory and reconciliation | Terraform under `infrastructure/terraform/` | Codified inventory of existing infrastructure and relationships, especially Proxmox VMs and selected Docker mirrors. | +| Secret policy and inventory | `SECURITY_SECRETS_INVENTORY.md` + local secret files in `secrets/` | What secrets exist and where they are expected. | + +## Practical meaning + +### Docker runtime decisions + +Change Compose files and `services-up.sh` when changing runtime behavior. + +Do **not** assume Terraform Docker resources are the deployment source for day-to-day service runtime. + +### Infrastructure inventory decisions + +Use Terraform when documenting/reconciling existing: + +- Proxmox VM config and identifiers. +- Physical host metadata. +- Select Docker container details that are intentionally mirrored. + +Do **not** treat Terraform as a full replacement for Compose operations in this repo. + +## Declared config vs observed/runtime state + +- **Declared config**: files in this repository (Compose, Terraform, docs). +- **Observed/runtime state**: live Docker/Proxmox reality and Terraform state snapshots. + +Brownfield workflows reconcile these two safely and incrementally. + +## Guardrails for contributors and Codex + +- Do not mass-import or mass-reconcile everything at once. +- Keep imports/reconciliation scoped to one object (or small set) at a time. +- Keep `ignore_changes` surgical and justified. +- Prefer shaped outputs (inventory-ready) over raw provider object dumps. +- Do not commit `.tfstate`, real `.tfvars`, or real secret files. + +See [docs/terraform-workflows.md](terraform-workflows.md) for step-by-step procedures. diff --git a/docs/terraform-workflows.md b/docs/terraform-workflows.md new file mode 100644 index 0000000..37d216a --- /dev/null +++ b/docs/terraform-workflows.md @@ -0,0 +1,78 @@ +# Terraform Workflows (Brownfield / Reconciliation) + +Terraform in this repository is primarily used for **importing and reconciling existing infrastructure**. + +This is a brownfield workflow: real infrastructure exists first, then code/state are brought into alignment. + +## Core workflow pattern + +1. Define/import one existing object. +2. Inspect current provider state. +3. Reconcile hand-maintained `.tf` configuration. +4. Use targeted `ignore_changes` only when necessary. +5. Iterate until plan is sane/no-op for intended scope. +6. Avoid casual apply operations. + +## Docker mirror workflow (documentation-oriented) + +Directory: `infrastructure/terraform/docker/` + +Use when intentionally mirroring selected running containers as structured documentation. + +### Steps + +1. Add minimal `docker_container` resource block (or uncomment/import-ready block). +2. Add `import {}` block or run `terraform import` for the container. +3. Run plan and inspect generated/state values. +4. Keep only meaningful, maintainable arguments in hand-edited files. +5. Use generated files as draft input, not final truth. +6. Re-run plan until intended scope is clean. + +## Proxmox VM workflow + +Directory: `infrastructure/terraform/proxmox/` + +Use for existing Proxmox VMs and metadata reconciliation. + +### Steps + +1. Add/import one VM resource at a time. +2. Confirm provider import ID format and vm/node mapping. +3. Inspect with `terraform state show` / plan output. +4. Move useful arguments into stable hand-maintained files. +5. Keep lifecycle ignore rules narrow and explicit. +6. Iterate per VM until plan stabilizes. + +## Physical host metadata workflow + +Physical host metadata currently lives in Proxmox Terraform locals/outputs and is used as documentation inventory context. + +When updating: + +1. update locals with factual host metadata, +2. ensure outputs remain documentation-friendly, +3. avoid leaking sensitive internal data not needed for repository goals. + +## Generated config guidance + +`infrastructure/terraform/scripts/reconcile_from_plan.sh` can generate Terraform draft configuration via `-generate-config-out`. + +Treat generated files as: + +- a starting point, +- reviewed manually, +- reduced to meaningful attributes, +- reformatted and split into maintainable files. + +## Safety reminders + +- Do not commit `.tfstate*` or real `.tfvars`. +- Do not commit credentials. +- Do not run `terraform apply`/`destroy` casually. +- Keep changes incremental and reviewable. + +## Related docs + +- [docs/source-of-truth.md](source-of-truth.md) +- [docs/infrastructure-inventory.md](infrastructure-inventory.md) +- [infrastructure/terraform/README.md](../infrastructure/terraform/README.md) diff --git a/infrastructure/terraform/README.md b/infrastructure/terraform/README.md index 53f03d4..8622aff 100644 --- a/infrastructure/terraform/README.md +++ b/infrastructure/terraform/README.md @@ -1,54 +1,60 @@ -# Terraform foundations +# Terraform in This Repository -This directory introduces Terraform in a conservative, incremental way for this homelab repo. +Terraform here is used as a **structured inventory + reconciliation layer** for existing infrastructure. -## Purpose in this repository +It does **not** replace Docker Compose as runtime deployment authority. -Terraform is used here to **document and gradually adopt management** of existing infrastructure without disrupting running services. +## What Terraform is currently used for -Current intent: -- Start with imported live Docker resources so infrastructure is visible and reproducible in code. -- Add Proxmox inventory/configuration later once provider details and import IDs are confirmed. -- Keep this phase local-state and learning-oriented (no remote backend yet). +- Proxmox VM import/reconciliation for existing VMs. +- Physical host metadata represented in Terraform locals/outputs. +- Select Docker container mirror resources for documentation-oriented tracking. +- Outputs that can support documentation and later downstream tooling. -## Tool boundaries +## What Terraform is not used for (today) -- **Docker Compose**: day-to-day application/service runtime definitions already used by this repo. -- **Terraform**: infrastructure state capture and controlled resource management (starting with imports). -- **Ansible**: follow-on host/configuration management after Terraform inventory and targets are stable. -- **NixOS**: host OS/system-level declarative configuration, separate from per-service compose workflows. +- Replacing `services-up.sh` / Compose for day-to-day app runtime orchestration. +- Broad, immediate greenfield provisioning of the whole stack. +- Casual `apply` operations across all infrastructure. -## Layout +## Directory map -- `docker/`: Docker provider scaffold and incremental import workflow. -- `proxmox/`: placeholder scaffold for later Proxmox adoption. -- `modules/`: placeholder module directories for future shared patterns. +- `proxmox/` — imported/reconciled VM resources and host metadata outputs. +- `docker/` — selective Docker container import/mirror resources. +- `bootstrap/` — backend/provider bootstrap scaffolding. +- `modules/` — placeholder module directories for future stable abstractions. +- `scripts/reconcile_from_plan.sh` — helper to convert generated plan config into reviewable draft files. -## Incremental adoption plan +## Brownfield workflow standard -1. Import Docker containers one-by-one into Terraform state. -2. Reconcile and stabilize Docker Terraform configuration until `terraform plan` is clean. -3. Add Proxmox inventory/configuration scaffolding and imports later. -4. Introduce Ansible workflow after Terraform-managed inventory is trustworthy. +1. Import one existing object. +2. Inspect state/plan output. +3. Reconcile hand-maintained Terraform code. +4. Keep `ignore_changes` narrowly scoped. +5. Iterate to no-op/sane plan for intended scope. +6. Avoid casual apply. +See detailed steps in [../../docs/terraform-workflows.md](../../docs/terraform-workflows.md). -## Plan-to-config helper script +## Safe validation commands -Use `scripts/reconcile_from_plan.sh` to automate Terraform configuration generation from `terraform plan` output (via Terraform's `-generate-config-out`). - -From a Terraform module directory (for example `infrastructure/terraform/docker`): +From Terraform directories, preferred checks are: ```bash -../../scripts/reconcile_from_plan.sh --output-file zz_generated_from_plan.auto.tf +terraform fmt -check -recursive +terraform init -backend=false -input=false +terraform validate ``` -Notes: -- Best used with an import-first workflow that already contains `import {}` blocks. -- The script writes generated config into a `.auto.tf` file and runs `terraform fmt` on it. -- Always review generated arguments before apply. +## Secrets and state safety -## Safety notes +- Do not commit `.tfstate*`. +- Do not commit real `.tfvars` values. +- Keep credentials in local, untracked inputs only. -- State files are intentionally gitignored for safety and portability. -- Do **not** run `terraform apply` until imported resources are fully reconciled and plan output is reviewed as no-op for intended targets. -- No remote backend is configured yet by design. +## Related docs + +- [../../docs/source-of-truth.md](../../docs/source-of-truth.md) +- [../../docs/infrastructure-inventory.md](../../docs/infrastructure-inventory.md) +- [docker/README.md](docker/README.md) +- [proxmox/README.md](proxmox/README.md) diff --git a/infrastructure/terraform/docker/README.md b/infrastructure/terraform/docker/README.md index 1a99988..e119b52 100644 --- a/infrastructure/terraform/docker/README.md +++ b/infrastructure/terraform/docker/README.md @@ -1,48 +1,43 @@ -# Docker Terraform scaffold +# Terraform Docker Mirror Layer -This directory is for **incremental, import-first Terraform adoption** of existing Docker containers. +This directory tracks selected existing Docker containers in Terraform for inventory/documentation purposes. -## What this directory is for +## Purpose -- Document existing live Docker resources in Terraform. -- Import containers one-by-one. -- Reconcile Terraform code with real runtime values until plan is clean. +- Mirror specific running containers as Terraform resources. +- Reconcile imported state into maintainable code. +- Produce structured outputs/reminders that support documentation workflows. -## Initialize +## Boundary with Docker Compose -From this directory: +Docker Compose + `services-up.sh` remain runtime composition authority. -```bash -terraform init -``` +Terraform resources here are **not** the primary day-to-day deployment mechanism for app services. -## Safe incremental import workflow +## Current contents -1. Add one `docker_container` resource block in `main.tf` for an already-running container. -2. Import it into state: +- `main.tf` — import-first workflow notes and minimal scaffolding. +- `searxng-webapp.tf` — generated/reconciled example container resource. +- `outputs.tf` — documentation-oriented reminders/outputs. +- `terraform.tfvars.example` — safe template for local values. - ```bash - terraform import docker_container. - ``` +## Import/reconciliation workflow -3. Inspect imported state: +1. Start with one existing container. +2. Import with `import {}` block or `terraform import`. +3. Inspect state / generated config. +4. Reduce generated attributes to meaningful, stable arguments. +5. Keep lifecycle `ignore_changes` narrow and justified. +6. Iterate until plan is clean for the intended resource. - ```bash - terraform state show docker_container. - ``` +## Guardrails -4. Copy required/meaningful arguments from state output into `main.tf`. -5. Run `terraform plan` and refine until there are no unintended changes. +- Do not attempt to mirror all containers in one pass. +- Do not commit local state or real credentials. +- Treat generated config as draft input that needs review. -## Reconciliation guidance +## Related docs -- Keep one resource block per imported container. -- Prefer explicit values for arguments that affect recreation. -- Avoid broad changes; reconcile each container independently. -- Do **not** run `terraform apply` until plan output is intentionally clean. - -## State and secrets handling - -- State files are ignored via `../.gitignore` because they may contain environment-specific metadata. -- Do not commit real `.tfvars` files with machine-specific values. -- Use `terraform.tfvars.example` as a safe starter template. +- [../README.md](../README.md) +- [../../../docs/source-of-truth.md](../../../docs/source-of-truth.md) +- [../../../docs/terraform-workflows.md](../../../docs/terraform-workflows.md) diff --git a/infrastructure/terraform/proxmox/README.md b/infrastructure/terraform/proxmox/README.md index b70f961..4fda99b 100644 --- a/infrastructure/terraform/proxmox/README.md +++ b/infrastructure/terraform/proxmox/README.md @@ -1,36 +1,44 @@ -# Proxmox Terraform scaffold +# Terraform Proxmox Inventory Layer -This directory is a **placeholder scaffold** for future Proxmox Terraform adoption. +This directory codifies existing Proxmox infrastructure using an import-first reconciliation model. -## What this directory is for +## Purpose -- Prepare provider/version/variable structure now. -- Delay real Proxmox resource management until import strategy is validated. +- Track existing Proxmox VMs in Terraform. +- Reconcile imported VM configuration into maintainable, explicit files. +- Represent physical host metadata as structured Terraform locals/outputs. +- Support documentation inventory and future downstream tooling. -## Initialize +## Current repository status -From this directory: +This directory already contains imported/reconciled VM resources (for example `docker`, `server-nixos`, `nix-cache`, `pbs`, `pihole`) plus host metadata locals/outputs. -```bash -terraform init -``` +This means it is no longer just a scaffold; treat it as active infrastructure inventory code. -## Current status +## Workflow standard (brownfield) -- No live Proxmox resources are defined yet. -- Provider auth variables are placeholders only. -- Import IDs and resource schemas must be verified against provider docs before adding resources. +1. Import one existing VM at a time. +2. Confirm provider-specific import ID format. +3. Inspect state/plan details. +4. Keep hand-maintained `.tf` files focused and readable. +5. Use `ignore_changes` only where drift noise is unavoidable. +6. Stop when plan is sane/no-op for intended scope. -## Future safe workflow +## File organization expectations -1. Add one resource block for an existing VM/object. -2. Import it with the provider-specific ID format. -3. Inspect with `terraform state show`. -4. Reconcile `.tf` arguments until `terraform plan` is clean. -5. Repeat incrementally. +- Prefer one-resource-per-file patterns when practical. +- Keep shared metadata in `locals`/outputs with clear descriptions. +- Keep generated comments/config under ongoing cleanup rather than assuming generated output is final. ## Safety notes -- Do not commit real credentials in `.tf` files or tracked `.tfvars`. -- State files are ignored by the Terraform-level `.gitignore`. -- Do not run `terraform apply` until plan is intentionally no-op for existing resources. +- Do not run broad applies casually. +- Do not commit real credentials or `.tfstate*`. +- Keep changes incremental and reviewable. + +## Related docs + +- [../README.md](../README.md) +- [../../../docs/source-of-truth.md](../../../docs/source-of-truth.md) +- [../../../docs/terraform-workflows.md](../../../docs/terraform-workflows.md) +- [../../../docs/infrastructure-inventory.md](../../../docs/infrastructure-inventory.md)