beatzaplenty/docker
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: overhaul repo documentation and workflow guides

Browse Source
This commit is contained in:
beatz174-bit
2026-04-21 09:28:55 +10:00
parent 020d6ecb79
commit c7dd9f2229
13 changed files with 539 additions and 174 deletions
Download Patch File Download Diff File Expand all files Collapse all files
DEPLOYMENT.md
+21 -20
View File
@@ -1,15 +1,17 @@
# Deployment prerequisites (required) # Deployment Prerequisites
Before running `docker compose up`, you **must** provision runtime secrets. Before running compose operations, provision local secret material.
## 1) Create non-committed secret files ## 1) Create non-committed secret env file
```bash ```bash
cp secrets/.env.secrets.example secrets/stack-secrets.env cp secrets/.env.secrets.example secrets/stack-secrets.env
chmod 600 secrets/stack-secrets.env chmod 600 secrets/stack-secrets.env
``` ```
Create these Docker secret files (all ignored by git): ## 2) Create required Docker secret files
All files below are expected locally and are gitignored:
- `secrets/nextcloud_db_root_password.txt` - `secrets/nextcloud_db_root_password.txt`
- `secrets/nextcloud_db_password.txt` - `secrets/nextcloud_db_password.txt`
@@ -26,23 +28,22 @@ Recommended permissions:
chmod 600 secrets/*.txt chmod 600 secrets/*.txt
``` ```
## 2) Rotate previously committed credentials ## 3) Validate composed configuration
These values were previously hardcoded and must be rotated in upstream systems immediately: Use the repository composition entrypoint:
- Database credentials (Nextcloud, Passbolt, InfluxDB).
- Nextcloud SMTP app password.
- Authelia reset JWT secret, session secret, storage encryption key.
- Traefik CrowdSec LAPI key.
- Gotify admin password.
- Prometheus Uptime Kuma basic-auth password.
## 3) Start stack
After secrets are provisioned:
```bash ```bash
docker compose -f core/docker-compose.yml up -d ./services-up.sh --profile all config
docker compose -f monitoring/prometheus/docker-compose.yml up -d
docker compose -f apps/nextcloud/docker-compose.yml up -d
``` ```
This confirms compose rendering with shared env/network inputs before any runtime operation.
## 4) Rotate previously committed credentials
If migrating from older states where secrets were committed, rotate upstream values immediately (DB credentials, app passwords, auth keys, and API tokens).
## Related docs
- [SECURITY_SECRETS_INVENTORY.md](SECURITY_SECRETS_INVENTORY.md)
- [docs/docker-environment.md](docs/docker-environment.md)
- [docs/source-of-truth.md](docs/source-of-truth.md)
README.md
+63 -4
View File
@@ -1,8 +1,67 @@
# Docker + Traefik Homelab Stack # Homelab Docker + Terraform Inventory Repository
This repository defines a multi-compose Docker environment with Traefik as ingress, app workloads, and a monitoring/alerting plane. This repository is both:
## High-Level Architecture 1. **operational** (Docker Compose application/runtime definition), and
2. **documentary/inventory-oriented** (Terraform capture of Proxmox VMs, host metadata, and selected Docker objects).
If you only read one section, read **[Source-of-truth boundaries](docs/source-of-truth.md)** first.
---
## Quick navigation
- Architecture overview: [docs/architecture.md](docs/architecture.md)
- Repository layout: [docs/repo-structure.md](docs/repo-structure.md)
- Source-of-truth boundaries and guardrails: [docs/source-of-truth.md](docs/source-of-truth.md)
- Docker environment composition and `services-up.sh`: [docs/docker-environment.md](docs/docker-environment.md)
- Terraform workflows (brownfield import/reconciliation): [docs/terraform-workflows.md](docs/terraform-workflows.md)
- Infrastructure inventory intent and outputs: [docs/infrastructure-inventory.md](docs/infrastructure-inventory.md)
- Deployment prerequisites and secrets setup: [DEPLOYMENT.md](DEPLOYMENT.md)
- Secrets inventory: [SECURITY_SECRETS_INVENTORY.md](SECURITY_SECRETS_INVENTORY.md)
Terraform subtrees:
- Terraform root docs: [infrastructure/terraform/README.md](infrastructure/terraform/README.md)
- Terraform Docker mirror: [infrastructure/terraform/docker/README.md](infrastructure/terraform/docker/README.md)
- Terraform Proxmox inventory: [infrastructure/terraform/proxmox/README.md](infrastructure/terraform/proxmox/README.md)
---
## Operating model
### Docker Compose (runtime authority)
- Compose files under `core/`, `apps/`, and `monitoring/` describe runtime services.
- `services-up.sh` composes the environment by discovering compose files and applying common env/network inputs.
- For service runtime behavior, start from Compose files and `services-up.sh` (not Terraform).
### Terraform (inventory and reconciliation authority)
- Terraform under `infrastructure/terraform/` is used to codify and reconcile existing infrastructure.
- Current repo usage emphasizes **brownfield import-first workflows** and safe reconciliation.
- Terraform captures:
- Proxmox VM configuration for existing VMs.
- Physical host metadata in locals/outputs.
- Documentation-oriented Docker container mirroring (limited, selective).
Terraform here is **not** a replacement for Docker Compose deployment.
---
## Guardrails
- Do not run destructive Terraform commands casually.
- Do not treat generated Terraform config as final without manual review.
- Do not commit real secrets, credentials, or local state.
- Keep one-resource-per-file patterns where already established in Terraform subdirectories.
- Prefer shaping outputs for documentation/tooling consumption over dumping raw provider objects.
See [docs/source-of-truth.md](docs/source-of-truth.md) and [docs/terraform-workflows.md](docs/terraform-workflows.md) for concrete do/don't guidance.
---
## High-level architecture
```mermaid ```mermaid
flowchart TB flowchart TB
@@ -43,4 +102,4 @@ flowchart TB
Prometheus --> Gotify Prometheus --> Gotify
``` ```
For a request-flow/network view and architecture notes, see [docs/architecture.md](docs/architecture.md). For request-flow and network detail, see [docs/architecture.md](docs/architecture.md).
SECURITY_SECRETS_INVENTORY.md
+29 -11
View File
@@ -1,6 +1,14 @@
# Security Secrets Inventory # Security Secrets Inventory
This inventory is aligned with `secrets/.env.secrets.example` and documents only the values that are expected to be set in the non-committed secrets env file (`secrets/stack-secrets.env`). This inventory tracks values expected in local secret material and where they are consumed.
## Scope and authority
- Canonical example template: `secrets/.env.secrets.example`
- Runtime-loaded secret env file: `secrets/stack-secrets.env` (local, non-committed)
- Docker secret files: `secrets/*.txt` (local, non-committed)
For setup steps, see [DEPLOYMENT.md](DEPLOYMENT.md).
## Secrets expected in `secrets/.env.secrets.example` ## Secrets expected in `secrets/.env.secrets.example`
@@ -22,16 +30,26 @@ This inventory is aligned with `secrets/.env.secrets.example` and documents only
| `INFLUXDB_INIT_USERNAME` | `monitoring/prometheus/docker-compose.yml` | InfluxDB initial username. | | `INFLUXDB_INIT_USERNAME` | `monitoring/prometheus/docker-compose.yml` | InfluxDB initial username. |
| `PIHOLE_PASSWORD` | `monitoring/prometheus/docker-compose.yml` | Exporter auth / Pi-hole integration password. | | `PIHOLE_PASSWORD` | `monitoring/prometheus/docker-compose.yml` | Exporter auth / Pi-hole integration password. |
## Managed outside `.env.secrets.example` ## Secrets managed outside `.env.secrets.example`
The following sensitive values are intentionally not duplicated in `secrets/.env.secrets.example` because they are provided via Docker secrets (`*_FILE`) or other mounted secret files: The following sensitive values are intentionally provided via Docker secret files (`*_FILE`) or other mounted secret files instead of the shared `.env` template:
- Database/root passwords for Nextcloud, Passbolt, and supporting services that are wired through Docker secrets. - Database/root passwords for Nextcloud, Passbolt, and supporting services.
- Redis runtime password (`--requirepass`) loaded from a Docker secret. - Redis runtime password loaded from a Docker secret.
- `DOCKER_INFLUXDB_INIT_PASSWORD` loaded from Docker secret in monitoring. - `DOCKER_INFLUXDB_INIT_PASSWORD` loaded from Docker secret in monitoring.
- Uptime Kuma basic auth password loaded via `password_file` in Prometheus config. - Uptime Kuma basic-auth password loaded via `password_file` in Prometheus config.
- Core stack secrets injected via env substitution in committed config files, such as: - Core stack secret values injected via env substitution (for example Authelia and CrowdSec values).
- `AUTHELIA_JWT_SECRET`
- `AUTHELIA_SESSION_SECRET` ## Commit safety rules
- `AUTHELIA_STORAGE_ENCRYPTION_KEY`
- `CROWDSEC_LAPI_KEY` Never commit:
- `secrets/stack-secrets.env`
- `secrets/*.txt` real secret files
- real Terraform `.tfvars` containing credentials
- Terraform state files with sensitive runtime metadata
## Related docs
- [DEPLOYMENT.md](DEPLOYMENT.md)
- [docs/source-of-truth.md](docs/source-of-truth.md)
docs/architecture.md
+15 -47
View File
@@ -2,13 +2,9 @@
## Overview ## Overview
This stack uses **Traefik v3** as the internet-facing ingress for application and operations UIs. Service routing is primarily label-driven from Docker Compose files, with a shared `traefik` bridge network for reverse-proxied traffic and a `monitor` network for internal telemetry components. This stack uses **Traefik v3** as internet-facing ingress for application and operations UIs. Service routing is label-driven from Docker Compose files, with shared Docker networks (`traefik`, `monitor`) connecting reverse-proxied and telemetry services.
TLS is terminated at Traefik using ACME HTTP challenge (`myresolver`), with additional hardening via: TLS is terminated at Traefik (ACME HTTP challenge), with hardening via middleware chains, Authelia forward-auth for selected routes, CrowdSec integration, and mTLS options for private-admin paths.
- a default middleware chain (security headers, CrowdSec bouncer, error pages),
- Authelia forward-auth middleware on selected routes,
- mTLS TLS options (`mtls-private-admin`) on private-admin endpoints.
## Network / Request Flow ## Network / Request Flow
@@ -18,7 +14,7 @@ flowchart LR
T -->|HTTP->HTTPS redirect| T T -->|HTTP->HTTPS redirect| T
T -->|ACME HTTP challenge| LE[Let's Encrypt ACME] T -->|ACME HTTP challenge| LE[Let's Encrypt ACME]
subgraph TraefikNet["Docker network: traefik (172.21.0.0 slash 16)"] subgraph TraefikNet[Docker network: traefik]
A[Authelia] A[Authelia]
CS[CrowdSec LAPI] CS[CrowdSec LAPI]
EP[Error Pages] EP[Error Pages]
@@ -76,51 +72,23 @@ flowchart LR
T --> DSP T --> DSP
``` ```
## Key Components ## Key components
- **Ingress & security plane:** Traefik, Authelia, CrowdSec, Error Pages. - **Ingress/security plane:** Traefik, Authelia, CrowdSec, Error Pages.
- **User-facing applications:** Nextcloud, Passbolt, Gitea, Gramps Web (Family Tree), SearXNG. - **User-facing apps:** Nextcloud, Passbolt, Gitea, Gramps Web, SearXNG.
- **Monitoring/ops:** Prometheus, Grafana, InfluxDB, Node-RED, Uptime Kuma, Portainer, Gotify. - **Monitoring/ops:** Prometheus, Grafana, InfluxDB, Node-RED, Uptime Kuma, Portainer, Gotify.
- **Support plane:** Docker Socket Proxy (shared Docker API gateway for Traefik/automation/ops tools). - **Support plane:** Docker Socket Proxy for controlled Docker API access.
## Remote Hosts Observed ## Relationship to Terraform inventory
Prometheus scrape targets indicate additional infrastructure outside the local Compose deployment, including hostnames for: Terraform in `infrastructure/terraform/` captures infrastructure inventory and reconciliation state for Proxmox VMs, physical host metadata, and selected Docker mirrors.
- `raspberrypi.tail13f623.ts.net` Use architecture docs together with:
- `pve.sweet.home`
- `pbs.sweet.home`
- `pihole`
- `server`
- `nix-cache`
- `kuma.lan.ddnsgeek.com`
## Runtime Inventory Input - [docs/source-of-truth.md](source-of-truth.md)
- [docs/terraform-workflows.md](terraform-workflows.md)
- [docs/infrastructure-inventory.md](infrastructure-inventory.md)
Prometheus runtime inventory snapshots are exported with `scripts/export_prometheus_inventory.py` and committed under `docs/runtime/`. The latest human-readable summary is in [docs/prometheus-inventory.md](prometheus-inventory.md). ## Notes on runtime vs declared state
These artifacts are an observed-runtime input for architecture diagrams/docs and should be combined with repository configuration, not treated as sole source of truth. Runtime scrape targets and health signals are useful observed-state inputs, but they do not replace declared config authority from Compose/Terraform sources.
## Assumptions / Unknowns
The repository provides enough detail to infer **container-level architecture**, but not full **Proxmox host/VM topology**.
Unknowns (left intentionally as placeholders):
- **Proxmox physical hosts:** _unknown from repo contents._
- **VM/LXC inventory and placement:** _unknown from repo contents._
- **Which services run on which Proxmox node(s):** _unknown from repo contents._
- **Inter-host VLAN/subnet layout beyond Docker bridges:** _unknown from repo contents._
If you want, this section can be replaced with a concrete Proxmox topology once you add an inventory source (e.g., Terraform, Ansible inventory, or a diagram export).
### Data sources
- Existing repository architecture docs for declared topology
### Notes from inventory
- The `up` query indicates scrape success from Prometheus perspective only.
- Use static repository architecture docs and deployment configs with this runtime export for complete diagrams.
<!-- END GENERATED PROMETHEUS SECTION -->
docs/docker-environment.md
+61
View File
@@ -0,0 +1,61 @@
# Docker Environment Composition
This repo uses multi-file Docker Compose with a wrapper script as the composition entrypoint.
## Composition source of truth
`services-up.sh` is the composition authority for this repository.
It:
1. discovers compose files under `apps/`, `monitoring/`, and `core/`,
2. prepends shared baseline files,
3. applies `default-environment.env` and `secrets/stack-secrets.env`,
4. invokes `docker compose` with a stable project name.
Because of this, when validating or understanding runtime composition, prefer running:
```bash
./services-up.sh --profile all config
```
## Inputs used by `services-up.sh`
- `default-network.yml`
- discovered `docker-compose.yml` / `docker-compose.yaml` files under `core/`, `apps/`, `monitoring/`
- `default-environment.env`
- `secrets/stack-secrets.env` (local, not committed)
## Typical workflows
### Validate final composed model
```bash
./services-up.sh --profile all config
```
Use this to review merged services, networks, volumes, profiles, and environment substitution.
### Validate one compose file directly
```bash
docker compose -f apps/nextcloud/docker-compose.yml config
```
Use this when focused on one service family.
### Deployment prerequisites
Before runtime operations, follow [DEPLOYMENT.md](../DEPLOYMENT.md) to create required local secret files.
## What not to do
- Do not treat archived compose files in `archive/` as active runtime definitions.
- Do not hardcode secrets in committed compose files.
- Do not bypass `services-up.sh` when trying to understand full active composition.
## Related docs
- [docs/source-of-truth.md](source-of-truth.md)
- [docs/repo-structure.md](repo-structure.md)
- [docs/architecture.md](architecture.md)
docs/documentation-strategy.md
+36
View File
@@ -0,0 +1,36 @@
# Documentation Strategy
This repository's documentation should help both humans and Codex agents make safe, accurate changes.
## Principles
1. **Authority first**: identify authoritative files clearly (Compose + `services-up.sh` for runtime; Terraform for structured inventory/reconciliation).
2. **Task-oriented docs**: include practical workflow steps, not only conceptual text.
3. **No speculation**: document implemented behavior, not aspirational designs without code evidence.
4. **Cross-linking**: root README and topic docs should point to each other for discoverability.
5. **Safety clarity**: explicitly note what should not be committed or applied casually.
## Documentation quality checklist
Before merging doc changes, check:
- Is this statement verifiable from current repo files?
- Does this conflict with existing docs?
- Does this clarify source-of-truth boundaries?
- Does this improve a real workflow for contributors/Codex?
- Are sensitive details excluded?
## Recommended update cadence
Update docs when any of these change:
- `services-up.sh` composition behavior,
- major Compose directory or profile structure,
- Terraform workflow conventions,
- inventory output shapes,
- secret handling conventions.
## Audience-specific outcomes
- Humans should quickly understand how to operate the repo safely.
- Future Codex runs should quickly identify authoritative files, guardrails, and reconciliation workflows.
docs/infrastructure-inventory.md
+51
View File
@@ -0,0 +1,51 @@
# Infrastructure Inventory Model
This repository treats infrastructure inventory as first-class documentation.
## Intent
The goal is not only deployment configuration, but also a maintainable map of:
- what hosts/VMs exist,
- how they are identified,
- what selected runtime objects are mirrored into Terraform,
- what outputs can be consumed by docs and future tooling.
## Current inventory sources
### 1) Terraform Proxmox layer
`infrastructure/terraform/proxmox/` contains imported/reconciled VM resources and local metadata for physical hosts.
This is currently the most structured host/VM inventory in the repo.
### 2) Terraform Docker layer
`infrastructure/terraform/docker/` contains selective Docker container resources used as documentation-oriented mirrors.
These resources should match existing running containers, not redefine runtime composition strategy.
### 3) Compose runtime definitions
Compose files define intended service runtime composition, networking, labels, and integration.
### 4) Architecture docs
`docs/architecture.md` provides a human-readable topology view based on repository configuration and observed runtime signals.
## Output shaping expectations
When adding Terraform outputs for documentation/tooling:
- prefer concise inventory maps/lists,
- include stable identifiers and roles,
- avoid raw giant provider objects where possible,
- include descriptions so future consumers understand intent.
## Limitations today
- No full generated inventory document pipeline is present yet.
- Some Terraform files still include generated boilerplate comments requiring ongoing cleanup.
- Ansible/NixOS operational layers are not yet implemented in a way that provides authoritative inventory in this repo.
These limitations are expected for the current adoption stage.
docs/repo-structure.md
+37
View File
@@ -0,0 +1,37 @@
# Repository Structure
This page explains where to find authoritative files quickly.
## Top-level directories
- `core/` — core platform/security services (Traefik, Authelia, CrowdSec, error pages).
- `apps/` — user/business applications (Nextcloud, Passbolt, Gitea, Gramps, SearXNG).
- `monitoring/` — observability and operational tooling (Prometheus, Grafana, InfluxDB, Node-RED, etc.).
- `infrastructure/terraform/` — brownfield Terraform inventory/reconciliation layers.
- `docs/` — repository-level architecture and workflow documentation.
- `archive/` — historical compose/config artifacts not part of active runtime composition.
- `secrets/` — local secret material and templates; never commit real values.
## Key top-level files
- `services-up.sh` — runtime composition entrypoint for multi-compose environment.
- `default-network.yml` — shared docker network definitions used across compose files.
- `default-environment.env` — non-secret default env values for compose rendering.
- `DEPLOYMENT.md` — prerequisite setup before runtime operations.
- `SECURITY_SECRETS_INVENTORY.md` — tracked secret-variable inventory and usage map.
## Terraform layout
- `infrastructure/terraform/README.md` — Terraform purpose and boundaries in this repo.
- `infrastructure/terraform/proxmox/` — imported/reconciled Proxmox VM resources and host metadata.
- `infrastructure/terraform/docker/` — selective Docker container documentation mirrors.
- `infrastructure/terraform/bootstrap/` — backend/provider bootstrap scaffolding.
- `infrastructure/terraform/scripts/reconcile_from_plan.sh` — helper for `terraform plan -generate-config-out` reconciliation workflow.
## Fast path for future Codex runs
1. Read [README.md](../README.md).
2. Read [docs/source-of-truth.md](source-of-truth.md).
3. Read [docs/docker-environment.md](docker-environment.md).
4. Read [docs/terraform-workflows.md](terraform-workflows.md).
5. Only then edit Compose/Terraform files.
docs/source-of-truth.md
+47
View File
@@ -0,0 +1,47 @@
# Source-of-Truth Boundaries
This repository has multiple layers. Knowing the authority for each layer prevents accidental drift.
## Boundary summary
| Layer | Primary authority | Purpose |
|---|---|---|
| Application/runtime container composition | `services-up.sh` + Compose files under `core/`, `apps/`, `monitoring/` | What runs in the Docker environment and how services are wired. |
| Docker shared baseline inputs | `default-network.yml`, `default-environment.env`, `secrets/stack-secrets.env` | Shared network/env material applied during compose rendering. |
| Infrastructure inventory and reconciliation | Terraform under `infrastructure/terraform/` | Codified inventory of existing infrastructure and relationships, especially Proxmox VMs and selected Docker mirrors. |
| Secret policy and inventory | `SECURITY_SECRETS_INVENTORY.md` + local secret files in `secrets/` | What secrets exist and where they are expected. |
## Practical meaning
### Docker runtime decisions
Change Compose files and `services-up.sh` when changing runtime behavior.
Do **not** assume Terraform Docker resources are the deployment source for day-to-day service runtime.
### Infrastructure inventory decisions
Use Terraform when documenting/reconciling existing:
- Proxmox VM config and identifiers.
- Physical host metadata.
- Select Docker container details that are intentionally mirrored.
Do **not** treat Terraform as a full replacement for Compose operations in this repo.
## Declared config vs observed/runtime state
- **Declared config**: files in this repository (Compose, Terraform, docs).
- **Observed/runtime state**: live Docker/Proxmox reality and Terraform state snapshots.
Brownfield workflows reconcile these two safely and incrementally.
## Guardrails for contributors and Codex
- Do not mass-import or mass-reconcile everything at once.
- Keep imports/reconciliation scoped to one object (or small set) at a time.
- Keep `ignore_changes` surgical and justified.
- Prefer shaped outputs (inventory-ready) over raw provider object dumps.
- Do not commit `.tfstate`, real `.tfvars`, or real secret files.
See [docs/terraform-workflows.md](terraform-workflows.md) for step-by-step procedures.
docs/terraform-workflows.md
+78
View File
@@ -0,0 +1,78 @@
# Terraform Workflows (Brownfield / Reconciliation)
Terraform in this repository is primarily used for **importing and reconciling existing infrastructure**.
This is a brownfield workflow: real infrastructure exists first, then code/state are brought into alignment.
## Core workflow pattern
1. Define/import one existing object.
2. Inspect current provider state.
3. Reconcile hand-maintained `.tf` configuration.
4. Use targeted `ignore_changes` only when necessary.
5. Iterate until plan is sane/no-op for intended scope.
6. Avoid casual apply operations.
## Docker mirror workflow (documentation-oriented)
Directory: `infrastructure/terraform/docker/`
Use when intentionally mirroring selected running containers as structured documentation.
### Steps
1. Add minimal `docker_container` resource block (or uncomment/import-ready block).
2. Add `import {}` block or run `terraform import` for the container.
3. Run plan and inspect generated/state values.
4. Keep only meaningful, maintainable arguments in hand-edited files.
5. Use generated files as draft input, not final truth.
6. Re-run plan until intended scope is clean.
## Proxmox VM workflow
Directory: `infrastructure/terraform/proxmox/`
Use for existing Proxmox VMs and metadata reconciliation.
### Steps
1. Add/import one VM resource at a time.
2. Confirm provider import ID format and vm/node mapping.
3. Inspect with `terraform state show` / plan output.
4. Move useful arguments into stable hand-maintained files.
5. Keep lifecycle ignore rules narrow and explicit.
6. Iterate per VM until plan stabilizes.
## Physical host metadata workflow
Physical host metadata currently lives in Proxmox Terraform locals/outputs and is used as documentation inventory context.
When updating:
1. update locals with factual host metadata,
2. ensure outputs remain documentation-friendly,
3. avoid leaking sensitive internal data not needed for repository goals.
## Generated config guidance
`infrastructure/terraform/scripts/reconcile_from_plan.sh` can generate Terraform draft configuration via `-generate-config-out`.
Treat generated files as:
- a starting point,
- reviewed manually,
- reduced to meaningful attributes,
- reformatted and split into maintainable files.
## Safety reminders
- Do not commit `.tfstate*` or real `.tfvars`.
- Do not commit credentials.
- Do not run `terraform apply`/`destroy` casually.
- Keep changes incremental and reviewable.
## Related docs
- [docs/source-of-truth.md](source-of-truth.md)
- [docs/infrastructure-inventory.md](infrastructure-inventory.md)
- [infrastructure/terraform/README.md](../infrastructure/terraform/README.md)
infrastructure/terraform/README.md
+41 -35
View File
@@ -1,54 +1,60 @@
# Terraform foundations # Terraform in This Repository
This directory introduces Terraform in a conservative, incremental way for this homelab repo. Terraform here is used as a **structured inventory + reconciliation layer** for existing infrastructure.
## Purpose in this repository It does **not** replace Docker Compose as runtime deployment authority.
Terraform is used here to **document and gradually adopt management** of existing infrastructure without disrupting running services. ## What Terraform is currently used for
Current intent: - Proxmox VM import/reconciliation for existing VMs.
- Start with imported live Docker resources so infrastructure is visible and reproducible in code. - Physical host metadata represented in Terraform locals/outputs.
- Add Proxmox inventory/configuration later once provider details and import IDs are confirmed. - Select Docker container mirror resources for documentation-oriented tracking.
- Keep this phase local-state and learning-oriented (no remote backend yet). - Outputs that can support documentation and later downstream tooling.
## Tool boundaries ## What Terraform is not used for (today)
- **Docker Compose**: day-to-day application/service runtime definitions already used by this repo. - Replacing `services-up.sh` / Compose for day-to-day app runtime orchestration.
- **Terraform**: infrastructure state capture and controlled resource management (starting with imports). - Broad, immediate greenfield provisioning of the whole stack.
- **Ansible**: follow-on host/configuration management after Terraform inventory and targets are stable. - Casual `apply` operations across all infrastructure.
- **NixOS**: host OS/system-level declarative configuration, separate from per-service compose workflows.
## Layout ## Directory map
- `docker/`: Docker provider scaffold and incremental import workflow. - `proxmox/` — imported/reconciled VM resources and host metadata outputs.
- `proxmox/`: placeholder scaffold for later Proxmox adoption. - `docker/` — selective Docker container import/mirror resources.
- `modules/`: placeholder module directories for future shared patterns. - `bootstrap/` — backend/provider bootstrap scaffolding.
- `modules/` — placeholder module directories for future stable abstractions.
- `scripts/reconcile_from_plan.sh` — helper to convert generated plan config into reviewable draft files.
## Incremental adoption plan ## Brownfield workflow standard
1. Import Docker containers one-by-one into Terraform state. 1. Import one existing object.
2. Reconcile and stabilize Docker Terraform configuration until `terraform plan` is clean. 2. Inspect state/plan output.
3. Add Proxmox inventory/configuration scaffolding and imports later. 3. Reconcile hand-maintained Terraform code.
4. Introduce Ansible workflow after Terraform-managed inventory is trustworthy. 4. Keep `ignore_changes` narrowly scoped.
5. Iterate to no-op/sane plan for intended scope.
6. Avoid casual apply.
See detailed steps in [../../docs/terraform-workflows.md](../../docs/terraform-workflows.md).
## Plan-to-config helper script ## Safe validation commands
Use `scripts/reconcile_from_plan.sh` to automate Terraform configuration generation from `terraform plan` output (via Terraform's `-generate-config-out`). From Terraform directories, preferred checks are:
From a Terraform module directory (for example `infrastructure/terraform/docker`):
```bash ```bash
../../scripts/reconcile_from_plan.sh --output-file zz_generated_from_plan.auto.tf terraform fmt -check -recursive
terraform init -backend=false -input=false
terraform validate
``` ```
Notes: ## Secrets and state safety
- Best used with an import-first workflow that already contains `import {}` blocks.
- The script writes generated config into a `.auto.tf` file and runs `terraform fmt` on it.
- Always review generated arguments before apply.
## Safety notes - Do not commit `.tfstate*`.
- Do not commit real `.tfvars` values.
- Keep credentials in local, untracked inputs only.
- State files are intentionally gitignored for safety and portability. ## Related docs
- Do **not** run `terraform apply` until imported resources are fully reconciled and plan output is reviewed as no-op for intended targets.
- No remote backend is configured yet by design. - [../../docs/source-of-truth.md](../../docs/source-of-truth.md)
- [../../docs/infrastructure-inventory.md](../../docs/infrastructure-inventory.md)
- [docker/README.md](docker/README.md)
- [proxmox/README.md](proxmox/README.md)
infrastructure/terraform/docker/README.md
+29 -34
View File
@@ -1,48 +1,43 @@
# Docker Terraform scaffold # Terraform Docker Mirror Layer
This directory is for **incremental, import-first Terraform adoption** of existing Docker containers. This directory tracks selected existing Docker containers in Terraform for inventory/documentation purposes.
## What this directory is for ## Purpose
- Document existing live Docker resources in Terraform. - Mirror specific running containers as Terraform resources.
- Import containers one-by-one. - Reconcile imported state into maintainable code.
- Reconcile Terraform code with real runtime values until plan is clean. - Produce structured outputs/reminders that support documentation workflows.
## Initialize ## Boundary with Docker Compose
From this directory: Docker Compose + `services-up.sh` remain runtime composition authority.
```bash Terraform resources here are **not** the primary day-to-day deployment mechanism for app services.
terraform init
```
## Safe incremental import workflow ## Current contents
1. Add one `docker_container` resource block in `main.tf` for an already-running container. - `main.tf` — import-first workflow notes and minimal scaffolding.
2. Import it into state: - `searxng-webapp.tf` — generated/reconciled example container resource.
- `outputs.tf` — documentation-oriented reminders/outputs.
- `terraform.tfvars.example` — safe template for local values.
```bash ## Import/reconciliation workflow
terraform import docker_container.<resource_name> <container_id_or_container_name>
```
3. Inspect imported state: 1. Start with one existing container.
2. Import with `import {}` block or `terraform import`.
3. Inspect state / generated config.
4. Reduce generated attributes to meaningful, stable arguments.
5. Keep lifecycle `ignore_changes` narrow and justified.
6. Iterate until plan is clean for the intended resource.
```bash ## Guardrails
terraform state show docker_container.<resource_name>
```
4. Copy required/meaningful arguments from state output into `main.tf`. - Do not attempt to mirror all containers in one pass.
5. Run `terraform plan` and refine until there are no unintended changes. - Do not commit local state or real credentials.
- Treat generated config as draft input that needs review.
## Reconciliation guidance ## Related docs
- Keep one resource block per imported container. - [../README.md](../README.md)
- Prefer explicit values for arguments that affect recreation. - [../../../docs/source-of-truth.md](../../../docs/source-of-truth.md)
- Avoid broad changes; reconcile each container independently. - [../../../docs/terraform-workflows.md](../../../docs/terraform-workflows.md)
- Do **not** run `terraform apply` until plan output is intentionally clean.
## State and secrets handling
- State files are ignored via `../.gitignore` because they may contain environment-specific metadata.
- Do not commit real `.tfvars` files with machine-specific values.
- Use `terraform.tfvars.example` as a safe starter template.
infrastructure/terraform/proxmox/README.md
+31 -23
View File
@@ -1,36 +1,44 @@
# Proxmox Terraform scaffold # Terraform Proxmox Inventory Layer
This directory is a **placeholder scaffold** for future Proxmox Terraform adoption. This directory codifies existing Proxmox infrastructure using an import-first reconciliation model.
## What this directory is for ## Purpose
- Prepare provider/version/variable structure now. - Track existing Proxmox VMs in Terraform.
- Delay real Proxmox resource management until import strategy is validated. - Reconcile imported VM configuration into maintainable, explicit files.
- Represent physical host metadata as structured Terraform locals/outputs.
- Support documentation inventory and future downstream tooling.
## Initialize ## Current repository status
From this directory: This directory already contains imported/reconciled VM resources (for example `docker`, `server-nixos`, `nix-cache`, `pbs`, `pihole`) plus host metadata locals/outputs.
```bash This means it is no longer just a scaffold; treat it as active infrastructure inventory code.
terraform init
```
## Current status ## Workflow standard (brownfield)
- No live Proxmox resources are defined yet. 1. Import one existing VM at a time.
- Provider auth variables are placeholders only. 2. Confirm provider-specific import ID format.
- Import IDs and resource schemas must be verified against provider docs before adding resources. 3. Inspect state/plan details.
4. Keep hand-maintained `.tf` files focused and readable.
5. Use `ignore_changes` only where drift noise is unavoidable.
6. Stop when plan is sane/no-op for intended scope.
## Future safe workflow ## File organization expectations
1. Add one resource block for an existing VM/object. - Prefer one-resource-per-file patterns when practical.
2. Import it with the provider-specific ID format. - Keep shared metadata in `locals`/outputs with clear descriptions.
3. Inspect with `terraform state show`. - Keep generated comments/config under ongoing cleanup rather than assuming generated output is final.
4. Reconcile `.tf` arguments until `terraform plan` is clean.
5. Repeat incrementally.
## Safety notes ## Safety notes
- Do not commit real credentials in `.tf` files or tracked `.tfvars`. - Do not run broad applies casually.
- State files are ignored by the Terraform-level `.gitignore`. - Do not commit real credentials or `.tfstate*`.
- Do not run `terraform apply` until plan is intentionally no-op for existing resources. - Keep changes incremental and reviewable.
## Related docs
- [../README.md](../README.md)
- [../../../docs/source-of-truth.md](../../../docs/source-of-truth.md)
- [../../../docs/terraform-workflows.md](../../../docs/terraform-workflows.md)
- [../../../docs/infrastructure-inventory.md](../../../docs/infrastructure-inventory.md)