docs: overhaul repo documentation and workflow guides
This commit is contained in:
beatz174-bit
+15
-47
@@ -2,13 +2,9 @@
|
||||
|
||||
## Overview
|
||||
|
||||
This stack uses **Traefik v3** as the internet-facing ingress for application and operations UIs. Service routing is primarily label-driven from Docker Compose files, with a shared `traefik` bridge network for reverse-proxied traffic and a `monitor` network for internal telemetry components.
|
||||
This stack uses **Traefik v3** as internet-facing ingress for application and operations UIs. Service routing is label-driven from Docker Compose files, with shared Docker networks (`traefik`, `monitor`) connecting reverse-proxied and telemetry services.
|
||||
|
||||
TLS is terminated at Traefik using ACME HTTP challenge (`myresolver`), with additional hardening via:
|
||||
|
||||
- a default middleware chain (security headers, CrowdSec bouncer, error pages),
|
||||
- Authelia forward-auth middleware on selected routes,
|
||||
- mTLS TLS options (`mtls-private-admin`) on private-admin endpoints.
|
||||
TLS is terminated at Traefik (ACME HTTP challenge), with hardening via middleware chains, Authelia forward-auth for selected routes, CrowdSec integration, and mTLS options for private-admin paths.
|
||||
|
||||
## Network / Request Flow
|
||||
|
||||
@@ -18,7 +14,7 @@ flowchart LR
|
||||
T -->|HTTP->HTTPS redirect| T
|
||||
T -->|ACME HTTP challenge| LE[Let's Encrypt ACME]
|
||||
|
||||
subgraph TraefikNet["Docker network: traefik (172.21.0.0 slash 16)"]
|
||||
subgraph TraefikNet[Docker network: traefik]
|
||||
A[Authelia]
|
||||
CS[CrowdSec LAPI]
|
||||
EP[Error Pages]
|
||||
@@ -76,51 +72,23 @@ flowchart LR
|
||||
T --> DSP
|
||||
```
|
||||
|
||||
## Key Components
|
||||
## Key components
|
||||
|
||||
- **Ingress & security plane:** Traefik, Authelia, CrowdSec, Error Pages.
|
||||
- **User-facing applications:** Nextcloud, Passbolt, Gitea, Gramps Web (Family Tree), SearXNG.
|
||||
- **Ingress/security plane:** Traefik, Authelia, CrowdSec, Error Pages.
|
||||
- **User-facing apps:** Nextcloud, Passbolt, Gitea, Gramps Web, SearXNG.
|
||||
- **Monitoring/ops:** Prometheus, Grafana, InfluxDB, Node-RED, Uptime Kuma, Portainer, Gotify.
|
||||
- **Support plane:** Docker Socket Proxy (shared Docker API gateway for Traefik/automation/ops tools).
|
||||
- **Support plane:** Docker Socket Proxy for controlled Docker API access.
|
||||
|
||||
## Remote Hosts Observed
|
||||
## Relationship to Terraform inventory
|
||||
|
||||
Prometheus scrape targets indicate additional infrastructure outside the local Compose deployment, including hostnames for:
|
||||
Terraform in `infrastructure/terraform/` captures infrastructure inventory and reconciliation state for Proxmox VMs, physical host metadata, and selected Docker mirrors.
|
||||
|
||||
- `raspberrypi.tail13f623.ts.net`
|
||||
- `pve.sweet.home`
|
||||
- `pbs.sweet.home`
|
||||
- `pihole`
|
||||
- `server`
|
||||
- `nix-cache`
|
||||
- `kuma.lan.ddnsgeek.com`
|
||||
Use architecture docs together with:
|
||||
|
||||
## Runtime Inventory Input
|
||||
- [docs/source-of-truth.md](source-of-truth.md)
|
||||
- [docs/terraform-workflows.md](terraform-workflows.md)
|
||||
- [docs/infrastructure-inventory.md](infrastructure-inventory.md)
|
||||
|
||||
Prometheus runtime inventory snapshots are exported with `scripts/export_prometheus_inventory.py` and committed under `docs/runtime/`. The latest human-readable summary is in [docs/prometheus-inventory.md](prometheus-inventory.md).
|
||||
## Notes on runtime vs declared state
|
||||
|
||||
These artifacts are an observed-runtime input for architecture diagrams/docs and should be combined with repository configuration, not treated as sole source of truth.
|
||||
|
||||
## Assumptions / Unknowns
|
||||
|
||||
The repository provides enough detail to infer **container-level architecture**, but not full **Proxmox host/VM topology**.
|
||||
|
||||
Unknowns (left intentionally as placeholders):
|
||||
|
||||
- **Proxmox physical hosts:** _unknown from repo contents._
|
||||
- **VM/LXC inventory and placement:** _unknown from repo contents._
|
||||
- **Which services run on which Proxmox node(s):** _unknown from repo contents._
|
||||
- **Inter-host VLAN/subnet layout beyond Docker bridges:** _unknown from repo contents._
|
||||
|
||||
If you want, this section can be replaced with a concrete Proxmox topology once you add an inventory source (e.g., Terraform, Ansible inventory, or a diagram export).
|
||||
|
||||
### Data sources
|
||||
|
||||
- Existing repository architecture docs for declared topology
|
||||
|
||||
### Notes from inventory
|
||||
|
||||
- The `up` query indicates scrape success from Prometheus perspective only.
|
||||
- Use static repository architecture docs and deployment configs with this runtime export for complete diagrams.
|
||||
|
||||
<!-- END GENERATED PROMETHEUS SECTION -->
|
||||
Runtime scrape targets and health signals are useful observed-state inputs, but they do not replace declared config authority from Compose/Terraform sources.
|
||||
|
||||
@@ -0,0 +1,61 @@
|
||||
# Docker Environment Composition
|
||||
|
||||
This repo uses multi-file Docker Compose with a wrapper script as the composition entrypoint.
|
||||
|
||||
## Composition source of truth
|
||||
|
||||
`services-up.sh` is the composition authority for this repository.
|
||||
|
||||
It:
|
||||
|
||||
1. discovers compose files under `apps/`, `monitoring/`, and `core/`,
|
||||
2. prepends shared baseline files,
|
||||
3. applies `default-environment.env` and `secrets/stack-secrets.env`,
|
||||
4. invokes `docker compose` with a stable project name.
|
||||
|
||||
Because of this, when validating or understanding runtime composition, prefer running:
|
||||
|
||||
```bash
|
||||
./services-up.sh --profile all config
|
||||
```
|
||||
|
||||
## Inputs used by `services-up.sh`
|
||||
|
||||
- `default-network.yml`
|
||||
- discovered `docker-compose.yml` / `docker-compose.yaml` files under `core/`, `apps/`, `monitoring/`
|
||||
- `default-environment.env`
|
||||
- `secrets/stack-secrets.env` (local, not committed)
|
||||
|
||||
## Typical workflows
|
||||
|
||||
### Validate final composed model
|
||||
|
||||
```bash
|
||||
./services-up.sh --profile all config
|
||||
```
|
||||
|
||||
Use this to review merged services, networks, volumes, profiles, and environment substitution.
|
||||
|
||||
### Validate one compose file directly
|
||||
|
||||
```bash
|
||||
docker compose -f apps/nextcloud/docker-compose.yml config
|
||||
```
|
||||
|
||||
Use this when focused on one service family.
|
||||
|
||||
### Deployment prerequisites
|
||||
|
||||
Before runtime operations, follow [DEPLOYMENT.md](../DEPLOYMENT.md) to create required local secret files.
|
||||
|
||||
## What not to do
|
||||
|
||||
- Do not treat archived compose files in `archive/` as active runtime definitions.
|
||||
- Do not hardcode secrets in committed compose files.
|
||||
- Do not bypass `services-up.sh` when trying to understand full active composition.
|
||||
|
||||
## Related docs
|
||||
|
||||
- [docs/source-of-truth.md](source-of-truth.md)
|
||||
- [docs/repo-structure.md](repo-structure.md)
|
||||
- [docs/architecture.md](architecture.md)
|
||||
@@ -0,0 +1,36 @@
|
||||
# Documentation Strategy
|
||||
|
||||
This repository's documentation should help both humans and Codex agents make safe, accurate changes.
|
||||
|
||||
## Principles
|
||||
|
||||
1. **Authority first**: identify authoritative files clearly (Compose + `services-up.sh` for runtime; Terraform for structured inventory/reconciliation).
|
||||
2. **Task-oriented docs**: include practical workflow steps, not only conceptual text.
|
||||
3. **No speculation**: document implemented behavior, not aspirational designs without code evidence.
|
||||
4. **Cross-linking**: root README and topic docs should point to each other for discoverability.
|
||||
5. **Safety clarity**: explicitly note what should not be committed or applied casually.
|
||||
|
||||
## Documentation quality checklist
|
||||
|
||||
Before merging doc changes, check:
|
||||
|
||||
- Is this statement verifiable from current repo files?
|
||||
- Does this conflict with existing docs?
|
||||
- Does this clarify source-of-truth boundaries?
|
||||
- Does this improve a real workflow for contributors/Codex?
|
||||
- Are sensitive details excluded?
|
||||
|
||||
## Recommended update cadence
|
||||
|
||||
Update docs when any of these change:
|
||||
|
||||
- `services-up.sh` composition behavior,
|
||||
- major Compose directory or profile structure,
|
||||
- Terraform workflow conventions,
|
||||
- inventory output shapes,
|
||||
- secret handling conventions.
|
||||
|
||||
## Audience-specific outcomes
|
||||
|
||||
- Humans should quickly understand how to operate the repo safely.
|
||||
- Future Codex runs should quickly identify authoritative files, guardrails, and reconciliation workflows.
|
||||
@@ -0,0 +1,51 @@
|
||||
# Infrastructure Inventory Model
|
||||
|
||||
This repository treats infrastructure inventory as first-class documentation.
|
||||
|
||||
## Intent
|
||||
|
||||
The goal is not only deployment configuration, but also a maintainable map of:
|
||||
|
||||
- what hosts/VMs exist,
|
||||
- how they are identified,
|
||||
- what selected runtime objects are mirrored into Terraform,
|
||||
- what outputs can be consumed by docs and future tooling.
|
||||
|
||||
## Current inventory sources
|
||||
|
||||
### 1) Terraform Proxmox layer
|
||||
|
||||
`infrastructure/terraform/proxmox/` contains imported/reconciled VM resources and local metadata for physical hosts.
|
||||
|
||||
This is currently the most structured host/VM inventory in the repo.
|
||||
|
||||
### 2) Terraform Docker layer
|
||||
|
||||
`infrastructure/terraform/docker/` contains selective Docker container resources used as documentation-oriented mirrors.
|
||||
|
||||
These resources should match existing running containers, not redefine runtime composition strategy.
|
||||
|
||||
### 3) Compose runtime definitions
|
||||
|
||||
Compose files define intended service runtime composition, networking, labels, and integration.
|
||||
|
||||
### 4) Architecture docs
|
||||
|
||||
`docs/architecture.md` provides a human-readable topology view based on repository configuration and observed runtime signals.
|
||||
|
||||
## Output shaping expectations
|
||||
|
||||
When adding Terraform outputs for documentation/tooling:
|
||||
|
||||
- prefer concise inventory maps/lists,
|
||||
- include stable identifiers and roles,
|
||||
- avoid raw giant provider objects where possible,
|
||||
- include descriptions so future consumers understand intent.
|
||||
|
||||
## Limitations today
|
||||
|
||||
- No full generated inventory document pipeline is present yet.
|
||||
- Some Terraform files still include generated boilerplate comments requiring ongoing cleanup.
|
||||
- Ansible/NixOS operational layers are not yet implemented in a way that provides authoritative inventory in this repo.
|
||||
|
||||
These limitations are expected for the current adoption stage.
|
||||
@@ -0,0 +1,37 @@
|
||||
# Repository Structure
|
||||
|
||||
This page explains where to find authoritative files quickly.
|
||||
|
||||
## Top-level directories
|
||||
|
||||
- `core/` — core platform/security services (Traefik, Authelia, CrowdSec, error pages).
|
||||
- `apps/` — user/business applications (Nextcloud, Passbolt, Gitea, Gramps, SearXNG).
|
||||
- `monitoring/` — observability and operational tooling (Prometheus, Grafana, InfluxDB, Node-RED, etc.).
|
||||
- `infrastructure/terraform/` — brownfield Terraform inventory/reconciliation layers.
|
||||
- `docs/` — repository-level architecture and workflow documentation.
|
||||
- `archive/` — historical compose/config artifacts not part of active runtime composition.
|
||||
- `secrets/` — local secret material and templates; never commit real values.
|
||||
|
||||
## Key top-level files
|
||||
|
||||
- `services-up.sh` — runtime composition entrypoint for multi-compose environment.
|
||||
- `default-network.yml` — shared docker network definitions used across compose files.
|
||||
- `default-environment.env` — non-secret default env values for compose rendering.
|
||||
- `DEPLOYMENT.md` — prerequisite setup before runtime operations.
|
||||
- `SECURITY_SECRETS_INVENTORY.md` — tracked secret-variable inventory and usage map.
|
||||
|
||||
## Terraform layout
|
||||
|
||||
- `infrastructure/terraform/README.md` — Terraform purpose and boundaries in this repo.
|
||||
- `infrastructure/terraform/proxmox/` — imported/reconciled Proxmox VM resources and host metadata.
|
||||
- `infrastructure/terraform/docker/` — selective Docker container documentation mirrors.
|
||||
- `infrastructure/terraform/bootstrap/` — backend/provider bootstrap scaffolding.
|
||||
- `infrastructure/terraform/scripts/reconcile_from_plan.sh` — helper for `terraform plan -generate-config-out` reconciliation workflow.
|
||||
|
||||
## Fast path for future Codex runs
|
||||
|
||||
1. Read [README.md](../README.md).
|
||||
2. Read [docs/source-of-truth.md](source-of-truth.md).
|
||||
3. Read [docs/docker-environment.md](docker-environment.md).
|
||||
4. Read [docs/terraform-workflows.md](terraform-workflows.md).
|
||||
5. Only then edit Compose/Terraform files.
|
||||
@@ -0,0 +1,47 @@
|
||||
# Source-of-Truth Boundaries
|
||||
|
||||
This repository has multiple layers. Knowing the authority for each layer prevents accidental drift.
|
||||
|
||||
## Boundary summary
|
||||
|
||||
| Layer | Primary authority | Purpose |
|
||||
|---|---|---|
|
||||
| Application/runtime container composition | `services-up.sh` + Compose files under `core/`, `apps/`, `monitoring/` | What runs in the Docker environment and how services are wired. |
|
||||
| Docker shared baseline inputs | `default-network.yml`, `default-environment.env`, `secrets/stack-secrets.env` | Shared network/env material applied during compose rendering. |
|
||||
| Infrastructure inventory and reconciliation | Terraform under `infrastructure/terraform/` | Codified inventory of existing infrastructure and relationships, especially Proxmox VMs and selected Docker mirrors. |
|
||||
| Secret policy and inventory | `SECURITY_SECRETS_INVENTORY.md` + local secret files in `secrets/` | What secrets exist and where they are expected. |
|
||||
|
||||
## Practical meaning
|
||||
|
||||
### Docker runtime decisions
|
||||
|
||||
Change Compose files and `services-up.sh` when changing runtime behavior.
|
||||
|
||||
Do **not** assume Terraform Docker resources are the deployment source for day-to-day service runtime.
|
||||
|
||||
### Infrastructure inventory decisions
|
||||
|
||||
Use Terraform when documenting/reconciling existing:
|
||||
|
||||
- Proxmox VM config and identifiers.
|
||||
- Physical host metadata.
|
||||
- Select Docker container details that are intentionally mirrored.
|
||||
|
||||
Do **not** treat Terraform as a full replacement for Compose operations in this repo.
|
||||
|
||||
## Declared config vs observed/runtime state
|
||||
|
||||
- **Declared config**: files in this repository (Compose, Terraform, docs).
|
||||
- **Observed/runtime state**: live Docker/Proxmox reality and Terraform state snapshots.
|
||||
|
||||
Brownfield workflows reconcile these two safely and incrementally.
|
||||
|
||||
## Guardrails for contributors and Codex
|
||||
|
||||
- Do not mass-import or mass-reconcile everything at once.
|
||||
- Keep imports/reconciliation scoped to one object (or small set) at a time.
|
||||
- Keep `ignore_changes` surgical and justified.
|
||||
- Prefer shaped outputs (inventory-ready) over raw provider object dumps.
|
||||
- Do not commit `.tfstate`, real `.tfvars`, or real secret files.
|
||||
|
||||
See [docs/terraform-workflows.md](terraform-workflows.md) for step-by-step procedures.
|
||||
@@ -0,0 +1,78 @@
|
||||
# Terraform Workflows (Brownfield / Reconciliation)
|
||||
|
||||
Terraform in this repository is primarily used for **importing and reconciling existing infrastructure**.
|
||||
|
||||
This is a brownfield workflow: real infrastructure exists first, then code/state are brought into alignment.
|
||||
|
||||
## Core workflow pattern
|
||||
|
||||
1. Define/import one existing object.
|
||||
2. Inspect current provider state.
|
||||
3. Reconcile hand-maintained `.tf` configuration.
|
||||
4. Use targeted `ignore_changes` only when necessary.
|
||||
5. Iterate until plan is sane/no-op for intended scope.
|
||||
6. Avoid casual apply operations.
|
||||
|
||||
## Docker mirror workflow (documentation-oriented)
|
||||
|
||||
Directory: `infrastructure/terraform/docker/`
|
||||
|
||||
Use when intentionally mirroring selected running containers as structured documentation.
|
||||
|
||||
### Steps
|
||||
|
||||
1. Add minimal `docker_container` resource block (or uncomment/import-ready block).
|
||||
2. Add `import {}` block or run `terraform import` for the container.
|
||||
3. Run plan and inspect generated/state values.
|
||||
4. Keep only meaningful, maintainable arguments in hand-edited files.
|
||||
5. Use generated files as draft input, not final truth.
|
||||
6. Re-run plan until intended scope is clean.
|
||||
|
||||
## Proxmox VM workflow
|
||||
|
||||
Directory: `infrastructure/terraform/proxmox/`
|
||||
|
||||
Use for existing Proxmox VMs and metadata reconciliation.
|
||||
|
||||
### Steps
|
||||
|
||||
1. Add/import one VM resource at a time.
|
||||
2. Confirm provider import ID format and vm/node mapping.
|
||||
3. Inspect with `terraform state show` / plan output.
|
||||
4. Move useful arguments into stable hand-maintained files.
|
||||
5. Keep lifecycle ignore rules narrow and explicit.
|
||||
6. Iterate per VM until plan stabilizes.
|
||||
|
||||
## Physical host metadata workflow
|
||||
|
||||
Physical host metadata currently lives in Proxmox Terraform locals/outputs and is used as documentation inventory context.
|
||||
|
||||
When updating:
|
||||
|
||||
1. update locals with factual host metadata,
|
||||
2. ensure outputs remain documentation-friendly,
|
||||
3. avoid leaking sensitive internal data not needed for repository goals.
|
||||
|
||||
## Generated config guidance
|
||||
|
||||
`infrastructure/terraform/scripts/reconcile_from_plan.sh` can generate Terraform draft configuration via `-generate-config-out`.
|
||||
|
||||
Treat generated files as:
|
||||
|
||||
- a starting point,
|
||||
- reviewed manually,
|
||||
- reduced to meaningful attributes,
|
||||
- reformatted and split into maintainable files.
|
||||
|
||||
## Safety reminders
|
||||
|
||||
- Do not commit `.tfstate*` or real `.tfvars`.
|
||||
- Do not commit credentials.
|
||||
- Do not run `terraform apply`/`destroy` casually.
|
||||
- Keep changes incremental and reviewable.
|
||||
|
||||
## Related docs
|
||||
|
||||
- [docs/source-of-truth.md](source-of-truth.md)
|
||||
- [docs/infrastructure-inventory.md](infrastructure-inventory.md)
|
||||
- [infrastructure/terraform/README.md](../infrastructure/terraform/README.md)
|
||||
Reference in New Issue
Block a user