Add strict read-only Dynu DNS inventory integration

This commit is contained in:
beatz174-bit
2026-04-21 12:31:52 +10:00
parent c77db36865
commit 580e9b9aed
9 changed files with 653 additions and 0 deletions
+1
View File
@@ -17,6 +17,7 @@ If you only read one section, read **[Source-of-truth boundaries](docs/source-of
- Docker environment composition and `services-up.sh`: [docs/docker-environment.md](docs/docker-environment.md) - Docker environment composition and `services-up.sh`: [docs/docker-environment.md](docs/docker-environment.md)
- Terraform workflows (brownfield import/reconciliation): [docs/terraform-workflows.md](docs/terraform-workflows.md) - Terraform workflows (brownfield import/reconciliation): [docs/terraform-workflows.md](docs/terraform-workflows.md)
- Infrastructure inventory intent and outputs: [docs/infrastructure-inventory.md](docs/infrastructure-inventory.md) - Infrastructure inventory intent and outputs: [docs/infrastructure-inventory.md](docs/infrastructure-inventory.md)
- Dynu DNS read-only inventory workflow: [docs/dynu-dns-inventory.md](docs/dynu-dns-inventory.md)
- Ansible bootstrap workflows: [docs/ansible-workflows.md](docs/ansible-workflows.md) - Ansible bootstrap workflows: [docs/ansible-workflows.md](docs/ansible-workflows.md)
- Deployment prerequisites and secrets setup: [docs/deployment-prerequisites.md](docs/deployment-prerequisites.md) - Deployment prerequisites and secrets setup: [docs/deployment-prerequisites.md](docs/deployment-prerequisites.md)
- Secrets inventory: [docs/security-secrets.md](docs/security-secrets.md) - Secrets inventory: [docs/security-secrets.md](docs/security-secrets.md)
+61
View File
@@ -0,0 +1,61 @@
# Dynu DNS Read-Only Inventory
This repository includes a **read-only** Dynu DNS inventory workflow for `lan.ddnsgeek.com`.
> This integration is intentionally read-only. No Dynu mutations are permitted in this repo at this stage.
## Scope
- Fetch live DNS/domain data from Dynu using **GET requests only**.
- Correlate Dynu hostnames with Traefik `Host(...)` rules found in compose files.
- Generate local inventory artifacts for documentation.
## Safety Guard Rails
- Scripts fail unless `DYNU_READ_ONLY=true`.
- No Dynu write methods (`POST`, `PUT`, `PATCH`, `DELETE`) are implemented.
- No Terraform Dynu provider/resources/modules are introduced.
- No Ansible Dynu mutation tasks are introduced.
- API secrets are read from environment variables and are never logged.
## Required Environment Variables
- `DYNU_API_KEY` (required)
- `DYNU_BASE_URL` (optional, defaults to `https://api.dynu.com`)
- `DYNU_READ_ONLY` (**must** be `true`)
## Commands
Run directly:
```bash
DYNU_READ_ONLY=true DYNU_API_KEY=... python3 scripts/dynu/fetch_dynu_dns.py
DYNU_READ_ONLY=true python3 scripts/dynu/correlate_dynu_with_traefik.py
```
Or run the wrapper:
```bash
DYNU_READ_ONLY=true DYNU_API_KEY=... scripts/dynu/build_dns_inventory.sh
```
## Artifacts
- `data/dns/dynu_live.json` (generated, untracked by default due to repo `data/` ignore)
- `data/dns/dynu_traefik_inventory.json` (generated, untracked by default)
- `docs/generated/dns-inventory.md` (generated documentation artifact)
Because `data/` is gitignored in this repository, JSON outputs are intentionally local-only unless ignore behavior changes in the future.
## Ansible Wrapper (Read-Only)
A syntax-safe wrapper playbook is provided at:
- `infrastructure/ansible/playbooks/dns-inventory.yml`
It only executes the local read-only scripts and does not call write-capable Dynu APIs.
## Not Managed Yet
Dynu DNS records are **not** managed by Terraform or Ansible in this repository at this stage.
No configuration in this repository sends Dynu mutation requests.
+7
View File
@@ -0,0 +1,7 @@
# DNS Inventory (Dynu + Traefik)
> This integration is intentionally read-only. No Dynu mutations are permitted in this repo at this stage.
This file is generated by `python3 scripts/dynu/correlate_dynu_with_traefik.py` after running `python3 scripts/dynu/fetch_dynu_dns.py`.
No live Dynu data is committed by default because `data/` paths are gitignored in this repository. Generate this document locally when needed.
+10
View File
@@ -40,6 +40,16 @@ Compose files define intended service runtime composition, networking, labels, a
`docs/architecture.md` provides a human-readable topology view based on repository configuration and observed runtime signals. `docs/architecture.md` provides a human-readable topology view based on repository configuration and observed runtime signals.
### 6) Dynu DNS read-only inventory
`scripts/dynu/` and `docs/dynu-dns-inventory.md` provide a strictly read-only DNS inventory workflow:
- fetch Dynu DNS data with GET-only API usage,
- correlate Dynu hostnames with Traefik `Host(...)` labels in Compose sources,
- generate local JSON and markdown artifacts for documentation pipelines.
Dynu write operations are intentionally blocked in this repository stage.
## Output shaping expectations ## Output shaping expectations
When adding Terraform outputs for documentation/tooling: When adding Terraform outputs for documentation/tooling:
+2
View File
@@ -24,6 +24,7 @@ This is intentionally a **foundation stage**, not full production automation.
- `inventory/group_vars/` - shared/group variables. - `inventory/group_vars/` - shared/group variables.
- `inventory/host_vars/` - per-host variables. - `inventory/host_vars/` - per-host variables.
- `playbooks/ping.yml` - minimal syntax/connection test playbook. - `playbooks/ping.yml` - minimal syntax/connection test playbook.
- `playbooks/dns-inventory.yml` - local-only Dynu DNS read-only inventory wrapper.
- `collections/requirements.yml` - lightweight baseline collections. - `collections/requirements.yml` - lightweight baseline collections.
- `roles/` - reserved for future incremental role adoption. - `roles/` - reserved for future incremental role adoption.
@@ -37,6 +38,7 @@ ansible-lint --version
ansible-galaxy collection install -r infrastructure/ansible/collections/requirements.yml -p infrastructure/ansible/collections ansible-galaxy collection install -r infrastructure/ansible/collections/requirements.yml -p infrastructure/ansible/collections
ansible-inventory -i infrastructure/ansible/inventory/hosts.yml --list ansible-inventory -i infrastructure/ansible/inventory/hosts.yml --list
ansible-playbook -i infrastructure/ansible/inventory/hosts.yml infrastructure/ansible/playbooks/ping.yml --syntax-check ansible-playbook -i infrastructure/ansible/inventory/hosts.yml infrastructure/ansible/playbooks/ping.yml --syntax-check
ansible-playbook -i infrastructure/ansible/inventory/hosts.yml infrastructure/ansible/playbooks/dns-inventory.yml --syntax-check
``` ```
## Secrets and safety ## Secrets and safety
@@ -0,0 +1,26 @@
---
# This integration is intentionally read-only.
# No Dynu mutations are permitted in this repo at this stage.
- name: Build Dynu DNS read-only inventory artifacts
hosts: localhost
connection: local
gather_facts: false
vars:
repo_root: "{{ playbook_dir }}/../../.."
tasks:
- name: Assert read-only guard variable is set
ansible.builtin.assert:
that:
- lookup('ansible.builtin.env', 'DYNU_READ_ONLY') == 'true'
fail_msg: "Refusing to run: DYNU_READ_ONLY must be exactly 'true'."
- name: Fetch Dynu DNS (GET-only script)
ansible.builtin.command: python3 scripts/dynu/fetch_dynu_dns.py
args:
chdir: "{{ repo_root }}"
- name: Correlate Dynu with Traefik and generate docs
ansible.builtin.command: python3 scripts/dynu/correlate_dynu_with_traefik.py
args:
chdir: "{{ repo_root }}"
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env bash
set -euo pipefail
# This integration is intentionally read-only.
# No Dynu mutations are permitted in this repo at this stage.
if [[ "${DYNU_READ_ONLY:-}" != "true" ]]; then
echo "Refusing to run: DYNU_READ_ONLY must be exactly 'true'." >&2
exit 2
fi
python3 scripts/dynu/fetch_dynu_dns.py
python3 scripts/dynu/correlate_dynu_with_traefik.py
+307
View File
@@ -0,0 +1,307 @@
#!/usr/bin/env python3
"""Correlate Dynu DNS data with Traefik host rules in compose sources.
This integration is intentionally read-only.
No Dynu mutations are permitted in this repo at this stage.
"""
from __future__ import annotations
import json
import os
import re
import sys
from collections import defaultdict
from datetime import datetime, timezone
from pathlib import Path
from typing import Dict, List, Tuple
BASE_DOMAIN = "lan.ddnsgeek.com"
DYN_DATA = Path("data/dns/dynu_live.json")
OUT_JSON = Path("data/dns/dynu_traefik_inventory.json")
OUT_MD = Path("docs/generated/dns-inventory.md")
HOST_RULE_RE = re.compile(r"Host\((.*?)\)")
DOMAIN_RE = re.compile(r"[`\"']([^`\"']+)[`\"']")
class ReadOnlyError(RuntimeError):
pass
def require_read_only() -> None:
if os.environ.get("DYNU_READ_ONLY") != "true":
raise ReadOnlyError(
"Refusing to run: DYNU_READ_ONLY must be exactly 'true'. "
"This integration is intentionally read-only."
)
def compose_files(root: Path) -> List[Path]:
files = [root / "default-network.yml"]
for area in ("apps", "monitoring", "core"):
base = root / area
if not base.exists():
continue
for p in sorted(base.glob("*/*")):
if p.is_file() and p.name in {"docker-compose.yml", "docker-compose.yaml"}:
files.append(p)
return files
def parse_hosts_from_label(label_value: str) -> List[str]:
found = []
for fragment in HOST_RULE_RE.findall(label_value):
for host in DOMAIN_RE.findall(fragment):
h = host.strip().strip(".").lower()
if h:
found.append(h)
return sorted(set(found))
def extract_traefik_hosts(path: Path) -> List[Dict[str, str]]:
lines = path.read_text(encoding="utf-8").splitlines()
entries: List[Dict[str, str]] = []
in_services = False
current_service = ""
current_labels_indent = None
for raw in lines:
line = raw.rstrip("\n")
stripped = line.strip()
if stripped == "services:":
in_services = True
current_service = ""
current_labels_indent = None
continue
if not in_services:
continue
service_match = re.match(r"^(\s{2})([A-Za-z0-9_.-]+):\s*$", line)
if service_match:
current_service = service_match.group(2)
current_labels_indent = None
continue
if re.match(r"^\S", line):
in_services = False
current_service = ""
current_labels_indent = None
continue
labels_match = re.match(r"^(\s+)labels:\s*$", line)
if labels_match and current_service:
current_labels_indent = len(labels_match.group(1))
continue
if current_labels_indent is None:
continue
indent = len(line) - len(line.lstrip(" "))
if indent <= current_labels_indent:
current_labels_indent = None
continue
if "traefik.http.routers." not in line or ".rule" not in line:
continue
label_value = ""
list_match = re.match(r"^\s*-\s*([\"']?)(.+)\1\s*$", stripped)
if list_match:
payload = list_match.group(2)
label_value = payload.split("=", 1)[1] if "=" in payload else payload
else:
map_match = re.match(r"^\s*([\"']?[^:]+\1):\s*(.+)$", line)
if map_match:
label_value = map_match.group(2).strip().strip("\"'")
for fqdn in parse_hosts_from_label(label_value):
entries.append(
{
"fqdn": fqdn,
"stack": path.parts[0],
"service": current_service,
"source_compose_file": str(path),
}
)
return entries
def load_dynu(path: Path) -> Dict[str, List[Dict[str, str]]]:
payload = json.loads(path.read_text(encoding="utf-8"))
if payload.get("base_domain") != BASE_DOMAIN:
raise RuntimeError(
f"Dynu JSON base_domain mismatch. Expected {BASE_DOMAIN}, got {payload.get('base_domain')}"
)
index: Dict[str, List[Dict[str, str]]] = defaultdict(list)
for domain in payload.get("domains", []):
for record in domain.get("records", []):
host = str(record.get("hostname", "")).strip(".").lower()
if host:
index[host].append(
{
"type": str(record.get("type", "")),
"value": str(record.get("value", "")),
"target": str(record.get("target") or ""),
"ttl": str(record.get("ttl") if record.get("ttl") is not None else ""),
}
)
for host in index:
index[host] = sorted(index[host], key=lambda x: (x["type"], x["value"], x["target"], x["ttl"]))
return index
def write_markdown(data: Dict) -> None:
matched = [x for x in data["inventory"] if x["status"] == "matched"]
missing = [x for x in data["inventory"] if x["status"] == "missing_in_dynu"]
dns_only = [x for x in data["inventory"] if x["status"] == "dns_only"]
lines = [
"# DNS Inventory (Dynu + Traefik)",
"",
"> This integration is intentionally read-only. No Dynu mutations are permitted in this repo at this stage.",
"",
f"- Base domain: `{data['base_domain']}`",
f"- Dynu fetched at: `{data['dynu_fetched_at']}`",
f"- Inventory generated at: `{data['generated_at']}`",
"",
"## Summary",
"",
f"- Traefik hostnames discovered: **{data['summary']['traefik_hostnames']}**",
f"- Dynu hostnames discovered: **{data['summary']['dynu_hostnames']}**",
f"- Matched: **{data['summary']['matched']}**",
f"- Missing in Dynu: **{data['summary']['missing_in_dynu']}**",
f"- Dynu DNS only: **{data['summary']['dns_only']}**",
f"- Duplicate Traefik hostnames: **{data['summary']['duplicate_traefik_hostnames']}**",
"",
"## Dynu Records",
"",
"| Hostname | Type | Value | TTL |",
"|---|---|---|---|",
]
for row in data["dynu_records_table"]:
lines.append(f"| `{row['hostname']}` | `{row['type']}` | `{row['value']}` | `{row['ttl']}` |")
lines.extend(["", "## Correlation", "", "| Hostname | Status | Service(s) | Source compose file(s) | DNS records |", "|---|---|---|---|---|"])
for row in data["inventory"]:
svc = ", ".join(sorted({f"{e['stack']}/{e['service']}" for e in row.get('traefik_entries', [])})) or "-"
src = ", ".join(sorted({e['source_compose_file'] for e in row.get('traefik_entries', [])})) or "-"
dns = ", ".join([f"{r['type']}:{r['value']}" for r in row.get("dynu_records", [])]) or "-"
lines.append(f"| `{row['fqdn']}` | `{row['status']}` | {svc} | {src} | {dns} |")
def section(title: str, rows: List[Dict]) -> None:
lines.extend(["", f"## {title}", ""])
if not rows:
lines.append("_None._")
return
for row in rows:
lines.append(f"- `{row['fqdn']}`")
section("Matched records", matched)
section("Traefik hostnames missing in Dynu", missing)
section("Dynu DNS records not mapped to known Traefik services", dns_only)
OUT_MD.parent.mkdir(parents=True, exist_ok=True)
OUT_MD.write_text("\n".join(lines) + "\n", encoding="utf-8")
def main() -> int:
try:
require_read_only()
except ReadOnlyError as exc:
print(str(exc), file=sys.stderr)
return 2
if not DYN_DATA.exists():
print(f"Missing {DYN_DATA}. Run fetch_dynu_dns.py first.", file=sys.stderr)
return 3
dyn_payload = json.loads(DYN_DATA.read_text(encoding="utf-8"))
dynu_index = load_dynu(DYN_DATA)
repo_root = Path(__file__).resolve().parents[2]
hosts = []
for cf in compose_files(repo_root):
hosts.extend(extract_traefik_hosts(cf.relative_to(repo_root)))
by_fqdn: Dict[str, List[Dict[str, str]]] = defaultdict(list)
for entry in hosts:
if entry["fqdn"].endswith(BASE_DOMAIN):
by_fqdn[entry["fqdn"]].append(entry)
duplicate_hosts = {k for k, v in by_fqdn.items() if len(v) > 1}
combined_fqdns = sorted(set(by_fqdn.keys()) | set(dynu_index.keys()))
inventory = []
for fqdn in combined_fqdns:
traefik_entries = sorted(
by_fqdn.get(fqdn, []),
key=lambda x: (x["stack"], x["service"], x["source_compose_file"]),
)
dns_records = dynu_index.get(fqdn, [])
if traefik_entries and dns_records:
status = "matched"
elif traefik_entries and not dns_records:
status = "missing_in_dynu"
else:
status = "dns_only"
inventory.append(
{
"fqdn": fqdn,
"status": status,
"duplicate": fqdn in duplicate_hosts,
"traefik_entries": traefik_entries,
"dynu_records": dns_records,
}
)
dynu_rows = []
for fqdn in sorted(dynu_index.keys()):
for rec in dynu_index[fqdn]:
dynu_rows.append(
{
"hostname": fqdn,
"type": rec["type"],
"value": rec["value"],
"ttl": rec["ttl"],
}
)
output = {
"source": "dynu+traefik",
"read_only": True,
"base_domain": BASE_DOMAIN,
"dynu_fetched_at": dyn_payload.get("fetched_at"),
"generated_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
"summary": {
"traefik_hostnames": len(by_fqdn),
"dynu_hostnames": len(dynu_index),
"matched": sum(1 for x in inventory if x["status"] == "matched"),
"missing_in_dynu": sum(1 for x in inventory if x["status"] == "missing_in_dynu"),
"dns_only": sum(1 for x in inventory if x["status"] == "dns_only"),
"duplicate_traefik_hostnames": len(duplicate_hosts),
},
"inventory": inventory,
"dynu_records_table": dynu_rows,
}
OUT_JSON.parent.mkdir(parents=True, exist_ok=True)
OUT_JSON.write_text(json.dumps(output, indent=2, sort_keys=True) + "\n", encoding="utf-8")
write_markdown(output)
print(f"Wrote {OUT_JSON}")
print(f"Wrote {OUT_MD}")
return 0
if __name__ == "__main__":
raise SystemExit(main())
+226
View File
@@ -0,0 +1,226 @@
#!/usr/bin/env python3
"""Fetch Dynu DNS inventory in strict read-only mode.
This integration is intentionally read-only.
No Dynu mutations are permitted in this repo at this stage.
"""
from __future__ import annotations
import json
import os
import sys
from datetime import datetime, timezone
from pathlib import Path
from typing import Any, Dict, Iterable, List, Optional
from urllib.error import HTTPError, URLError
from urllib.parse import urlencode
from urllib.request import Request, urlopen
BASE_DOMAIN = "lan.ddnsgeek.com"
DEFAULT_BASE_URL = "https://api.dynu.com"
OUT_PATH = Path("data/dns/dynu_live.json")
class DynuReadOnlyError(RuntimeError):
"""Raised when read-only safety guardrails are not met."""
def require_read_only() -> None:
if os.environ.get("DYNU_READ_ONLY") != "true":
raise DynuReadOnlyError(
"Refusing to run: DYNU_READ_ONLY must be exactly 'true'. "
"This integration is intentionally read-only."
)
def get_json(base_url: str, api_key: str, path: str, query: Optional[Dict[str, Any]] = None) -> Any:
"""HTTP GET helper. Any non-GET usage is blocked by design."""
url = f"{base_url.rstrip('/')}{path}"
if query:
url = f"{url}?{urlencode(query)}"
req = Request(
url=url,
headers={
"accept": "application/json",
"API-Key": api_key,
},
method="GET",
)
try:
with urlopen(req, timeout=30) as resp:
body = resp.read().decode("utf-8")
except HTTPError as exc:
detail = exc.read().decode("utf-8", errors="replace")
raise RuntimeError(f"Dynu API GET failed at {path}: HTTP {exc.code} {detail}") from exc
except URLError as exc:
raise RuntimeError(f"Dynu API GET failed at {path}: {exc}") from exc
try:
return json.loads(body)
except json.JSONDecodeError as exc:
raise RuntimeError(f"Dynu API returned non-JSON response at {path}") from exc
def list_domains(base_url: str, api_key: str) -> List[Dict[str, Any]]:
first = get_json(base_url, api_key, "/v2/dns")
def extract_domains(payload: Any) -> List[Dict[str, Any]]:
if isinstance(payload, list):
return [x for x in payload if isinstance(x, dict)]
if isinstance(payload, dict):
for key in ("domains", "items", "data", "dnsDomains"):
val = payload.get(key)
if isinstance(val, list):
return [x for x in val if isinstance(x, dict)]
return []
domains = extract_domains(first)
if isinstance(first, dict):
page = first.get("pageNumber") or first.get("page") or 1
total = first.get("totalPages") or first.get("pages")
if isinstance(page, int) and isinstance(total, int) and total > page:
for p in range(page + 1, total + 1):
nxt = get_json(base_url, api_key, "/v2/dns", {"page": p})
domains.extend(extract_domains(nxt))
return domains
def list_records(base_url: str, api_key: str, domain_id: Any) -> List[Dict[str, Any]]:
payload = get_json(base_url, api_key, f"/v2/dns/{domain_id}/record")
if isinstance(payload, list):
return [x for x in payload if isinstance(x, dict)]
if isinstance(payload, dict):
for key in ("dnsRecords", "records", "items", "data"):
val = payload.get(key)
if isinstance(val, list):
return [x for x in val if isinstance(x, dict)]
return []
def normalize_hostname(record: Dict[str, Any], domain_name: str) -> str:
node_name = record.get("nodeName") or record.get("hostname") or record.get("node") or ""
node_name = str(node_name).strip().strip(".")
base = domain_name.strip().strip(".")
if not node_name or node_name in {"@", base}:
return base
if node_name.endswith(base):
return node_name
return f"{node_name}.{base}"
def normalize_records(records: Iterable[Dict[str, Any]], domain_name: str) -> List[Dict[str, Any]]:
normalized = []
for rec in records:
record_type = rec.get("recordType") or rec.get("type") or ""
value = (
rec.get("value")
or rec.get("ipv4Address")
or rec.get("ipv6Address")
or rec.get("host")
or rec.get("textData")
or ""
)
target = rec.get("host") or rec.get("target")
priority = rec.get("priority")
ttl = rec.get("ttl")
raw_subset = {
k: rec.get(k)
for k in (
"id",
"nodeName",
"recordType",
"state",
"group",
"host",
"ipv4Address",
"ipv6Address",
"textData",
"ttl",
"priority",
"weight",
"port",
)
if k in rec
}
normalized.append(
{
"id": rec.get("id"),
"hostname": normalize_hostname(rec, domain_name),
"type": str(record_type),
"value": str(value),
"target": target,
"ttl": ttl,
"priority": priority,
"raw": raw_subset,
}
)
return sorted(normalized, key=lambda x: (x["hostname"], x["type"], x["value"], str(x.get("id") or "")))
def main() -> int:
try:
require_read_only()
except DynuReadOnlyError as exc:
print(str(exc), file=sys.stderr)
return 2
api_key = os.environ.get("DYNU_API_KEY")
if not api_key:
print("Missing DYNU_API_KEY. Refusing to call Dynu API.", file=sys.stderr)
return 2
base_url = os.environ.get("DYNU_BASE_URL", DEFAULT_BASE_URL)
domains = list_domains(base_url, api_key)
target = [d for d in domains if str(d.get("name", "")).strip(".").lower() == BASE_DOMAIN]
if not target:
print(
f"Could not find required domain '{BASE_DOMAIN}' in Dynu /v2/dns response.",
file=sys.stderr,
)
return 3
normalized_domains = []
for d in target:
domain_id = d.get("id")
if domain_id is None:
print(f"Domain entry for {BASE_DOMAIN} is missing 'id'; cannot fetch records.", file=sys.stderr)
return 4
records = list_records(base_url, api_key, domain_id)
normalized_domains.append(
{
"name": str(d.get("name", BASE_DOMAIN)).strip().strip("."),
"id": domain_id,
"records": normalize_records(records, str(d.get("name", BASE_DOMAIN))),
}
)
normalized_domains.sort(key=lambda x: (x["name"], str(x.get("id") or "")))
output = {
"source": "dynu",
"read_only": True,
"fetched_at": datetime.now(timezone.utc).replace(microsecond=0).isoformat(),
"base_domain": BASE_DOMAIN,
"domains": normalized_domains,
}
OUT_PATH.parent.mkdir(parents=True, exist_ok=True)
OUT_PATH.write_text(json.dumps(output, indent=2, sort_keys=True) + "\n", encoding="utf-8")
print(f"Wrote {OUT_PATH}")
return 0
if __name__ == "__main__":
raise SystemExit(main())