Archived
archived old monitoring system
This commit is contained in:
git committed
1 parent
f17a8ee22c
commit
dccff1d61c
25 files changed
+2396
-1897
No files matched your search
@@ -0,0 +1,7 @@
|
||||
FROM nodered/node-red:latest
|
||||
|
||||
USER root
|
||||
RUN apk add --no-cache docker-cli docker-cli-compose
|
||||
#RUN addgroup -g 131 -S docker && addgroup node-red docker
|
||||
|
||||
USER node-red
|
||||
@@ -0,0 +1,160 @@
|
||||
# Node-RED update logging for Grafana
|
||||
|
||||
This guide adds structured update-event logging to your existing Node-RED + Telegraf + Prometheus + Grafana stack without introducing Loki.
|
||||
|
||||
## Goal
|
||||
|
||||
Track and surface (in Grafana) the latest update attempts from Node-RED, including:
|
||||
|
||||
- when an update attempt started,
|
||||
- target container/project,
|
||||
- success/failure,
|
||||
- optional failure reason,
|
||||
- elapsed duration.
|
||||
|
||||
## 1) Add a reusable logger function in Node-RED
|
||||
|
||||
Create a **Function** node named `Build update log event` and use:
|
||||
|
||||
```javascript
|
||||
const nowIso = new Date().toISOString();
|
||||
const startedAt = msg.update_started_at || Date.now();
|
||||
const durationMs = Math.max(0, Date.now() - startedAt);
|
||||
|
||||
const payload = msg.payload || {};
|
||||
const labels = payload.labels || {};
|
||||
|
||||
const status = (msg.update_status || payload.status || "unknown").toString().toLowerCase();
|
||||
const success = status === "success" ? 1 : 0;
|
||||
const failed = status === "failed" ? 1 : 0;
|
||||
|
||||
msg.payload = {
|
||||
ts: nowIso,
|
||||
flow: "docker-updates",
|
||||
event: msg.update_event || "attempt",
|
||||
container: msg.container || labels.container || "unknown",
|
||||
project: labels.com_docker_compose_project || msg.project || "unknown",
|
||||
host: msg.host || "unknown",
|
||||
status,
|
||||
success,
|
||||
failed,
|
||||
duration_ms: durationMs,
|
||||
code: Number.isFinite(Number(payload.code)) ? Number(payload.code) : 0,
|
||||
error: (msg.update_error || payload.error || "").toString().slice(0, 300)
|
||||
};
|
||||
|
||||
// one JSON line per event for file output
|
||||
msg.payload = JSON.stringify(msg.payload);
|
||||
return msg;
|
||||
```
|
||||
|
||||
### Wiring recommendation
|
||||
|
||||
Use the same logger function in these branches:
|
||||
|
||||
- before a pull/update command (`update_status=started`, `update_event=attempt`),
|
||||
- success path (`update_status=success`, `update_event=completed`),
|
||||
- failure path (`update_status=failed`, `update_event=completed`, and include `msg.update_error`).
|
||||
|
||||
Then route each branch into a **File** node configured as:
|
||||
|
||||
- Filename: `/data/update-events.ndjson`
|
||||
- Action: append to file
|
||||
- Add newline: enabled
|
||||
|
||||
## 2) Make update state explicit in existing update flow
|
||||
|
||||
In your current update flow (already present in `flows.json`), add/change **Change** nodes around your shell/docker nodes:
|
||||
|
||||
- At update start:
|
||||
- `msg.update_started_at = $millis()`
|
||||
- `msg.update_status = "started"`
|
||||
- `msg.update_event = "attempt"`
|
||||
- At success:
|
||||
- `msg.update_status = "success"`
|
||||
- `msg.update_event = "completed"`
|
||||
- At failure:
|
||||
- `msg.update_status = "failed"`
|
||||
- `msg.update_event = "completed"`
|
||||
- `msg.update_error = msg.payload.stderr` (or equivalent error field)
|
||||
|
||||
## 3) Let Telegraf ingest Node-RED event logs
|
||||
|
||||
Append this to `monitoring/telegraf/telegraf.conf`:
|
||||
|
||||
```toml
|
||||
[[inputs.tail]]
|
||||
files = ["/var/log/node-red/update-events.ndjson"]
|
||||
from_beginning = false
|
||||
name_override = "node_red_update_event"
|
||||
data_format = "json_v2"
|
||||
|
||||
[[inputs.tail.json_v2]]
|
||||
measurement_name = "node_red_update_event"
|
||||
|
||||
[[inputs.tail.json_v2.tag]]
|
||||
path = "flow"
|
||||
[[inputs.tail.json_v2.tag]]
|
||||
path = "event"
|
||||
[[inputs.tail.json_v2.tag]]
|
||||
path = "container"
|
||||
[[inputs.tail.json_v2.tag]]
|
||||
path = "project"
|
||||
[[inputs.tail.json_v2.tag]]
|
||||
path = "host"
|
||||
[[inputs.tail.json_v2.tag]]
|
||||
path = "status"
|
||||
|
||||
[[inputs.tail.json_v2.field]]
|
||||
path = "success"
|
||||
type = "int"
|
||||
[[inputs.tail.json_v2.field]]
|
||||
path = "failed"
|
||||
type = "int"
|
||||
[[inputs.tail.json_v2.field]]
|
||||
path = "duration_ms"
|
||||
type = "int"
|
||||
[[inputs.tail.json_v2.field]]
|
||||
path = "code"
|
||||
type = "int"
|
||||
```
|
||||
|
||||
And mount the Node-RED data directory into Telegraf (read-only) in `monitoring/prometheus/docker-compose.yml` under `telegraf.volumes`:
|
||||
|
||||
```yaml
|
||||
- ${PROJECT_ROOT}/monitoring/node-red/data:/var/log/node-red:ro
|
||||
```
|
||||
|
||||
## 4) Prometheus scrape (already in place)
|
||||
|
||||
No Prometheus scrape change is required as long as it already scrapes Telegraf (`telegraf:9273`).
|
||||
|
||||
## 5) Grafana queries to start with
|
||||
|
||||
Use your Prometheus data source and try:
|
||||
|
||||
- Latest success/failure by container:
|
||||
- `last_over_time(node_red_update_event_success[24h])`
|
||||
- `last_over_time(node_red_update_event_failed[24h])`
|
||||
- Failed updates in the last 24h:
|
||||
- `sum by (container, project) (increase(node_red_update_event_failed[24h]))`
|
||||
- Average update duration in last 24h:
|
||||
- `avg by (container, project) (avg_over_time(node_red_update_event_duration_ms[24h]))`
|
||||
|
||||
Recommended panels:
|
||||
|
||||
- **Table**: container, project, status (last value), duration_ms (last value)
|
||||
- **Time series**: failed count over time
|
||||
- **Stat**: total failed updates in last 24h
|
||||
|
||||
## 6) Validation checklist
|
||||
|
||||
1. Trigger a known update path (including one failure if possible).
|
||||
2. Check Node-RED log file:
|
||||
- `tail -n 20 monitoring/node-red/data/update-events.ndjson`
|
||||
3. Check Telegraf metrics endpoint for `node_red_update_event_` metrics.
|
||||
4. Confirm Grafana panel values match the latest Node-RED run.
|
||||
|
||||
## Optional next step
|
||||
|
||||
If you want searchable raw log text and richer log UX, add Loki + Promtail later. Keep this structured metrics path for high-signal alerting even after adding logs.
|
||||
@@ -0,0 +1,67 @@
|
||||
services:
|
||||
node-red:
|
||||
profiles: ["monitoring","all","node-red"]
|
||||
# image: nodered/node-red:latest
|
||||
build:
|
||||
context: ${PROJECT_ROOT}/monitoring/node-red
|
||||
container_name: node-red
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
- docker-socket-proxy
|
||||
environment:
|
||||
DOCKER_HOST: ${DOCKER_SOCKET_PROXY_HOST}
|
||||
TZ: ${TZ}
|
||||
PROJECT_ROOT: ${NODE_COMPOSE_ROOT}
|
||||
cap_drop:
|
||||
- ALL
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
# ports:
|
||||
# - "1880:1880"
|
||||
volumes:
|
||||
# - ${PROJECT_ROOT}/monitoring/node-red/data:/data
|
||||
- ${DOCKER_VOLUMES}/node-red-data:/data
|
||||
- ${PROJECT_ROOT}:/compose/docker:ro
|
||||
- /home/nixos/raspi:/compose/raspi:ro
|
||||
# - ${PROJECT_ROOT}:/usr/src/node-red:ro
|
||||
|
||||
# - ${PROJECT_ROOT}/default-environment.env:/usr/src/node-red/default-environment.env:ro
|
||||
# - ${PROJECT_ROOT}/default-network.yml:/usr/src/node-red/default-network.yml:ro
|
||||
# - ${PROJECT_ROOT}/core/docker-compose.yml:/usr/src/node-red/core/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/monitoring/prometheus/docker-compose.yml:/usr/src/node-red/monitoring/prometheus/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/monitoring/gotify/docker-compose.yml:/usr/src/node-red/monitoring/gotify/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/monitoring/grafana/docker-compose.yml:/usr/src/node-red/monitoring/grafana/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/monitoring/portainer/docker-compose.yml:/usr/src/node-red/monitoring/portainer/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/monitoring/uptime-kuma/docker-compose.yml:/usr/src/node-red/monitoring/uptime-kuma/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/apps/gitea/docker-compose.yml:/usr/src/node-red/apps/gitea/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/apps/gramps/docker-compose.yml:/usr/src/node-red/apps/gramps/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/apps/nextcloud/docker-compose.yml:/usr/src/node-red/apps/nextcloud/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/apps/passbolt/docker-compose.yml:/usr/src/node-red/apps/passbolt/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/apps/searxng/docker-compose.yml:/usr/src/node-red/apps/searxng/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/apps/shift-recorder/docker-compose.yml:/usr/src/node-red/apps/shift-recorder/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/apps/stockfill/docker-compose.yml:/usr/src/node-red/apps/stockfill/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/monitoring/node-red/docker-compose.yml:/usr/src/node-red/monitoring/node-red/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/core/test/docker-compose.yml:/usr/src/node-red/core/test/docker-compose.yml:ro
|
||||
# - ${PROJECT_ROOT}/secrets/stack-secrets.env:/usr/src/node-red/secrets/stack-secrets.env:ro
|
||||
|
||||
# - /run/current-system/sw/bin/docker:/usr/bin/docker:ro
|
||||
# depends_on:
|
||||
# - mosquitto
|
||||
# - influxdb
|
||||
networks:
|
||||
- monitor
|
||||
- traefik
|
||||
labels:
|
||||
- "traefik.enable=true"
|
||||
- "traefik.http.routers.node-red.rule=Host(`node-red.lan.ddnsgeek.com`)"
|
||||
# - "traefik.http.routers.node-red.service=api@internal"
|
||||
- "traefik.http.routers.node-red.entrypoints=websecure"
|
||||
- "traefik.http.routers.node-red.tls.certresolver=myresolver"
|
||||
- "traefik.http.routers.node-red.tls.options=mtls-private-admin@file"
|
||||
- "traefik.http.routers.node-red.middlewares=authelia"
|
||||
- "io.portainer.accesscontrol.public"
|
||||
- "traefik.docker.network=core_traefik"
|
||||
- "traefik.http.services.node-red.loadbalancer.server.port=1880"
|
||||
#volumes:
|
||||
# node-red-data:
|
||||
# external: true
|
||||
Reference in new issue
Block a user