Observability / Monitoring

[0xlua]

I want to setup comprehensive monitoring. These are some thoughts on the components.

Use Vector to collect data from various sources, optionally transform it and send it to a sink to be centrally stored or displayed.

Sources

I'm interested in the logs of the services I'm running. I could use Vector directly or something dedicated like flowgger.

OpenTelemetry for my own projects.

Sinks

These are possible sinks for Vector to send data to.

• GreptimeDB: Store & Query Logs
• InfluxDB: Similar to Greptime?
• Quickwit: Could also be used to display data

Visualisation

At the end, everything is displayed in OpenObserve.

• I want to investigate if Home Assistant could also be a good place to visualize the data and trigger Notifications.
• Grafana is always an option.
• Perses looks interesting too.

Notifications

In case of problems I want to be notified via ntfy / UnifiedPush.

For TrueNAS see here: https://www.truenas.com/docs/scale/25.10/scaleuireference/toptoolbar/alerts/alertsettingsscreen/

More Metrics

This setup would allow me to easily monitor logs. However I am interested in a few more metrics:

• Host metrics: CPU load & & temps, Memory Usage, ...
  • beszel as a standalone solution
  • dozzle for podman logs
• Service Metrics:
  • Failed login attempts?
  • Gluetun Status & IP
  • Bore Tunnel Status
  • Pocket ID
  • Stalwart: Logs, Traces and Metrics
  • Caddy