feat: Tempo 3, Access-gated Grafana, and a logs dashboard

.github/dependabot.yml

@@ -8,11 +8,6 @@ updates:
       monitoring-stack-updates:
         patterns:
           - "*"
-    ignore:
-      # Tempo 3.x is a re-architecture requiring a config migration; upgrade
-      # deliberately, not via a routine bump. See comment in compose.yml.
-      - dependency-name: "grafana/tempo"
-        update-types: ["version-update:semver-major"]
 
   - package-ecosystem: "docker"
     directory: "/demo"

.gitignore

@@ -5,3 +5,4 @@ backups/
 infra/.terraform/
 infra/terraform.tfstate*
 infra/*.tfvars
+!infra/*.tfvars.example

CHANGELOG.md

@@ -3,52 +3,50 @@
 Notable changes to this stack. Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/);
 versions follow [SemVer](https://semver.org/).
 
-## [Unreleased]
+## [0.2.0] - 2026-07-05
 
-Reliability hardening: the stack becomes operable, not just runnable.
+Makes the stack operable, not just runnable: authentication, self-monitoring,
+alert delivery, backups, and infrastructure as code.
 
 ### Added
 
-- Bearer-token auth on OTLP ingestion (`OTLP_AUTH_TOKEN`); senders use
-  `OTEL_EXPORTER_OTLP_HEADERS`.
-- Host self-monitoring: node-exporter, `HostDiskSpaceLow` alert (the
-  storage backstop — Loki/Tempo have no total-size cap), **Stack Health**
-  dashboard.
-- Alert delivery: Alertmanager routes to `ALERT_WEBHOOK_URL`; an
-  always-firing `Watchdog` heartbeats to `HEARTBEAT_URL` (dead man's
-  switch).
-- `just backup` / `just restore` — crash-consistent volume snapshots,
-  exercised end-to-end.
-- `docs/RUNBOOK.md` and `docs/ONBOARDING.md` with verified telemetry
-  templates (zero-code Python, plain OTLP, Loki Docker driver, Grafana
-  Alloy).
-- `infra/`: OpenTofu for the Cloudflare tunnel, ingress routes, and DNS.
+- **Authenticated ingestion**: OTLP now requires a bearer token
+  (`OTLP_AUTH_TOKEN`).
+- **Self-monitoring**: node-exporter, a **Stack Health** dashboard, and a
+  disk-space alert that backstops Loki and Tempo (neither can cap its own
+  size).
+- **Alert delivery**: Alertmanager posts to any webhook
+  (`ALERT_WEBHOOK_URL`), and an always-firing `Watchdog` pings a dead man's
+  switch (`HEARTBEAT_URL`) so even a dead host gets noticed.
+- **Backup and restore**: `just backup` / `just restore`, tested by wiping
+  every volume and recovering.
+- **The Cloudflare edge as code**: `infra/` (OpenTofu) manages the tunnel,
+  DNS, and an email-PIN Access policy in front of Grafana.
+- **Operator and onboarding docs**: a runbook plus copy-paste telemetry
+  templates for new projects.
+- **Logs Overview dashboard** for OTLP logs, an Active Alerts table on Stack
+  Health, and Stack Health as the Grafana home page.
 
 ### Changed
 
-- Prometheus storage now bounded by size as well as time
-  (`--storage.tsdb.retention.size=15GB`).
-
-### Fixed
-
-- Tempo held at 2.x: a routine bump to 3.0 crash-looped on the 2.x config
-  (3.0 is a re-architecture). Dependabot now skips Tempo majors so the
-  migration happens deliberately.
+- Prometheus storage is now capped by size as well as time (15 GB).
+- **Tempo upgraded to 3.0**, migrating off the removed ingester/compactor
+  config and verified in place against live traffic.
 
 ## [0.1.0] - 2026-07-03
 
 First tagged release: the stack is runnable, demoable, and validated in CI.
 
 ### Added
 
-- `just demo`: one-command demo — an auto-instrumented FastAPI service under
+- `just demo`: one-command demo: an auto-instrumented FastAPI service under
   constant load populates Grafana with correlated traces, metrics, and logs.
 - `just check`: validation gate (compose syntax, Prometheus config + alert
   rules, collector config, YAML, workflows, dashboard JSON), all in pinned
   containers. Run by GitHub Actions on every push/PR, plus a `just smoke`
   boot test.
 - Provisioned **Service Health (RED)** dashboard driven by Tempo span
-  metrics — works for any service that sends traces.
+  metrics: works for any service that sends traces.
 - Prometheus alert rules: target down, OTel export failures, high span
   error rate.
 - README: architecture diagram, demo quickstart, screenshot, design

README.md

@@ -1,17 +1,23 @@
 # Observability stack
 
-One host running Grafana, Loki, Tempo, Prometheus, and an OpenTelemetry
-Collector, wired so logs, traces, and metrics cross-link. Point any number of
-projects at it over OTLP and their telemetry lands in one place.
+Central monitoring for CML's research software. One host runs Grafana,
+Loki, Tempo, Prometheus, and an OpenTelemetry Collector, wired together so
+logs, traces, and metrics cross-reference each other. Projects send their
+telemetry here over OTLP (the OpenTelemetry protocol) and it lands in one
+place, queryable side by side.
 
 ## Architecture
 
-Everything enters through one gateway — the OTel Collector — so a project
-configures a single OTLP endpoint and a backend can be swapped without
-touching any app. Tempo derives RED metrics from spans, so a service that
-only sends traces still gets dashboards and error alerting. It all runs on
-one host on purpose: at CML's telemetry volume, distributed ingest would add
-operational weight for no gain. Rationale and alternatives:
+Everything enters through a single gateway, the OpenTelemetry Collector. A
+project only ever configures one endpoint, and we can swap a storage
+backend later without touching any application. Tempo also derives
+request-rate, error-rate, and duration ("RED") metrics from the traces it
+receives, so a service that sends nothing but traces still gets a working
+dashboard and error alerting.
+
+The stack deliberately runs on a single host. At CML's telemetry volume,
+distributed ingestion would add operational weight for no gain. The
+reasoning, and the alternatives we considered, are recorded in
 [ADR 0001](docs/adr/0001-observability-stack.md).
 
 ```mermaid
@@ -31,9 +37,10 @@ flowchart LR
     end
 ```
 
-Solid arrows are the write path; dotted arrows are Grafana reading at query
-time. Locally (`just up` / `just demo`) there is no tunnel — everything
-talks over the compose network and Grafana is on `localhost:3000`.
+Solid arrows show telemetry being written; dotted arrows show Grafana
+reading at query time. Locally (`just up` or `just demo`) there is no
+tunnel involved: everything talks over the compose network and Grafana is
+at `localhost:3000`.
 
 ## Demo: see it work in one command
 
@@ -42,21 +49,24 @@ cp .env.example .env
 just demo
 ```
 
-This starts the full stack plus a small auto-instrumented FastAPI service
-under constant load (`compose.demo.yml`). Give it a minute, then open
-Grafana at <http://localhost:3000> (admin / change-me):
+This starts the full stack plus a small FastAPI service under constant
+artificial load (`compose.demo.yml`). The service is instrumented with
+OpenTelemetry auto-instrumentation and fails about one request in ten, on
+purpose. Give it a minute, then open Grafana at <http://localhost:3000>
+(admin / change-me) and look at:
 
 - **Dashboards → Service Health (RED)** — request rate, error rate, and
-  latency for the demo service. Latency dots are exemplars: click one to
-  open that exact trace in Tempo.
-- **Explore → Loki** — `{service_name="demo-api"}`; expand an error line
-  and follow its `trace_id` to the trace.
-- **Alerting → Alert rules** — stack-health and error-rate rules,
-  evaluated by Prometheus.
+  latency. The dots on the latency panel are exemplars: click one and
+  Grafana opens the exact trace behind that measurement.
+- **Dashboards → Logs Overview** — log volume by service and level, an
+  error feed, and a live tail of everything arriving over OTLP.
+- **Alerting → Alert rules** — the stack-health and error-rate rules
+  Prometheus is evaluating.
 
 ![Service Health (RED) dashboard](docs/img/service-health.png)
 
-Tear it down with `just demo-down`.
+`just demo-down` removes the demo services again; the rest of the stack
+keeps running.
 
 ## Layout
 
@@ -83,47 +93,57 @@ infra/                  # OpenTofu: Cloudflare tunnel, ingress routes, DNS
 ```sh
 cp .env.example .env    # set GRAFANA_ADMIN_PASSWORD
 just up                 # core stack, local only
-just up-tunnel          # core stack + Cloudflare Tunnel (needs CLOUDFLARE_TUNNEL_TOKEN)
+just up-tunnel          # production: core stack + Cloudflare Tunnel
 ```
 
 Grafana: <http://localhost:3000> (admin / whatever you set).
 
-The Cloudflare side of the tunnel (hostnames, DNS, the tunnel itself) is
-code too: `infra/` holds a small OpenTofu config whose output is the
-`CLOUDFLARE_TUNNEL_TOKEN` the overlay needs — bootstrap instructions in
-[infra/main.tf](infra/main.tf).
+In production the stack sits behind a Cloudflare Tunnel, and that edge is
+code too. The tunnel, its hostnames, DNS, and the Cloudflare Access rule
+that puts an email one-time-PIN in front of Grafana all live in `infra/`
+as a small OpenTofu configuration. Applying it produces the
+`CLOUDFLARE_TUNNEL_TOKEN` that `just up-tunnel` needs; bootstrap steps are
+at the top of [infra/main.tf](infra/main.tf).
 
-`just check` validates everything (compose files, Prometheus config and
-alert rules, collector config, YAML, workflows, dashboard JSON) in pinned
-containers — no host installs. CI runs the same command on every push and
-PR, plus a smoke test that boots the stack and waits for Grafana to report
-healthy.
+`just check` validates the whole repository: compose files, Prometheus
+config and alert rules, collector config, YAML, workflows, and dashboard
+JSON. Every validator runs in a pinned container, so nothing needs to be
+installed on the host. CI runs the same command on every push and pull
+request, plus a smoke test that boots the stack and waits for Grafana to
+come up healthy.
 
 ## Sending telemetry from a project
 
-One OTLP endpoint (`<host>:4317` gRPC / `:4318` HTTP), bearer-token auth
-(`OTLP_AUTH_TOKEN`), and three naming conventions. Copy-paste templates for
-every ingestion form — zero-code Python/FastAPI, plain OTLP env vars, the
-Loki Docker driver, and Grafana Alloy for file logs — live in
-**[docs/ONBOARDING.md](docs/ONBOARDING.md)**.
+You need three things: the OTLP endpoint, the bearer token
+(`OTLP_AUTH_TOKEN`), and a few naming conventions. Copy-paste templates
+for each route — zero-code Python/FastAPI, plain OTLP environment
+variables, the Loki Docker driver, and Grafana Alloy for log files — are
+in **[docs/ONBOARDING.md](docs/ONBOARDING.md)**.
 
-Never publish 4317/4318 directly; the compose file binds them to
-`127.0.0.1` and the tunnel is the exposure path.
+Never publish ports 4317/4318 to the internet. The compose file binds them
+to `127.0.0.1`; the tunnel is the way in.
 
 ## Alerting
 
-Prometheus evaluates the rules in `config/alerts/` (target down, OTel
-export failures, >5% span error rate, disk >80%); Alertmanager delivers
-them to any webhook via `ALERT_WEBHOOK_URL`, and the always-firing
-`Watchdog` posts to `HEARTBEAT_URL` every 5 minutes — point that at a dead
-man's switch so you hear about it when the monitoring host itself dies.
-Both are optional; unset means alerts are visible in Grafana only.
-Operations (rotating tokens, disk pressure, backup/restore):
-**[docs/RUNBOOK.md](docs/RUNBOOK.md)**.
+Prometheus evaluates the rules in `config/alerts/`: scrape target down,
+OTel export failures, error rate above 5%, disk above 80%. Alertmanager
+delivers them to whatever webhook you set in `ALERT_WEBHOOK_URL` (ntfy,
+Slack, and so on).
+
+One rule, `Watchdog`, fires permanently by design and posts to
+`HEARTBEAT_URL` every five minutes. Point that at a dead man's switch such
+as healthchecks.io — a service that alerts when the pings *stop* — and you
+will also hear about the one failure the host cannot report itself: its own
+death. Both variables are optional; leave them unset and alerts are simply
+visible in Grafana.
+
+Day-to-day operations (rotating tokens, disk pressure, backup and restore)
+are covered in **[docs/RUNBOOK.md](docs/RUNBOOK.md)**.
 
 ## Storage
 
 Everything persists to local Docker volumes (`loki_data`, `tempo_data`,
-`prometheus_data`, `grafana_data`). Swap Loki / Tempo storage to S3-compatible
-(Backblaze B2, Cloudflare R2, Hetzner, MinIO) when you outgrow local disk —
-`compose.storage-s3.yml` documents the concrete shape of that change.
+`prometheus_data`, `grafana_data`). When local disk stops fitting, Loki
+and Tempo can move to any S3-compatible object store (Backblaze B2,
+Cloudflare R2, Hetzner, MinIO); `compose.storage-s3.yml` documents the
+concrete shape of that change.

compose.yml

@@ -43,11 +43,7 @@ services:
     logging: *default-logging
 
   tempo:
-    # Held at 2.x: Tempo 3.0 is a re-architecture (ingester/compactor removed,
-    # metrics-generator changed) needing a real config migration — see
-    # https://grafana.com/docs/tempo/latest/set-up-for-tracing/setup-tempo/migrate-to-3/
-    # Dependabot ignores 3.x majors until that migration is done deliberately.
-    image: grafana/tempo:2.10.4
+    image: grafana/tempo:3.0.2
     restart: unless-stopped
     volumes:
       - ./config/tempo.yaml:/etc/tempo/tempo.yaml:ro
@@ -123,6 +119,8 @@ services:
       GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD:?set GRAFANA_ADMIN_PASSWORD in .env}
       GF_USERS_ALLOW_SIGN_UP: "false"
       GF_SERVER_ROOT_URL: ${GRAFANA_ROOT_URL:-http://localhost:3000}
+      # Land on Stack Health instead of the empty welcome page.
+      GF_DASHBOARDS_DEFAULT_HOME_DASHBOARD_PATH: /var/lib/grafana/dashboards/stack-health.json
     logging: *default-logging
 
 volumes:

config/tempo.yaml

@@ -1,5 +1,8 @@
-# Tempo accepts OTLP directly. We only forward traces *from* the collector
+# Tempo 3.x single-binary. Traces arrive only from the collector
 # (collector → tempo:4317) so external apps never talk to Tempo directly.
+# Migrated from 2.x: the ingester/compactor blocks and the local-blocks
+# processor are gone (handled internally by the live-store/backend worker);
+# block retention now lives under overrides.
 
 server:
   http_listen_port: 3200
@@ -12,13 +15,6 @@ distributor:
         grpc:
           endpoint: 0.0.0.0:4317
 
-ingester:
-  max_block_duration: 5m
-
-compactor:
-  compaction:
-    block_retention: 168h  # 7 days — traces are bulky, tune to disk budget
-
 storage:
   trace:
     backend: local
@@ -28,7 +24,7 @@ storage:
       path: /var/tempo/blocks
 
 # Derive RED metrics (requests/errors/duration) + a service graph from traces.
-# These land in Prometheus and power Grafana's built-in Service Graph panel.
+# These land in Prometheus and power the Service Health dashboard.
 metrics_generator:
   registry:
     external_labels:
@@ -38,13 +34,13 @@ metrics_generator:
     remote_write:
       - url: http://prometheus:9090/api/v1/write
         send_exemplars: true
-  traces_storage:
-    path: /var/tempo/generator/traces
 
 overrides:
   defaults:
     metrics_generator:
-      processors: [service-graphs, span-metrics, local-blocks]
+      processors: [service-graphs, span-metrics]
+    compaction:
+      block_retention: 168h  # 7 days — traces are bulky, tune to disk budget
 
 usage_report:
   reporting_enabled: false