feat(monitoring): opt-in gateway distributed tracing + Tempo in telemetry playground

[udsmicrosoft]

Summary

Adds opt-in distributed tracing to the DocumentDB monitoring stack, alongside the existing metrics path, and wires it end-to-end in the local telemetry playground.

Metrics are scraped by Prometheus from each per-pod OTel sidecar. Traces cannot be scraped — each sidecar pushes spans (OTLP/gRPC) to a central trace backend. This PR adds that push path plus a Tempo backend in the playground.

Operator changes

• API: new spec.monitoring.tracing stanza — enabled, otlpEndpoint, sqlCommenterEnabled (TracingSpec), with regenerated CRD + deepcopy.
• Sidecar collector (internal/otel/config.go): adds a traces pipeline (otlp receiver → resource/batch → otlp/traces exporter) that pushes spans to tracing.otlpEndpoint. Metrics and traces pipelines coexist.
• Gateway env (sidecar-injector): when tracing is enabled the operator passes gatewayTracingEnabled / gatewaySqlCommenterEnabled plugin params, and the injector adds OTEL_TRACES_ENABLED=true (and optionally DOCUMENTDB_SQL_COMMENTER_ENABLED=true) to the gateway container.
• Tests: new unit coverage for the traces pipeline generation and the env-injection matrix (tracing on/off, sqlcommenter requires tracing).

Playground changes (documentdb-playground/telemetry/local/)

• Tempo trace backend deployed in the observability namespace (monolithic, OTLP in / query out).
• Grafana Tempo datasource with trace→metrics correlation.
• Demo CR enables spec.monitoring.tracing.
• deploy.sh accepts DOCUMENTDB_VERSION to pull a tracing-capable gateway image; validate.sh checks Tempo readiness + gateway spans.
• README + CHANGELOG updated.

Known limitation (why this is a draft)

Tracing requires a gateway image built with OTLP trace support (gateway ≥ 0.115). The operator's default gateway image (0.110.0) predates it, and no gateway:0.115.0 is published to GHCR yet. Until a tracing-capable operator gateway image exists:

• The pipeline is wired correctly and metrics keep working.
• No spans are produced (validate.sh reports this as a warning, not a failure).

Validation done

• Both Go modules build; go vet clean.
• Operator + sidecar-injector unit tests pass (including new tracing tests).
• YAML manifests, embedded collector/datasource configs, and CRD schema validated; shell scripts pass bash -n.
• Not yet validated on a live cluster with real spans (blocked on the gateway image above).

Branch is merged up to date with main (incl. the domain-grouped DocumentDBSpec API change).

[documentdb-triage-tool]

🤖 Auto-triaged by documentdb-triage-tool.

Applied: ecosystem, go, documentation, test, enhancement
Project fields suggested: Component playground · Priority P2 · Effort L · Status In Progress
Confidence: 0.30 (deterministic)

Reasoning

component from path globs (playground, controllers, docs, test, api, manifests); effort from diff stats (593+38 LOC, 18 files); LLM failed: Invalid response body while trying to fetch https://api.anthropic.com/v1/messages: Premature close

If a label is wrong, remove it manually and ping @patty-chow so the rules can be tuned. The bot will not re-label items that already have component labels.