Skip to content

feat(observability): deployment (OTel collector + Parca/eBPF)#5377

Draft
Ma77Ball wants to merge 66 commits into
apache:mainfrom
Ma77Ball:obs/pr3/deployment
Draft

feat(observability): deployment (OTel collector + Parca/eBPF)#5377
Ma77Ball wants to merge 66 commits into
apache:mainfrom
Ma77Ball:obs/pr3/deployment

Conversation

@Ma77Ball

@Ma77Ball Ma77Ball commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

Provides the local deployment wiring that receives and stores telemetry. Default-off and isolated from the application services.

  • Adds an OpenTelemetry Collector configuration and wires it into the single-node docker-compose stack.
  • Adds the Parca server configuration and the Parca eBPF agent for continuous profiling.
  • Updates the single-node up.sh and .env to start the observability backends.
  • Infrastructure only; the application runs unchanged whether or not these services are started.

Any related issues, documentation, or discussions?

Closes: #5369
Part of #4070. Stacked on #5376.

How was this PR tested?

  • Configuration-validation specs for the collector and Parca config.
  • sbt scalafmtCheckAll passes; the build runs in this PR's CI.

Was this PR authored or co-authored using generative AI tooling?

Co-authored with Claude Opus 4.8 in compliance with ASF

Ma77Ball and others added 3 commits June 5, 2026 04:49
…, SDK bootstrap (default-off)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ca/eBPF profiling

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tracing primitives

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions github-actions Bot added dependencies Pull requests that update a dependency file docs Changes related to documentations dev common labels Jun 5, 2026
@codecov-commenter

codecov-commenter commented Jun 5, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 71.22507% with 101 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.17%. Comparing base (7a38b6c) to head (d927cff).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines Patch % Lines
...ala/org/apache/texera/observability/OtelInit.scala 63.30% 43 Missing and 8 partials ⚠️
...org/apache/texera/observability/TexeraTracer.scala 0.00% 14 Missing ⚠️
...rg/apache/texera/observability/TexeraMetrics.scala 84.93% 4 Missing and 7 partials ⚠️
...la/org/apache/texera/observability/SpanAttrs.scala 66.66% 8 Missing and 2 partials ⚠️
...e/texera/observability/TexeraOtelLogAppender.scala 72.22% 4 Missing and 6 partials ⚠️
...org/apache/texera/observability/LogSanitizer.scala 96.42% 0 Missing and 1 partial ⚠️
.../texera/service/ComputingUnitManagingService.scala 0.00% 1 Missing ⚠️
...cala/org/apache/texera/service/ConfigService.scala 0.00% 1 Missing ⚠️
.../scala/org/apache/texera/service/FileService.scala 0.00% 1 Missing ⚠️
...ache/texera/service/WorkflowCompilingService.scala 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##               main    #5377      +/-   ##
============================================
+ Coverage     54.89%   56.17%   +1.27%     
+ Complexity     2962     1860    -1102     
============================================
  Files          1117      847     -270     
  Lines         43133    34932    -8201     
  Branches       4648     3543    -1105     
============================================
- Hits          23680    19623    -4057     
+ Misses        18064    14336    -3728     
+ Partials       1389      973     -416     
Flag Coverage Δ *Carryforward flag
access-control-service 70.14% <100.00%> (+0.14%) ⬆️
agent-service 34.36% <ø> (ø) Carriedforward from f6bf45d
amber 65.22% <71.96%> (+8.16%) ⬆️
computing-unit-managing-service 0.00% <0.00%> (ø)
config-service 50.76% <0.00%> (-0.80%) ⬇️
file-service 58.88% <0.00%> (-0.15%) ⬇️
frontend 48.01% <ø> (-0.61%) ⬇️ Carriedforward from f6bf45d
notebook-migration-service 78.57% <ø> (ø)
pyamber 90.15% <ø> (-0.05%) ⬇️ Carriedforward from f6bf45d
python 90.76% <ø> (ø) Carriedforward from f6bf45d
workflow-compiling-service 54.74% <0.00%> (-0.41%) ⬇️

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions github-actions Bot added the platform Non-amber Scala service paths label Jun 5, 2026
Ma77Ball and others added 18 commits June 14, 2026 18:28
Call OtelInit.init(<service.name>) in each service main so its logs
bridge to the OTel collector under its own service.name; cap noisy
framework loggers (pekko/iceberg/hadoop/kafka/jetty/jersey/grpc/
netty/hikari/awssdk) at WARN in each service config.

Services: access-control, config, file, computing-unit-managing,
workflow-compiling, computing-unit-master, texera-web, amber.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…e span

- WorkflowMetricsRecorder: emit workflow lifecycle metrics keyed by
  execution, driven from the ExecutionStateStore state-transition
  chokepoint; registered via WorkflowMetricsRecorder.init() in
  ComputingUnitMaster
- WorkflowService: wrap initExecutionService in a run-level TexeraTracer
  span so setup-path logs carry the trace id

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- bin/observability/docker-compose.yml: collector + parca-agent stack
- bin/single-node/docker-compose.yml: mount the otel-collector and parca
  configs and run the parca-agent sidecar

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jun 23, 2026

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @bobbai00, @Yicong-Huang, @aglinxinyuan
    You can notify them by mentioning @bobbai00, @Yicong-Huang, @aglinxinyuan in a comment.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

common dependencies Pull requests that update a dependency file dev docs Changes related to documentations engine platform Non-amber Scala service paths

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Observability] Deploy OpenTelemetry Collector and Parca/eBPF profiling backends

2 participants