Skip to content

ci: keep request replayer responsive under parallel tests - #4106

Draft
bm1549 wants to merge 2 commits into
bengl/ci-quickwinsfrom
brian.marks/fix-request-replayer-concurrency
Draft

ci: keep request replayer responsive under parallel tests#4106
bm1549 wants to merge 2 commits into
bengl/ci-quickwinsfrom
brian.marks/fix-request-replayer-concurrency

Conversation

@bm1549

@bm1549 bm1549 commented Aug 13, 2026

Copy link
Copy Markdown
Collaborator

Description

High-level change and expected CI impact report

Stacked on #4097.

test_extension_ci runs as many as 12 PHPT workers against one single-threaded request-replayer. If its worker gets stuck processing a request, /replay also blocks. The test process then waits on agent requests until the job reaches its two-hour timeout.

Run request-replayer with 16 PHP CLI workers. Its file-backed queues use read-modify-write operations, so each session now has a file lock around state changes. MessagePack decoding stays outside the lock. Requests for different sessions can still run at the same time, including the un-tokened PHPT path.

The CI service gates HTTP requests until the checked-out router and lock helper are installed. The installer retries missing files and transient copy failures. UDP metrics runs under a restart loop.

This is separate from #4097: that PR splits normal and valgrind passes and raises test parallelism; this PR keeps request-replayer responsive under that parallel workload.

Verification

  • Production-router regression test passes locally and in php:8.2-cli with 16 workers.
  • Removing the state lock makes the regression test fail with lost concurrent requests.
  • The exact CI image retained 200 concurrent requests and passed replay, metrics, stats, remote-config, agent-info, clear, and next-response checks.
  • Early traffic remained gated during a simulated delayed checkout; after installation, the service retained 100 concurrent requests without restarting its listener.
  • Killing the UDP metrics process produced a replacement process, which accepted and replayed a tagged metric.
  • The test harness passed 12 consecutive runs in the exact generate-templates image with one CPU and 256 MiB. Child logs use files rather than undrained pipes and are printed if the test fails.
  • PHP and shell syntax checks pass; git diff --check is clean.

Reviewer checklist

  • Test coverage seems ok.
  • Appropriate labels assigned.

@bm1549 bm1549 added the AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos label Aug 13, 2026
@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Aug 13, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🔄 Datadog auto-retried 1 job - 1 passed on retry View in Datadog

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 60.63% (+0.03%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: fa445ce | Docs | Datadog PR Page | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AI Generated Largely based on code generated by an AI or LLM. This label is the same across all dd-trace-* repos

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant