Fix error spam from filtered out log probes - #12207
Conversation
A log probe with a condition that rejects a hit no longer evaluates its capture expressions on that hit. Those captures are dropped before being sent, so evaluating them served no purpose, and any evaluation failure was reported as a debugger event for every single hit, with no rate limiting applied. A probe with a broken capture expression on a hot method could therefore flood the debugger UI with error events. Capture expressions are also skipped when the hit is rate limited, which matches how the log message is handled.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fec98ea077
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
|
🎯 Code Coverage (details) 🔗 Commit SHA: 44df0ec | Docs | Datadog PR Page | Give us feedback! |
![datadog-datadog-prod-us1[bot]](https://avatars.githubusercontent.com/u/365230?s=60&v=4){kind=small}
There was a problem hiding this comment.
More details
The new guard matches snapshot commit semantics: rejected or rate-limited hits cannot contribute capture values, while accepted sampled hits still evaluate them and condition-evaluation errors remain reportable through the existing error path.
🤖 Datadog Autotest · Commit fec98ea · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest
Gate capture expression evaluation on the effective send decision (LogStatus.shouldSend()) instead of the raw sampling flag, so probes in an active debug session keep their capture expressions when their own sampler rejects the hit.
🟢 Java Benchmark SLOs — All performance SLOs passed
PR vs. master results
Commit: Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion. |
1 participant
What Does This Do
Avoids evaluating a log probe’s capture expressions when we already know the hit won’t be sent. This includes hits rejected by the probe condition, dropped by rate limiting, or from a disabled debug session.
Motivation
Previously, capture expressions were evaluated on every probe hit, even when the probe condition failed. Since those captured values were never used, this was unnecessary work.
There was also a more serious issue when a capture expression failed to evaluate. The failure generated a debugger error event, but that event bypassed sampling. The sampler only runs when the probe condition passes or the condition itself fails, so these events kept the default "sampled" status.
As a result, none of the normal limits applied: the per-probe rate limit, the 1 error/second limit, or the global rate limit.
Additional Notes
Reproduced by a new system test, which sends 5 filtered out hits at more than 1s apart and requires at most 1 event.
Contributor Checklist
type:and (comp:orinst:) labels in addition to any other useful labelsclose,fix, or any linking keywords when referencing an issueUse
solvesinstead, and assign the PR milestone to the issueJira ticket: IDEAI-714