Send agentless Feature Flags exposures directly to EVP

[leoromanovsky]

Motivation

Java can evaluate Feature Flags from the managed CDN without a local Datadog receiver. The current writer drops exposures when no local EVP proxy exists. This loses experimentation data in the default agentless deployment.

flowchart TD
    E[Exposure event] --> S{Configuration source}
    S -->|Remote Configuration| L[Local EVP proxy]
    S -->|Agentless| R{Compatible local EVP route?}
    R -->|Yes| L
    R -->|No, API key available| D[Direct EVP intake<br/>/api/v2/exposures]
    R -->|No API key| X[Disable exposure delivery]
    L --> P[Event Platform exposures]
    D --> P

Loading

Feature Flags use normal Agent discovery. Direct delivery requires an API key. A definitive local rejection can also select direct delivery.

Changes and Decisions

This PR adds direct EVP capability for exposure events. It does not change flagevaluation delivery.

• Keep Remote Configuration exposure delivery on the local EVP proxy.
• In agentless mode, prefer a compatible local EVP proxy from the Datadog Agent or serverless-init.
• Use authenticated direct EVP intake when no compatible local route exists and an API key is available.
• Switch from local delivery to direct delivery after connection refusal or HTTP 403, 404, or 405.
• Do not replay after timeouts, resets, rate limits, or server errors. This prevents possible duplicate exposures.
• Initialize the exposure writer and CDN poller only after application provider activation.
• Preserve the existing exposure payload, batching, and deduplication behavior.
• Keep traces and OTLP metrics on the Datadog Agent or serverless-init. This change does not send them directly.

Validation

The shared exposure contract is enabled in DataDog/system-tests#7494.

• Tested dd-trace-java 9992388df3 with the Spring Boot weblog.
• The Datadog Agent, direct, and serverless-init:1.9.13 routes each passed.
• Each route evaluated the same flag five times and produced one deduplicated exposure.
• The direct route used no local receiver.
• The unused direct or sidecar route received no exposure.

[Copilot]

Pull request overview

This PR updates Feature Flagging exposure delivery to support agentless deployments by selecting an appropriate transport (local EVP proxy when available, with safe direct-intake fallback when necessary), and adds supporting test coverage across feature-flagging and communication modules.

Changes:

• Introduce an exposure-specific backend selection layer (ExposureBackendApiFactory) and an agentless fallback transport (AgentlessExposureBackendApi) to switch from local EVP proxy to direct intake on definitive rejections.
• Start exposure delivery earlier in agentless mode (before application provider activation) while keeping the configuration source lazy until activation.
• Improve HTTP failure observability for EVP proxy calls by surfacing status codes via HttpResponseException, and add targeted tests.

Reviewed changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
products/feature-flagging/feature-flagging-lib/src/test/java/com/datadog/featureflag/ExposureWriterTests.java	Adds coverage for agentless direct-intake exposure delivery (API key header + direct endpoint).
products/feature-flagging/feature-flagging-lib/src/test/java/com/datadog/featureflag/ExposureBackendApiFactoryTest.java	New unit tests for backend selection rules (remote-config vs agentless, local vs direct, invalid direct URL).
products/feature-flagging/feature-flagging-lib/src/test/java/com/datadog/featureflag/AgentlessExposureBackendApiTest.java	New unit tests validating fallback and replay behavior for definitive vs ambiguous failures.
products/feature-flagging/feature-flagging-lib/src/main/java/com/datadog/featureflag/ExposureWriterImpl.java	Switches exposure writer to use ExposureBackendApiFactory rather than the generic BackendApiFactory selection.
products/feature-flagging/feature-flagging-lib/src/main/java/com/datadog/featureflag/ExposureBackendApiFactory.java	New transport selection logic for exposures, including agentless direct-intake fallback (when API key exists).
products/feature-flagging/feature-flagging-lib/src/main/java/com/datadog/featureflag/AgentlessExposureBackendApi.java	New BackendApi wrapper that switches from local EVP to direct intake on definitive rejections.
products/feature-flagging/feature-flagging-lib/build.gradle.kts	Adds config module dependency for compile-time constants (and test dependency for new tests).
products/feature-flagging/feature-flagging-agent/src/test/java/com/datadog/featureflag/FeatureFlaggingSystemTest.java	Updates system test expectations for earlier exposure writer initialization in agentless mode.
products/feature-flagging/feature-flagging-agent/src/main/java/com/datadog/featureflag/FeatureFlaggingSystem.java	Initializes exposure writer pre-activation for agentless mode; adds state inspection helpers used by tests.
communication/src/test/java/datadog/communication/EvpProxyApiTest.java	New test ensuring non-2xx responses surface status codes and preserve expected request shape.
communication/src/main/java/datadog/communication/HttpResponseException.java	New exception type carrying HTTP status codes for failed requests.
communication/src/main/java/datadog/communication/EvpProxyApi.java	Throws HttpResponseException (instead of generic IOException) on non-success responses.
communication/src/main/java/datadog/communication/BackendApiFactory.java	Refactors backend creation into separate local-EVP vs direct-intake factory methods.

Suppressed comments (1)

products/feature-flagging/feature-flagging-lib/src/main/java/com/datadog/featureflag/ExposureWriterImpl.java:143

• ExposureBackendApiFactory#create() is explicitly nullable when no local EVP proxy or direct intake credentials are available, but the serializer thread treats this as an exceptional state and throws an uncaught IllegalArgumentException. This will surface as a background-thread crash even though the factory already logs that delivery is disabled. Prefer a clean shutdown (log + return) after errorCallback.run() and use an error message that matches the broader set of failure reasons (not only EVP proxy).

      evp = backendApiFactory.create();
      if (evp == null) {
        errorCallback.run();
        throw new IllegalArgumentException("EVP Proxy not available");
      }

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dd-octo-sts]

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.85 s	14.65 s	[+0.5%; +2.3%] (maybe worse)
startup:insecure-bank:tracing:Agent	13.64 s	13.73 s	[-1.6%; +0.3%] (no difference)
startup:petclinic:appsec:Agent	16.89 s	16.89 s	[-1.0%; +1.0%] (no difference)
startup:petclinic:iast:Agent	17.02 s	16.92 s	[-0.3%; +1.4%] (no difference)
startup:petclinic:profiling:Agent	16.70 s	16.28 s	[-2.0%; +7.2%] (no difference)
startup:petclinic:sca:Agent	16.72 s	16.69 s	[-0.8%; +1.2%] (no difference)
startup:petclinic:tracing:Agent	16.19 s	16.19 s	[-0.9%; +1.0%] (no difference)

Commit: 6b3be03f · CI Pipeline · Benchmarking Platform UI

---

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

[datadog-official]

Early local-EVP discovery can permanently disable exposure delivery without an API key: the writer exits before provider activation, yet activation reuses that dead writer even after the local route becomes available.

_{📊 Validated against 10 scenarios · Open Bits AI session}

_{🤖 Datadog Autotest · Commit 3716594 · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated no new comments.

Suppressed comments (2)

communication/src/main/java/datadog/communication/BackendApiFactory.java:56

• The FatalAgentMisconfigurationError message is misleading here: createDirectIntakeApi(...) can be used outside of "agentless mode" (e.g., direct intake fallback), but the error claims agentless mode is enabled. Using a generic message will be clearer for users troubleshooting missing credentials.

    if (apiKey == null || apiKey.isEmpty()) {
      throw new FatalAgentMisconfigurationError(
          "Agentless mode is enabled and API key is not set. Please set DD_API_KEY");
    }

products/feature-flagging/feature-flagging-lib/src/test/java/com/datadog/featureflag/ExposureWriterTests.java:156

• This test creates a new OkHttpClient instance that is never shut down, which can leak threads/resources across the test JVM. Reusing the existing SharedCommunicationObjects client avoids introducing an extra client lifecycle to manage here.

            new OkHttpClient.Builder().build(),

[Copilot]

Pull request overview

Copilot reviewed 14 out of 14 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

products/feature-flagging/feature-flagging-lib/src/main/java/com/datadog/featureflag/AgentlessExposureBackendApi.java:47

• The real transport still replays ambiguous failures despite the stated no-replay contract. selectedApi is an EvpProxyApi created with the shared five-retry policy, which retries timeouts, 429s, and 5xx responses before this catch runs; if it ultimately throws here, ExposureWriterImpl.flushIfNecessary() also retains buffer and posts it again on the next duty cycle (ExposureWriterTests.testFailuresAreRetried verifies that behavior). The fake backend in the new unit test bypasses both retry layers. Use a no-retry exposure transport and drop/clear batches whose delivery is ambiguous, then cover this through the real writer/HTTP stack.

      return selectedApi.post(
          uri, requestBody, responseParser, requestListener, requestCompression);
    } catch (final IOException exception) {
      if (selectedApi != localApi || !isDefinitiveRejection(exception)) {
        throw exception;