From 47df00160d76d47c5d618df398278d04b31155d1 Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Fri, 10 Jul 2026 17:49:00 -0700 Subject: [PATCH 1/4] docs: document the opt-in response cache Adds a Response Cache page to the adaptive plugin docs: when to use it, a plugins.toml quickstart, per-language plugin configuration, the complete-answers-only storage rules with streaming aggregation and native replay, exact-request key normalization, the response_cache mark and doctor surfaces, a field reference, and the unredacted-at-rest caveat for shared Redis deployments. Links the page from the adaptive About and Configuration pages. Signed-off-by: Zhongxuan Wang --- docs/configure-plugins/adaptive/about.mdx | 3 + .../adaptive/configuration.mdx | 6 +- .../adaptive/response-cache.mdx | 321 ++++++++++++++++++ 3 files changed, 328 insertions(+), 2 deletions(-) create mode 100644 docs/configure-plugins/adaptive/response-cache.mdx diff --git a/docs/configure-plugins/adaptive/about.mdx b/docs/configure-plugins/adaptive/about.mdx index e6570839d..3bdf9b30c 100644 --- a/docs/configure-plugins/adaptive/about.mdx +++ b/docs/configure-plugins/adaptive/about.mdx @@ -22,6 +22,7 @@ The plugin can coordinate: - Adaptive hints injected into outgoing model requests. - Tool-parallelism observation or scheduling behavior. - Adaptive Cache Governor (ACG) prompt-cache planning. +- Opt-in response caching for repeated LLM calls. - Component-local validation policy. ## Use Adaptive When @@ -51,6 +52,8 @@ If instrumentation is not in place yet, start with cache planning accomplishes. - [Adaptive Hints](/configure-plugins/adaptive/adaptive-hints) explains request hint injection and how downstream model paths can consume the hints. +- [Response Cache](/configure-plugins/adaptive/response-cache) explains the opt-in LLM response cache: + turning it on, what gets cached, and how savings are reported. State, telemetry, tool parallelism, and policy are whole-plugin configuration areas. They are documented on [Adaptive Configuration](/configure-plugins/adaptive/configuration) rather diff --git a/docs/configure-plugins/adaptive/configuration.mdx b/docs/configure-plugins/adaptive/configuration.mdx index 570729b34..abd8c843f 100644 --- a/docs/configure-plugins/adaptive/configuration.mdx +++ b/docs/configure-plugins/adaptive/configuration.mdx @@ -32,10 +32,12 @@ The top-level adaptive object contains: | `adaptive_hints` | Request hint-injection behavior. | | `tool_parallelism` | Tool scheduling observation or scheduling behavior. | | `acg` | Adaptive Cache Governor prompt-cache planning. | +| `response_cache` | Opt-in LLM response cache for repeated managed calls. | | `policy` | Adaptive-local handling for unknown fields and unsupported values. | -The requested area pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg) and -[Adaptive Hints](/configure-plugins/adaptive/adaptive-hints). State, telemetry, tool parallelism, and +The requested area pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg), +[Adaptive Hints](/configure-plugins/adaptive/adaptive-hints), and +[Response Cache](/configure-plugins/adaptive/response-cache). State, telemetry, tool parallelism, and policy remain whole-plugin settings: - Use `state.backend.kind = "in_memory"` for local experiments. diff --git a/docs/configure-plugins/adaptive/response-cache.mdx b/docs/configure-plugins/adaptive/response-cache.mdx new file mode 100644 index 000000000..fbce1f852 --- /dev/null +++ b/docs/configure-plugins/adaptive/response-cache.mdx @@ -0,0 +1,321 @@ +--- +title: "Response Cache" +description: "" +position: 5 +--- +{/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. +SPDX-License-Identifier: Apache-2.0 */} + + +Use the response cache when the same LLM request is made more than once and the +repeat should be served from a store instead of paying for the call again. A +repeat of an identical request returns the earlier answer instantly, at no +cost, and unchanged — usage fields intact, so a cached reuse is +shape-identical to a live call. + +The cache is an optional `response_cache` section of the +[Adaptive plugin](/configure-plugins/adaptive/configuration), not a standalone plugin +kind. It is off until the section is present, applies to +[managed LLM calls](/instrument-applications/instrument-llm-call) without any +call-site changes, and fails open: any cache error falls back to a normal live +call. + +## When To Use It + +- Development and test loops that replay the same prompts — repeats are + instant, free, and reproducible. +- CI suites that exercise real models — only the first run of each distinct + request pays. +- Demos and workshops — stable answers and no cost spikes. +- A shared team cache — point the `redis` backend at one store and identical + requests hit across processes and machines. +- Gateway deployments — enable caching for an agent without touching its code. + +Reuse requires the request to match exactly after normalization, so prompts +that embed volatile content (timestamps, random IDs) will not repeat. A stored +answer is also frozen for its TTL — size `ttl_seconds` to how fresh answers +must be, and use `bypass_rate` to re-run a sample of cacheable calls live. + +## `plugins.toml` Example + +```toml +version = 1 + +[[components]] +kind = "adaptive" +enabled = true + +[components.config] +version = 1 + +[components.config.response_cache] +ttl_seconds = 3600 # how long a stored answer stays reusable +namespace = "dev-harness" # separates environments, tenants, and experiments +bypass_rate = 0.0 # 0.1 would re-run 10% of cacheable calls live + +[components.config.response_cache.backend] +kind = "in_memory" # or "redis" for a shared cache +``` + +With this configuration the first occurrence of a request runs the provider +and stores the answer; an identical repeat within the TTL is served from the +store and skips the provider entirely. The request's provider surface is +auto-detected from its shape, so there is nothing else to configure. For file +discovery and precedence rules, see +[Plugin Configuration Files](/configure-plugins/plugin-configuration-files). + +## Plugin Configuration + +Use plugin configuration when the application should let NeMo Relay own the +cache lifecycle. + + + +```python +import nemo_relay + +adaptive_config = nemo_relay.adaptive.AdaptiveConfig( + response_cache=nemo_relay.adaptive.ResponseCacheConfig( + ttl_seconds=3600, + namespace="dev-harness", + ), +) + +plugin_config = nemo_relay.plugin.PluginConfig( + components=[nemo_relay.adaptive.ComponentSpec(adaptive_config)] +) + +report = nemo_relay.plugin.validate(plugin_config) +if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): + raise RuntimeError(report["diagnostics"]) + +await nemo_relay.plugin.initialize(plugin_config) + +# Managed calls need no changes. The first call runs the provider; +# an identical repeat is served from the cache. +request = nemo_relay.LLMRequest( + {}, {"model": "gpt-4o", "messages": [{"role": "user", "content": "What is NeMo Relay?"}]} +) +await nemo_relay.llm.execute("openai", request, call_model) # miss: runs call_model +await nemo_relay.llm.execute("openai", request, call_model) # hit: call_model is skipped +``` + + + +```js +const adaptive = require("nemo-relay-node/adaptive"); +const plugin = require("nemo-relay-node/plugin"); + +const adaptiveConfig = adaptive.defaultConfig(); +adaptiveConfig.response_cache = adaptive.responseCacheConfig({ + ttl_seconds: 3600, + namespace: "dev-harness", +}); + +const pluginConfig = plugin.defaultConfig(); +pluginConfig.components = [adaptive.ComponentSpec(adaptiveConfig)]; + +const report = plugin.validate(pluginConfig); +if (report.diagnostics.some((diagnostic) => diagnostic.level === "error")) { + throw new Error(JSON.stringify(report.diagnostics)); +} + +await plugin.initialize(pluginConfig); +// Managed LLM calls now serve identical repeats from the cache. +``` + + + +```rust +use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; +use nemo_relay_adaptive::plugin_component::ComponentSpec; +use nemo_relay_adaptive::{AdaptiveConfig, ResponseCacheConfig}; + +let mut adaptive = AdaptiveConfig::default(); +adaptive.response_cache = Some(ResponseCacheConfig { + ttl_seconds: 3600, + namespace: "dev-harness".into(), + ..ResponseCacheConfig::default() +}); + +let mut plugin_config = PluginConfig::default(); +plugin_config.components.push(ComponentSpec::new(adaptive).into()); + +let report = validate_plugin_config(&plugin_config); +assert!(!report.has_errors()); + +let active = initialize_plugins(plugin_config).await?; +// Managed LLM calls now serve identical repeats from the cache. +``` + + + + + +NeMo Relay plugin configuration keys use `snake_case` regardless of language or +file format. The `backend` helpers differ slightly by binding: Python uses +`nemo_relay.adaptive.BackendSpec.in_memory()` / `BackendSpec.redis(url)`, and +Node.js uses `adaptive.inMemoryBackend()` / `adaptive.redisBackend(url)`. Both +redis helpers set `key_prefix` to `"nemo_relay:"` explicitly, overriding the +default in the fields table; pass a `key_prefix` argument to choose your own. + + +## Manual API + +Use the manual runtime API when an integration needs to own the adaptive +lifecycle directly instead of activating the top-level plugin component. + + + +```python +import nemo_relay + +adaptive_config = nemo_relay.adaptive.AdaptiveConfig( + response_cache=nemo_relay.adaptive.ResponseCacheConfig(namespace="dev-harness"), +) + +runtime = nemo_relay.adaptive.AdaptiveRuntime(adaptive_config.to_dict()) +await runtime.register() +try: + # Run instrumented application work here. + runtime.wait_for_idle() +finally: + await runtime.shutdown() +``` + + + +The Node.js binding exposes the response cache through the adaptive plugin +component helpers. Use the Plugin Configuration example above when activating +the cache from Node.js. + + + +```rust +use nemo_relay_adaptive::{AdaptiveConfig, AdaptiveRuntime, ResponseCacheConfig}; + +let mut adaptive = AdaptiveConfig::default(); +adaptive.response_cache = Some(ResponseCacheConfig { + namespace: "dev-harness".into(), + ..ResponseCacheConfig::default() +}); + +let mut runtime = AdaptiveRuntime::new(adaptive).await?; +runtime.register().await?; + +// Run instrumented application work here. + +runtime.wait_for_idle(); +runtime.shutdown().await?; +``` + + + + +## What Gets Cached + +Only complete, replayable answers are stored: + +- A response with a non-null `error`, or a non-final `status` such as + `failed`, `cancelled`, `incomplete`, or `in_progress`, is never stored. A + failed upstream reply passes through to the caller untouched. +- Stateful requests bypass the cache entirely: a request that opts into + server-side persistence (a non-`false` `store` flag, or an OpenAI Responses + call without an explicit `store = false`), continues a stored interaction + (`previous_response_id`), or references server-side state (`conversation`, + `container`). Their answers depend on state the cache key cannot see. +- Streaming calls are cached too. On a miss the live chunks are forwarded to + the consumer while being assembled into one aggregate response — the same + shape a buffered call stores, so buffered and streaming calls share one + keyspace. On a hit the stored answer is replayed as provider-native chunks + (OpenAI Chat deltas, OpenAI Responses lifecycle events, or Anthropic + Messages events), so strict streaming clients parse it like a live stream. + Truncated or incomplete streams, and streams whose content cannot be + replayed faithfully (for example thinking blocks), are never stored; a + request whose surface cannot be inferred runs live, uncached. + +A hit returns the stored response unchanged. What the hit avoided — tokens and +estimated cost — is reported on the `response_cache` mark, never by editing +the response body. + +## Cache Keys + +Two requests hit the same entry when they are the same request after +normalization, under the default `key_strategy = "exact_request"`: + +- The request is decoded to its normalized form and fingerprinted with + SHA-256, so provider-shaped differences that mean the same thing collapse to + one key. When a request cannot be decoded faithfully, the raw body is + fingerprinted instead — that fallback can only cost a miss, never a wrong + hit. +- Field order and whitespace never matter (RFC 8785 canonicalization). +- Volatile request fields are always dropped from the key: `stream`, `user`, + `metadata`, `service_tier`, and `store`. Add project-specific noise fields + with `skip_keys`. +- Tool-call IDs are normalized, so randomly generated per-call IDs do not + fragment the keyspace. +- Request headers stay out of the key unless explicitly named in + `header_allowlist`; auth headers are rejected outright. +- The `namespace` and the provider name are folded into every key, so + environments and providers never share entries. + +## Observability + +Every cache decision emits a `response_cache` mark with +`data.status` set to one of: + +| Status | Meaning | +|---|---| +| `hit` | Served the stored answer; the provider was skipped. | +| `miss` | No stored answer; the call ran live and, on a clean result, the answer was stored. | +| `bypass` | The request is not cacheable, or the `bypass_rate` sampler chose to run live. | + +Mark metadata rides under `nemo_relay.response_cache.*`: `backend`, +`surface`, `key_hash` (the `sha256:…` fingerprint), `ttl_ms`, `age_ms` and +`saved_tokens` / `saved_cost_usd` on hits, and a `reason` on bypasses and +store-error misses (for example `sampled`, `stateful_store`, `store_error`, +`stream_no_codec`). Marks carry only fingerprints — never prompts, answers, or +credentials. + +`nemo-relay doctor` reports the cache state: `not configured` when the section +is absent, `configured but disabled (adaptive plugin disabled)` when the +adaptive component is off, `on; backend '' reachable` when healthy, and +a failure when the config is invalid or the backend is unreachable. + +## Fields + +| Field | Default | Notes | +|---|---|---| +| `ttl_seconds` | `3600` | How long a stored answer stays reusable, in seconds. | +| `namespace` | `""` | Folded into every key; separates environments, tenants, and experiments. | +| `priority` | `50` | LLM execution intercept priority. Lower values run earlier. | +| `bypass_rate` | `0.0` | Probability in `[0.0, 1.0]` of running a cacheable call live. A sampled call refreshes the stored answer, so this doubles as drift detection. | +| `cache_nondeterministic` | `true` | A repeat of a nondeterministic request serves one stored sample. Set `false` to cache only requests pinned deterministic (`temperature = 0`). | +| `key_strategy` | `"exact_request"` | The only supported strategy: reuse requires the same normalized request. | +| `header_allowlist` | `[]` | Headers folded into the key (case-insensitive). Auth headers are rejected. | +| `skip_keys` | `[]` | Extra top-level body keys dropped from the key, beyond the built-in volatile set. | +| `backend.kind` | `"in_memory"` | `"in_memory"`, or `"redis"` (requires building with the `redis-backend` feature). | +| `backend.config.max_bytes` | 256 MiB | In-memory size budget; the oldest entries are evicted first. | +| `backend.config.url` | — | Redis connection URL. Required for the `redis` backend. | +| `backend.config.key_prefix` | `"nemo-relay:llm-cache:"` | Prefix for keys in Redis. | + + +Cached entries are stored unredacted. PII sanitize guardrails rewrite emitted +telemetry, never payloads, so the store holds full request and response +bodies. A shared Redis backend must be trusted, access-controlled, and +namespaced per environment and tenant. + + +## Common Validation Failures + +- `ttl_seconds` is `0`, or `bypass_rate` is outside `[0.0, 1.0]`. +- `key_strategy` is not `"exact_request"`. +- `skip_keys` names an answer-determining field such as `messages`, `model`, + `tools`, or `tool_choice`. +- `header_allowlist` names an auth header such as `authorization` or + `x-api-key`. +- in_memory `max_bytes` set to zero or a non-integer. +- `backend.kind = "redis"` without `backend.config.url`, or in a build without + the `redis-backend` feature. +- A `redis` backend with an empty `namespace` warns: every caller of the + shared store would exchange cached answers. From 00ded9bb195b3b9104330e55fa869c3891f6efad Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Wed, 22 Jul 2026 17:41:22 -0700 Subject: [PATCH 2/4] docs(node): use response-cache camelCase API Signed-off-by: Zhongxuan Wang --- docs/configure-plugins/adaptive/response-cache.mdx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/configure-plugins/adaptive/response-cache.mdx b/docs/configure-plugins/adaptive/response-cache.mdx index fbce1f852..4130c1007 100644 --- a/docs/configure-plugins/adaptive/response-cache.mdx +++ b/docs/configure-plugins/adaptive/response-cache.mdx @@ -107,8 +107,8 @@ const adaptive = require("nemo-relay-node/adaptive"); const plugin = require("nemo-relay-node/plugin"); const adaptiveConfig = adaptive.defaultConfig(); -adaptiveConfig.response_cache = adaptive.responseCacheConfig({ - ttl_seconds: 3600, +adaptiveConfig.responseCache = adaptive.responseCacheConfig({ + ttlSeconds: 3600, namespace: "dev-harness", }); From d0fae8eba0902b8beba51d277a5422506d21cf3e Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Wed, 22 Jul 2026 18:22:54 -0700 Subject: [PATCH 3/4] docs(response-cache): qualify reuse claims Signed-off-by: Zhongxuan Wang --- .../adaptive/response-cache.mdx | 21 ++++++++++--------- 1 file changed, 11 insertions(+), 10 deletions(-) diff --git a/docs/configure-plugins/adaptive/response-cache.mdx b/docs/configure-plugins/adaptive/response-cache.mdx index 4130c1007..da46146d2 100644 --- a/docs/configure-plugins/adaptive/response-cache.mdx +++ b/docs/configure-plugins/adaptive/response-cache.mdx @@ -8,10 +8,10 @@ SPDX-License-Identifier: Apache-2.0 */} Use the response cache when the same LLM request is made more than once and the -repeat should be served from a store instead of paying for the call again. A -repeat of an identical request returns the earlier answer instantly, at no -cost, and unchanged — usage fields intact, so a cached reuse is -shape-identical to a live call. +repeat should be served from a store instead of calling the provider again. An +eligible request that matches an unexpired cache entry can be served from the +store without calling the provider. Cached responses preserve the stored +response shape and usage fields. The cache is an optional `response_cache` section of the [Adaptive plugin](/configure-plugins/adaptive/configuration), not a standalone plugin @@ -22,13 +22,14 @@ call. ## When To Use It -- Development and test loops that replay the same prompts — repeats are - instant, free, and reproducible. -- CI suites that exercise real models — only the first run of each distinct - request pays. -- Demos and workshops — stable answers and no cost spikes. +- Development and test loops that replay the same prompts — eligible repeats + can reuse stored responses. +- CI suites that exercise real models — cached repeats can avoid additional + provider calls. +- Demos and workshops — reuse stable stored answers while reducing provider + calls. - A shared team cache — point the `redis` backend at one store and identical - requests hit across processes and machines. + requests can reuse entries across processes and machines. - Gateway deployments — enable caching for an agent without touching its code. Reuse requires the request to match exactly after normalization, so prompts From 0a3872073629470a3d6b457d1c4fba71de1790c3 Mon Sep 17 00:00:00 2001 From: Zhongxuan Wang Date: Mon, 27 Jul 2026 12:49:31 -0700 Subject: [PATCH 4/4] docs(response-cache): correct configuration guidance Signed-off-by: Zhongxuan Wang --- .../adaptive/configuration.mdx | 4 +- .../adaptive/response-cache.mdx | 187 ++++++++++++------ 2 files changed, 127 insertions(+), 64 deletions(-) diff --git a/docs/configure-plugins/adaptive/configuration.mdx b/docs/configure-plugins/adaptive/configuration.mdx index abd8c843f..21719d06f 100644 --- a/docs/configure-plugins/adaptive/configuration.mdx +++ b/docs/configure-plugins/adaptive/configuration.mdx @@ -1,7 +1,7 @@ --- title: "Adaptive Configuration" sidebar-title: "Configuration" -description: "Configure the built-in Adaptive component, including state, telemetry, hints, and ACG." +description: "Configure the Adaptive component, including state, telemetry, hints, ACG, and response caching." position: 2 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -35,7 +35,7 @@ The top-level adaptive object contains: | `response_cache` | Opt-in LLM response cache for repeated managed calls. | | `policy` | Adaptive-local handling for unknown fields and unsupported values. | -The requested area pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg), +Dedicated pages cover [Adaptive Cache Governor (ACG)](/configure-plugins/adaptive/acg), [Adaptive Hints](/configure-plugins/adaptive/adaptive-hints), and [Response Cache](/configure-plugins/adaptive/response-cache). State, telemetry, tool parallelism, and policy remain whole-plugin settings: diff --git a/docs/configure-plugins/adaptive/response-cache.mdx b/docs/configure-plugins/adaptive/response-cache.mdx index da46146d2..333b842b3 100644 --- a/docs/configure-plugins/adaptive/response-cache.mdx +++ b/docs/configure-plugins/adaptive/response-cache.mdx @@ -1,6 +1,6 @@ --- title: "Response Cache" -description: "" +description: "Configure exact-match response caching for managed LLM calls." position: 5 --- {/* SPDX-FileCopyrightText: Copyright (c) 2026, NVIDIA CORPORATION & AFFILIATES. All rights reserved. @@ -10,15 +10,18 @@ SPDX-License-Identifier: Apache-2.0 */} Use the response cache when the same LLM request is made more than once and the repeat should be served from a store instead of calling the provider again. An eligible request that matches an unexpired cache entry can be served from the -store without calling the provider. Cached responses preserve the stored -response shape and usage fields. +store without calling the provider. Buffered hits preserve the stored response +shape and usage fields; streaming hits replay an equivalent provider-native +stream. The cache is an optional `response_cache` section of the [Adaptive plugin](/configure-plugins/adaptive/configuration), not a standalone plugin kind. It is off until the section is present, applies to -[managed LLM calls](/instrument-applications/instrument-llm-call) without any -call-site changes, and fails open: any cache error falls back to a normal live -call. +[managed LLM calls](/instrument-applications/instrument-llm-call) without +changing the execution API. By default, only requests with an explicit numeric +`temperature = 0` are eligible; set `cache_nondeterministic = true` to opt +sampled requests into caching. Runtime backend errors fail open to a normal +live call, while invalid configuration is rejected during validation. ## When To Use It @@ -34,8 +37,9 @@ call. Reuse requires the request to match exactly after normalization, so prompts that embed volatile content (timestamps, random IDs) will not repeat. A stored -answer is also frozen for its TTL — size `ttl_seconds` to how fresh answers -must be, and use `bypass_rate` to re-run a sample of cacheable calls live. +answer remains reusable until its TTL expires unless a sampled bypass +successfully refreshes it. Size `ttl_seconds` to how fresh answers must be, and +use `bypass_rate` to re-run a sample of cacheable calls live. ## `plugins.toml` Example @@ -53,16 +57,17 @@ version = 1 ttl_seconds = 3600 # how long a stored answer stays reusable namespace = "dev-harness" # separates environments, tenants, and experiments bypass_rate = 0.0 # 0.1 would re-run 10% of cacheable calls live +cache_nondeterministic = false # set true to cache nondeterministic requests [components.config.response_cache.backend] kind = "in_memory" # or "redis" for a shared cache ``` -With this configuration the first occurrence of a request runs the provider -and stores the answer; an identical repeat within the TTL is served from the -store and skips the provider entirely. The request's provider surface is -auto-detected from its shape, so there is nothing else to configure. For file -discovery and precedence rules, see +With this configuration the first eligible occurrence of a request runs the +provider and attempts to store a complete answer. A later eligible, identical +request within the TTL is served from the store when that write succeeds. The +request's provider surface is auto-detected from its shape, so there is nothing +else to configure. For file discovery and precedence rules, see [Plugin Configuration Files](/configure-plugins/plugin-configuration-files). ## Plugin Configuration @@ -92,10 +97,24 @@ if any(diagnostic["level"] == "error" for diagnostic in report["diagnostics"]): await nemo_relay.plugin.initialize(plugin_config) +def call_model(_request): + return { + "model": "gpt-4o", + "choices": [{ + "message": {"role": "assistant", "content": "NeMo Relay instruments agent calls."}, + "finish_reason": "stop", + }], + } + # Managed calls need no changes. The first call runs the provider; # an identical repeat is served from the cache. request = nemo_relay.LLMRequest( - {}, {"model": "gpt-4o", "messages": [{"role": "user", "content": "What is NeMo Relay?"}]} + {}, + { + "model": "gpt-4o", + "messages": [{"role": "user", "content": "What is NeMo Relay?"}], + "temperature": 0, + }, ) await nemo_relay.llm.execute("openai", request, call_model) # miss: runs call_model await nemo_relay.llm.execute("openai", request, call_model) # hit: call_model is skipped @@ -121,44 +140,62 @@ if (report.diagnostics.some((diagnostic) => diagnostic.level === "error")) { throw new Error(JSON.stringify(report.diagnostics)); } -await plugin.initialize(pluginConfig); -// Managed LLM calls now serve identical repeats from the cache. +void (async () => { + await plugin.initialize(pluginConfig); + try { + // Eligible deterministic repeats are served from the cache. + } finally { + plugin.clear(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; +}); ``` ```rust -use nemo_relay::plugin::{initialize_plugins, validate_plugin_config, PluginConfig}; -use nemo_relay_adaptive::plugin_component::ComponentSpec; +use nemo_relay::plugin::{ + clear_plugin_configuration, initialize_plugins, validate_plugin_config, PluginConfig, +}; +use nemo_relay_adaptive::plugin_component::{register_adaptive_component, ComponentSpec}; use nemo_relay_adaptive::{AdaptiveConfig, ResponseCacheConfig}; -let mut adaptive = AdaptiveConfig::default(); -adaptive.response_cache = Some(ResponseCacheConfig { - ttl_seconds: 3600, - namespace: "dev-harness".into(), - ..ResponseCacheConfig::default() -}); +#[tokio::main] +async fn main() -> Result<(), Box> { + let mut adaptive = AdaptiveConfig::default(); + adaptive.response_cache = Some(ResponseCacheConfig { + ttl_seconds: 3600, + namespace: "dev-harness".into(), + ..ResponseCacheConfig::default() + }); -let mut plugin_config = PluginConfig::default(); -plugin_config.components.push(ComponentSpec::new(adaptive).into()); + let mut plugin_config = PluginConfig::default(); + plugin_config.components.push(ComponentSpec::new(adaptive).into()); -let report = validate_plugin_config(&plugin_config); -assert!(!report.has_errors()); + register_adaptive_component()?; + let report = validate_plugin_config(&plugin_config); + assert!(!report.has_errors()); -let active = initialize_plugins(plugin_config).await?; -// Managed LLM calls now serve identical repeats from the cache. + let _active = initialize_plugins(plugin_config).await?; + // Eligible deterministic repeats are now served from the cache. + + clear_plugin_configuration()?; + Ok(()) +} ``` -NeMo Relay plugin configuration keys use `snake_case` regardless of language or -file format. The `backend` helpers differ slightly by binding: Python uses -`nemo_relay.adaptive.BackendSpec.in_memory()` / `BackendSpec.redis(url)`, and -Node.js uses `adaptive.inMemoryBackend()` / `adaptive.redisBackend(url)`. Both -redis helpers set `key_prefix` to `"nemo_relay:"` explicitly, overriding the -default in the fields table; pass a `key_prefix` argument to choose your own. +Canonical plugin documents and `plugins.toml` use `snake_case`. Python helpers +follow that convention, while Node.js convenience helpers use `camelCase` and +serialize it to the canonical shape. Python uses +`BackendSpec.redis(url, key_prefix=...)`; Node.js uses +`adaptive.redisBackend(url, keyPrefix)`. Both Redis helpers default the prefix +to `"nemo_relay:"`, overriding the value in the fields table. ## Manual API @@ -186,28 +223,51 @@ finally: -The Node.js binding exposes the response cache through the adaptive plugin -component helpers. Use the Plugin Configuration example above when activating -the cache from Node.js. +```js +const adaptive = require("nemo-relay-node/adaptive"); + +const adaptiveConfig = adaptive.defaultConfig(); +adaptiveConfig.responseCache = adaptive.responseCacheConfig({ + namespace: "dev-harness", +}); + +const runtime = new adaptive.AdaptiveRuntime(adaptiveConfig); +void (async () => { + await runtime.register(); + try { + // Run instrumented application work here. + runtime.waitForIdle(); + } finally { + await runtime.shutdown(); + } +})().catch((error) => { + console.error(error); + process.exitCode = 1; +}); +``` ```rust use nemo_relay_adaptive::{AdaptiveConfig, AdaptiveRuntime, ResponseCacheConfig}; -let mut adaptive = AdaptiveConfig::default(); -adaptive.response_cache = Some(ResponseCacheConfig { - namespace: "dev-harness".into(), - ..ResponseCacheConfig::default() -}); +#[tokio::main] +async fn main() -> Result<(), Box> { + let mut adaptive = AdaptiveConfig::default(); + adaptive.response_cache = Some(ResponseCacheConfig { + namespace: "dev-harness".into(), + ..ResponseCacheConfig::default() + }); -let mut runtime = AdaptiveRuntime::new(adaptive).await?; -runtime.register().await?; + let mut runtime = AdaptiveRuntime::new(adaptive).await?; + runtime.register().await?; -// Run instrumented application work here. + // Run instrumented application work here. -runtime.wait_for_idle(); -runtime.shutdown().await?; + runtime.wait_for_idle(); + runtime.shutdown().await?; + Ok(()) +} ``` @@ -235,9 +295,10 @@ Only complete, replayable answers are stored: replayed faithfully (for example thinking blocks), are never stored; a request whose surface cannot be inferred runs live, uncached. -A hit returns the stored response unchanged. What the hit avoided — tokens and -estimated cost — is reported on the `response_cache` mark, never by editing -the response body. +A buffered hit returns the stored response unchanged. A streaming hit replays +an equivalent provider-native stream. What the hit avoided — tokens and +estimated cost — is reported on the `response_cache` mark, never by editing a +buffered response body. ## Cache Keys @@ -257,8 +318,10 @@ normalization, under the default `key_strategy = "exact_request"`: fragment the keyspace. - Request headers stay out of the key unless explicitly named in `header_allowlist`; auth headers are rejected outright. -- The `namespace` and the provider name are folded into every key, so - environments and providers never share entries. +- The provider name is folded into every key, so different providers do not + share entries. The `namespace` is also folded into the key; assign distinct + values to isolate environments or tenants. Identical or empty namespaces can + share entries. ## Observability @@ -268,7 +331,7 @@ Every cache decision emits a `response_cache` mark with | Status | Meaning | |---|---| | `hit` | Served the stored answer; the provider was skipped. | -| `miss` | No stored answer; the call ran live and, on a clean result, the answer was stored. | +| `miss` | No stored answer; the call ran live, and a cacheable result is stored when the write succeeds. | | `bypass` | The request is not cacheable, or the `bypass_rate` sampler chose to run live. | Mark metadata rides under `nemo_relay.response_cache.*`: `backend`, @@ -288,10 +351,10 @@ a failure when the config is invalid or the backend is unreachable. | Field | Default | Notes | |---|---|---| | `ttl_seconds` | `3600` | How long a stored answer stays reusable, in seconds. | -| `namespace` | `""` | Folded into every key; separates environments, tenants, and experiments. | +| `namespace` | `""` | Folded into every key. Set distinct values to separate environments, tenants, and experiments. | | `priority` | `50` | LLM execution intercept priority. Lower values run earlier. | -| `bypass_rate` | `0.0` | Probability in `[0.0, 1.0]` of running a cacheable call live. A sampled call refreshes the stored answer, so this doubles as drift detection. | -| `cache_nondeterministic` | `true` | A repeat of a nondeterministic request serves one stored sample. Set `false` to cache only requests pinned deterministic (`temperature = 0`). | +| `bypass_rate` | `0.0` | Probability in `[0.0, 1.0]` of running a cacheable call live. A sampled call attempts to refresh the entry when its result is cacheable and the write succeeds. | +| `cache_nondeterministic` | `false` | Nondeterministic requests bypass the cache. Set `true` to cache and reuse sampled responses. | | `key_strategy` | `"exact_request"` | The only supported strategy: reuse requires the same normalized request. | | `header_allowlist` | `[]` | Headers folded into the key (case-insensitive). Auth headers are rejected. | | `skip_keys` | `[]` | Extra top-level body keys dropped from the key, beyond the built-in volatile set. | @@ -301,10 +364,10 @@ a failure when the config is invalid or the backend is unreachable. | `backend.config.key_prefix` | `"nemo-relay:llm-cache:"` | Prefix for keys in Redis. | -Cached entries are stored unredacted. PII sanitize guardrails rewrite emitted -telemetry, never payloads, so the store holds full request and response -bodies. A shared Redis backend must be trusted, access-controlled, and -namespaced per environment and tenant. +Cached responses are stored unredacted. PII sanitize guardrails rewrite emitted +telemetry, never payloads, so the store holds full response bodies. Requests +contribute only a SHA-256 fingerprint to the cache key. A shared Redis backend +must be trusted, access-controlled, and namespaced per environment and tenant. ## Common Validation Failures