docs: define and validate KV-affine Responses routing

[franciscojavierarceo]

Summary

• Add ADR-04 - KV-affine continuation for agentic Responses workloads, covering the end-to-end lifecycle from agentic-api rehydration and llm-d placement through vLLM APC lookup, generation, and KV-event publication.
• Make fleet-wide KV-cache reuse the primary objective. Exact llm-d routing is the first milestone; compact portable token artifacts are a later transport/rendering optimization.
• Document the required cache_salt contract and the isolation failure found during live validation. The implementation fix is intentionally split into fix: preserve Responses cache salt #96.
• Record the required coordinated changes in agentic-api, vLLM, llm-d-router, llm-d deployment configuration, and later llm-d-kv-cache work.
• Add the raw results and conclusions from the two-replica GPT-OSS-20B llm-d routing benchmark.

Tracking

• Epic: #69 - Maximize KV-cache reuse for agentic Responses workloads
• Milestone 1: #73 - Upstream exact Responses prefix routing through llm-d
• Milestone 2: #71 - Persist reliable Responses continuation checkpoints
• Milestone 3: #70 - Add portable vLLM token artifacts for Responses continuations
• Benchmark expansion: #74 - Benchmark KV-affine Responses routing and token artifacts
• Completed transport prerequisite: #72 - Add Responses WebSocket support for Codex continuations
• Separate implementation fix: #96 - Preserve Responses cache salt

Request lifecycle

Client
  -> agentic-api: resolve previous_response_id and rehydrate full logical history
  -> llm-d EPP: ask a vLLM replica's /tokenize endpoint for the exact rendered Responses tokens
  -> llm-d precise-prefix producer: compute canonical block keys and query the KV-event index
  -> llm-d scorer/picker: select the replica with the longest resident matching prefix
  -> selected vLLM replica: render the same request and perform the normal APC lookup
  -> vLLM scheduler/model runner: prefill only the uncached tail, decode, and generate output
  -> vLLM KV-event publisher: report newly stored blocks back to llm-d
  -> agentic-api: persist the response and return/stream it to the client

The vLLM-placement router must run after agentic-api rehydration. A router placed only before agentic-api cannot see the model-visible history represented by previous_response_id.

Measured result

The accepted benchmark used two GPT-OSS-20B vLLM replicas with APC enabled, 64-token blocks, distinct KV-event publishers, and 12 sequential two-turn pairs per routing profile. Every pair used a new unpredictable cache_salt; its continuation used only previous_response_id, forcing agentic-api to rehydrate 6,653 input tokens before llm-d placement.

The routing profiles differ only in how llm-d selects the continuation replica:

• Load-only considers backend load and queue/KV utilization but ignores which replica owns the cached prefix. With two equivalent replicas, returning to the warm replica is approximately a coin flip.
• Approximate prefix derives a pseudo-token prefix identity from the logical request. It can prefer a warm replica without asking vLLM for the exact rendered Responses tokens, but it can diverge when templates, tools, instructions, reasoning configuration, or branching history affect rendering.
• Precise Responses prefix runs after agentic-api rehydration, asks vLLM /tokenize for the exact model-visible tokens, converts them to llm-d canonical block keys, and selects the replica with the longest known matching prefix in the KV-event index.

Profile	Warm replica returns	Mean KV-cache hit	Mean continuation latency	95% CI
Load-only	6/12	49.54%	811.0 ms	+/-313.8 ms
Approximate prefix	12/12	99.08%	314.2 ms	+/-15.2 ms
Precise Responses prefix	12/12	99.08%	316.5 ms	+/-8.6 ms

Precise routing improved over load-only by 49.54 cache-hit percentage points and reduced mean continuation latency by 494.5 ms (60.97%).

The 99.08% hit rate means vLLM reused 6,592 of 6,653 continuation input tokens; APC matches complete blocks and still recomputes the unmatched or required tail. Load-only has a wide latency interval because its six warm returns took approximately 0.30-0.42 seconds while its six wrong-replica continuations took approximately 1.27-1.31 seconds. Both cache-aware profiles returned all 12 continuations to the warm replica, producing narrow intervals near 0.31 seconds.

Approximate and precise routing are effectively tied in this idle deterministic workload. The result proves that exact Responses identity reaches the maximum observed KV reuse without pseudo-token estimates; it does not yet prove an advantage over approximate routing under template changes, tools, concurrency, event lag, branching histories, or cache pressure.

Raw evidence is committed under docs/adr/results/adr-04/2026-07-13-n12.

Cache-salt isolation gate

Before accepting the routing comparison, the same 12,944-token prompt was sent to one fixed replica using salt A, salt B, and salt B again:

• Salt A: 0 cached tokens, 2,617.0 ms
• Different salt B: 0 cached tokens, 2,575.5 ms
• Repeated salt B: 12,928 cached tokens, 283.0 ms

This confirms that agentic-api forwards the salt, different salts are isolated, and identical salts can reuse the matching vLLM blocks.

Required upstream work

Project	Stage 1 requirement	Current status
agentic-api	Preserve cache_salt; point the execution LLM URL at llm-d after rehydration	Routed deployment validated; implementation split into #96
vLLM	Accept Responses requests at /tokenize and delegate to the same Responses renderer used for inference	Implemented on the benchmark branch; needs an upstream vLLM PR
llm-d-router	Dispatch Responses token production to vLLM /tokenize, preserve salt, reject incomplete identities, and propagate file-discovery rankIndex for same-host replicas	Implemented on the benchmark branch; needs an upstream llm-d-router PR
llm-d	Ship/pin the precise EPP configuration, matching 64-token block size, endpoint ranks, and KV-event ports	Deployment/configuration change; no new Stage 1 runtime core required
llm-d-kv-cache	Use the existing vLLM event adapter and canonical request-key index	No Stage 1 code change required; receipts and active-active behavior are later work
vLLM APC/core	Reuse existing block hashing, allocation, scheduler, and KV-event behavior	No Stage 1 core change required

Scope boundaries

This PR is documentation and benchmark evidence only. It completes the single-client Stage 1 proof; production acceptance still requires concurrent agent/tool traffic, cache pressure and eviction, event lag, branching conversations, replica restart, and active-active EPP testing.

Portable token artifacts remain useful after routing is correct: they can avoid repeatedly transferring, reconstructing, rendering, and tokenizing long logical histories. They must preserve the Stage 1 cache-hit ratio and cannot replace llm-d's fleet-level placement decision with an endpoint-local handle.

Test Plan

• uvx pre-commit run --all-files
• uv run --with-requirements docs/requirements.txt mkdocs build --strict
• xmllint --noout docs/adr/results/adr-04/2026-07-13-n12/routing-comparison.svg
• Render and visually inspect the committed SVG
• Recompute the committed summaries from all three raw benchmark CSV files
• Verify vLLM /tokenize parity across both replicas
• Verify salt isolation against a fixed replica
• Verify EPP KV-event subscriptions, index admissions, lookups, and absence of tokenization/ZMQ errors

[maralbahari]

@franciscojavierarceo thank you for the comprehensive ADR. I wanted to clarify my understanding of the overall follow proposed.

Turn 1: no cache yet full path

1. rehydrate items from DB (ordered)
2. send full prompt to vLLM
3. vLLM renders + tokenizes → generates output
4. vLLM returns response + prompt_token_ids = [t0...t_N]
5. agentic-api stores:
   • response items in DB
   • prefix_hash = sha256([t0...t_N])
   • prefix_token_count = N
   • tokenizer_fingerprint, renderer_fingerprint, template_fingerprint
     there would not be any raw token ids stored in agentic-api storage right?

Turn 2: cache path

1. rehydrate items from DB (ordered)
2. fresh full render locally → [t0, t1, ... t_N, t_N+1, t_N+2]
3. strict-prefix check:
   sha256(fresh_render[:N]) == stored prefix_hash?
   → YES: proceed with replay
   → NO: fall back to full prompt (back to turn 1 path)
4. marginal suffix = fresh_render[N:] = [t_N+1, t_N+2]
5. send to vLLM:
   {
   prompt_cache_ref: {
   handle: "vllm_prefix_...",
   prefix_hash: "sha256:...",
   prefix_token_count: N,
   tokenizer_fingerprint: "...",
   renderer_fingerprint: "..."
   },
   append_token_ids: [t_N+1, t_N+2]
   }
6. vLLM validates handle, skips render+tokenize for prefix
7. APC kicks in, skips prefill for cached blocks
8. only append_token_ids go through prefill
9. decode starts
10. agentic-api stores:
    • new response items in DB
    • prefix_hash = sha256(prompt_token_ids returned by vLLM)
    • prefix_token_count = N + len(append_token_ids) + len(output_token_ids)
      here there would not be raw token_id storage by agentic-api for the marginal input either right we always store the hash?

currently we cannot interface this design we would need your feature branch from vllm fork to land into vllm upstream. are you planning to open PR soon on vllm?

I was also wondering the benchmarking figure mentioned in this ADR how is it generated? like prototyping with llm-d ogx rehydration or using ur forked vllm feature branch with agentic-api ? or is it just simple benchmark result on vllm upstream with prefix caching enabled vs disabled?

[franciscojavierarceo]

@maralbahari the overall shape is close, with two important corrections.

first, i would not want production turn 2 to do a fresh full render on every request. that is useful for prototype validation, CI, canary sampling, and fallback/reseed, but if it happens on every hot-path request we give back much of the render/tokenize win. the production path should validate the stored prefix metadata/handle, then render only the marginal suffix through a renderer-owned incremental path or a stored safe-boundary segment.

second, for agentic-api storage, the hot path should store compact prefix metadata: prefix hash, token count, model/tokenizer/renderer/template fingerprints, safe-boundary proof, and eventually router-visible block hashes for llm-d. raw token ids are useful as an optional cold-path diagnostic/reseed artifact, but i do not think we should require reading/writing full token arrays synchronously every turn. same for marginal input ids: they are execution material for vLLM, but not something the DB hot path should need to persist as raw ids unless we explicitly choose to keep a debug/checkpoint span.

on vLLM: yes, this design depends on a vLLM primitive that is not upstream today. the benchmark used my fork/branch with the prompt_cache_ref + append_token_ids prototype, not stock upstream vLLM. i think the vLLM work should probably land as one or more focused PRs: diagnostic token-id visibility first, then the replay primitive/handle fallback path.

on the figure: it was generated by a standalone benchmark harness against the forked vLLM server on the DGX with GPT-OSS-20B and APC enabled. it was not llm-d, not OGX rehydration, and not simply upstream prefix caching on/off. the key comparison was full-history streaming request versus minimal prompt-cache-ref replay. the important result was that APC was already hitting, so the measured win was mostly reduced prompt reconstruction / render-tokenize / request-size overhead, not recovering a missed KV prefill cache. i updated the ADR to make that provenance and storage split clearer.

[bbrowning]

Something important to point out here is that in vLLM, subsequent turns of a conversation are not a strict append-only situation by adding new token ids at the end of an existing list of token ids. Reasoning handling is a good example, where depending on where we are in the turns we conditionally drop some older reasoning that would distract the model from its current task at hand. Other examples are things like handling of multiple system messages for specific models, where some models can handle system messages that arrive in later turns and some can not. Or toggling thinking on or off in subsequent turns, changing reasoning effort between turns, etc. This is not an exhaustive list, and these are model-specific things that may or may not invalidate our prefix caching for any given request.

The point is, vLLM is the thing that knows how to properly turn a request into token ids and has model-dependent logic to do that. In many happy path cases that can look append-only, but in the real world it often is not with backtracking across at least one turn boundary for many popular models for every user turn due to how reasoning trimming is done.

Basically, the full vLLM render pipeline has to run for a given request to get the correct token ids for that turn's input to the model. There may be some things that could be cached internally, in vLLM, for happy path situations specific to each model that are append-only for subsequent turns. But, any logic to do that would need to live in vLLM itself right next to all the logic that decides which inputs to keep, drop, munge, or otherwise change as it's constructing the next set of input tokens.