anthropicprovider: add opt-in prompt caching (cache_control)#496
anthropicprovider: add opt-in prompt caching (cache_control)#496lifeofzero wants to merge 1 commit into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds opt-in Anthropic prompt caching support to anthropicprovider by emitting cache_control breakpoints in the outbound request payload, enabling the provider to actually produce cache hits that it already reports in token usage.
Changes:
- Adds
AgentConfig.PromptCaching(default off) to control whether prompt-caching breakpoints are applied. - Applies two
cache_controlbreakpoints to AnthropicMessageNewParams: end of system blocks and end of the last message’s last content block. - Adds internal tests that assert the marshalled JSON wire format contains (or does not contain)
cache_controlas expected.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| provider/anthropicprovider/agent.go | Adds PromptCaching config and applies Anthropic cache_control breakpoints during request construction. |
| provider/anthropicprovider/caching_internal_test.go | Adds JSON wire-format tests to validate cache_control behavior (off by default; correct breakpoints when enabled). |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| if n := len(params.Messages); n > 0 { | ||
| blocks := params.Messages[n-1].Content | ||
| if len(blocks) > 0 { | ||
| setCacheControl(&blocks[len(blocks)-1]) | ||
| } | ||
| } |
There was a problem hiding this comment.
Good catch, and fixed in 492b515 — but it turned out to be the small half of the bug.
You're right that a trailing thinking block swallows the breakpoint: thinking has no cache_control field on the wire, so caching would be on and not one message token cached. Silent, which is exactly the failure this PR exists to prevent. (It isn't reachable today — buildMessageParam has no thinking case — but it's latent the moment one lands.)
Chasing it surfaced the bigger problem. ContentBlockParamUnion has 17 variants; the switch covered five. So ten cacheable variants were being silently skipped — including every server-tool result (web_search_tool_result, code_execution_tool_result, tool_search_tool_result, …), which are the largest blocks in a modern agent's context. Precisely the ones you most want cached, quietly uncached.
Rather than extend the list to 17 and have it rot on the 18th, setCacheControl now reflects over the union and makes the SDK's own types the source of truth: a block is cacheable exactly when its param struct carries a CacheControl field. thinking and redacted_thinking are the only two that don't. It returns a bool, and applyPromptCaching walks backward to the last block that accepts a breakpoint — an earlier breakpoint still caches nearly the entire prefix, so degraded beats silent. Cost is one reflect walk over a 17-field struct per request, next to a network call to an LLM.
Three tests, each verified to fail against the previous code:
SkipsTrailingThinkingBlock— your case.FallsBackToEarlierMessage— an all-thinking final message still gets a breakpoint.CoversEveryCacheableVariant— drives itself off the union's reflected shape, so a new SDK variant is covered the day it lands. Reverting to a hand-written switch fails it loudly, naming all ten missed variants.
|
Thanks for identifying this gap and for the thorough implementation and tests. We would like this to follow the approach used by the .NET Anthropic SDK. Rather than adding an agent-level Could you rework this PR accordingly?
References:
This keeps prompt caching explicit and opt-in, gives callers control over cost and breakpoint placement, and aligns the Go and .NET provider APIs. The work already done here—particularly around tool-result and unsupported thinking blocks—should remain useful for the per-content implementation. |
|
Thanks, this makes sense and I'm happy to rework it this way. Explicit per-content cache control is the better fit: it matches how Anthropic models Plan for the reworked PR:
As you noted, the tool-result handling and the unsupported-block cases (thinking / redacted_thinking carry no One optional question, and I'm fine either way: the common agentic-loop pattern is always the same shape (cache the system prompt, the last turn, and large stable tool results). Would a small opt-in helper layered on top of |
|
This is a genuinely excellent PR — I read it closely and wanted to flag what stands out, plus one question. What's especially strong:
One question (not a blocker): when (Not a maintainer — just another contributor who learned something reading this. Deferring to the team on the merge.) |
Let's leave higher level constructs for a future PR. |
Adds Anthropic prompt caching as an explicit, per-content-block primitive
rather than an agent-wide toggle, matching how the .NET Anthropic SDK models
cache_control (a property of the individual content, not a mode of the client).
The caller decides exactly which blocks become cache breakpoints.
content := anthropicprovider.WithCacheControl(
message.NewTextContent(bigSystemContext),
anthropicprovider.WithTTL(anthropicprovider.Ephemeral1h),
)
The marker is stored in the content's ContentHeader.AdditionalProperties under
a package-private key -- the same hook .NET's AIContent uses for provider
metadata -- and read back in buildMessageParam, which transfers it onto whatever
Anthropic block the content becomes (text, image, document, tool-use,
tool-result). A content whose block type has no cache_control field on the wire
(thinking / redacted_thinking) is a clean no-op, never a panic.
Caching is opt-in by design: a cache WRITE costs more than a plain input token,
so it is a net loss on a short single-turn prompt and nothing turns it on unless
the caller asks.
cache_control is applied to the block via reflection over the SDK's
ContentBlockParamUnion rather than a hand-written variant switch: the union has
17 variants today (every server-tool result among them), and reflection makes
the SDK's own type definitions the source of truth so a new variant is handled
the day it appears rather than silently left uncached. A canary test asserts the
"exactly one non-nil pointer field = the variant" invariant across all variants.
TTL supports the ephemeral 5-minute (default) and 1-hour tiers via
WithTTL(Ephemeral5m|Ephemeral1h). The TTL type is the SDK's own
anthropic.CacheControlEphemeralTTL, consistent with this package already handing
raw SDK types across its public surface (MessageNewParams), rather than wrapping
it in a parallel type callers would have to convert.
Anthropic permits at most 4 cache_control breakpoints per request; the caller is
responsible for staying within that budget. This primitive does not dedupe or
cap, and higher-level automatic placement is intentionally left to a future
change. MessageNewParams remains the low-level escape hatch for breakpoints on
tools or the system prompt.
492b515 to
3b8f648
Compare
|
Reworked per @qmuntal's guidance to make caching explicit and per-content, matching the .NET What changed
The reflection-based block marking that a few of you looked at is retained (generalized to carry the TTL), and the variant-coverage canary test is kept and extended. On the 4-breakpoint question (thanks @PratikDhanave for catching it) — with automatic placement gone, the old "2 unconditional + your manual ones" interaction no longer exists. The cap is still real, so I documented it on
|
The gap
anthropicproviderreads cached-token counts back —CacheReadInputTokensandCacheCreationInputTokensatagent.go:188— but never setscache_control. There is not a single occurrence ofcache_controlorephemeralin the provider.So it faithfully reports cache hits it is structurally incapable of producing. Prompt caching is simply unreachable from this provider today.
Why it matters here more than most places
A tool-using agent re-sends its entire context on every turn: the tool definitions, the system prompt, and every tool result accumulated so far. That repeated prefix is the bill.
Measured on a real 5-agent workflow against an MCP toolserver:
One agent alone made 21 tool calls whose results were 5–20KB each, and carried all of them forward through 8 turns.
What this adds
AgentConfig.PromptCaching(default off), placing two breakpoints:tools → system → messages, so this single breakpoint covers the tool definitions and the instructions — the part that is byte-identical on every turn.Uses 2 of Anthropic's 4 allowed breakpoints, leaving room for finer control via the existing
MessageNewParamsoption.Two deliberate decisions, in case you'd rather go the other way
Opt-in, not on by default. A cache write costs more than a plain input token, so caching is a net loss for a short single-turn agent with a small prompt — precisely the agent whose author is least likely to be measuring. Defaulting it on would quietly raise the bill for the cheapest use case in order to lower it for the most expensive one. I think that trade should be a decision, not a surprise.
Every content-block variant is handled, not just text. The trailing block on a tool-using turn is usually a
tool_result, not atextblock. Handling only text would leave the expensive half of the context uncached while the config still reported caching as on — caching that claims success and saves nothing, which is worse than no caching at all.Tests
Three, asserting the wire format (marshalled JSON) rather than SDK struct internals —
cache_controlis a contract with the API, and that is the thing that has to be right.All three are mutation-verified:
PromptCachingflagTestPromptCaching_OffByDefaultfailsOfToolResultcase (text-only breakpoint)TestPromptCaching_MarksTrailingToolResultBlockfails with "the expensive half of the context would go uncached"Full suite green,
go vetandgofmtclean.Related: #493 / #495 (token usage on spans) — that PR is what made this measurable in the first place.