Skip to content

fix(observability): first cache write after below-threshold calls is first_write, not miss_avoidable#701

Merged
philmerrell merged 1 commit into
developfrom
fix/cache-miss-avoidable-below-threshold
Jul 20, 2026
Merged

fix(observability): first cache write after below-threshold calls is first_write, not miss_avoidable#701
philmerrell merged 1 commit into
developfrom
fix/cache-miss-avoidable-below-threshold

Conversation

@philmerrell

Copy link
Copy Markdown
Contributor

Problem

The prompt-cache classifier (PR #697) produces a false positive: in a session where every prior call was uncached (prompt below the model's minimum cacheable prefix — e.g. ~4096 tokens for Claude Haiku 4.5), the first call that crosses the threshold does cacheWrite>0 / cacheRead=0 within the TTL window and gets classified miss_avoidable. That inflates the AvoidableMiss and WastedUsd EMF metrics and the admin Session Cost Anatomy page.

Verified live in dev-ai session 9a1f25b2-8ea4-487f-b0e7-80a60deff945: calls 1–5 uncached at 3.5–4k tokens, call 6 falsely flagged miss_avoidable with write=4122/read=0, calls 7–8 clean hits.

Fix

classify_cache_status gains an optional previous_cached_prefix_tokens parameter (the previous call's cacheRead + cacheWrite total, which _derive_cache_observability already computed for waste pricing). When the immediately preceding call had zero cache activity, there was no cache entry to read from — so the threshold-crossing write is classified first_write (the expected initial population) and excluded from waste pricing.

Reusing first_write (rather than adding a new status) keeps the EMF schema, dashboard, and admin anatomy page enums unchanged. None (unknown split) preserves the previous behavior.

Tests

  • Classifier: below-threshold-then-crossing → first_write; unknown previous prefix stays miss_avoidable.
  • Derivation: previous row with zero cache activity → first_write, wastedUsd == 0.0.
  • Full backend suite: 4771 passed, 3 skipped.

🤖 Generated with Claude Code

…d calls as miss_avoidable

When every prior call in a session was uncached (prompt below the
model's minimum cacheable prefix, e.g. ~4096 tokens on Claude Haiku 4.5),
the first call that crosses the threshold does cacheWrite>0/cacheRead=0
and was classified miss_avoidable — inflating the AvoidableMiss and
WastedUsd EMF metrics and the admin Session Cost Anatomy page. Verified
live in dev-ai session 9a1f25b2 (calls 1-5 uncached at 3.5-4k tokens,
call 6 falsely flagged with write=4122/read=0).

classify_cache_status now takes the previous call's cached-prefix token
total: when the immediately preceding call had zero cache activity there
was no entry to read from, so the write is classified first_write (the
expected initial population) and excluded from waste pricing. Unknown
(None) keeps the previous behavior.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@philmerrell
philmerrell merged commit b76bf39 into develop Jul 20, 2026
4 checks passed
@philmerrell
philmerrell deleted the fix/cache-miss-avoidable-below-threshold branch July 20, 2026 14:39
@philmerrell philmerrell mentioned this pull request Jul 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant