fix: preserve Bedrock prompt-cache hits across turns (deterministic ordering + byte-stable compaction)

[philmerrell]

Problem

A cost audit of a $1.60 prod conversation (Chief of Staff agent + Memory Space, session aecd387d…1166) found 75% of spend was Bedrock prompt-cache re-writes ($1.19 of $1.60). 6 of 7 user turns started with cacheRead=0 and a full prefix re-write at $2.50/MTok — four of them well inside the 5-minute cache TTL. Three distinct miss mechanisms were identified:

1. Nondeterministic skill/tool ordering. batch_get_skills returns DynamoDB batch_get_item order (arbitrary), and RBAC effective permissions are materialized from unordered sets. These lists feed the <available_skills> system-prompt block and toolConfig — an order flip between turns fails Bedrock's exact-prefix match. Confirmed in prod: persisted agent_skills.last_injected_xml flip-flopped between turns, correlating with sub-TTL misses.
2. Per-turn tool-content truncation. _apply_compaction truncated tool results on every restore with a sliding protected window, mutating older history every turn (observed inter-turn prefix shrinkages of −382/−1,035/−1,513 tokens). Truncation economics are backwards at this scale: clamping ~1k tokens saves ~$0.0002/call in reads but costs $0.08–$0.35 per forced re-write.
3. Structural lookback miss on wide tool fan-outs. Strands' auto cache strategy keeps a single message cachePoint (moved every call, all others stripped), and Anthropic's cache lookup checks only ~20 content blocks behind a breakpoint. An 18-way parallel tool batch appended ~38 blocks in one hop → previous checkpoint out of range → full 134k re-write mid-turn, with nothing read because no system/tools checkpoint existed.

Fix (4 commits)

1. Deterministic ordering — batch_get_skills sorts by skill_id; build_skills_runtime re-sorts defensively; AppRoleService sorts tools/models/skills in effective permissions.

2. Byte-stable compaction (truncation anchor) — new persisted truncation_anchor: truncation applies only below the anchor, making restored history a pure function of (stored history, persisted state). The anchor advances only at checkpoint advances (the slice already pays one re-write) or when the prompt cache has already expired (cache_ttl_seconds, default 300s — the re-write is free then). Legacy sessions default the anchor to their checkpoint.

3. Layered cachePoints — cache_tools="default" plus a system-prompt cachePoint (via SystemContentBlock list; 3 of Bedrock's 4 allowed points, position-asserted by test). A message-level miss now reads the stable ~28k-token tools+system prefix at $0.20/MTok instead of re-writing it — this is the floor protection for miss mode 3, worth ~$0.30 in the audited session. Gated on bedrock_cache_points_supported() (Anthropic-on-Bedrock only) so non-Claude models don't receive verbatim cachePoints they'd reject.

4. Prompt-cache observability — makes this whole bug class self-diagnosing:

• PrefixFingerprintHook (BeforeModelCallEvent) hashes the three cacheable prefix segments per model call — toolConfigHash, systemPromptHash, historyHash — persisted on each C# cost row. A miss now attributes itself: system hash changed → ordering/injection drift; toolConfig changed → tool list drift; history changed → mutation; all stable → TTL or lookback.
• Derived cacheStatus per call (first_write | hit | miss_ttl_expired | miss_avoidable) + cacheGapSeconds + wastedUsd, computed at metadata-write time against the session's previous cost row.
• Session rollups: totalCacheReadTokens / totalCacheWriteTokens / avoidableMissCount / wastedUsd alongside totalCost.
• Admin cost-anatomy endpoint: GET /admin/costs/sessions/{session_id}/calls (require_admin).
• CloudWatch EMF metrics (AgentCoreStack/PromptCache: CacheReadTokens, CacheWriteTokens, AvoidableMiss) emitted as raw-JSON stdout lines — no agent, no SDK calls.
• CI determinism guard: builds the agent twice from shuffled skill/tool inputs and asserts identical system-prompt and toolConfig hashes.

Impact

On the audited conversation, working turn-to-turn caching converts 287k tokens from cache-write to cache-read: **$0.66 saved on a $1.60 conversation (~40%)**, and the ordering bug alone was taxing every multi-turn conversation with >1 skill granted. Layered cachePoints additionally cap the blast radius of any residual message-level miss.

Verification

• Byte-stability contract tests (test_compaction_stability.py): same stored history + same persisted state ⇒ byte-identical agent.messages across restores; anchor advance rules; legacy migration
• cachePoint position tests (test_bedrock_cache_points.py), fingerprint hook tests, cacheStatus derivation tests, EMF format tests, determinism guard — 61 new tests
• Ordering regression tests (shuffled fetch order ⇒ stable output)
• Session/skills/repository suites: 314 passed; import-boundary tests pass
• Acceptance after deploy: re-run the audited scenario; every turn-start call within 5 min of the prior turn should show cacheRead > 0, and any miss row should carry a fingerprint diff explaining itself

Follow-ups (deliberately not in this PR)

• SPA admin page consuming the cost-anatomy endpoint
• CDK CloudWatch dashboard/alarm on the EMF metrics (keeps this PR backend-only — one backend.yml deploy ships it)
• Upstream Strands issue: auto cache strategy should keep a rolling pair of message cachePoints so wide parallel tool batches don't outrun the ~20-block lookback
• ContextOffloader adoption (S3 storage) to shrink tool results at ingestion; MCP server payload trims

🤖 Generated with Claude Code